From an architectural standpoint, we can break up Drupal into three pieces--its core, modules, and themes.
When we discuss Drupal 8 core, we can interpret it in two ways. A more restrictive interpretation sees it as a functionality covered in all the code it ships with, without modules and themes. The more widespread interpretation sees it as the total code base it ships with (out of the box).
Although the most widespread interpretation is the latter (not least because it differentiates all the functionalities its standard installation contains versus all others provided by contributed modules and themes), it is interesting to consider the first one as well, even if just for a minute. In this way, we can distinguish, architecturally, the base code from the modules and themes that provide various functionalities and layouts (both from the core installation and from external modules and themes). Along these lines will be also the hooks and events that glue everything together, allowing us to intercept and inject ties to our own functionality.
The core libraries are made up of code belonging to the Drupal project and those from the wider PHP community, which Drupal borrows under open source licensing. This latter approach is new in Drupal 8 and has been regarded by many as a positive shift toward getting off the Drupal island and embracing outside libraries, frameworks, and communities.
Essentially, the core libraries provide the functions and services used throughout Drupal. Facilities for interacting with the database, translating between languages, sanitizing user data, building forms, encoding data, and many such utilities are found in Drupal's core libraries.
The modules (both core and contributed) are where most of the actual functionality and business logic is. If enabled, they can provide functionality or extend the existing functionality. Most of the core modules are needed and cannot be disabled due to their importance in the standard Drupal installation. However, contributed ones can be installed and uninstalled as needed.
The themes (both core and contributed) are an important part of the theme system and are used in the presentation logic. They provide HTML templates within which content and data can be rendered to the user, as well as CSS styling and even client-side scripting for some nice visual interactions. Themes can extend other themes and can also contain some PHP logic to process the data before being rendered.