Defining a Node.js module

Modules are the basic building blocks for constructing Node.js applications. A Node.js module encapsulates functions, hiding details inside a well-protected container, and exposing an explicitly declared API.

When Node.js was created, the ES6 module system, of course, did not yet exist. Ryan Dahl, therefore, based on the Node.js module system on the CommonJS standard. The examples we've seen so far are modules written to that format. With ES2015/ES2016, a new module format was created for use with all JavaScript implementations. This new module format is used by both front-end engineers in their in-browser JavaScript code and by Node.js engineers, and for any other JavaScript implementation.

Because ES6 modules are now the standard module format, the Node.js Technical Steering Committee (TSC) committed to first-class support for ES6 modules alongside the CommonJS format. Starting with Node.js 14.x, the Node.js TSC delivered on that promise.

Every source file used in an application on the Node.js platform is a module. Over the next few sections, we'll examine the different types of modules, starting with the CommonJS module format. 

Throughout this book, we'll identify traditional Node.js modules as CommonJS modules, and the new module format as ES6 modules.

To start our exploration of Node.js modules, we must, of course, start at the beginning.

Examining the traditional Node.js module format

We already saw CommonJS modules in action in the previous chapter. It's now time to see what they are and how they work. 

In the ls.js example in Chapter 2, Setting Up Node.js, we wrote the following code to pull in the fs module, giving us access to its functions:

const fs = require('fs'); 

The require function is given a module identifier, and it searches for the module named by that identifier. If found, it loads the module definition into the Node.js runtime and making its functions available. In this case, the fs object contains the code (and data) exported by the fs module. The fs module is part of the Node.js core and provides filesystem functions.

By declaring fs as const, we have a little bit of assurance against making coding mistakes. We could mistakenly assign a value to fs, and then the program would fail, but as a const we know the reference to the fs module will not be changed.

The file, ls.js, is itself a module because every source file we use on Node.js is a module. In this case, it does not export anything but is instead a script that consumes other modules.

What does it mean to say the fs object contains the code exported by the fs module? In a CommonJS module, there is an object, module, provided by Node.js, with which the module's author describes the module. Within this object is a field, module.exports, containing the functions and data exported by the module. The return value of the require function is the object. The object is the interface provided by the module to other modules. Anything added to the module.exports object is available to other pieces of code, and everything else is hidden. As a convenience, the module.exports object is also available as exports. 

The module object contains several fields that you might find useful. Refer to the online Node.js documentation for details.

Because exports is an alias of module.exports, the following two lines of code are equivalent:

exports.funcName = function(arg, arg1) { ... };
module.exports.funcName = function(arg, arg2) { .. }; 

Whether you use module.exports or exports is up to you. However, do not ever do anything like the following:

exports = function(arg, arg1) { ... };

Any assignment to exports will break the alias, and it will no longer be equivalent to module.exports. Assignments to exports.something are okay, but assigning to exports will cause failure. If your intent is to assign a single object or function to be returned by require, do this instead:

module.exports = function(arg, arg1) { ... };

Some modules do export a single function because that's how the module author envisioned delivering the desired functionality.

When we said ls.js does not export anything, we meant that ls.js did not assign anything to module.exports.

To give us a brief example, let's create a simple module, named simple.js:

var count = 0;
exports.next = function() { return ++count; };
exports.hello = function() {
  return "Hello, world!";
};

We have one variable, count, which is not attached to the exports object, and a function, next, which is attached. Because count is not attached to exports, it is private to the module. 

Any module can have private implementation details that are not exported and are therefore not available to any other code.

Now, let's use the module we just wrote:

$ node
> const s = require('./simple');
undefined
> s.hello();
'Hello, world!'
> s.next();
1
> s.next();
2
> s.next();
3
> console.log(s.count);
undefined
undefined
> 

The exports object in the module is the object that is returned by require('./simple'). Therefore, each call to s.next calls the next function in simple.js. Each returns (and increments) the value of the local variable, count. An attempt to access the private field, count, shows it's unavailable from outside the module.

This is how Node.js solves the global object problem of browser-based JavaScript. The variables that look like they are global variables are only global to the module containing the variable. These variables are not visible to any other code.

The Node.js package format is derived from the CommonJS module system (http://commonjs.org). When developed, the CommonJS team aimed to fill a gap in the JavaScript ecosystem. At that time, there was no standard module system, making it trickier to package JavaScript applications. The require function, the exports object, and other aspects of Node.js modules come directly from the CommonJS Modules/1.0 spec.

The module object is a global-to-the-module object injected by Node.js. It also injects two other variables: __dirname and __filename. These are useful for helping code in a module know where it is located in the filesystem. Primarily, this is used for loading other files using a path relative to the module's location. 

For example, one can store assets like CSS or image files in a directory relative to the module. An app framework can then make the files available via an HTTP server. In Express, we do so with this code snippet:

app.use('/assets/vendor/jquery', express.static( 
 path.join(__dirname, 'node_modules', 'jquery'))); 

This says that HTTP requests on the /assets/vendor/jquery URL are to be handled by the static handler in Express, from the contents of a directory relative to the directory containing the module. Don't worry about the details because we'll discuss this more carefully in a later chapter. Just notice that __dirname is useful to calculate a filename relative to the location of the module source code.

To see it in action, create a file named dirname.js containing the following:

console.log(`dirname: ${__dirname}`);
console.log(`filename: ${__filename}`);

This lets us see the values we receive:

$ node dirname.js 
dirname: /home/david/Chapter03
filename: /home/david/Chapter03/dirname.js

Simple enough, but as we'll see later these values are not directly available in ES6 modules.

Now that we've got a taste for CommonJS modules, let's take a look at ES2015 modules.

Examining the ES6/ES2015 module format

ES6 modules are a new module format designed for all JavaScript environments. While Node.js has always had a good module system, browser-side JavaScript has not. That meant the browser-side community had to use non-standardized solutions. The CommonJS module format was one of those non-standard solutions, which was borrowed for use in Node.js. Therefore, ES6 modules are a big improvement for the entire JavaScript world, by getting everyone on the same page with a common module format and mechanisms.

An issue we have to deal with is the file extension to use for ES6 modules. Node.js needs to know whether to parse using the CommonJS or ES6 module syntax. To distinguish between them, Node.js uses the file extension .mjs to denote ES6 modules, and .js to denote CommonJS modules. However, that's not the entire story since Node.js can be configured to recognize the .js files as ES6 modules. We'll give the exact particulars later in this chapter.

The ES6 and CommonJS modules are conceptually similar. Both support exporting data and functions from a module, and both support hiding implementation inside a module. But they are very different in many practical ways.

Let's start with defining an ES6 module. Create a file named simple2.mjs in the same directory as the simple.js example that we looked at earlier:

let count = 0;
export function next() { return ++count; }
function squared() { return Math.pow(count, 2); }
export function hello() {
    return "Hello, world!";
}
export default function() { return count; }
export const meaning = 42;
export let nocount = -1;
export { squared };

This is similar to simple.js but with a few additions to demonstrate further features. As before count is a private variable that isn't exported, and next is an exported function that increments count.

The export keyword declares what is being exported from an ES6 module. In this case, we have several exported functions and two exported variables. The export keyword can be put in front of any top-level declaration, such as variable, function, or class declarations:

 export function next() { .. }

The effect of this is similar to the following:

module.exports.next = function() { .. }

The intent of both is essentially the same: to make a function or other object available to code outside the module. But instead of explicitly creating an object, module.exports, we're simply declaring what is to be exported. A statement such as export function next() is a named export, meaning the exported function (as here) or object has a name, and that code outside the module uses that name to access the object. As we see here, named exports can be functions or objects, and they may also be class definitions.

The default export from a module, defined with export default, can be done once per module. The default export is what code outside the module accesses when using the module object itself, rather than when using one of the exports from the module.

You can also declare something, such as the squared function, and then export it later.

Now let's see how to use the ES2015 module. Create a simpledemo.mjs file with the following:

import * as simple2 from './simple2.mjs';

console.log(simple2.hello());
console.log(`${simple2.next()} ${simple2.squared()}`);
console.log(`${simple2.next()} ${simple2.squared()}`);
console.log(`${simple2.default()} ${simple2.squared()}`);
console.log(`${simple2.next()} ${simple2.squared()}`);
console.log(`${simple2.next()} ${simple2.squared()}`);
console.log(`${simple2.next()} ${simple2.squared()}`);
console.log(simple2.meaning);

The import statement does what it says: it imports objects exported from a module. Because it uses the import * as foo syntax, it imports everything from the module, attaching everything to an object, in this case named simple2. This version of the import statement is most similar to a traditional Node.js require statement because it creates an object with fields containing the objects exported from the module.

This is how the code executes:

$ node simpledemo.mjs 
Hello, world!
1 1
2 4
2 4
3 9
4 16
5 25
42

In the past, the ES6 module format was hidden behind an option flag, --experimental-module, but as of Node.js 13.2 that flag is no longer required. Accessing the default export is accomplished by accessing the field named default. Accessing an exported value, such as the meaning field, is done without parentheses because it is a value and not a function.

Now to see a different way to import objects from a module, create another file, named simpledemo2.mjs, containing the following:

import { 
    default as simple, hello, next, meaning 
} from './simple2.mjs';
console.log(hello());
console.log(next());
console.log(next());
console.log(simple());
console.log(next());
console.log(next());
console.log(next());
console.log(meaning);

In this case, the import is treated similarly to an ES2015 destructuring assignment. With this style of import, we specify exactly what is to be imported, rather than importing everything. Furthermore, instead of attaching the imported things to a common object, and therefore executing simple2.next(), the imported things are executed using their simple name, as in next().

The import for default as simple is the way to declare an alias of an imported thing. In this case, it is necessary so that the default export has a name other than default. 

Node.js modules can be used from the ES2015 .mjs code. Create a file named ls.mjs containing the following:

import { promises as fs } from 'fs';

async function listFiles() {
    const files = await fs.readdir('.');
    for (const file of files) {
        console.log(file);
    }
}

listFiles().catch(err => { console.error(err); });

This is a reimplementation of the ls.js example in Chapter 2, Setting Up Node.js. In both cases, we're using the promises submodule of the fs package. To do this with the import statement, we access the promises export from the fs module, and use the as clause to rename fs.promises to fs. This way we can use an async function rather than deal with callbacks.

Otherwise, we have an async function, listFiles, that performs filesystem operations to read filenames from a directory. Because listFiles is async, it returns a Promise, and we must catch any errors using a .catch clause.

Executing the script gives the following:

$ node ls.mjs
ls.mjs
module1.js
module2.js
simple.js
simple2.mjs
simpledemo.mjs
simpledemo2.mjs

The last thing to note about ES2015 module code is that the import and export statements must be top-level code. Try putting an export inside a simple block like this:

{
   export const meaning = 42;
}

That innocent bit of code results in an error:

$ node badexport.mjs 
file:///home/david/Chapter03/badexport.mjs:2
    export const meaning = 42;
    ^^^^^^

SyntaxError: Unexpected token 'export'
    at Loader.moduleStrategy (internal/modules/esm/translators.js:83:18)
    at async link (internal/modules/esm/module_job.js:36:21)

While there are a few more details about the ES2015 modules, these are their most important attributes.

Remember that the objects injected into CommonJS modules are not available to ES6 modules. The __dirname and __filename objects are the most important, since there are many cases where we compute a filename relative to the currently executing module. Let us explore how to handle that issue.

Injected objects in ES6 modules

Just as for CommonJS modules, certain objects are injected into ES6 modules. Furthermore, ES6 modules do not receive the __dirname, and __filename objects or other objects that are injected into CommonJS modules.

The import.meta meta-property is the only value injected into ES6 modules. In Node.js it contains a single field, url. This is the URL from which the currently executing module was loaded.

Using import.meta.url, we can compute __dirname and __filename.

Computing the missing __dirname variable in ES6 modules

If we make a duplicate of dirname.js as dirname.mjs, so it will be interpreted as an ES6 module, we get the following:

$ cp dirname.js dirname.mjs
$ node dirname.mjs 
console.log(`dirname: ${__dirname}`);
 ^
ReferenceError: __dirname is not defined
 at file:///home/david/Chapter03/dirname.mjs:1:25
 at ModuleJob.run (internal/modules/esm/module_job.js:109:37)
 at async Loader.import (internal/modules/esm/loader.js:132:24)

Since __dirname and __filename are not part of the JavaScript specification, they are not available within ES6 modules. Enter the import.meta.url object, from which we can compute __dirname and __filename. To see it in action, create a dirname-fixed.mjs file containing the following:

import { fileURLToPath } from 'url';
import { dirname } from 'path';

console.log(`import.meta.url: ${import.meta.url}`);

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

console.log(`dirname: ${__dirname}`);
console.log(`filename: ${__filename}`);

We are importing a couple of useful functions from the url and path core packages. While we could take the import.meta.url object and do our own computations, these functions already exist. The computation is to extract the pathname portion of the module URL, to compute __filename, and then use dirname to compute __dirname.

$ node dirname-fixed.mjs 
import.meta.url: file:///home/david/Chapter03/dirname-fixed.mjs
dirname: /home/david/Chapter03
filename: /home/david/Chapter03/dirname-fixed.mjs

And we see the file:// URL of the module, and the computed values for __dirname and __filename using the built-in core functions.

We've talked about both the CommonJS and ES6 module formats, and now it's time to talk about using them together in an application.

Using CommonJS and ES6 modules together

Node.js supports two module formats for JavaScript code: the CommonJS format originally developed for Node.js, and the new ES6 module format. The two are conceptually similar, but there are many practical differences. Because of this, we will face situations of using both in the same application and will need to know how to proceed.

First is the question of file extensions and recognizing which module format to use. The ES6 module format is used in the following situations:

• Files where the filename ends in .mjs.
• If the package.json has a field named type with the value module, then filenames ending with .js.
• If the node binary is executed with the --input-type=module flag, then any code passed through the --eval or --print argument, or piped in via STDIN (the standard input), is interpreted as ES6 module code.

That's fairly straight-forward. ES6 modules are in files named with the .mjs extension, unless you've declared in the package.json that the package defaults to ES6 modules, in which case files named with the .js extension are also interpreted as ES6 modules.

The CommonJS module format is used in the following situations:

• Files where the file name ends in .cjs.
• If the package.json does not contain a type field, or if it contains a type field with a value of commonjs, the filenames will end with .js.
• If the node binary is executed with the --input-type flag or with the --type-type=commonjs flag, then any code passed through the --eval or --print argument, or piped in via STDIN (the standard input), is interpreted as CommonJS module code.

Again this is straight-forward, with Node.js defaulting to CommonJS modules for the .js files. If the package is explicitly declared to default to CommonJS modules, then Node.js will interpret the .js files as CommonJS.

The Node.js team strongly recommends that package authors include a type field in package.json, even if the type is commonjs.

Consider a package.json with this declaration:

{
    "type": "module" ...
}

This, of course, informs Node.js that the package defaults to ES6 modules. Therefore, this command interprets the module as an ES6 module:

$ node my-module.js

This command will do the same, even without the package.json entry:

$ node --input-type=module my-module.js

If instead, the type field had the commonjs, or the --input-type flag specified as commonjs, or if both those were completely missing, then my-module.js would be interpreted as a CommonJS module.

These rules also apply to the import statement, the import() function, and the require() function. We will cover those commands in more depth in a later section. In the meantime, let's learn how the import() function partly resolves the inability to use ES6 modules in a CommonJS module. 

Using ES6 modules from CommonJS using import()

The import statement in ES6 modules is a statement, and not a function like require(). This means that import can only be given a static string, and you cannot compute the module identifier to import. Another limitation is that import only works in ES6 modules, and therefore a CommonJS module cannot load an ES6 module. Or, can it?

Since the import() function is available in both CommonJS and ES6 modules, that means we should be able to use it to import ES6 modules in a CommonJS module. 

To see how this works, create a file named simple-dynamic-import.js containing the following:

async function simpleFn() {
    const simple2 = await import('./simple2.mjs');
    console.log(simple2.hello());
    console.log(simple2.next());
    console.log(simple2.next());
    console.log(`count = ${simple2.default()}`);
    console.log(`Meaning: ${simple2.meaning}`);
}

simpleFn().catch(err => { console.error(err); });

This is a CommonJS module that's using an ES6 module we created earlier. It simply calls a few of the functions, nothing exciting except that it is using an ES6 module when we said earlier import only works in ES6 modules. Let's see this module in action:

$ node simple-dynamic-import.js 
Hello, world!
1
2
count = 2
Meaning: 42

This is a CommonJS module successfully executing code contained in an ES6 module simply by using import().

Notice that import() was called not in the global scope of the module, but inside an async function. As we saw earlier, the ES6 module keyword statements like export and import must be called in the global scope. However, import() is an asynchronous function, limiting our ability to use it in the global scope.

The import statement is itself an asynchronous process, and by extension the import() function is asynchronous, while the Node.js require() function is synchronous. 

In this case, we executed import() inside an async function using the await keyword. Therefore, even if import() were used in the global scope, it would be tricky getting a global-scope variable to hold the reference to that module. To see, why let's rewrite that example as simple-dynamic-import-fail.js:

const simple2 = import('./simple2.mjs');
console.log(simple2);
console.log(simple2.hello());
console.log(simple2.next());
console.log(simple2.next());
console.log(`count = ${simple2.default()}`);
console.log(`Meaning: ${simple2.meaning}`);

It's the same code but running in the global scope. In the global scope, we cannot use the await keyword, so we should expect that simple2 will contain a pending Promise. Running the script gives us this failure:

$ node simple-dynamic-import-fail.js 
Promise { <pending> }
/home/david/Chapter03/simple-dynamic-import-fail.js:4
console.log(simple2.hello());
 ^
TypeError: simple2.hello is not a function
 at Object.<anonymous> (/home/david/Chapter03/simple-dynamic-import-fail.js:4:21)
 at Module._compile (internal/modules/cjs/loader.js:1139:30)
 at Object.Module._extensions..js (internal/modules/cjs/loader.js:1159:10)
 at Module.load (internal/modules/cjs/loader.js:988:32)
 at Function.Module._load (internal/modules/cjs/loader.js:896:14)
 at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
 at internal/main/run_main_module.js:17:47

We see that simple2 does indeed contain a pending Promise, meaning that import() has not yet finished. Since simple2 does not contain a reference to the module, attempts to call the exported function fail.

The best we could do in the global scope is to attach the .then and .catch handlers to the import() function call. That would wait until the Promise transitions to either a success or failure state, but the loaded module would be inside the callback function. We'll see this example later in the chapter.

Let's now see how modules hide implementation details.

Hiding implementation details with encapsulation in CommonJS and ES6 modules

We've already seen a couple of examples of how modules hide implementation details with the simple.js example and the programs we examined in Chapter 2, Setting up Node.js. Let's take a closer look.

Node.js modules provide a simple encapsulation mechanism to hide implementation details while exposing an API. To review, in CommonJS modules the exposed API is assigned to the module.exports object, while in ES6 modules the exposed API is declared with the export keyword. Everything else inside a module is not available to code outside the module.

In practice, CommonJS modules are treated as if they were written as follows:

(function(exports, require, module, __filename, __dirname) {
// Module code actually lives in here
}); 

Thus, everything within the module is contained within an anonymous private namespace context. This is how the global object problem is resolved: everything in a module that looks global is actually contained within a private context. This also explains how the injected variables are actually injected into the module. They are parameters to the function that creates the module.

The other advantage is code safety. Because the private code in a module is stashed in a private namespace, it is impossible for code outside the module to access the private code or data.

Let's take a look at a practical demonstration of the encapsulation. Create a file named module1.js, containing the following:

const A = "value A";
const B = "value B";
exports.values = function() {
   return { A: A, B: B };
}

Then, create a file named module2.js, containing the following:

const util = require('util');
const A = "a different value A";
const B = "a different value B";
const m1 = require('./module1');
console.log(`A=${A} B=${B} values=${util.inspect(m1.values())}`);
console.log(`${m1.A} ${m1.B}`);
const vals = m1.values();
vals.B = "something completely different";
console.log(util.inspect(vals));
console.log(util.inspect(m1.values()));

Using these two modules we can see how each module is its own protected bubble.

Then run it as follows:

$ node module2.js 
A=a different value A B=a different value B values={ A: 'value A', B: 'value B' }
undefined undefined
{ A: 'value A', B: 'something completely different' }
{ A: 'value A', B: 'value B' }

This artificial example demonstrates encapsulation of the values in module1.js from those in module2.js. The A and B values in module1.js don't overwrite A and B in module2.js because they're encapsulated within module1.js. The values function in module1.js does allow code in module2.js access to the values; however, module2.js cannot directly access those values. We can modify the object module2.js received from module1.js. But doing so does not change the values within module1.js.

In Node.js modules can also be data, not just code.

Using JSON modules

Node.js supports using require('./path/to/file-name.json') to import a JSON file in a CommonJS module. It is equivalent to the following code:

const fs = require('fs');
module.exports = JSON.parse(
        fs.readFileSync('/path/to/file-name.json', 'utf8'));

That is, the JSON file is read synchronously, and the text is parsed as JSON. The resultant object is available as the object exported from the module. Create a file named data.json, containing the following:

{ 
    "hello": "Hello, world!", 
    "meaning": 42 
}

Now create a file named showdata.js containing the following:

const data = require('./data.json');
console.log(data);

It will execute as follows:

$ node showdata.js 
{ hello: 'Hello, world!', meaning: 42 }

The console.log function outputs information to the Terminal. When it receives an object, it prints out the object content like this. And this demonstrates that require correctly read the JSON file since the resulting object matched the JSON.

In an ES6 module, this is done with the import statement and requires a special flag. Create a file named showdata-es6.mjs containing the following:

import * as data from './data.json';
console.log(data);

So far that is equivalent to the CommonJS version of this script, but using import rather than require.

$ node --experimental-modules --experimental-json-modules showdata-es6.mjs 
(node:12772) ExperimentalWarning: The ESM module loader is experimental.
[Module] { default: { hello: 'Hello, world!', meaning: 42 } }

Currently using import to load a JSON file is an experimental feature. Enabling the feature requires these command-line arguments, causing this warning to be printed. We also see that instead of data being an anonymous object, it is an object with the type Module.

Now let's look at how to use ES6 modules on some older Node.js releases.

Supporting ES6 modules on older Node.js versions

Initially, ES6 module support was an experimental feature in Node.js 8.5 and became a fully supported feature in Node.js 14. With the right tools, we can use it on earlier Node.js implementations. 

For an example of using Babel to transpile ES6 code for older Node.js versions, see https://blog.revillweb.com/using-es2015-es6-modules-with-babel-6-3ffc0870095b.

The better method of using ES6 modules on Node.js 6.x is the esm package. Simply do the following:

$ nvm install 6
Downloading and installing node v6.14.1...
Downloading https://nodejs.org/dist/v6.14.1/node-v6.14.1-darwin-x64.tar.xz...
######################################################################## 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v6.14.1 (npm v3.10.10)
$ nvm use 6
Now using node v6.14.1 (npm v3.10.10)
$ npm install esm
... npm output
$ node --require esm simpledemo.mjs 
Hello, world!
1 1
2 4
2 4
3 9
4 16
5 25
42

There are two ways to use this module: 

• In a CommonJS module, invoke require('esm').
• On the command line, use --require esm, as shown here.

In both cases, the effect is the same, to load the esm module. This module only needs to be loaded once, and we do not have to call any of its methods. Instead esm retrofits ES6 module support into the Node.js runtime, and is compatible with version 6.x and later.

So, we can use this module to retrofit ES6 module support; it does not retrofit other features such as async functions. Successfully executing the ls.mjs example requires support for both the async functions and arrow functions. Since Node.js 6.x does not support either, the ls.mjs example will load correctly, but will still fail because it uses other unsupported features.

$ node --version
v6.14.1
$ node --require esm ls.mjs 
/Users/David/chap03/ls.mjs:5
(async () => {
       ^

SyntaxError: Unexpected token (
 at exports.runInThisContext (vm.js:53:16)
 at Module._compile (module.js:373:25)

It is, of course, possible to use Babel in such cases to convert the full set of ES2015+ features to run on older Node.js releases.

For more information about esm, see: 
https://medium.com/web-on-the-edge/es-modules-in-node-today-32cff914e4b. The article describes an older release of the esm module, at the time named @std/esm.

Th current documentation for the esm package is available at: https://www.npmjs.com/package/esm.

In this section, we've learned about how to define a Node.js module and various ways to use both CommonJS and ES6 modules. But we've left out some very important things: what is the module identifier and all the ways to locate and use modules. In the next section, we cover these topics.