Using npm – the Node.js package management system

As described in Chapter 2, Setting up Node.js, npm is a package management and distribution system for Node.js. It has become the de facto standard for distributing modules (packages) for use with Node.js. Conceptually, it's similar to tools such as apt-get (Debian), rpm/yum (Red Hat/Fedora), MacPorts/Homebrew (macOS), CPAN (Perl), or PEAR (PHP). Its purpose is to publish and distributing Node.js packages over the internet using a simple command-line interface. In recent years, it has also become widely used for distributing front-end libraries like jQuery and Bootstrap that are not Node.js modules. With npm, you can quickly find packages to serve specific purposes, download them, install them, and manage packages you've already installed.

The npm application extends on the package format for Node.js, which in turn is largely based on the CommonJS package specification. It uses the same package.json file that's supported natively by Node.js, but with additional fields for additional functionality.

The npm package format

An npm package is a directory structure with a package.json file describing the package. This is exactly what was referred to earlier as a directory module, except that npm recognizes many more package.json tags than Node.js does. The starting point for npm's package.json file is the CommonJS Packages/1.0 specification. The documentation for the npm package.json implementation is accessed using the following command:

$  npm help package.json

A basic package.json file is as follows:

{ "name": "packageName", 
   "version": "1.0", 
   "main": "mainModuleName".
   "bin": "./path/to/program"
} 

Npm recognizes many more fields than this, and we'll go over some of them in the coming sections. The file is in JSON format, which, as a JavaScript programmer, you should be familiar with.

There is a lot to cover concerning the npm package.json format, and we'll do so over the following sections.

Accessing npm helpful documentation

The main npm command has a long list of subcommands for specific package management operations. These cover every aspect of the life cycle of publishing packages (as a package author), and downloading, using, or removing packages (as an npm consumer).

You can view the list of these commands just by typing npm (with no arguments). If you see one you want to learn more about, view the help information:

$ npm help <command>

The help text will be shown on your screen.

Help information is also available on the npm website at: https://docs.npmjs.com/cli-documentation/.

Before we can look for and install Node.js packages, we must have a project directory initialized.

Initializing a Node.js package or project with npm init

The npm tool makes it easy to initialize a Node.js project directory. Such a directory contains at the minimum a package.json file and one or more Node.js JavaScript files. 

All Node.js project directories are therefore modules, going by the definition we learned earlier. However, in many cases, a Node.js project is not meant to export any functionality but instead is an application. Such a project will likely require other Node.js packages, and those packages will be declared in the package.json file so that they're easy to install using npm. The other common use case of a Node.js project is a package of functionality meant to be used by other Node.js packages or applications. These also consist of a package.json file plus one or more Node.js JavaScript files, but in this case, they're Node.js modules that export functions and can be loaded using require, import(), or import.

What this means is the key to initializing a Node.js project directory is creating the package.json file.

While the package.json file can be created by hand – it's just a JSON file after all - the npm tool provides a convenient method:

$ mkdir example-package
$ cd example-package/
$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (example-package) 
version: (1.0.0) 
description: This is an example of initializing a Node.js project
entry point: (index.js) 
test command: mocha
git repository: 
keywords: example, package
author: David Herron <david@davidherron.com>
license: (ISC) 
About to write to /home/david/example-package/package.json:

{
 "name": "example-package",
 "version": "1.0.0",
 "description": "This is an example of initializing a Node.js project",
 "main": "index.js",
 "scripts": {
     "test": "mocha"
 },
 "keywords": [
     "example",
     "package"
 ],
 "author": "David Herron <david@davidherron.com>",
 "license": "ISC"
}

Is this OK? (yes) yes

In a blank directory, run npm init, answer the questions, and as quick as that you have the starting point for a Node.js project.

This is, of course, a starting point, and as you write the code for your project it will often be necessary to use other packages.

Finding npm packages

By default, npm packages are retrieved over the internet from the public package registry maintained on http://npmjs.com. If you know the module name, it can be installed simply by typing the following:

$ npm install moduleName  

But what if you don't know the module name? How do you discover the interesting modules? The website http://npmjs.com publishes a searchable index of the modules in the registry. The npm package also has a command-line search function to consult the same index:

Of course, upon finding a module, it's installed as follows:

$ npm install acoustid  

The npm repository uses a few package.json fields to aid in finding packages.

The package.json fields that help finding packages

For a package to be easily found in the npm repository requires a good package name, package description, and keywords. The npm search function scans those package attributes and presents them in search results.

The relevant package.json fields are as follows:

{ ...
  "description": "My wonderful package that walks dogs",
  "homepage": "http://npm.dogs.org/dogwalker/",
  "author": "dogwhisperer@dogs.org",
  "keywords": [ "dogs", "dog walking" ]
... }

The npm view command shows us information from package.json file for a given package, and with the --json flag we're shown the raw JSON.

The name tag is of course the package name, and it is used in URLs and command names, so choose one that's safe for both. If you desire to publish a package in the public npm repository, it's helpful to check whether a particular name is already being used by searching on https://npmjs.com or by using the npm search command.

The description tag is a short description that's meant as a brief/terse description of the package. 

It is the name and description tags that are shown in npm search results.

The keywords tag is where we list attributes of the package. The npm website contains pages listing all packages using a particular keyword. These keyword indexes are useful when searching for a package since it lists the related packages in one place, and therefore when publishing a package it's useful to land on the correct keyword pages.

Another source is the contents of the README.md file. This file should be added to the package to provide basic package documentation. This file is shown on the package page on npmjs.com, and therefore it is important for this file to convince potential users of your package to actually use it. As the file name implies, this is a Markdown file.

Once you have found a package to use, you must install it in order to use the package.

Installing an npm package

The npm install command makes it easy to install packages upon finding one of your dreams, as follows:

$ npm install express
/home/david/projects/notes/
- express@4.13.4
...  

The named module is installed in node_modules in the current directory. During the installation process, the package is set up. This includes installing any packages it depends on and running the preinstall and postinstall scripts. Of course, installing the dependent packages also involves the same installation process of installing dependencies and executing pre-install and post-install scripts. 

Some packages in the npm repository have a package scope prepended to the package name. The package name in such cases is presented as @scope-name/package-name, or, for example, @akashacms/plugins-footnotes. In such a package, the name field in package.json contains the full package name with its @scope.

We'll discuss dependencies and scripts later. In the meantime, we notice that a version number was printed in the output, so let's discuss package version numbers.

Installing a package by version number

Version number matching in npm is powerful and flexible. With it, we can target a specific release of a given package or any version number range. By default, npm installs the latest version of the named package, as we did in the previous section. Whether you take the default or specify a version number, npm will determine what to install.

The package version is declared in the package.json file, so let's look at the relevant fields:

{ ...
 "version": "1.2.1",
 "dist-tags": {
 "latest": "1.2.1"
 },
... }

The version field obviously declares the current package version. The dist-tags field lists symbolic tags that the package maintainer can use to aid their users in selecting the correct version. This field is maintained by the npm dist-tag command.

The npm install command supports these variants:

$ npm install package-name@tag
$ npm install package-name@version
$ npm install package-name@version-range

The last two are what they sound like. You can specify express@4.16.2 to target a precise version, or express@">4.1.0 < 5.0" to target a range of Express V4 versions. We might use that specific expression because Express 5.0 might include breaking changes.

The version match specifiers include the following choices:

• Exact version match: 1.2.3
• At least version N: >1.2.3
• Up to version N: <1.2.3
• Between two releases: >=1.2.3 <1.3.0

The @tag attribute is a symbolic name such as @latest, @stable, or @canary. The package owner assigns these symbolic names to specific version numbers and can reassign them as desired. The exception is @latest, which is updated whenever a new release of the package is published.

For more documentation, run these commands: npm help json and npm help npm-dist-tag.

In selecting the correct package to use, sometimes we want to use packages that are not in the npm repository.

Installing packages from outside the npm repository

As awesome as the npm repository is, we don't want to push everything we do through their service. This is especially true for internal development teams who cannot publish their code for all the world to see. Fortunately, Node.js packages can be installed from other locations. Details about this are in npm help package.json in the dependencies section. Some examples are as follows:

• URL: You can specify any URL that downloads a tarball, that is, a .tar.gz file. For example, GitHub or GitLab repositories can easily export a tarball URL. Simply go to the Releases tab to find them.
• Git URL: Similarly, any Git repository can be accessed with the right URL, for example:

$ npm install git+ssh://user@hostname:project.git#git-tag

• GitHub shortcut: For GitHub repositories, you can list just the repository specifier, such as expressjs/express. A tag or a commit can be referenced using expressjs/express#tag-name.
• GitLab, BitBucket, and GitHub URL shortcuts: In addition to the GitHub shortcut, npm supports a special URL format for specific Git services with URLs like github:user/repo, bitbucket:user/repo, and gitlab:user/repo.
• Local filesystem: You can install from a local directory using a URL with the: file:../../path/to/dir.

Sometimes we need to install a package for use by several projects, without requiring that each project installs the package.

Global package installs

In some instances, you want to install a module globally, so that it can be used from any directory. For example, the Grunt or Babel build tools are widely useful, and conceivably you will find it useful if these tools are installed globally. Simply add the -g option:

$ npm install -g grunt-cli  

If you get an error, and you're on a Unix-like system (Linux/Mac), you may need to run this with sudo:

$ sudo npm install -g grunt-cli

This variant, of course, runs npm install with elevated permissions.

The npm website offers a guideline with more information at https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally.

If a local package install lands in node_modules, where does a global package install land? On a Unix-like system, it lands in PREFIX/lib/node_modules, and on Windows, it lands in PREFIX/node_modules. In this case, PREFIX means the directory where Node.js is installed. You can inspect the location of the directory as follows:

$ npm config get prefix
/opt/local

The algorithm used by Node.js for the require function automatically searches the directory for packages if the package is not found elsewhere.

ES6 modules do not support global packages.

Many believe it is not a good idea to install packages globally, which we will look at next.

Avoiding global module installation

Some in the Node.js community now frown on installing packages globally. One rationale is that a software project is more reliable if all its dependencies are explicitly declared. If a build tool such as Grunt is required but is not explicitly declared in package.json, the users of the application would have to receive instructions to install Grunt, and they would have to follow those instructions. 

Users being users, they might skip over the instructions, fail to install the dependency, and then complain the application doesn't work. Surely, most of us have done that once or twice.

It's recommended to avoid this potential problem by installing everything locally via one mechanism—the npm install command.

There are two strategies we use to avoid using globally installed Node.js packages. For the packages that install commands, we can configure the PATH variable, or use npx to run the command. In some cases, a package is used only during development and can be declared as such in package.json.

Maintaining package dependencies with npm

The npm install command by itself, with no package name specified, installs the packages listed in the dependencies section of package.json. Likewise, the npm update command compares the installed packages against the dependencies and against what's available in the npm repository and updates any package that is out of date in regards to the repository. 

These two commands make it easy and convenient to set up a project, and to keep it up to date as dependencies are updated. The package author simply lists all the dependencies, and npm installs or updates the dependencies required for using the package. What happens is npm looks in package.json for the dependencies or devDependencies fields, and it works out what to do from there.

You can manage the dependencies manually by editing package.json. Or you can use npm to assist you with editing the dependencies. You can add a new dependency like so:

$ npm install akasharender --save  

With the --save flag, npm will add a dependencies tag to package.json:

"dependencies": { 
    "akasharender": "^0.7.8" 
} 

With the added dependency, when your application is installed, npm will now install the package along with any other dependencies listed in package.json file.

The devDependencies lists modules used during development and testing. The field is initialized the same as the preceding one, but with the --save-dev flag. The devDependencies can be used to avoid some cases where one might instead perform a global package install.

By default, when npm install is run, modules listed in both dependencies and devDependencies are installed. Of course, the purpose of having two dependency lists is to control when each set of dependencies is installed.

$ npm install --production  

This installs the "production" version, which means to install only the modules listed in dependencies and none of the devDependencies modules. For example, if we use a build tool like Babel in development, the tool should not be installed in production.

While we can manually maintain dependencies in package.json, npm can handle this for us.

Automatically updating package.json dependencies

With npm@5 (also known as npm version 5), one change was that it's no longer required to add --save to the npm install command. Instead, npm by default acts as if you ran the command with --save, and will automatically add the dependency to package.json. This is meant to simplify using npm, and it is arguably more convenient that npm now does this. At the same time, it can be very surprising and inconvenient for npm to go ahead and modify package.json for you. The behavior can be disabled by using the --no-save flag, or it can be permanently disabled using the following:

$ npm config set save false

The npm config command supports a long list of settable options for tuning the behavior of npm. See npm help config for the documentation and npm help 7 config for the list of options.

Now let's talk about the one big use for package dependencies: to fix or avoid bugs.

Fixing bugs by updating package dependencies

Bugs exist in every piece of software. An update to the Node.js platform may break an existing package, as might an upgrade to packages used by the application. Your application may trigger a bug in a package it uses. In these and other cases, fixing the problem might be as simple as updating a package dependency to a later (or earlier) version.

First, identify whether the problem exists in the package or in your code. After determining it's a problem in another package, investigate whether the package maintainers have already fixed the bug. Is the package hosted on GitHub or another service with a public issue queue? Look for an open issue on this problem. That investigation will tell you whether to update the package dependency to a later version. Sometimes, it will tell you to revert to an earlier version; for example, if the package maintainer introduced a bug that doesn't exist in an earlier version.

Sometimes, you will find that the package maintainers are unprepared to issue a new release. In such a case, you can fork their repository and create a patched version of their package. In such a case, your package might use a Github URL referencing your patched package.

One approach to fixing this problem is pinning the package version number to one that's known to work. You might know that version 6.1.2 was the last release against which your application functioned and that starting with version 6.2.0 your application breaks. Hence, in package.json:

"dependencies": {
    "module1": "6.1.2"
}

This freezes your dependency on the specific version number. You're free, then, to take your time updating your code to work against later releases of the module. Once your code is updated, or the upstream project is updated, change the dependency appropriately.

When listing dependencies in package.json, it's tempting to be lazy, but that leads to trouble.

Explicitly specifying package dependency version numbers

As we've said several times in this chapter, explicitly declaring your dependencies is A Good Thing. We've already touched on this, but it's worth reiterating and to see how npm makes this easy to accomplish.

The first step is ensuring that your application code is checked into a source code repository. You probably already know this, and even have the best of intentions to ensure that everything is checked in. With Node.js, each module should have its own repository rather than putting every single last piece of code in one repository.

Each module can then progress on its own timeline. A breakage in one module is easy to back out by changing the version dependency in package.json.

The next step is to explicitly declare all dependencies of every module. The goal is simplifying and automating the process of setting up every module. Ideally, on the Node.js platform, the module setup is as simple as running npm install.

Any additional required steps can be forgotten or executed incorrectly. An automated setup process eliminates several kinds of potential mistakes.

With the dependencies and devDependencies sections of package.json, we can explicitly declare not only the dependencies but the precise version numbers.

The lazy way of declaring dependencies is putting * in the version field. That uses the latest version in the npm repository. This will seem to work, until that one day the maintainers of that package introduce a bug. You'll type npm update, and all of a sudden your code doesn't work. You'll head over to the GitHub site for the package, look in the issue queue, and possibly see that others have already reported the problem you're seeing. Some of them will say that they've pinned on the previous release until this bug is fixed. What that means is their package.json file does not depend on * for the latest version, but on a specific version number before the bug was created.

Don't do the lazy thing, do the smart thing.

The other aspect of explicitly declaring dependencies is to not implicitly depend on global packages. Earlier, we said that some people in the Node.js community caution against installing modules in the global directories. This might seem like an easy shortcut to sharing code between applications. Just install it globally, and you don't have to install the code in each application.

But, doesn't that make deployment harder? Will the new team member be instructed on all the special files to install here and there to make the application run? Will you remember to install that global module on all destination machines?

For Node.js, that means listing all the module dependencies in package.json, and then the installation instructions are simply npm install, followed perhaps by editing a configuration file.

While most packages in the npm repository are libraries with an API, some are tools we can run from the command line.

Packages that install commands

Some packages install command-line programs. A side effect of installing such packages is a new command that you can type at the shell prompt or use in shell scripts. An example is the hexy program that we briefly used in Chapter 2, Setting Up Node.js. Another example is the widely used Grunt or Babel build tools.

The recommendation to explicitly declare all dependencies in package.json applies to command-line tools as well as any other package. Therefore these packages will typically be installed locally. This requires special care in setting up the PATH environment variable correctly. As you should already be aware, the PATH variable is used on both Unix-like systems and Windows to list the directories in which the command-line shell searches for commands.

The command can be installed to one of two places:

• Global install: It is installed either to a directory, such as /usr/local, or to the bin directory where Node.js was installed. The npm bin -g command tells you the absolute pathname for this directory. In this case, it's unlikely you'll have to modify the PATH environment variable.
• Local install: Installs to node_modules/.bin in the package where the module is being installed, the npm bin command tells you the absolute pathname for the directory. Because the directory is inconveniently located to run commands, a change to the PATH variable is useful.

To run the command, simply type the command name at a shell prompt. This works correctly if the directory where the command is installed happens to be in the PATH variable. Let's look at how to configure the PATH variable to handle locally installed commands.

Configuring the PATH variable to handle locally installed commands

Assume we have installed the hexy command like so:

$ npm install hexy

As a local install, this creates a command as node_modules/.bin/hexy. We can attempt to use it as follows:

$ hexy package.json 
-bash: hexy: command not found

But this breaks because the command is not in a directory listed in the PATH. The workaround is to use the full pathname or relative pathname:

$ ./node_modules/.bin/hexy package.json
... hexy output 

But obviously typing the full or partial pathname is not a user-friendly way to execute the command. We want to use the commands installed by modules, and we want a simple process for doing so. This means, we must add an appropriate value in the PATH variable, but what is it?

For global package installations, the executable lands in a directory that is probably already in your PATH variable, like /usr/bin or /usr/local/bin. Local package installations require special handling. The full path for the node_modules/.bin directory varies for each project, and obviously it won't work to add the full path for every node_modules/.bin directory to your PATH.

Adding ./node_modules/.bin to the PATH variable (or, on Windows, .\node_modules\.bin) works great. Any time your shell is in the root of a Node.js project, it will automatically find locally installed commands from Node.js packages.

How we do this depends on the command shell you use and your operating system.

On a Unix-like system, the command shells are bash and csh. Your PATH variable would be set up in one of these ways:

$ export PATH=./node_modules/.bin:${PATH}     # bash
$ setenv PATH ./node_modules/.bin:${PATH}     # csh

The next step is adding the command to your login scripts so the variable is always set. On bash, add the corresponding line to ~/.bashrc, and on csh add it to ~/.cshrc.

Once this is accomplished the command-line tool executes correctly.

Configuring the PATH variable on Windows

On Windows, this task is handled through a system-wide settings panel:

This pane of the System Properties panel is found by searching for PATH in the Windows Settings screen. Click on the Environment Variables button, then select the Path variable, and finally click on the Edit button. On the screen here, click the New button to add an entry to this variable, and enter .\node_modules\.bin as shown. You'll have to restart any open command shell windows. Once you do, the effect will be as shown previously.

As easy as it is to modify the PATH variable, we don't want to do this in all circumstances.

Avoiding modifications to the PATH variable

What if you don't want to add these variables to your PATH at all times? The npm-path module may be of interest. This is a small program that computes the correct PATH variable for your shell and operating system. See the package at https://www.npmjs.com/package/npm-path.

Another option is to use the npx command to execute such commands. This tool is automatically installed alongside the npm command. This command either executes commands from a locally installed package or it silently installs commands in a global cache:

$ npx hexy package.json

Using npx is this easy.

Of course, once you've installed some packages, they'll go out of date and need to be updated.

Updating packages you've installed when they're outdated

The coder codes, updating their package, leaving you in the dust unless you keep up.

To find out whether your installed packages are out of date, use the following command:

$ npm outdated  

The report shows the current npm packages, the currently installed version, as well as the current version in the npm repository. Updating outdated packages is very simple:

$ npm update express
$ npm update 

Specifying a package name updates just the named package. Otherwise, it updates every package that would be printed by npm outdated.

Npm handles more than package management, it has a decent built-in task automation system.

Automating tasks with scripts in package.json

The npm command handles not just installing packages, it can also be used to automate running tasks related to the project. In package.json, we can add a field, scripts, containing one or more command strings. Originally scripts were meant to handle tasks related to installing an application, such as compiling native code, but they can be used for much more. For example, you might have a deployment task using rsync to copy files to a server. In package.json, you can add this:

{ ...
  "scripts: {
    "deploy": "rsync --archive --delete local-dir user@host:/path/to/dest-dir
  }
... }

What's important here is that we can add any script we like, and the scripts entry records the command to run:

$ npm run deploy

Once it has been recorded in scripts, running the command is this easy.

There is a long list of "lifecycle events" for which npm has defined script names. These include the following:

• install, for when the package is installed
• uninstall, for when it is uninstalled
• test, for running a test suite
• start and stop, for controlling a server defined by the package

Package authors are free to define any other script they like. 

For the full list of predefined script names, see the documentation: https://docs.npmjs.com/misc/scripts

Npm also defines a pattern for scripts that run before or after another script, namely to prepend pre or post to the script name. Therefore the pretest script runs before the test script, and the posttest script runs afterward.

A practical example is to run a test script in a prepublish script to ensure the package is tested before publishing it to the npm repository:

{
  "scripts": {
      "test": "cd test && mocha",
      "prepublish": "npm run test"
  }
}

With this combination, if the test author types npm publish, the prepublish script will cause the test script to run, which in turn uses mocha to run the test suite.

It is a well-known best practice to automate all administrative tasks, if only so that you never forget how to run those tasks. Creating the scripts entries for every such task not only prevents you from forgetting how to do things, but it also documents the administrative tasks for the benefit of others.

Next, let's talk about how to ensure the Node.js platform on which a package is executed supports the required features.

Declaring Node.js version compatibility

It's important that your Node.js software runs on the correct version of Node.js. The primary reason being that the Node.js platform features required by your package are available every time your package is run. Therefore, the package author must know which Node.js releases are compatible with the package, and then describe in package.json that compatibility.

This dependency is declared in package.json using the engines tag:

"engines": { 
    "node": ">= 8.x <=10.x" 
} 

The version string is similar to what we can use in dependencies and devDependencies. In this case, we've defined that the package is compatible with Node.js 8.x, 9.x, and 10.x. 

Now that we know how to construct a package, let's talk about publishing packages.

Publishing an npm package

All those packages in the npm repository came from people like you with an idea of a better way of doing something. It is very easy to get started with publishing packages.

Online docs about publishing packages can be found at https://docs.npmjs.com/getting-started/publishing-npm-packages.

Also consider this: https://xkcd.com/927/.

You first use the npm adduser command to register yourself with the npm repository. You can also sign up with the website. Next, you log in using the npm login command.

Finally, while sitting in the package root directory, use the npm publish command. Then, stand back so that you don't get stampeded by the crush of thronging fans, or, maybe not. There are several zillion packages in the repository, with hundreds of packages added every day. To get yours to stand out, you will require some marketing skill, which is another topic beyond the scope of this book.

It is suggested that your first package be a scoped package, for example, @my-user-name/my-great-package. 

We've learned a lot in this section about using npm to manage and publish packages. But npm is not the only game in town for managing Node.js packages.