Tips for writing nodejs scripts using only built-in modules and async

Jonathan Creamer -

For the last few years, I've spent a huge majority of my time writing scripts in nodejs as my default scripting language. I've dabbled with bash a bit too, but it's still so weird, and hard to test, and I'm such a Typescript fan that I generally prefer to go the typescript route as often as possible.

Below is a list of tips to make writing nodejs scripts fun, fast, and easy.

Use ES Modules

Yes! You can do it now! However, thar be dragons.

In general tho, if you want to use ES modules in node, you can...

  1. Name your file with .mjs on the end of it
  2. Use a .js file in a place where the closest package.json has "type": "module"
  3. Use node --eval and pass --input-type=module

Similar rules apply to keeping a file use commonjs

  1. Name your file with .cjs
  2. Use a .js file in a place where the closest package.json has "type": "commonjs"
  3. Use node --eval and pass --input-type=commonjs

There are a couple of provisos, a, a couple of quid pro quos

When you switch to using ES Modules in node, you have to:

  • Always include the extension in your imports...
import { foo } from "./bar.js";
  • You have to use import() instead of require
  • This doesn't work... __dirname

But, this does...

import path from 'node:path';
import { fileURLToPath } from 'node:url';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

You might also run into issues with some of the node_modules that you import, so, just be careful. But, if you're generally sticking to native node modules and utilities anyways, then you should be pretty good to go.

Use top level await

This became available in 14.8. For the longest time, I'd always write a wrapper every time I got ready to write a node.js script like...

const main = async () => {
  // await some stuff in here
};

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

I did that so often that I wrote VSCode snippets for it. Heh.

But, cool thing is, you can just await anywhere in a JS file now.

NOTE: you can only do this in a "module" context, i.e. see "Use ES Modules" above

import fs from 'node:fs/promises';

const pkg = await fs.readFile('./package.json');

console.log(JSON.parse(pkg).name);

Use the new-ish promise based APIs along with the node prefix

As of node v14.18 and v16, you can now require native node modules, i.e. those things like path, fs, etc which are built into node, by using the node: protocol in the import or require statement. This will prevent script writers from accidentally importing 3rd party versions of utilities.

import fs from "node:fs/promises";
import stream from "node:stream/promises";
import dns from "node:dns/promises";
import timers from "node:timers/promises";

Also, for everything else that takes a standard node (err, results) => {} callback, you can always fall back on the promisify method.

import fs from 'node:fs/promises';
import { exec as _exec } from 'node:child_process';
import { promisify } from 'node:util';

const exec = promisify(_exec);

const out = await exec('git status');

console.log(out.stdout);

Use child_process effectively

The child_process module is one of the most useful modules in a script writer's backpack. You can run any arbitrary thing on the machine using its spawn or exec methods. However, knowing when, and how to use either of them, is essential.

The most fundamental difference between spawn and exec is...

spawn creates and returns a child process that you can interact with easily via streams

exec creates a runs a subprocess returning the output and error streams with a max of 200kb

Therefore, most of the time exec will do what you need, but if you need some sort of interactivity or have a bunch of output, then you can use spawn.

One of the previous examples above used exec to call git status. Another example could be an abstraction over some sort of unix method like find.

import fs from 'node:fs/promises';
import { exec as _exec } from 'node:child_process';
import { promisify } from 'node:util';

const exec = promisify(_exec);
const [,, filename, directory] = process.argv;

const out = await exec(`find ${directory} -type f -name ${filename} `);

console.log(out.stdout);

// run with...
// node ./script.mjs filename ./directory

Here's an example of using spawn to do something.

In this case, we're going to spawn an npm install in a different package in our repo.

import { spawn } from 'node:child_process';

const install = () => {
  return new Promise((resolve, reject) => {
    const proc = spawn('npm', ['install'], {
      stdio: ['inherit', 'pipe', 'pipe'],
      env: {
        ...process.env,
      }
    });
    
    proc.stdout.on('data', (data) => {
      if (data.includes('npm WARN')) {
        // Do something
      }
    });
    
    proc.stderr.on('data', (data) => {
      console.error(`stderr: ${data}`);
    });
    
    proc.on('close', (code) => {
      if (code !== 0) {
        return reject(code);
      }
    
      resolve();
    });
  });
};

try {
  await install();
} catch(e) {
  console.error("Something went wrong with the install.");
  process.exit(1);
}

First thing to note here is that we're going create a function and use the Promise constructor in the return. That way we can await it at the top level.

Let's take a look at the options we're passing to spawn. (You can see the entire API of spawn here) The first is the top level command, in this case npm. The second option is an array of the arguments we want to pass to npm in this case just install. But, if we wanted pass multiple things to npm, we'd need to add them each as elements to that array such as ['install', '--save-dev', '--legacy-peer-deps'].

The third argument is the options object. By default, when you spawn a process, the stdio option is set to pipe.

The pipe option means that the proc which is returned from spawn will have proc.stdout, proc.stdin, and proc.stderr on it. These are streams so that you can use methods like on and one to respond to events such as data.

The stdin argument can either take pipe, inherit, or an array corresponding to which parts of the process you want to control...

If you don't need to use stdin or stderr for example, you can switch from pipe to inherit.

stdin: [
  'inherit' /* stdin */,
  'pipe' /* stdout */,
  'inherit' /* stderr */ 
],

Passing stdin like that 👆 would provide proc.stdout  but not proc.stdin or proc.stderr.

Lastly, you can watch for the process you're spawning to close by using proc.on('close', () => {});.

One more nice option is that you can control which working directory the process is spawned in via the cwd option. This allows you to use either process.cwd(), or set some other path where you actually want to spawn the process.

Use Typescript for an even cooler script writing experience

Generally I prefer to use Typescript when writing just about anything these days, including scripts. There's a couple of different ways when writing scripts like this which you can use to leverage Typescript for the extra security blanket of checks in your code.

First of all, if you're going to be using Typescript and Node, make sure that you have the @types/node package installed as a "devDependency". That will ensure that all the native modules have correct type information.

Intellisense for parameters
Intellisense for options

That in and of itself is super useful.

What's really worth noting though is that, you don't HAVE to have your file end in a .ts extension. You can add a couple of options to a tsconfig.json. They are "allowJs" and "checkJs". Set them both to true, and you're good to go.

Typescript as of the 4.7 release now supports ES Modules too by reading the "type": "module" that can be added to package.json files. And you can also manually tell Typescript to use modules by setting the "module" option to node16 or nodenext. All that is documented here.

If you want to use .ts instead of using allowJs and checkJs, you'll probably also want to install something like @babel/register, or ts-node, maybe even esbuild-runner. I still personaly prefer the Babel route.

Conclusion

In the world of Frontend and DevOps aka DivOps, writing scripts is inevitable. Writing scripts in Javascript / Typescript allows you to use the JS skills you have as a Frontend dev to interact with the machine to do all kinds of automations, pipelines, etc. And bonus points because you can also write unit tests too. Go forth and write all your scripts in JavaScript!