Inside the pain of monorepos and hoisting

Working in a monorepo comes with a long list of pros, and a few cons. One of the most painful cons, when it comes to working in a specifically JavaScript based monorepo, is the pain that comes from hoisting.

Here's a video that talks about it, and you can also keep reading to learn more!

What's this hoisting thing?

As we all know, node_modules is a deep dark place with lots and lots of stuff. That problem is even more massive in a large monorepo.

Let's take a step back for a sec and take a look at what happens when you require something.

node module resolution

If you take a read through the docs, you'll find this...

require(X) from module at path Y
1. If X is a core module,
   a. return the core module
   b. STOP
2. If X begins with '/'
   a. set Y to be the filesystem root
3. If X begins with './' or '/' or '../'
   a. LOAD_AS_FILE(Y + X)
   b. LOAD_AS_DIRECTORY(Y + X)
   c. THROW "not found"
4. If X begins with '#'
   a. LOAD_PACKAGE_IMPORTS(X, dirname(Y))
5. LOAD_PACKAGE_SELF(X, dirname(Y))
6. LOAD_NODE_MODULES(X, dirname(Y))
7. THROW "not found"

What this is basically saying is...

If you require X, see if it exists in node, things like fs, child_process, etc.

If you start require('/X'); you're at the file system root.

If you require('./X'); then see if .X is file, then a directory.

...and this is the interesting one...

If you do require('X'); and it's not a node thing, then keep traversing the file system looking in every node_modules along the way.

Hoisting

Package managers such as yarn and npm implemented a hoisting algorithm as a part of their different workspaces implementations.

What this hoisting does is, it scans your package.json files across your workspaces and figures out what the most common versions of dependencies are.

If you have 50 packages, and 47 of them are using react 16.9.0, but 3 are using 16.13.0, it'll "hoist" the common version of react, 16.9.0 to the top level ./node_modules directory. That way you don't have to have 50 different versions of react in your project, the import React or require('react') calls will simply pull from the root node_modules/react one in the case of 16.9.0, or ./packages/*/node_modules/react for the 3 cases of 16.13.0.

Voila, space saved.

However, the plot thickens...

Monorepo structure

So, most monorepo structures have a root package.json, and a packages folder.

./packages/foo/
./packages/foo/node_modules
./packages/bar/
./packages/bar/node_modules
./package.json
./node_modules

Let's say we're working with the ./packages/foo/ and it does an import React from 'react';.

Following the node module resolution from above, it'll eventually look in the ./packages/foo/node_modules directory. If it's not there, it'll look at ../packages/node_modules/, which definitely should not be there, and then it'll look at the ./node_modules directory and see if it's there.

Ok, so that seems fine, where can we go wrong here?

Well, a couple of different bad things can happen here.

Let's say the foo/package.json has "react": "^16.9.0".

Then, let's say over in bar, they forgot to add react in the package.json, but in bar/src/index.ts someone does an import React..., and it also happens that bar used a feature that only exists in the 16.13.0 version of React.

What's gonna happen? Welp, because of the fact that bar doesn't have a react in its package.json, the node module resolution algorithm will kick in and look up to the ../../node_modules/ directory and grab react from there.

But OH NOES, that code won't work at runtime because the code written in bar needs that fancy new feature in React 16.13! This is a problem that is also nicely summed up in the Rush docs as a "Phantom Dependency" and a "Doppleganger".

Real world ouchies

Here's a real world example of how this came into play in a recent release pipeline failure.

In the monorepo supported by my team, the most common version of jest-environment-jsdom that was hoisted to the root node_modules folder in the repo was 24.9.0.

A pull request when in which added jest-environment-jsdom 26.10.0 to one of the packages in the ./packages folder. Well, what happened was, there were a couple of other places using that same version across the repo so, yarn in its attempt to de-duplicate by hoisting decided to switch the hoisted version to 26.10.0!

Here's where things got bad.

Let's say the package which added jest-environment-jsdom was called, cool-button. The pull request for adding the new dependency will get pushed to CI, and the CI server does a check for all the places where cool-button is used, and start running tests, builds, etc on the dependencies in the repo to make sure that the changes cool-button doesn't break any of it's downstream dependencies.

Ok, PR is green, all looks great!

Well, let's say there's another package called cool-color-picker. And cool-color-picker had some tests which was more or less like...

it('should pull from memory storage when localStorage isnt there', () => {
  expect(storageUtilThing.get('item')).toBeTruthy();
})

Well, in this crazy insane case... cool-color-picker was relying on the hoisted jest-environment-jsdom version, 24.9.0. Well, that particular version used jsdom 11.11.0. In that particular version of jsdom there was NO local storage support in the jsdom environment, so the test would pass.

However! In the new version of 26.10.0 of jest-environment-jsdom, the version of jsdom gets bumped to 16.x.x and it just so happens every so ironically that version 11.12.0 of jsdom implemented localStorage. See Changelog.

All the sudden, the test is now failing because the test had previously assumed that there was no localStorage support in jsdom without some kind of 3rd party mocking like jest-localstorage-mock.

What's worse here is, because cool-color-picker isn't in the dependency graph of cool-button, it never got tested as a part of the pull request.

Therefore, the change landed and attempted to go through the release pipeline.

THANKFULLY, although also painfully, the release pipeline currently does a "build the world" strategy regardless of what changed.

It's here where the release pipeline which always builds all the packages that the failure happened and broke releases for a few hours until we figured out what happened which involved LOTS of code spelunking.

So what next?

Well, in an ideal world we'll soon be switching to a "strict" package manager like the one we're currently building called midgard-yarn-strict, which will eventually migrate much of it's feature set into the implementation of Isolated Mode in NPM. Yarn v2 or later can also solve this problem with plug and play, or pnpm as well, but we're currently focusing on the former of the solutions with NPM and midgard yarn strict.

A strict package manager such as these options will do a couple of things.

First of all, the packages are forced to correctly declare their dependencies in the package.json so as to eliminate the phantom dependency / doppleganger problem.

Secondly, the root node_modules folder can be moved to a different location on disk, and pointed to with symlinks.

./packages/foo/
./packages/foo/node_modules/react  -> node_modules/.store/react-16.9.0
./packages/bar/
./packages/bar/node_modules/react -> node_modules/.store/react-16.13.0
./package.json

# This breaks the node module resoution
./node_modules/.store
./node_modules/.store/react-16.13.0
./node_modules/.store/react-16.9.0

By moving the node_modules folder somewhere else, and simply symlinking all of the package's dependencies to the new location, the hoisting algorithm breaks.

So, in the previous case where the bar package had forgotten to declare react in its package, bar can no longer rely on hoisting!

Other things you can do to help are implementing a couple of solutions like DepCheck to make based on scanning your code that your dependencies are all correctly declared. There's also a tool called TypeSync which does something similar for the @types packages to make sure they're present. The hoisting problem with@types packages presents a whole different set of complexities because their usage isn't in code necessarily, but only in the TypeScript setup.

Conclusion

Hoisting helped a lot in solving some problems for many repos out there. However, the moment your monorepo starts to scale a bit, you'll inevitably run into these problems.