Turning a page in Ethereum JavaScript history

Farewell, node-gyp

If you’ve written software for Ethereum, it’s very likely at some point during your journey you’ve faced the dreadful beast, aka node-gyp. Somewhat Ethereum-aligned in values, node-gyp has been very democratic in the way it targets its victims. A nightmare for beginners, and a frustrating productivity sink for advanced developers.

In case you’ve seen this error or a variation of it, and don’t know what it’s about, here’s a recap. Ethereum makes pretty heavy use of cryptography, and in the JavaScript world (Ethereum’s biggest dev ecosystem) cryptography has mostly been handled using node-gyp-based dependencies. Node-gyp is a Node.js utility that allows the inclusion of native dependencies that are compiled on project setup. This has some benefits, like execution speed, but a significant number of downsides:

• Setting up a project takes very long, as every dependency is compiled when we run npm install. secp256k1 is actually compiled multiple times each time we run installon an Ethereum repository.
• It requires native build tools to work on any Ethereum project. On Mac OS, this means downloading Xcode’s 3gb. Good luck if you’re at an in-person hackathon.
• It’s brittle. Every new major release of Node.js breaks this setup, and triggers very unfriendly errors like the one above.
• Its errors are, ironically, very cryptic. The first time one runs into these it’s very hard to figure out what’s going on.

This is why as part of our work with the Ethereum Foundation we started working on ethereum-cryptography, a project to eradicate node-gyp from Ethereum. This library contains all the cryptographic primitives that are needed to build Ethereum software, through pure-JavaScript dependencies.

The vast majority of the dependencies are not new, they’re actually the same dependencies currently in use in browser contexts, but switching the ecosystem’s cryptography dependencies is no small undertaking. Pulling this off safely and effectively has been the main reason why this project has taken a while to happen. To reduce the risk surface we worked with Trail of Bits on an awesome audit of the library and iterated the design based on their recommendations. If you’re into cryptography, the report is an interesting read.

Arguably, ethereum-cryptography is now a safer way of solving Ethereum’s cryptography needs than directly using the dependencies, as it narrows down the scope of what’s possible to do cryptographically to APIs that don’t have unnecessary footguns.

To make sure things are as fast as possible, the library uses Node.js’s new N-API where it can. This new native addon doesn’t require compiling and doesn’t break across Node.js versions, but it’s very new and only some of the primitives are available.

The installation time improvements are stark. Buidler at one point was taking 3 minutes to install, and now it’s down to ~20 seconds. The installation time for just web3.js has gone down ~40 seconds.

Deployment

Once the library was ready, we enrolled the Tenderly team to help us deploy across the ecosystem. To make sure impact is immediate and not in the future as people migrate to new versions of their favorite tools and libraries, we analyzed the full extent to which node-gyp dependencies are used across the ecosystem. Not just in each project, but also in which versions within each project, so that new patch releases are published for old versions too. This way we made sure to make every Ethereum developer’s life easier, even if they’re stuck on old versions. This resulted in more than fifteen pull requests across eight different software projects. With last week’s ganache-core release node-gyp is now largely vanished from Ethereum, regardless of which tools and libraries you use.

How to upgrade your project to stop using node-gyp

We had patch versions released for all major versions of every single core Ethereum dependency that used node-gyp in some way. You should be able to receive a backward-compatible update to your dependencies regardless of what version you’re using. All you have to do is bump your dependencies to the latest release without changing the major version. You can do this by making sure you’re using ^ at the beginning of your npm package dependency versions, eliminating your package-lock.json file and running npm install in your project.

Maintenance

Keeping node-gyp away from Ethereum now is on the maintainers of each project. While we’ve done the heavy lifting, we can’t prevent maintainers from reintroducing node-gyp dependencies in future releases. If you see a node-gyp compilation happening when working with Ethereum JavaScript software, please open a Github issue in the offending project asking the maintainers to remove it. Now that ethereum-cryptography exists there’s an easy and proven path to get it done.

Thank you 🙏

This has been a long road with many collaborators we’d like to thank in no specific order: Holger Drews and the rest of the EF JavaScript team, Nebojsa Urosevic (Tenderly), Nick D’Andrea (Truffle), Trail of Bits, Makoto Inoue (ENS), Michael Bradley (Embark), Brian McKelvey (WebSocket-Node) and Greg Markou (ChainSafe/Web3.js).