Lerna bootstrap fails due to concurrency (with npm)

[Reinmar]

The latest RC version fails when running lerna bootstrap. I'm not sure when it was introduced, though, because only today builds started failing on Travis while RC3 was released 8 days ago. So, I may be actually running into two separate issues. Both turned to have the same solution, though, switching off concurrency.

The issue, at least now, can be reproduced in the following way:

p@m /workspace/test> git clone git@github.com:ckeditor/ckeditor5.git
Cloning into 'ckeditor5'...
remote: Counting objects: 6484, done.
remote: Compressing objects: 100% (71/71), done.
remote: Total 6484 (delta 32), reused 0 (delta 0), pack-reused 6407
Receiving objects: 100% (6484/6484), 1.16 MiB | 336.00 KiB/s, done.
Resolving deltas: 100% (3672/3672), done.
p@m /workspace/test> cd ckeditor5/
(master cbe80fd) p@m /workspace/test/ckeditor5> lerna bootstrap
Lerna v2.0.0-rc.3
Bootstrapping 1 packages
Preinstalling packages
Installing external dependencies
Installing hoisted dependencies into root
Pruning hoisted dependencies
FileSystemUtilities.rename    ("./package.json", "./package.json.lerna_backup")
Error: ENOENT: no such file or directory, rename '/workspace/test/ckeditor5/package.json' -> '/workspace/test/ckeditor5/package.json.lerna_backup'
    at Error (native)
NpmUtilities.installInDir     (".", ["guppy-pre-commit@^0.4.0"], {})
Error: ENOENT: no such file or directory, rename '/workspace/test/ckeditor5/package.json' -> '/workspace/test/ckeditor5/package.json.lerna_backup'
    at Error (native)
Errored while running BootstrapCommand.execute
Error: ENOENT: no such file or directory, rename '/workspace/test/ckeditor5/package.json' -> '/workspace/test/ckeditor5/package.json.lerna_backup'
    at Error (native)

When I try with --concurrency=1 the issue doesn't seem to appear:

(master cbe80fd) p@m /workspace/test/ckeditor5> lerna bootstrap --concurrency=1
Lerna v2.0.0-rc.3
Bootstrapping 1 packages
Preinstalling packages
Installing external dependencies
Installing hoisted dependencies into root
Pruning hoisted dependencies
Symlinking packages and binaries
Postinstalling packages
Prepublishing packages
Successfully bootstrapped 1 packages.

It also proved to help with a bit more complicated situation I had on Travis: https://travis-ci.org/ckeditor/ckeditor5/builds/226105896. It worked well after disabling concurrency: https://travis-ci.org/ckeditor/ckeditor5/builds/226101027.

I can see that similar issues were already reported: #397 and #616. However, this may be the first scenario in which the issue can be easily reproduced.

Your Environment

Executable	Version
lerna --version	2.0.0-rc.3
npm --version	4.0.3
node --version	6.9.1

OS	Version
macOS Sierra	10.12.4

[evocateur]

@Reinmar Thanks for the repro case!

[lukebatchelor]

Interesting, I ran into this issue a while back and was getting very consistent failures. Tried playing with various combinations of yarn, lerna and node to see why this was happening.

Tried it again today and was seeing the same issue, but cleaned my yarn cache and suddenly everything is working

package	version
lerna	2.0.0-rc.4
yarn	0.20.3
node	7.9.0

yarn cache clean

[Reinmar]

I got a bit different error today, but from what I checked it's also related to concurrency:

https://gist.github.com/Reinmar/b9978b60d09fec1ffc9f55f8d1262349

(this is npm-debug.log from one of the packages in which npm i --global-style failed)

[evocateur]

@Reinmar That error seems to indicate there is invalid JSON in the package.json file? Did the lerna-debug.log give any hints?

[Reinmar]

Unfortunately, I didn't save it as I went straight to npm-debug.log (which is usually more helpful so Lerna could mention that to the user). In the console, as I recall, there was just an info about crashing processes.

I know that the npm log indicates that some JSON was missing/invalid, but I've seen the same error many times in the past. Usually, a second lerna bootstrap run would help with that. So, my assumption now, when I discovered that lerna bootstrap --concurrency=1 helps even more consistently is that:

• one Lerna process starts installing package X,
• a second process tries to install the same package at the same time,
• there's some mutex failure and the second process thinks that the package is already available in npm's cache (/Users/me/.npm) but it isn't or is temporarily invalid at that moment,
• the second process crashes while trying to read this file.

And why does it work when running lerna bootstrap for the second time (even concurrently)? Because the cache is already hot and all JSONs are usually already unpacked there.

Anyhow reasonable?

[evocateur]

That sounds very reasonable. Definitely worthy of further investigation.

[jwb]

Do you think it would be fruitful to make an integration test that demonstrates this behavior?

[evocateur]

That would definitely be a part of the fix.

[Reinmar]

I wonder if npm 5 will solve this issue:

A new, standardised lockfile feature meant for cross-package-manager compatibility (package-lock.json), and a new format and semantics for shrinkwrap. (#16441)

And there's the whole cache rewrite too.

[DavideDaniel]

nope. npm5 does not.

[Brantron]

We've been able to solve this issue by setting --mutex network on the bootstrap command and running our multiple packages in stages. Here's a slimmed down version of our config

language: node_js
sudo: required
dist: trusty
node_js:
  - '8'

cache:
  directories:
    - $HOME/.npm
    - $HOME/.yarn-cache
    - packages/browser-extension/node_modules
    - packages/frontend/node_modules

before_install:
  - npm i -g yarn@"1.5.1" --cache-min 999999999
  - npm i -g lerna --cache-min 999999999

jobs:
  include:
    - stage: frontend
      script: ./scripts/frontend
      env:
        - SHARD_COUNT=5 SHARD_INDEX=0
    - stage: frontend
      script: ./scripts/frontend
      env:
        - SHARD_COUNT=5 SHARD_INDEX=1
    - stage: frontend
      script: ./scripts/frontend
      env:
        - SHARD_COUNT=5 SHARD_INDEX=2
    - stage: frontend
      script: ./scripts/frontend
      env:
        - SHARD_COUNT=5 SHARD_INDEX=3
    - stage: frontend
      script: ./scripts/frontend
      env:
        - SHARD_COUNT=5 SHARD_INDEX=4
    - stage: extension
      script: ./scripts/extension

where scripts frontend is

#!/bin/sh
lerna bootstrap --scope=frontend --mutex network;
(cd packages/frontend && yarn lint);
(cd packages/frontend && yarn test:ci --silent --maxWorkers=4 --coverage)

and scripts extension is

#!/bin/sh
lerna bootstrap --scope=browser-extension --concurrency=1 --mutex network;
(cd packages/browser-extension && yarn lint);
(cd packages/browser-extension && yarn test:ci --silent --maxWorkers=4 --coverage)

[jony89]

having the same issue :

lerna info version 2.9.0
lerna info versioning independent
lerna info Bootstrapping 4 packages
lerna info lifecycle preinstall
lerna info Installing external dependencies
lerna info hoist Finished installing in root
lerna ERR! execute callback with error
lerna ERR! Error: Command failed: npm install --global-style
lerna ERR! npm ERR! cb() never called!
lerna ERR!
lerna ERR! npm ERR! This is an error with npm itself. Please report this error at:
lerna ERR! npm ERR! https://github.com/npm/npm/issues
lerna ERR!
lerna ERR! at Promise.all.then.arr (/usr/local/lib/node_modules/lerna/node_modules/execa/index.js:236:11)
lerna ERR! at
lerna WARN complete Waiting for 3 child processes to exit. CTRL-C to exit immediately.

Tried even npm clear cache --force

npm : v5.8.0
node : v9.10.0
on mac

[Brantron]

We finally have a reliable workaround!!!

install: travis_retry make ci-install

where the makefile does

ci-install:
	yarn;
	lerna bootstrap --mutex file:/tmp/.yarn-mutex --concurrency=1;

The key is travis_retry where it will attempt the installation 3 times. So far we have 100% success.

[Aqours]

Same issue when run command lerna exec --parallel --stream -- npm i on node 12.13.0 and npm 6.12.0. And it works fine on node 10.10.0 and npm 6.4.1

[amitdahan]

Experiencing this as well.
Project consistently fails for the same local package.
After some troubleshooting, it seems some "latest"s in said package's dependencies, which I changed to ^1.0.0 (* also worked, but ^1.0.0 better fit our intent), caused this failure (which only surfaced during lerna bootstrap with concurrency set to more than 1).

I read that latest isn't supported in lerna, but only for local dependencies. This is not the case, so I'm not sure why this happened.

[plastictoast]

Same issue when run command lerna exec --parallel --stream -- npm i on node 12.13.0 and npm 6.12.0. And it works fine on node 10.10.0 and npm 6.4.1

In our case I narrowed the bug down to npm version 6.10.2. I can successfully run 6.10.1 with concurrency=8, whereas using 6.10.2 it fails every time. Looking at the changelog there were changes in this version to cache file ownership, so this sounds likely.

We're using node 12.13.0, although I'm not sure this is relevant.

(To be clear, I'm referring to the cache lock issue, e.g. ENOENT: no such file or directory, lchown '/Users/xxx/.npm/_locks/staging-5e1c07f26fab6251.lock')

[Aqours]

@plastictoast Great. It's seems the issue is caused by shared cache. Disable npm cache works fine, I find this workaround. lerna exec --parallel --stream -- npm i --cache ./your-cache-directory works fine.

[morewry]

We had this issue at times years ago, but hadn' in a long time. Started having it again every 2nd or 3rd build after we updated our Node.js and npm versions.

npm from various 6.5.x and 6.9.x to 6.13.4.
Node.js from 10.13.0 to 12.14.0.

In our case, we can't use lerna exec to run a vanilla npm install and use the flag to disable the cache, since that will try to install private internal dependencies that aren't published and error.

With 60 packages, our bootstraps need all the help they can get, so disabling concurrency is a perceptible difference 🤕. Nervous that it sound like the issues goes back to npm 6.10.2...

[UrielCh]

Hi

I have a 200+ sub modules project using lerna. here

---

if I call:
lerna exec --stream -- npm i --cache ./.cache
to populate the node_modules, lerna won't create any symlink.

---

if I call:
lerna bootstrap --concurrency=1 --ignore-scripts --force-local
then
lerna exec --stream -- npm i --cache ./.cache

The first bootstrap call will create the symlink but the npm call will overwrite them.

---

if I call:
lerna exec --stream -- npm i --cache ./.cache
then
lerna bootstrap --concurrency=1 --ignore-scripts --force-local

my node_modules will be populate with copy of the files
my prebuild script will compile my typescript files (using some old version of my modules)
and then the bootstrap will replace file copy by symlink, but the typescript transpiling had already be done, and failed, on case of breaking change.

---

so...

is there a wait to provide an think of npm i --cache ./.cache params to make works directly.

a single call to lerna bootstrap --concurrency=1 --force-local works, but take 20 minutes...

[morewry]

Recently tried enabling npm ci for all installs, including local ones, and haven't had the error since!

[jared-laethem]

Has any consideration been given to fixing this problem by implementing a retry on the npm installs?

I ran into a variation of this issue, a problem where an npm installs failed while running lerna bootstrap, which was resolved by setting parallelism setting to one. Only logging I got was a stack trace (no error message) pointing back to the execa library. I patched in some additonal logging into execa, enough to see that my npm install was exiting with a a SIGKILL.

Just wondering if an easy but effective fix/workaround here would be to retry once from lerna if an npm install fails with a SIGKILL?

[JamesHenry]

Hi Folks 👋

Please take a look at our published roadmap for Lerna v7 here: #3410

One of the key items covered at length on there (please do read it for full context) is that now that we find ourselves in late 2022, it no longer makes sense for lerna to supplement package management concerns (such as installation, boostrapping, linking etc) which are covered reliably for monorepo workspaces by the three main package managers: npm, yarn and pnpm. lerna bootstrap et al were developed in completely different era of the JavaScript ecosystem.

If you have any specific concerns please do join in on that discussion, and provide as much context as possible.

Many thanks 🙏