Semantic release blocked by always-failing Travis CI on merge push

[johnjbarton]

We have semantic-release setup but it never triggers because the Travis CI always fails on push.

I think the sequence is:

1. PR is ready, Merge button pushed.
2. Github signal merge commit pushed to master
3. Travis gives event, starts build with TRAVIS_EVENT_TYPE = "push"
4. test fail to contact browserstack browsers.
5. CI exits before running semantic-release.

The travis Build History is red for branch master going back months:
https://travis-ci.org/github/karma-runner/karma/builds?utm_medium=notification&utm_source=github_status

[Den-dp]

Looks like this one is a blocker for releasing a fix #3451 for #3447

[webmaster128]

The different CI behaviour between master and PRs is because

3.a Grunt task test:client is executed, running with test/client/karma.conf.js
3.b Browser selection is done depending on process.env.TRAVIS_SECURE_ENV_VARS === 'false' in https://github.com/karma-runner/karma/blob/master/test/client/karma.conf.js#L72-L76
3.c TRAVIS_SECURE_ENV_VARS is "false" for PRs from forks because "Encrypted environment variables have been removed for security reasons. See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions"

There are started but not captured browser stack sessions: https://travis-ci.org/github/karma-runner/karma/jobs/668845791#L702-L755. Can someone with the correct account look into those sessions?

[johnjbarton]

TRAVIS_SECURE_ENV_VARS is "false" for PRs from forks

I don't understand "PRs from forks". We seem to succeed on Travis when PR commits are tested, but then fail on Travis when those same commits are tests after merging into master. Is one or the other of these "PRs from forks"?

Maybe you are saying that TRAVIS_WITHOUT_BS is true for PR commit tests but false for tests after merging? So that the TRAVIS_WITHOUT_BS==false path fails so merge commits fail.

There are started but not captured browser stack sessions...
Yes, that is the issue here. But what does it mean to "look into those sessions"?

[webmaster128]

I don't understand "PRs from forks".

Each pull request has a source branch. This can be a branch from the original repository karma-runner/karma or from a fork, like in this case:

This is what I called a "PR from a fork". It appears that you create your own PRs from a personal fork as well, which this is not strictly necessary when you have write permissions and could create PR branches here directly.

Now there are secure environment variables in Travis, like BROWSER_STACK_ACCESS_KEY. Those are set in the Travis interface and are used in the build.

If those secure variables were available in PRs from forks, a random guy from the internet could submit a change like

diff --git a/.travis.yml b/.travis.yml
index 5295ece..96cca22 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -44,6 +44,10 @@ before_script:
 
 script:
   - npm run travis
+  - echo "browser stack $BROWSER_STACK_ACCESS_KEY"
+  - echo "npm $NPM_TOKEN"
+  - echo "github $GH_TOKEN"
+  - echo "🦹"
 
 after_success:
   # run automated release process with semantic-release

and let Travis esecute it automatically. Suddenly all your credentials are leaked. In order to prevent that, Travis makes those secure environment variables unavailable in PRs from forks. All builds created from this repo directly (i.e. master builds, branch builds and PRs from branches) are considered trusted and have access to secure variables. The original documentation by Travis is available here.

Maybe you are saying that TRAVIS_WITHOUT_BS is true for PR commit tests but false for tests after merging?

Right. And this is because TRAVIS_WITHOUT_BS depends on TRAVIS_SECURE_ENV_VARS.

Yes, that is the issue here. But what does it mean to "look into those sessions"?

The Travis log of a master build shows URLs of browser stack sessions that were started, like e.g. https://automate.browserstack.com/builds/7c418d1aa417fa54d51f6d48bebdb5f85fd82b8c/sessions/aecaf26c351de3a75a637fa7731c51fe40326a94. I cannot access those links as I get a Login form only. I think there might be useful information available behind those links.

[johnjbarton]

Ok excellent, thanks! I can't use that link either but I know how to reset the travis values to ones I can access. I will be able to do that Monday.

[johnjbarton]

Ok we've made progress. I switched the browserstack username to one controlled by karmarunnerbot, the project identity. Keen then figured out how to configure the karma-runner project for open-source support on browserstack. With that we do pass post-commit ("push") tests: https://travis-ci.org/github/karma-runner/karma/jobs/671745396?utm_medium=notification&utm_source=github_status

Unfortunately we are not done. The install of semantic-release fails with an obscure error:

npm ERR! enoent ENOENT: no such file or directory, rename '/home/travis/build/karma-runner/karma/node_modules/watchify/node_modules/fsevents/node_modules/debug' -> '/home/travis/build/karma-runner/karma/node_modules/@semantic-release/github/node_modules/debug'
npm ERR! enoent This is related to npm not being able to find a file.

I attempted to reproduce this locally but it looks like the issue is OS dependent. When installing on a mac I see

[fsevents] Success: "/work/karma/node_modules/watchify/node_modules/fsevents/lib/binding/Release/node-v64-darwin-x64/fse.node" is installed via remote

So the install works on my mac, so I can't debug the issue there.

The package-lock.json shows two different versions of fsevents: 1.2.9 imported by chokidar 2.1.6 and 2.0.7 imported by chokidar 3.0.2. Normally this is not a problem but since fsevents seems to have an OS-dependent feature, two different major versions may step on each other.

[johnjbarton]

Looks like this is a known issue browserify/watchify#374

[devoto13]

To me it looks more like NPM having hard time to build a dependency tree because semantic-release packages are installed ad-hoc. Let's see if #3455 has the same problem.

UPD Yup, packages are installed successfully.

[johnjbarton]

It looks like grunt-browserify pulls both browserify and watchify but we never invoke watchify.

[johnjbarton]

Well, I thought all the problems were solved. Both browserstack and saucelabs now work in the new config (except for one flaky proxy test which needs to be fixed).

The release fails on git push tag. This implies that the GH_TOKEN is incorrect. I re-added it but it did not fix us:

  command: 'git push --tags https://[secure]@github.com/karma-runner/karma.git HEAD:master',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: '',
  stderr: 'remote: error: GH006: Protected branch update failed for refs/heads/master.        \n' +
    'remote: error: 3 of 3 required status checks are expected. At least 1 approving review is required by reviewers with write access.        \n' +
    'To https://github.com/karma-runner/karma.git\n' +
    ' ! [remote rejected] HEAD -> master (protected branch hook declined)\n' +
    "error: failed to push some refs to 'https://[secure]@github.com/karma-runner/karma.git'",

I checked and karmarunnerbot has write access.

The master branch on karma-runner/karma has "branch protection" unlike karma-jasmine. So maybe that is why we fail. I set karmarunnerbot as a branch exception.

[johnjbarton]

At least 1 approving review is required by reviewers with write access. 

That will never be true. So the semantic-release can't work for this kind of protected branch thing.

[johnjbarton]

AFAICT the karmarunnerbot has to be a project Admin, not Maintainer. (The github protected branches are confusing and buggy0.