Switch integration tests runner from Jest to Mocha

[medikoo]

There are various problems with Jest:

• It runs tests in copied/hacked Node.js environment which introduces some issues (not patchable)
  • stdout and stderr are output of out sync stdout and stderr are written to console out of sync jestjs/jest#6718
  • Some public Node.js API's are replaced with non standard alternatives -> Broken access to Node.js modules cache jestjs/jest#6725
  • Due to non standard and uneven modules cache handling it's prone to OOM errors: Jest memory problems in node environment jestjs/jest#6399 (comment)
• Tests progress is not output until tests finalize, which is inconvenient in long going tests (as integration ones) -> Report progress of individual test cases jestjs/jest#6616 (not easily patchable)
• Any stdout/stderr output is hidden until test finalizes -> Delayed output from tests jestjs/jest#5281 (patchable with useStderr option)
• Default runner exposes various bugs (patchable via using alternative jest-circus runner)
  • Crashes in beforeAll do not prevent execution of tests -> Tests are run even when beforeAll throws error jestjs/jest#2713, failing beforeAll() causes even passing tests in the scope to fail jestjs/jest#6695
  • Crashes in afterAll are not exposed -> afterAll inside a describe hides errors jestjs/jest#6692
  • only and skip are not fully respected -> Nested BeforeAlls execute when marking higher level tests as only jestjs/jest#4166
• Source code is result of transpilation which makes it more difficult to debug eventual issues in test runner.

Mocha is also not free from problems (still all known to me are patchable). It also doesn't provide test run parallelization and proper isolation, but we have achieved that through script/test-isolated.js

---

Additionally:

• Improved script/test-isolated.js to handle well case of integration tests
• Updated tests to not change cwd during run (it interfered with cwd change validation check we implied into Mocha runner)
• Fixed service name resolution in basic integration test. It got broken with 868c8a3 (until then it was simply aws-nodejs, after it reflected folder name). Uncommented also test run in Travis CI

Is it a breaking change?: NO

[pmuens]

Just looked through the code and it looks good 👍

Great to see only one test runner being used. While testing the log format was a little bit off and the logs were only shown once the test was done. I'm fine with that but I really enjoyed the live log-streaming of Jest (with lines in the codebase). This helped a lot debugging the integration tests since I could e.g. immediately cat the serverless.yml file in the temporary directory without waiting for the test to finish.

Since this is something more Jest vs. Mocha related I'd say that we're good and there's no need to complicate things via Monkey-patching.

[pmuens]

Is there any way to set the timeout globally for all integration tests?

[medikoo]

There's nothing in Mocha that allows to set directory specific settings, and I don't think it's good to have it set to this value globally (e.g. that's how it was set with Jest)

Still I think it's not that big drawback, also timeout across integration tests may differ. Here we deal with CF deploys which are very long, but in case of packaging where there are only SDK calls such long timeout is not needed.

[medikoo]

I'm fine with that but I really enjoyed the live log-streaming of Jest

Jest by default doesn't output anything (apart of progress bar) until test finalizes. We actually kinda hacked jest with --useStderr option, to output on the go (see jestjs/jest#5281 (comment)), and it's result of that which you enjoyed :)

Still there's actually a problem with this output even here. We're still not notified of individual test fails while test goes (no live stream of that), and what's worse it mixes output from all tests being run.

While in our current case it's not that eminent (yet), In some scenarios it prevents to reliably debug a test fail, as we can't clearly see which exactly logs came from given test run.

With newly configured Mocha version, in parallel run output for given test is flushed once test finalizes (so you can easily see which log output is from which test, as they appear one after another)

While when single test file is run, all output is live streamed on the go, together with tests results info (that part is missing on Jest side, no matter what -> jestjs/jest#6616).

So if you want to have output on the go, just run one test file that focuses on functionality you're working on, and you'll have then even better output than Jest gave.

Alternatively you may also run multiple test files but sequentially via npx mocha file1.test.js file2.test.js that will also give output on the go.

Providing output on the go with parallelization is involved, to be reliable will have to involve some window splitting I believe. I'm not sure if ability to consume mixed output from different test runs can be considered as worthwhile feature.

(with lines in the codebase).

Yes, this is how Jest decorates console.* writers. I think it can be achieved via some dedicated package that focuses just on that, and we can configure that if we will really miss that feature.

[pmuens]

Thanks for the in-depth explanation @medikoo 👍