Appveyor build is flaky

[not-an-aardvark]

For the past few weeks, the Appveyor build has been occasionally been failing with messages like this:

It's not clear what's going on. (It says "no coverage files found", but I'm not sure why that's the case, or why it only happens sometimes.)

[not-an-aardvark]

The "ERROR: no coverage files found" message occurs when this command is run and no coverage/ folder was generated by the above command. However, I'm not sure why that would be happening.

[aladdin-add]

it happened occasionally and I've never seen it in my local, so I guess it might be related to mocha timeout. can we set it to be bigger to see if it happens again?

[not-an-aardvark]

I noticed that the error usually happens when the progress bar is at this stage:

[▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬..]

I think that's the point where the tests in tests/bin/eslint.js tests run. It makes sense that the tests/bin/eslint.js tests could be causing the problem, because:

• Those tests use child_process.fork(), which could have odd side-effects
• Those tests take longer than other tests

[not-an-aardvark]

I don't think the issue is a timeout, because when I apply this diff:

diff --git a/tests/bin/eslint.js b/tests/bin/eslint.js
index 7742eb71..830672e5 100644
--- a/tests/bin/eslint.js
+++ b/tests/bin/eslint.js
@@ -46,7 +46,9 @@ function getOutput(runningProcess) {
     return awaitExit(runningProcess).then(() => ({ stdout, stderr }));
 }
 
-describe("bin/eslint.js", () => {
+describe("bin/eslint.js", function() {
+    this.timeout(200); // eslint-disable-line no-invalid-this
+
     const forkedProcesses = new Set();
 
     /**

...I get a bunch of error messages that the test timed out. On Appveyor, there are no error messages.

[not-an-aardvark]

It's also interesting that the test bar stops at [▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬..] and it never reaches the end, and then the "ERROR: no coverage files found" is printed immediately afterwards. Under normal circumstances, a coverage/ folder is created after the tests finish, and then istanbul check-coverage is run to verify the coverage. If the coverage folder doesn't have any coverage info yet (e.g. because the tests are still running), then istanbul will output "ERROR: no coverage files found".

So my guess is that when we do child_process.fork() in tests/bin/eslint.js and then the forked process stops, somehow that is getting interpreted as if the parent test process stopped, so the Makefile.js script continues even though the tests haven't finished.

[not-an-aardvark]

Looks like this is still a problem (e.g. the build failed for #9589)

[not-an-aardvark]

Should this be closed? I haven't seen the build fail for awhile.

[ilyavolodin]

I think it's fine to close it. Occasional errors in the build happens. We've had them on Travis during last release, for example. I don't think there a way to completely eliminate them. As long as they are within reasonable boundaries, we just have to live with them.

[aladdin-add]

a timeout error was happening today. https://ci.appveyor.com/project/nzakas/eslint/build/9677