JUnit formatter should output passing <test> entries for rules that pass

[brettdh]

Related: #9093, #9094

#9094 introduced junit output for the all-rules-passing case, which fixed a problem with Jenkins. However, CircleCI displays no test summary for eslint if the Junit XML contains only empty <testsuite>s and no <test>s.

Tell us about your environment

• ESLint Version: 4.9.0
• Node Version: 6.11.2
• npm Version: 5.5.1

./node_modules/.bin/eslint index.js ./src/* ./__tests__/* --format junit

What did you expect to happen?
A non-zero number of (passing) <test>s in the XML output - one per rule per file (<testsuite>).

What actually happened? Please include the actual, raw output from ESLint.
Many <testsuite>s, all of which contained zero <test>s.

[platinumazure]

Does the formatter need to expose a test element per rule per file? Or just one passing test per suite/file?

Having a test element for every file sounds verbose to me, but I don't know JUnit.

[j-f1]

What about putting just one “linting passed” test in the suite if the file has no errors or warnings?

[brettdh]

Note: I mistakenly wrote <test>; the correct element is <testcase>.

It seems cleaner (and yes, verbose) to me to have a <testcase> per rule per file, so that the test result XML contains a <testcase> for each test that was actually run.

Glancing at the code in #9094, it looks like there's a <testcase> per message per file, which I understand to be failed test cases. It seems more consistent to have individual success elements to match the individual failure elements. This also lets you look back and see the first build that introduced a rule.

A single passing <testcase> per suite on success would be better than none, though. I'm not sure how tools like CircleCI would behave if <testcase>s disappear after being fixed rather than just being set to "passed" as is the case for unit test reporters like jest-junit.

[platinumazure]

@brettdh Would it be possible to link to a section of the JUnit specification (if there is one) to describe the correct behavior here? I'm honestly wondering if we should have only one testcase per rule per file even on failure (and put multiple error messages in a description, or assertion/failure, or something), and then have one testcase with no failures to indicate success...

[brettdh]

That would make test results a lot less useful. Tools like Jenkins and CircleCI use the individual test cases to track failures over time by testcase name, and they assume a <testcase> is one test case - since it's the smallest unit of testing that the xml report allows.

A quick google search turns up lots of possible specs (including some XSD files). Not sure if any are authoritative. Maybe one of these?

[not-an-aardvark]

Thanks for your interest in improving ESLint. Unfortunately, it looks like this issue didn't get consensus from the team, so I'm closing it. We define consensus as having three 👍s from team members, as well as a team member willing to champion the proposal. This is a high bar by design -- we can't realistically accept and maintain every feature request in the long term, so we only accept feature requests which are useful enough that there is consensus among the team that they're worth adding.

Since ESLint is pluggable and can load custom formatters at runtime, the lack of consensus among the ESLint team doesn't need to be a blocker for you using this in your project, if you'd find it useful. It just means that you would need to implement the formatter yourself, rather than using a bundled formatter that is packaged with ESLint.