Fix the stability of the Playwright E2E tests on the CI

[jmcameron]

The Playwright E2E tests are currently disabled in the Semaphore CI. This is because the E2E tests unstable in the CI although they work locally (usually) on my laptop.

There are several issues

• Running a E2E test that involves many tests seems to produce more and more instability. The E2E tests are currently split into 2 parts (stock tests and non-stock tests). The E2E stock tests, which number about 80, seem to run stably most of the time. But the rest which number of 400 rarely pass successfully even thought they (usually) run successfully locally.
• The way Playwright works for waiting for page loads, etc, seems to work well when only one directory is tested, and generally work well locally. But in the CI, tests often time out. This may be because the machine is slow, but I suspect we do not have something quite correct about how we are waiting for page loads, dialogs to come up, and seeing grid results when submitting a search.
• Playwright is currently actively supported for the current Angular. We are lucky it seems to work with our old Angular. It is possible that some of these problems are due to this issue.

Suggestions

• Work with the tests in the CI to get them running reliably.
• It may be necessary to further divide the E2E tests so that no particular run is handling more that 100 or so tests. This may help significantly. This can be done by moving groups of tests in to subdirectories and then using the current mechanism to run each directory/group separately.
• Many tests are currently being skipped because the run fine when the E2E test is run on a single directory, but fail when run with other directories locally. But some tests are skipped because the work fine singly and with other tests locally but fail regularly on the CI. These tests need to be re-enabled and we need to figure out how to improve the page waiting mechanisms.

[jniles]

I think #7219 will really help with this. In my experience with other frameworks having setup/teardown scripts gives much more predictable behavior/load/resource consumption than with before() or beforeAll() hooks, and that might help the CIs behave more predictably.

[jniles]

@jmcameron what progress is made towards this?

[jmcameron]

@jmcameron what progress is made towards this?

From the suggestions above:

• Work with the tests in the CI to get them running reliably.
  • The Jenkins server is running regularly now, once an hour. I've been taking notes on which tests fail regularly and plan to skip some soon.
• It may be necessary to further divide the E2E tests so that no particular run is handling more that 100 or so tests.
  • This has been done.
• Many tests are currently being skipped because the run fine when the E2E test is run on a single directory, but fail when run with other directories locally. But some tests are skipped because the work fine singly and with other tests locally but fail regularly on the CI. These tests need to be re-enabled and we need to figure out how to improve the page waiting mechanisms.
  • Have not followed up on this

Another item that needs to be done it to set up the global login so each test does not need to login as part of the test. See Issue #7219