testcafe CLI exits before asynchronous task in reportTaskDone of custom reporter completes

[ajeetd]

I have created a contrived example to reproduce this easily. Although no one would do this exactly, with a time out, it demonstrates an asynchronous task that takes an indeterminate amount of time to complete.

Reproduction steps:

1. Create a custom reporter containing:

module.exports = function () {  
    return {
        timeout: function (ms) {
            return new Promise(resolve => setTimeout(resolve, ms));
        },

        async reportTaskDone (/* endTime, passed, warnings */) {
                console.log('Expect to be here for 60 seconds');
                await this.timeout(60000);
                console.log('Should see this after 60 seconds');
        }
    }
}

2. Execute: testcafe chrome Tests.js -r customReporter , where Tests.js contains tests and customReporter is the custom reporter from 1.

Actual Result

TestCafe exits almost immediately after tests complete and before 'Should see this after 60 seconds' is output.

Expected Result

Should see both console logs output with an approximate 60 seconds interval between them.

This issue was discussed here: https://testcafe-discuss.devexpress.com/t/no-built-in-reporting-of-results-especially-failures-by-email/993/12

Specifically see comment:

AndreyBelymApr '18
Hello @neekfenwick!

I had a talk with @amoskovkin yesterday, and we came to the conclusion that we can turn reportTaskDone into an async method easily. Transforming other reporter methods will be harder to implement, because the reporting subsystem is event-driven and it doesn't support asynchronous event handles.

So with the asynchronous reportTaskDone you will be able to prevent process termination until email server will finish message sending. Is it sufficient for you?

Currently the only available workaround is to write your own custom test runner using TestCafe API2. In your own runner you can terminate process only if all tests are finished and email server finished its job.

And I see this open task here: #2753 but I think some clarification is needed because the documentation for custom reporter functions seems to indicate async/await is already supported: https://devexpress.github.io/testcafe/documentation/extending-testcafe/reporter-plugin/reporter-methods.html

The changelog here: https://www.libupdate.com/libs/192a126f-308f-4b25-86e8-ecbecbabd350 for V1.0.0 also indicates this is implemented: https://www.libupdate.com/libs/192a126f-308f-4b25-86e8-ecbecbabd350 :

TestCafe v1.0.0 also introduces asynchronous API for reporter plugins.

Similarly to request hooks, if any of the custom reporter's methods (reportTaskStart, reportFixtureStart, reportTestDone or reportTaskDone) returns a Promise, this Promise is now awaited.

Side effects may show up in certain cases.

Since the reporter methods are now asynchronous, add the async keyword to their declarations.

Reporters can call asynchronous fs functions, invoke a child_process, or perform asynchronous network requests (to send an email, use REST API, connect to a database, etc).

[AndreyBelym]

Thank you for your report. I've reproduced the problem. It's a bug, and we will fix it in the current development sprint.

[ajeetd]

Thank you Andrey. Look forward to it and I will be ready to test the fix. Your sprints look like they are open ended with respect to dates (e.g. current one https://github.com/DevExpress/testcafe/milestone/50), do you provide ETAs?

[AndreyBelym]

Thank you, I've set the due date for the milestone. The current sprint ends on July 10 and we will release a new TestCafe version at the end of the sprint.

[lock]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.