[Feature] Add page.waitForRequest() and page.waitForResponse()

[matthaywardwebdesign]

Hey!

After using Puppeteer for a little while now in a number of different scenarios (SSR, automated testing) a common piece of functionality that often came up as needed, or at very minimum a nice to have, was the ability to wait for a specific network request to be made, or alternatively for a specific network response to come back. As a result I propose the addition of the following two functions.

page.waitForRequest( match[, options])

• match <string> A regular expression to match against the request URL
• options <Object> Optional parameters
  • method <string|Array> Either a single HTTP request method (GET, POST, etc) as a string or an array of values ["GET", "POST"]. Defaults to all HTTP request methods.
  • timeout <number> Maximum time to wait in milliseconds

And additionally

page.waitForResponse( match[, options])

With a similar set of options. I'd probably also additionally add support for filtering by response code.

An example

await page.waitForRequest( 'https?://example.com/(users|companies)', { method: 'POST' });

Happy to submit a PR if you all think it's a worthwhile addition!

Thanks!

[dowsanjack]

I think this would be pretty helpful

[dowsanjack]

Kindly submit a Pull request 😄

[aslushnikov]

This looks reasonable. Feel free to submit a PR.

Let's pass RegExp object as the first argument if you want a regex, and a string if you want an exact url match.

// Wait for regex to match
const request = await page.waitForRequest(new RegExp('https?://example.com/(users|companies)'), { method: 'POST' });
// Wait for exact url
const request = await page.waitForRequest('https://example.com/');

[matthaywardwebdesign]

@aslushnikov @dowsanjack Thanks! Will get to it within the next couple of days :)

[StephanBijzitter]

Agreeing with @aslushnikov, but since there's some minor differences between RegExp objects and literals, please do test both :-)

[yanivefraim]

@matthaywardwebdesign - are you working on this? Can I help? (:

[yanivefraim]

@aslushnikov - I started implementing this feature, and I have a questions regarding the api:
Do we want to get an options object, with timeout, for example? (or is it enough to use default timeout)

[aslushnikov]

Do we want to get an options object, with timeout, for example?

@yanivefraim yes, options object with timeout will come handy. It will also make API consistent with other waitFor* methods.

[rodoabad]

@aslushnikov it looks like the second argument isn't really made for options like the issue was requesting?

const firstResponse = await page.waitForResponse('https://example.com/resource');
const finalResponse = await page.waitForResponse(response => response.url() === 'https://example.com' && response.status() === 200);
return finalResponse.ok();

Do we need to always return return finalResponse.ok();?