Puppeteer does not follow javascript redirects - leads to infinite loading of page without timeout

[DaveRingelnatz]

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 1.11.0

• Platform / OS version: Mac OS, 10.13.6

• URLs (if applicable):

  • http://clkuk.tradedoubler.com/click?p=190137&a=3063988&g=22260412&url=https://eu.puma.com/de/de/home/&epi=dede-shopping-stripe

• Node.js version: 6.1.0

What steps will reproduce the problem?

const puppeteer = require('puppeteer');

let browser;
let page;

(async() => {
  let config = {
        args: ['--disable-gpu', '--no-sandbox', '--disable-setuid-sandbox', '--ignoreHTTPSErrors']
    };

  browser = await puppeteer.launch(config);
  console.log(await browser.version());
  page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on('request', interceptedRequest => {
    if (interceptedRequest.resourceType() === 'media') {
      log("request aborted");
      interceptedRequest.abort();
    }
    else {
      interceptedRequest.continue();
    }
  });

  async function collectLinks() {
    // don't use toJSON method of website -> delete it if existing
    delete Array.prototype.toJSON;

    // HTML4 elements
    let a = Array.from(document.querySelectorAll('a')).map((val) => val.href);

    // compose report object
    let dataObject = {};
    dataObject['a'] = JSON.stringify(a);
    //console.log(dataObject);
    return dataObject;
  }

  let response = await page.goto('http://clkuk.tradedoubler.com/click?p=190137&a=3063988&g=22260412&url=https://eu.puma.com/de/de/home/&epi=dede-shopping-stripe');
  await page.waitFor(5000);
  const status = response.status();
  if ((status >= 300) && (status <= 399)) {
    console.log('Redirect from', response.url(), 'to', response.headers()['location'])
  }
  let dataObject = await page.evaluate(collectLinks).catch(console.log);
  console.log(dataObject);
  console.log(response.request().redirectChain());
  await page.close();
  await browser.close();
})();

What is the expected result?

The page.evaluate function should either throw an error or a timeout should occur. But neither happens. Alternative solution: puppeteer should follow the javascript redirects; that would also solve the problem.

What happens instead?

The script is running in an endless loop, cause the page is not fully loaded or rather the redirects in form of javascript redirects is not followed. If I open the page above in my Chrome I got redirected to this url. If the evaluate script would run on this url, everything would be fine. Unfortunately the redirects don't work with puppeteer for some reasons.

[samgooi4189]

Having the same issue.

The URL I visit is https://drive.google.com/?tab=io.
Puppeteer is throwing error, because there is a Javascript redirect rather than 302 redirect.

[DaveRingelnatz]

I would be happy if puppeteer would throw an error in my case. :)
My problem is the endless loop I stuck in.

[aslushnikov]

I can reproduce this with the following script:

const puppeteer = require('puppeteer');

(async() => {
  const browser = await puppeteer.launch({
    ignoreHTTPSErrors: true,
  });
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on('request', r => r.continue());

  await page.goto('http://clkuk.tradedoubler.com/click?p=190137&a=3063988&g=22260412&url=https://eu.puma.com/de/de/home/&epi=dede-shopping-stripe');
  await page.waitFor(5000);
  await page.evaluate(() => 7 * 8); // hangs here
  await browser.close();
})();

The core issue here is again request interception - #3471, since our heuristic fails to match interception with requestWillBeSent.

I'll try to relax the heuristic a little more for now and see if it still holds in the wild.