feat: introduce puppeteer-firefox (#3628)

This adds a proof-of-concept of `puppeteer-firefox`.
This consists of two parts:
- `//experimental/juggler` - patches to apply to Firefox.
- `//experimental/puppeteer-firefox` - front-end code to
be merged with Puppeteer.

As things become more stable, we'll gradually move it out of
the experimental folder.

.eslintignore

@@ -6,3 +6,4 @@ utils/testrunner/examples/
 node6/*
 node6-test/*
 node6-testrunner/*
+experimental/

experimental/README.md

@@ -0,0 +1,8 @@
+#### What's this?
+
+This is a **highly experimental** Puppeteer API to drive Firefox browser.
+
+Beware, alligators live here.
+
+`/juggler` - firefox-puppeteer backend
+`/puppeteer-firefox` - puppeteer API for Firefox

experimental/juggler/.cirrus.yml

@@ -0,0 +1,38 @@
+task:
+  timeout_in: 120m
+  env:
+    CIRRUS_WORKING_DIR: /usr/local/src
+    SOURCE: /usr/local/src/
+    GS_AUTH: ENCRYPTED[c4b5b0404f5bfdc1b663a1eb5b70f3187b5d470d02eec3265b06b8e0d30226781523630931c1da6db06714c0d359f71f]
+    PATH: /root/.cargo/bin:$PATH:$SOURCE/gcloud/google-cloud-sdk/bin
+    SHELL: /bin/bash
+  container:
+    dockerfile: Dockerfile
+    # image: ubuntu
+    cpu: 8
+    memory: 24
+  name: linux
+  # install_deps_script: apt-get update && apt-get install -y wget python clang llvm git curl
+  install_gcloud_script: ./scripts/install_gcloud.sh
+  check_gcloud_script:
+    - echo "REVISION: $(git rev-parse HEAD)"
+    - gsutil cp FIREFOX_SHA gs://juggler-builds/$(git rev-parse HEAD)/
+  clone_firefox_script: ./scripts/fetch_firefox.sh
+  apply_patches_script:
+    - cd $SOURCE/firefox
+    - git config --global user.name "cirrus-ci-builder"
+    - git config --global user.email "aslushnikov@gmail.com"
+    - git am ../patches/*
+    - ln -s $PWD/../juggler testing/juggler
+  bootstrap_firefox_script:
+    - cd $SOURCE/firefox
+    - ./mach bootstrap --application-choice=browser --no-interactive
+  build_firefox_script:
+    - cd $SOURCE/firefox
+    - ./mach build
+  package_firefox_script:
+    - cd $SOURCE/firefox
+    - ./mach package
+  upload_build_to_gcloud_script:
+    - bash $SOURCE/scripts/upload_linux.sh
+

experimental/juggler/.gitignore

@@ -0,0 +1 @@
+firefox/

experimental/juggler/Dockerfile

@@ -0,0 +1,27 @@
+FROM ubuntu:trusty
+
+MAINTAINER Andrey Lushnikov <aslushnikov@gmail.com>
+ENV SHELL=/bin/bash
+
+# Install generic deps
+RUN apt-get update -y && apt-get install -y wget python clang llvm git curl
+
+# Install gcc7 (mach requires 6.1+)
+RUN apt-get update -y && \
+    apt-get upgrade -y && \
+    apt-get dist-upgrade -y && \
+    apt-get install build-essential software-properties-common -y && \
+    add-apt-repository ppa:ubuntu-toolchain-r/test -y && \
+    apt-get update -y && \
+    apt-get install gcc-7 g++-7 -y && \
+    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7 && \
+    update-alternatives --config gcc
+
+# Install llvm 3.9.0 (mach requires 3.9.0+)
+RUN echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main" >> /etc/apt/sources.list && \
+    echo "deb-src http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main" >> /etc/apt/sources.list && \
+    apt-get install clang-3.9 lldb-3.9 -y
+
+# Install python 3.6 (mach requires 3.5+)
+RUN add-apt-repository ppa:deadsnakes/ppa -y && \
+    apt-get update -y && apt-get install python3.6 -y

experimental/juggler/FIREFOX_SHA

@@ -0,0 +1 @@
+663997bb1dd09a5d93135b1707feb59024eb9db4

experimental/juggler/README.md

@@ -0,0 +1,77 @@
+# Juggler
+
+> Juggler - Firefox Automation Protocol for implementing the Puppeteer API.
+
+## Protocol
+
+See [`//src/Protocol.js`](https://github.com/GoogleChrome/puppeteer/blob/master/experimental/juggler/src/Protocol.js).
+
+## Building FF with Juggler
+
+1. Clone Juggler repository
+```bash
+git clone https://github.com/aslushnikov/juggler
+cd juggler
+```
+
+2. Checkout [pinned Firefox revision](https://github.com/aslushnikov/juggler/blob/master/FIREFOX_SHA) from mozilla [github mirror](https://github.com/mozilla/gecko-dev) into `//firefox` folder.
+
+```bash
+SOURCE=$PWD bash scripts/fetch_firefox.sh
+```
+
+3. Apply juggler patches to Firefox source code
+
+```bash
+cd firefox
+git am ../patches/*
+ln -s $PWD/../src $PWD/testing/juggler
+```
+
+4. Bootstrap host environment for Firefox build and compile firefox locally
+
+```bash
+# OPTIONAL - bootstrap host environment.
+./mach bootstrap --application-choice=browser --no-interactive
+# Compile browser
+./mach build
+```
+
+## Running Firefox with Juggler
+
+Juggle adds a `-juggler` CLI flag that accepts a port to expose a remote protocol on.
+Pass `0` to pick a random port - Juggler will print its port to STDOUT.
+
+```
+./mach run -- -juggler 0
+```
+
+## Uploading builds to Google Storage
+
+Firefox builds with Juggler support are uploaded to gs://juggler-builds/ bucket.
+
+Project maintainers can upload builds.
+To upload a build, do the following:
+
+1. Install [gcloud](https://cloud.google.com/sdk/install) if you haven't yet.
+2. Authenticate in the cloud and select project
+
+```bash
+gcloud auth login
+gcloud config set project juggler-builds
+```
+
+3. Make sure **firefox is compiled**; after that, package a build for a redistribution:
+
+```bash
+cd firefox
+./mach package
+```
+
+4. Archive build and copy to the gbucket
+
+We want to ship `*.zip` archives so that it's easy to decompress them on the node-side.
+
+- Linux: `./scripts/upload_linux.sh`
+- Mac: `./scripts/upload_mac.sh`
+

experimental/juggler/patches/0001-Introduce-nsIWebProgressListener2-onFrameLocationCha.patch

@@ -0,0 +1,160 @@
+From fb96032ad20cb0dc5fbabe52a80d13d6e6808bb8 Mon Sep 17 00:00:00 2001
+From: Andrey Lushnikov <lushnikov@chromium.org>
+Date: Tue, 27 Nov 2018 13:37:12 -0800
+Subject: [PATCH 1/3] Introduce nsIWebProgressListener2::onFrameLocationChange
+ event
+
+The event is fired when subframes commit navigation.
+Juggler uses this event to track same-document iframe navigations.
+---
+ docshell/base/nsDocShell.cpp                  |  1 +
+ .../statusfilter/nsBrowserStatusFilter.cpp    | 10 ++++++++
+ uriloader/base/nsDocLoader.cpp                | 20 ++++++++++++++++
+ uriloader/base/nsDocLoader.h                  |  5 ++++
+ uriloader/base/nsIWebProgress.idl             |  7 +++++-
+ uriloader/base/nsIWebProgressListener2.idl    | 23 +++++++++++++++++++
+ 6 files changed, 65 insertions(+), 1 deletion(-)
+
+diff --git a/docshell/base/nsDocShell.cpp b/docshell/base/nsDocShell.cpp
+index ea0926732..3f738d39c 100644
+--- a/docshell/base/nsDocShell.cpp
++++ b/docshell/base/nsDocShell.cpp
+@@ -1349,6 +1349,7 @@ nsDocShell::SetCurrentURI(nsIURI* aURI, nsIRequest* aRequest,
+     mLSHE->GetIsSubFrame(&isSubFrame);
+   }
+
++  FireOnFrameLocationChange(this, aRequest, aURI, aLocationFlags);
+   if (!isSubFrame && !isRoot) {
+     /*
+      * We don't want to send OnLocationChange notifications when
+diff --git a/toolkit/components/statusfilter/nsBrowserStatusFilter.cpp b/toolkit/components/statusfilter/nsBrowserStatusFilter.cpp
+index c4d04dcc4..bb9e40cca 100644
+--- a/toolkit/components/statusfilter/nsBrowserStatusFilter.cpp
++++ b/toolkit/components/statusfilter/nsBrowserStatusFilter.cpp
+@@ -188,6 +188,16 @@ nsBrowserStatusFilter::OnStateChange(nsIWebProgress *aWebProgress,
+     return NS_OK;
+ }
+
++
++NS_IMETHODIMP
++nsBrowserStatusFilter::OnFrameLocationChange(nsIWebProgress *aWebProgress,
++                                             nsIRequest *aRequest,
++                                             nsIURI *aLocation,
++                                             uint32_t aFlags)
++{
++  return NS_OK;
++}
++
+ NS_IMETHODIMP
+ nsBrowserStatusFilter::OnProgressChange(nsIWebProgress *aWebProgress,
+                                         nsIRequest *aRequest,
+diff --git a/uriloader/base/nsDocLoader.cpp b/uriloader/base/nsDocLoader.cpp
+index 524681ad8..68d3f976c 100644
+--- a/uriloader/base/nsDocLoader.cpp
++++ b/uriloader/base/nsDocLoader.cpp
+@@ -1330,6 +1330,26 @@ nsDocLoader::FireOnLocationChange(nsIWebProgress* aWebProgress,
+   }
+ }
+
++void
++nsDocLoader::FireOnFrameLocationChange(nsIWebProgress* aWebProgress,
++                                       nsIRequest* aRequest,
++                                       nsIURI *aUri,
++                                       uint32_t aFlags)
++{
++  NOTIFY_LISTENERS(nsIWebProgress::NOTIFY_FRAME_LOCATION,
++    nsCOMPtr<nsIWebProgressListener2> listener2 =
++      do_QueryReferent(info.mWeakListener);
++    if (!listener2)
++      continue;
++    listener2->OnFrameLocationChange(aWebProgress, aRequest, aUri, aFlags);
++  );
++
++  // Pass the notification up to the parent...
++  if (mParent) {
++    mParent->FireOnFrameLocationChange(aWebProgress, aRequest, aUri, aFlags);
++  }
++}
++
+ void
+ nsDocLoader::FireOnStatusChange(nsIWebProgress* aWebProgress,
+                                 nsIRequest* aRequest,
+diff --git a/uriloader/base/nsDocLoader.h b/uriloader/base/nsDocLoader.h
+index 2dc1d0cae..05f8b2877 100644
+--- a/uriloader/base/nsDocLoader.h
++++ b/uriloader/base/nsDocLoader.h
+@@ -167,6 +167,11 @@ protected:
+                               nsIURI *aUri,
+                               uint32_t aFlags);
+
++    void FireOnFrameLocationChange(nsIWebProgress* aWebProgress,
++                              nsIRequest* aRequest,
++                              nsIURI *aUri,
++                              uint32_t aFlags);
++
+     MOZ_MUST_USE bool RefreshAttempted(nsIWebProgress* aWebProgress,
+                             nsIURI *aURI,
+                             int32_t aDelay,
+diff --git a/uriloader/base/nsIWebProgress.idl b/uriloader/base/nsIWebProgress.idl
+index 0549f32e1..3078e35d7 100644
+--- a/uriloader/base/nsIWebProgress.idl
++++ b/uriloader/base/nsIWebProgress.idl
+@@ -84,17 +84,22 @@ interface nsIWebProgress : nsISupports
+    * NOTIFY_REFRESH
+    *   Receive onRefreshAttempted events.
+    *   This is defined on nsIWebProgressListener2.
++   *
++   * NOTIFY_FRAME_LOCATION
++   *   Receive onFrameLocationChange events.
++   *   This is defined on nsIWebProgressListener2.
+    */
+   const unsigned long NOTIFY_PROGRESS       = 0x00000010;
+   const unsigned long NOTIFY_STATUS         = 0x00000020;
+   const unsigned long NOTIFY_SECURITY       = 0x00000040;
+   const unsigned long NOTIFY_LOCATION       = 0x00000080;
+   const unsigned long NOTIFY_REFRESH        = 0x00000100;
++  const unsigned long NOTIFY_FRAME_LOCATION = 0x00000200;
+
+   /**
+    * This flag enables all notifications.
+    */
+-  const unsigned long NOTIFY_ALL            = 0x000001ff;
++  const unsigned long NOTIFY_ALL            = 0x000002ff;
+
+   /**
+    * Registers a listener to receive web progress events.
+diff --git a/uriloader/base/nsIWebProgressListener2.idl b/uriloader/base/nsIWebProgressListener2.idl
+index 87701f8d2..8a69e6b29 100644
+--- a/uriloader/base/nsIWebProgressListener2.idl
++++ b/uriloader/base/nsIWebProgressListener2.idl
+@@ -66,4 +66,27 @@ interface nsIWebProgressListener2 : nsIWebProgressListener {
+                              in nsIURI aRefreshURI,
+                              in long aMillis,
+                              in boolean aSameURI);
++
++  /**
++   * Called when the location of the window or its subframes changes.  This is not
++   * when a load is requested, but rather once it is verified that the load is
++   * going to occur in the given window.  For instance, a load that starts in a
++   * window might send progress and status messages for the new site, but it
++   * will not send the onLocationChange until we are sure that we are loading
++   * this new page here.
++   *
++   * @param aWebProgress
++   *        The nsIWebProgress instance that fired the notification.
++   * @param aRequest
++   *        The associated nsIRequest.  This may be null in some cases.
++   * @param aLocation
++   *        The URI of the location that is being loaded.
++   * @param aFlags
++   *        This is a value which explains the situation or the reason why
++   *        the location has changed.
++   */
++  void onFrameLocationChange(in nsIWebProgress aWebProgress,
++                        in nsIRequest aRequest,
++                        in nsIURI aLocation,
++                        [optional] in unsigned long aFlags);
+ };
+-- 
+2.19.0.605.g01d371f741-goog
+

experimental/juggler/patches/0002-Add-Juggler-to-gecko-build-system.patch

@@ -0,0 +1,24 @@
+From c6f975dbc28b902cc271f79dedc42073ab1bde7d Mon Sep 17 00:00:00 2001
+From: Andrey Lushnikov <lushnikov@chromium.org>
+Date: Tue, 27 Nov 2018 13:39:00 -0800
+Subject: [PATCH 2/3] Add Juggler to gecko build system
+
+---
+ toolkit/toolkit.mozbuild | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/toolkit/toolkit.mozbuild b/toolkit/toolkit.mozbuild
+index 4a0e5f172..b8abc1e72 100644
+--- a/toolkit/toolkit.mozbuild
++++ b/toolkit/toolkit.mozbuild
+@@ -163,6 +163,7 @@ if CONFIG['ENABLE_MARIONETTE']:
+     DIRS += [
+         '/testing/firefox-ui',
+         '/testing/marionette',
++        '/testing/juggler',
+     ]
+
+ if CONFIG['ENABLE_GECKODRIVER'] and not CONFIG['MOZ_TSAN']:
+-- 
+2.19.0.605.g01d371f741-goog
+

experimental/juggler/patches/0003-Add-Juggler-to-mozilla-packaging-script.patch

@@ -0,0 +1,43 @@
+From 1449495af094fbc5e1bb351f8387c3a341977763 Mon Sep 17 00:00:00 2001
+From: Andrey Lushnikov <lushnikov@chromium.org>
+Date: Thu, 29 Nov 2018 11:40:32 -0800
+Subject: [PATCH 3/3] Add Juggler to mozilla packaging script
+
+---
+ browser/installer/allowed-dupes.mn    | 6 ++++++
+ browser/installer/package-manifest.in | 5 +++++
+ 2 files changed, 11 insertions(+)
+
+diff --git a/browser/installer/allowed-dupes.mn b/browser/installer/allowed-dupes.mn
+index 5685a30d9..32ba241b8 100644
+--- a/browser/installer/allowed-dupes.mn
++++ b/browser/installer/allowed-dupes.mn
+@@ -154,3 +154,9 @@ browser/defaults/settings/main/example.json
+ # Bug 1463748 - Fork and pref-off the new error pages
+ browser/chrome/browser/content/browser/aboutNetError-new.xhtml
+ browser/chrome/browser/content/browser/aboutNetError.xhtml
++
++# Juggler/marionette files
++chrome/juggler/content/content/floating-scrollbars.css
++browser/chrome/devtools/skin/floating-scrollbars-responsive-design.css
++chrome/juggler/content/server/stream-utils.js
++chrome/marionette/content/stream-utils.js
+diff --git a/browser/installer/package-manifest.in b/browser/installer/package-manifest.in
+index 5b828784a..a5d9f9741 100644
+--- a/browser/installer/package-manifest.in
++++ b/browser/installer/package-manifest.in
+@@ -338,6 +338,11 @@
+ @RESPATH@/defaults/pref/marionette.js
+ #endif
+
++@RESPATH@/chrome/juggler@JAREXT@
++@RESPATH@/chrome/juggler.manifest
++@RESPATH@/components/juggler.manifest
++@RESPATH@/components/juggler.js
++
+ @RESPATH@/components/nsAsyncShutdown.manifest
+ @RESPATH@/components/nsAsyncShutdown.js
+
+-- 
+2.19.0.605.g01d371f741-goog
+