Manual chunks function (#2831)

* Merge compact and non-compact import.meta.url mechanisms

* Extract more common code

* Add option to configure import.meta.url resolution

* Switch to using a plugin hook

* Extract more common code

* Move functionality into default plugin

* Improve SystemJS handling

* Refactor import.meta.url handling

* Fix, refactor and test asset emission

* Extract ModuleLoader

* Use sets for colouring hashes

* Simplify alias generation

* Attach aliases to modules

* Transform loading into a process that dynamically accepts new entry points
and manual chunks

* Pass alias objects through the module loader

* Implement basic chunk emission

* Allow duplicate entry points again

* Rename addEntry -> emitEntryChunk

* Simplify alias handling by immediately assigning a chunkAlias to entry
points and introducing a manualChunkAlias for colouring to resolve this
confusing double use of chunkAlias

* * Allow manual chunks to contain nested entry points
* Allow manual chunks to contain entry points without name or with the same name
* Throw if an emitted chunk is not found
* Throw if there is a conflict between manual chunk entries
* Allow nesting of manual chunks without requiring a specific order

* Manual chunks never conflict with entry points:
- if the alias matches, the manual chunk becomes the entry chunk
- otherwise a facade is created

* Return correct file name if a facade is created for an emitted chunk

* Start using central error handlers

* Improve plugin driver type, add generic resolveFileUrl hook

* Test new resolveFileUrl hook, make meta properties tree-shakeable

* Move setAssetSource failure tests to function

* Test and extract all errors thrown when emitting assets

* Extract and refine error when chunk id cannot be found

* Fail if filename is not yet available

* Fail when adding a chunk after loading has finished

* Do not access process.cwd() unchecked

* Move isExternal to module loader

* Fix typing of resolveId context hook

* Refine worker test

* Revert to "wrong" resolveId type as this will probably be fixed in a separate PR

* Suppress .js extensions for AMD, fix issue with empty dynamically imported chunks

* Use generated chunk naming scheme for emitted chunks and rename context
function to `emitChunk`

* Allow emitted chunks to be named

* Add paint worklet example

* Update documentation

* Add reference ids to resolveFileUrl and replace type

* Do not require `input` to be set if a dynamic entry is emitted

* Use facade module id as [name] for dynamic imports

* Update documentation

* Refine pluginDriver types (ported from add-entry branch)

* Make sure resolveId is always passed either string or null

* Unify parameter names

* Add new this.resolve context function

* Get rid of dynamic import alias

* Make sure resolveDynamicImport behaves the same as resolveId if an object
is returned

* Test new warnings and errors

* Mark this.resolveId and this.isExternal as deprecated

* Extract more errors

* Use error message generators instead of error generators

* Fix documentation ordering

* Mark utility functions deprecated in types

* Use relative ids in error messages

* Allow manualChunks to be a function

* Only add manual chunks in a single location

* Add vendor chunk example

docs/999-big-list-of-options.md

@@ -145,7 +145,7 @@ Alternatively, supply a function that will turn an external module ID into a glo
 
 When given as a command line argument, it should be a comma-separated list of `id:variableName` pairs:
 
-```bash
+```
 rollup -i src/main.js ... -g jquery:$,underscore:_
 ```
 
@@ -263,11 +263,31 @@ Default: `false`
 This will inline dynamic imports instead of creating new chunks to create a single bundle. Only possible if a single input is provided.
 
 #### manualChunks
-Type: `{ [chunkAlias: string]: string[] }`
+Type: `{ [chunkAlias: string]: string[] } | ((id: string) => string | void)`
+
+Allows the creation of custom shared common chunks. When using the object form, each property represents a chunk that contains the listed modules and all their dependencies if they are part of the module graph unless they are already in another manual chunk. The name of the chunk will be determined by the property key.
+
+Note that it is not necessary for the listed modules themselves to be be part of the module graph, which is useful if you are working with `rollup-plugin-node-resolve` and use deep imports from packages. For instance
+
+```
+manualChunks: {
+  lodash: ['lodash']
+}
+```
+
+will put all lodash modules into a manual chunk even if you are only using imports of the form `import get from 'lodash/get'`.
 
-Allows the creation of custom shared common chunks. Provides an alias for the chunk and the list of modules to include in that chunk. Modules are bundled into the chunk along with their dependencies. If a module is already in a previous chunk, then the chunk will reference it there. Modules defined into chunks this way are considered to be entry points that can execute independently to any parent importers.
+When using the function form, each resolved module id will be passed to the function. If a string is returned, the module and all its dependency will be added to the manual chunk with the given name. For instance this will create a `vendor` chunk containing all dependencies inside `node_modules`:
+
+```javascript
+manualChunks(id) {
+  if (id.includes('node_modules')) {
+    return 'vendor';
+  }
+}
+```
 
-Note that manual chunks can change the behaviour of the application if side-effects are triggered before the corresponding modules are actually used.
+Be aware that manual chunks can change the behaviour of the application if side-effects are triggered before the corresponding modules are actually used.
 
 #### onwarn
 Type: `(warning: RollupWarning, defaultHandler: (warning: string | RollupWarning) => void) => void;`

src/Graph.ts

@@ -12,6 +12,7 @@ import { ModuleLoader, UnresolvedModuleWithAlias } from './ModuleLoader';
 import {
 	Asset,
 	InputOptions,
+	ManualChunksOption,
 	ModuleJSON,
 	OutputBundle,
 	RollupCache,
@@ -191,13 +192,14 @@ export default class Graph {
 			this,
 			this.moduleById,
 			this.pluginDriver,
-			options.external
+			options.external,
+			typeof options.manualChunks === 'function' && options.manualChunks
 		);
 	}
 
 	build(
 		entryModules: string | string[] | Record<string, string>,
-		manualChunks: Record<string, string[]> | void,
+		manualChunks: ManualChunksOption | void,
 		inlineDynamicImports: boolean
 	): Promise<Chunk[]> {
 		// Phase 1 – discovery. We load the entry module and find which
@@ -208,7 +210,9 @@ export default class Graph {
 
 		return Promise.all([
 			this.moduleLoader.addEntryModules(normalizeEntryModules(entryModules), true),
-			manualChunks && this.moduleLoader.addManualChunks(manualChunks)
+			manualChunks &&
+				typeof manualChunks === 'object' &&
+				this.moduleLoader.addManualChunks(manualChunks)
 		]).then(([{ entryModules, manualChunkModulesByAlias }]) => {
 			if (entryModules.length === 0) {
 				throw new Error('You must supply options.input to rollup');

src/ModuleLoader.ts

@@ -4,6 +4,7 @@ import Graph from './Graph';
 import Module from './Module';
 import {
 	ExternalOption,
+	GetManualChunk,
 	IsExternal,
 	ModuleJSON,
 	ResolvedId,
@@ -36,7 +37,7 @@ export interface UnresolvedModuleWithAlias {
 }
 
 interface UnresolvedEntryModuleWithAlias extends UnresolvedModuleWithAlias {
-	isManualChunkEntry?: boolean;
+	manualChunkAlias?: string;
 }
 
 function normalizeRelativeExternalId(importer: string, source: string) {
@@ -50,6 +51,7 @@ export class ModuleLoader {
 		{ module: Module | null; name: string }
 	>();
 	private readonly entryModules: Module[] = [];
+	private readonly getManualChunk: GetManualChunk;
 	private readonly graph: Graph;
 	private latestLoadModulesPromise: Promise<any> = Promise.resolve();
 	private readonly manualChunkModules: Record<string, Module[]> = {};
@@ -60,7 +62,8 @@ export class ModuleLoader {
 		graph: Graph,
 		modulesById: Map<string, Module | ExternalModule>,
 		pluginDriver: PluginDriver,
-		external: ExternalOption
+		external: ExternalOption,
+		getManualChunk: GetManualChunk | null
 	) {
 		this.graph = graph;
 		this.modulesById = modulesById;
@@ -72,6 +75,7 @@ export class ModuleLoader {
 			const ids = new Set(Array.isArray(external) ? external : external ? [external] : []);
 			this.isExternal = id => ids.has(id);
 		}
+		this.getManualChunk = typeof getManualChunk === 'function' ? getManualChunk : () => null;
 	}
 
 	addEntryModuleAndGetReferenceId(unresolvedEntryModule: UnresolvedModuleWithAlias): string {
@@ -127,17 +131,17 @@ export class ModuleLoader {
 		for (const alias of Object.keys(manualChunks)) {
 			const manualChunkIds = manualChunks[alias];
 			for (const unresolvedId of manualChunkIds) {
-				unresolvedManualChunks.push({ alias, unresolvedId, isManualChunkEntry: true });
+				unresolvedManualChunks.push({ alias: null, unresolvedId, manualChunkAlias: alias });
 			}
 		}
 		const loadNewManualChunkModulesPromise = Promise.all(
 			unresolvedManualChunks.map(this.loadEntryModule)
 		).then(manualChunkModules => {
-			for (const module of manualChunkModules) {
-				if (!this.manualChunkModules[module.manualChunkAlias]) {
-					this.manualChunkModules[module.manualChunkAlias] = [];
-				}
-				this.manualChunkModules[module.manualChunkAlias].push(module);
+			for (let index = 0; index < manualChunkModules.length; index++) {
+				this.addToManualChunk(
+					unresolvedManualChunks[index].manualChunkAlias,
+					manualChunkModules[index]
+				);
 			}
 		});
 
@@ -164,6 +168,17 @@ export class ModuleLoader {
 		).then((result: ResolveIdResult) => this.normalizeResolveIdResult(result, importer, source));
 	}
 
+	private addToManualChunk(alias: string, module: Module) {
+		if (module.manualChunkAlias !== null && module.manualChunkAlias !== alias) {
+			error(errCannotAssignModuleToChunk(module.id, alias, module.manualChunkAlias));
+		}
+		module.manualChunkAlias = alias;
+		if (!this.manualChunkModules[alias]) {
+			this.manualChunkModules[alias] = [];
+		}
+		this.manualChunkModules[alias].push(module);
+	}
+
 	private awaitLoadModulesPromise<T>(loadNewModulesPromise: Promise<T>): Promise<T> {
 		this.latestLoadModulesPromise = Promise.all([
 			loadNewModulesPromise,
@@ -218,6 +233,10 @@ export class ModuleLoader {
 
 		const module: Module = new Module(this.graph, id);
 		this.modulesById.set(id, module);
+		const manualChunkAlias = this.getManualChunk(id);
+		if (typeof manualChunkAlias === 'string') {
+			this.addToManualChunk(manualChunkAlias, module);
+		}
 
 		timeStart('load modules', 3);
 		return Promise.resolve(
@@ -329,11 +348,7 @@ export class ModuleLoader {
 		return resolvedId;
 	}
 
-	private loadEntryModule = ({
-		alias,
-		unresolvedId,
-		isManualChunkEntry
-	}: UnresolvedEntryModuleWithAlias): Promise<Module> =>
+	private loadEntryModule = ({ alias, unresolvedId }: UnresolvedModuleWithAlias): Promise<Module> =>
 		this.pluginDriver
 			.hookFirst('resolveId', [unresolvedId, undefined])
 			.then((resolveIdResult: ResolveIdResult) => {
@@ -351,13 +366,6 @@ export class ModuleLoader {
 				if (typeof id === 'string') {
 					return this.fetchModule(id, undefined).then(module => {
 						if (alias !== null) {
-							if (isManualChunkEntry) {
-								if (module.manualChunkAlias !== null && module.manualChunkAlias !== alias) {
-									error(errCannotAssignModuleToChunk(module.id, alias, module.manualChunkAlias));
-								}
-								module.manualChunkAlias = alias;
-								return module;
-							}
 							if (module.chunkAlias !== null && module.chunkAlias !== alias) {
 								error(errCannotAssignModuleToChunk(module.id, alias, module.chunkAlias));
 							}

src/rollup/types.d.ts

@@ -195,7 +195,6 @@ export type RenderChunkHook = (
 	| string
 	| null;
 
-// TODO this should probably return ResolveIdResult
 export type ResolveDynamicImportHook = (
 	this: PluginContext,
 	specifier: string | ESTree.Node,
@@ -315,9 +314,12 @@ export interface TreeshakingOptions {
 	pureExternalModules?: boolean;
 }
 
+export type GetManualChunk = (id: string) => string | void;
+
 export type ExternalOption = string[] | IsExternal;
 export type GlobalsOption = { [name: string]: string } | ((name: string) => string);
 export type InputOption = string | string[] | { [entryAlias: string]: string };
+export type ManualChunksOption = { [chunkAlias: string]: string[] } | GetManualChunk;
 
 export interface InputOptions {
 	acorn?: any;
@@ -331,7 +333,7 @@ export interface InputOptions {
 	external?: ExternalOption;
 	inlineDynamicImports?: boolean;
 	input?: InputOption;
-	manualChunks?: { [chunkAlias: string]: string[] };
+	manualChunks?: ManualChunksOption;
 	moduleContext?: ((id: string) => string) | { [id: string]: string };
 	onwarn?: WarningHandler;
 	perf?: boolean;

src/utils/mergeOptions.ts

@@ -56,6 +56,7 @@ const getOnWarn = (
 		? warning => config.onwarn(warning, defaultOnWarnHandler)
 		: defaultOnWarnHandler;
 
+// TODO Lukas manual chunks should receive the same treatment
 const getExternal = (config: GenericConfigObject, command: GenericConfigObject) => {
 	const configExternal = config.external;
 	return typeof configExternal === 'function'

test/chunking-form/samples/manual-chunks-function/_config.js

@@ -0,0 +1,12 @@
+module.exports = {
+	description: 'allows to define manual chunks via a function',
+	options: {
+		input: ['main-a'],
+		manualChunks(id) {
+			if (id[id.length - 5] === '-') {
+				console.log(id, id[id.length - 4]);
+				return `chunk-${id[id.length - 4]}`;
+			}
+		}
+	}
+};

test/chunking-form/samples/manual-chunks-function/_expected/amd/chunk-a.js

@@ -0,0 +1,9 @@
+define(['./generated-chunk-b', './generated-chunk-c'], function (__chunk_1, __chunk_2) { 'use strict';
+
+	console.log('dep1');
+
+	console.log('dep-a');
+
+	console.log('main-a');
+
+});

test/chunking-form/samples/manual-chunks-function/_expected/amd/generated-chunk-b.js

@@ -0,0 +1,7 @@
+define(function () { 'use strict';
+
+	console.log('dep2');
+
+	console.log('dep-b');
+
+});

test/chunking-form/samples/manual-chunks-function/_expected/amd/generated-chunk-c.js

@@ -0,0 +1,5 @@
+define(['./generated-chunk-b'], function (__chunk_1) { 'use strict';
+
+	console.log('dep-c');
+
+});

test/chunking-form/samples/manual-chunks-function/_expected/cjs/chunk-a.js

@@ -0,0 +1,10 @@
+'use strict';
+
+require('./generated-chunk-b.js');
+require('./generated-chunk-c.js');
+
+console.log('dep1');
+
+console.log('dep-a');
+
+console.log('main-a');

test/chunking-form/samples/manual-chunks-function/_expected/cjs/generated-chunk-b.js

@@ -0,0 +1,5 @@
+'use strict';
+
+console.log('dep2');
+
+console.log('dep-b');

test/chunking-form/samples/manual-chunks-function/_expected/cjs/generated-chunk-c.js

@@ -0,0 +1,5 @@
+'use strict';
+
+require('./generated-chunk-b.js');
+
+console.log('dep-c');

test/chunking-form/samples/manual-chunks-function/_expected/es/chunk-a.js

@@ -0,0 +1,8 @@
+import './generated-chunk-b.js';
+import './generated-chunk-c.js';
+
+console.log('dep1');
+
+console.log('dep-a');
+
+console.log('main-a');

test/chunking-form/samples/manual-chunks-function/_expected/es/generated-chunk-b.js

@@ -0,0 +1,3 @@
+console.log('dep2');
+
+console.log('dep-b');

test/chunking-form/samples/manual-chunks-function/_expected/es/generated-chunk-c.js

@@ -0,0 +1,3 @@
+import './generated-chunk-b.js';
+
+console.log('dep-c');

test/chunking-form/samples/manual-chunks-function/_expected/system/chunk-a.js

@@ -0,0 +1,15 @@
+System.register(['./generated-chunk-b.js', './generated-chunk-c.js'], function (exports, module) {
+	'use strict';
+	return {
+		setters: [function () {}, function () {}],
+		execute: function () {
+
+			console.log('dep1');
+
+			console.log('dep-a');
+
+			console.log('main-a');
+
+		}
+	};
+});

test/chunking-form/samples/manual-chunks-function/_expected/system/generated-chunk-b.js

@@ -0,0 +1,12 @@
+System.register([], function (exports, module) {
+	'use strict';
+	return {
+		execute: function () {
+
+			console.log('dep2');
+
+			console.log('dep-b');
+
+		}
+	};
+});

test/chunking-form/samples/manual-chunks-function/_expected/system/generated-chunk-c.js

@@ -0,0 +1,11 @@
+System.register(['./generated-chunk-b.js'], function (exports, module) {
+	'use strict';
+	return {
+		setters: [function () {}],
+		execute: function () {
+
+			console.log('dep-c');
+
+		}
+	};
+});

test/chunking-form/samples/manual-chunks-function/dep-a.js

@@ -0,0 +1,4 @@
+import './dep-c';
+import './dep1';
+
+console.log('dep-a');

test/chunking-form/samples/manual-chunks-function/dep-b.js

@@ -0,0 +1,3 @@
+import './dep2';
+
+console.log('dep-b');

test/chunking-form/samples/manual-chunks-function/dep-c.js

@@ -0,0 +1,3 @@
+import './dep2';
+
+console.log('dep-c');

test/chunking-form/samples/manual-chunks-function/dep1.js

@@ -0,0 +1 @@
+console.log('dep1');

test/chunking-form/samples/manual-chunks-function/dep2.js

@@ -0,0 +1 @@
+console.log('dep2');

test/chunking-form/samples/manual-chunks-function/main-a.js

@@ -0,0 +1,4 @@
+import './dep-a';
+import './dep-b';
+
+console.log('main-a');

test/function/samples/manual-chunks-conflict/_config.js

@@ -9,6 +9,6 @@ module.exports = {
 	},
 	error: {
 		code: 'INVALID_CHUNK',
-		message: `Cannot assign dep.js to the "dep1" chunk as it is already in the "dep2" chunk.`
+		message: `Cannot assign dep.js to the "dep2" chunk as it is already in the "dep1" chunk.`
 	}
 };