diff --git a/docs/dashboard/cicd/best-practices.md b/docs/dashboard/cicd/best-practices.md new file mode 100644 index 00000000000..10a0487e973 --- /dev/null +++ b/docs/dashboard/cicd/best-practices.md @@ -0,0 +1,21 @@ + + + + +### [Read this on the main serverless docs site](https://www.serverless.com/framework/docs/dashboard/cicd/best-practices/) + + + +# Serverless CI/CD Best Practices + +Serverless Framework Pro provides a lot of capabilities out of the box to help you manage and deploy +your services. As your teams grow and the number of services grow, it can be difficult to know +the best way to organize your services for scale. + +To help you manage and deploy your services at scale, check out the +[Serverless CI/CD Workflow Guide](https://serverless.com/learn/guides/cicd/) for our recommendations +on organizing your apps, servies, repos and automating your release process. diff --git a/docs/providers/aws/cli-reference/deploy.md b/docs/providers/aws/cli-reference/deploy.md index 8679a2ae84f..23fda45ca63 100644 --- a/docs/providers/aws/cli-reference/deploy.md +++ b/docs/providers/aws/cli-reference/deploy.md @@ -64,3 +64,11 @@ serverless deploy --package /path/to/package/directory ``` With this example, the packaging step will be skipped and the framework will start deploying the package from the `/path/to/package/directory` directory. + +### Environment variables + +- `SLS_AWS_MONITORING_FREQUENCY` allows the adjustment of the deployment monitoring frequency time in ms, default is `5000`. + +```bash +SLS_AWS_MONITORING_FREQUENCY=10000 serverless deploy +``` diff --git a/docs/providers/aws/events/apigateway.md b/docs/providers/aws/events/apigateway.md index 1339f517194..d2cde1bb532 100644 --- a/docs/providers/aws/events/apigateway.md +++ b/docs/providers/aws/events/apigateway.md @@ -284,6 +284,8 @@ Please note that since you can't send multiple values for [Access-Control-Allow- Configuring the `cors` property sets [Access-Control-Allow-Origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin), [Access-Control-Allow-Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers), [Access-Control-Allow-Methods](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Methods),[Access-Control-Allow-Credentials](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials) headers in the CORS preflight response. +Please note that the [Access-Control-Allow-Credentials](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials)-Header is omitted when not explicitly set to `true`. + To enable the `Access-Control-Max-Age` preflight response header, set the `maxAge` property in the `cors` object: ```yml diff --git a/docs/providers/aws/events/sqs.md b/docs/providers/aws/events/sqs.md index 6d9d960b0ed..7ec00228f3b 100644 --- a/docs/providers/aws/events/sqs.md +++ b/docs/providers/aws/events/sqs.md @@ -20,8 +20,6 @@ The ARN for the queue can be specified as a string, the reference to the ARN of **Note:** The `sqs` event will hook up your existing SQS Queue to a Lambda function. Serverless won't create a new queue for you. -**IMPORTANT:** AWS is [not supporting FIFO queue](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) to trigger Lambda function so your queue(s) **must be** a standard queue. - ```yml functions: compute: diff --git a/docs/providers/aws/events/streams.md b/docs/providers/aws/events/streams.md index 17d89c9413a..835de5c3b6d 100644 --- a/docs/providers/aws/events/streams.md +++ b/docs/providers/aws/events/streams.md @@ -108,3 +108,23 @@ functions: arn: arn:aws:kinesis:region:XXXXXX:stream/foo batchWindow: 10 ``` + +## Setting the ParallelizationFactor + +The configuration below sets up a Kinesis stream event for the `preprocess` function which has a parallelization factor of 10 (default is 1). + +The `parallelizationFactor` property specifies the number of concurrent Lambda invocations for each shard of the Kinesis Stream. + +For more information, read the [AWS release announcement](https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/) for this property. + +**Note:** The `stream` event will hook up your existing streams to a Lambda function. Serverless won't create a new stream for you. + +```yml +functions: + preprocess: + handler: handler.preprocess + events: + - stream: + arn: arn:aws:kinesis:region:XXXXXX:stream/foo + parallelizationFactor: 10 +``` diff --git a/docs/providers/aws/events/websocket.md b/docs/providers/aws/events/websocket.md index bc5aa990a93..61cfd906eeb 100644 --- a/docs/providers/aws/events/websocket.md +++ b/docs/providers/aws/events/websocket.md @@ -212,3 +212,16 @@ provider: ``` The log streams will be generated in a dedicated log group which follows the naming schema `/aws/websocket/{service}-{stage}`. + +The default log level will be INFO. You can change this to error with the following: + +```yml +# serverless.yml +provider: + name: aws + logs: + websocket: + level: ERROR +``` + +Valid values are INFO, ERROR. diff --git a/docs/providers/aws/guide/serverless.yml.md b/docs/providers/aws/guide/serverless.yml.md index 0b67fdc919d..799a10cacfb 100644 --- a/docs/providers/aws/guide/serverless.yml.md +++ b/docs/providers/aws/guide/serverless.yml.md @@ -136,14 +136,15 @@ provider: apiGateway: true lambda: true # Optional, can be true (true equals 'Active'), 'Active' or 'PassThrough' logs: - restApi: # Optional configuration which specifies if API Gateway logs are used. This can either be set to true to use defaults, or configured via subproperties. + restApi: # Optional configuration which specifies if API Gateway logs are used. This can either be set to `true` to use defaults, or configured via subproperties. accessLogging: true # Optional configuration which enables or disables access logging. Defaults to true. format: 'requestId: $context.requestId' # Optional configuration which specifies the log format to use for access logging. executionLogging: true # Optional configuration which enables or disables execution logging. Defaults to true. level: INFO # Optional configuration which specifies the log level to use for execution logging. May be set to either INFO or ERROR. fullExecutionData: true # Optional configuration which specifies whether or not to log full requests/responses for execution logging. Defaults to true. role: arn:aws:iam::123456:role # Optional IAM role for ApiGateway to use when managing CloudWatch Logs - websocket: true # Optional configuration which specifies if Websockets logs are used + websocket: # Optional configuration which specifies if Websocket logs are used. This can either be set to `true` to use defaults, or configured via subproperties. + level: INFO # Optional configuration which specifies the log level to use for execution logging. May be set to either INFO or ERROR. frameworkLambda: true # Optional, whether to write CloudWatch logs for custom resource lambdas as added by the framework package: # Optional deployment packaging configuration diff --git a/docs/providers/azure/guide/credentials.md b/docs/providers/azure/guide/credentials.md index 2b8fcd059c5..d5ca1d364b8 100644 --- a/docs/providers/azure/guide/credentials.md +++ b/docs/providers/azure/guide/credentials.md @@ -48,7 +48,7 @@ This will give you a code and prompt you to visit [aka.ms/devicelogin](https://a $ az account list { "cloudName": "AzureCloud", - "id": "c6e5c9a2-a4dd-4c05-81b4-6bed04f913ea", + "id": "", "isDefault": true, "name": "My Azure Subscription", "registeredProviders": [], diff --git a/docs/providers/azure/guide/quick-start.md b/docs/providers/azure/guide/quick-start.md index 00d3c37591d..313b86acba1 100644 --- a/docs/providers/azure/guide/quick-start.md +++ b/docs/providers/azure/guide/quick-start.md @@ -156,10 +156,29 @@ The getting started walkthrough illustrates the interactive login experience, wh ```bash # Login to Azure $ az login - # Set Azure Subscription for which to create Service Principal + ``` + This will yield something like: + ```json + [ + { + "cloudName": "", + "id": "", + "isDefault": true, + "name": "", + "state": "", + "tenantId": "", + "user": { + "name": "", + "type": "" + } + } + ] + ``` +3. Set Azure Subscription for which to create Service Principal + ```bash $ az account set -s ``` -3. Generate Service Principal for Azure Subscription +4. Generate Service Principal for Azure Subscription ```bash # Create SP with unique name $ az ad sp create-for-rbac --name @@ -174,7 +193,7 @@ The getting started walkthrough illustrates the interactive login experience, wh "tenant": "" } ``` -4. Set environment variables +5. Set environment variables **Bash** diff --git a/lib/classes/PluginManager.test.js b/lib/classes/PluginManager.test.js index 5c2a2377cfe..1fb70906770 100644 --- a/lib/classes/PluginManager.test.js +++ b/lib/classes/PluginManager.test.js @@ -65,7 +65,7 @@ describe('PluginManager', () => { } functions() { - this.deployedFunctions = this.deployedFunctions + 1; + this.deployedFunctions += 1; } } @@ -88,7 +88,7 @@ describe('PluginManager', () => { } functions() { - this.deployedFunctions = this.deployedFunctions + 1; + this.deployedFunctions += 1; } } @@ -135,14 +135,14 @@ describe('PluginManager', () => { functions() { return new BbPromise(resolve => { - this.deployedFunctions = this.deployedFunctions + 1; + this.deployedFunctions += 1; return resolve(); }); } resources() { return new BbPromise(resolve => { - this.deployedResources = this.deployedResources + 1; + this.deployedResources += 1; return resolve(); }); } @@ -190,11 +190,11 @@ describe('PluginManager', () => { } functions() { - this.deployedFunctions = this.deployedFunctions + 1; + this.deployedFunctions += 1; } resources() { - this.deployedResources = this.deployedResources + 1; + this.deployedResources += 1; } } @@ -241,11 +241,11 @@ describe('PluginManager', () => { } functions() { - this.deployedFunctions = this.deployedFunctions + 1; + this.deployedFunctions += 1; } resources() { - this.deployedResources = this.deployedResources + 1; + this.deployedResources += 1; } } diff --git a/lib/plugins/aws/customResources/generateZip.js b/lib/plugins/aws/customResources/generateZip.js new file mode 100644 index 00000000000..11e9879bf75 --- /dev/null +++ b/lib/plugins/aws/customResources/generateZip.js @@ -0,0 +1,47 @@ +'use strict'; + +const os = require('os'); +const path = require('path'); +const { memoize } = require('lodash'); +const BbPromise = require('bluebird'); +const childProcess = BbPromise.promisifyAll(require('child_process')); +const fse = BbPromise.promisifyAll(require('fs-extra')); +const { version } = require('../../../../package'); +const getTmpDirPath = require('../../../utils/fs/getTmpDirPath'); +const createZipFile = require('../../../utils/fs/createZipFile'); + +const srcDirPath = path.join(__dirname, 'resources'); +const cachedZipFilePath = path.join( + os.homedir(), + '.serverless/cache/custom-resources', + version, + 'custom-resources.zip' +); + +module.exports = memoize(() => + fse + .lstatAsync(cachedZipFilePath) + .then( + stats => { + if (stats.isFile()) return true; + return false; + }, + error => { + if (error.code === 'ENOENT') return false; + throw error; + } + ) + .then(isCached => { + if (isCached) return cachedZipFilePath; + const ensureCachedDirDeferred = fse.ensureDirAsync(path.dirname(cachedZipFilePath)); + const tmpDirPath = getTmpDirPath(); + return fse + .copyAsync(srcDirPath, tmpDirPath) + .then(() => childProcess.execAsync('npm install', { cwd: tmpDirPath })) + .then(() => ensureCachedDirDeferred) + .then(() => createZipFile(tmpDirPath, cachedZipFilePath)) + .then(() => cachedZipFilePath); + }) +); + +module.exports.cachedZipFilePath = cachedZipFilePath; diff --git a/lib/plugins/aws/customResources/index.js b/lib/plugins/aws/customResources/index.js index b97b5cc1d59..0597957c77f 100644 --- a/lib/plugins/aws/customResources/index.js +++ b/lib/plugins/aws/customResources/index.js @@ -4,17 +4,7 @@ const path = require('path'); const crypto = require('crypto'); const BbPromise = require('bluebird'); const fse = BbPromise.promisifyAll(require('fs-extra')); -const childProcess = BbPromise.promisifyAll(require('child_process')); -const getTmpDirPath = require('../../../utils/fs/getTmpDirPath'); -const createZipFile = require('../../../utils/fs/createZipFile'); - -function copyCustomResources(srcDirPath, destDirPath) { - return fse.ensureDirAsync(destDirPath).then(() => fse.copyAsync(srcDirPath, destDirPath)); -} - -function installDependencies(dirPath) { - return childProcess.execAsync('npm install', { cwd: dirPath }); -} +const generateZip = require('./generateZip'); function addCustomResourceToService(awsProvider, resourceName, iamRoleStatements) { let functionName; @@ -28,15 +18,12 @@ function addCustomResourceToService(awsProvider, resourceName, iamRoleStatements const shouldWriteLogs = providerConfig.logs && providerConfig.logs.frameworkLambda; const { Resources } = providerConfig.compiledCloudFormationTemplate; const customResourcesRoleLogicalId = awsProvider.naming.getCustomResourcesRoleLogicalId(); - const srcDirPath = path.join(__dirname, 'resources'); - const destDirPath = path.join( + const zipFilePath = path.join( serverless.config.servicePath, '.serverless', - awsProvider.naming.getCustomResourcesArtifactDirectoryName() + awsProvider.naming.getCustomResourcesArtifactName() ); - const tmpDirPath = path.join(getTmpDirPath(), 'resources'); const funcPrefix = `${serverless.service.service}-${cliOptions.stage}`; - const zipFilePath = `${destDirPath}.zip`; serverless.utils.writeFileDir(zipFilePath); // check which custom resource should be used @@ -72,10 +59,9 @@ function addCustomResourceToService(awsProvider, resourceName, iamRoleStatements // TODO: check every once in a while if external packages are still necessary serverless.cli.log('Installing dependencies for custom CloudFormation resources...'); - return copyCustomResources(srcDirPath, tmpDirPath) - .then(() => installDependencies(tmpDirPath)) - .then(() => createZipFile(tmpDirPath, zipFilePath)) - .then(outputFilePath => { + return generateZip().then(cachedZipFilePath => { + const zipFileBasename = path.basename(zipFilePath); + return fse.copyAsync(cachedZipFilePath, zipFilePath).then(() => { let S3Bucket = { Ref: awsProvider.naming.getDeploymentBucketLogicalId(), }; @@ -83,7 +69,7 @@ function addCustomResourceToService(awsProvider, resourceName, iamRoleStatements S3Bucket = serverless.service.package.deploymentBucket; } const s3Folder = serverless.service.package.artifactDirectoryName; - const s3FileName = outputFilePath.split(path.sep).pop(); + const s3FileName = zipFileBasename; const S3Key = `${s3Folder}/${s3FileName}`; const cfnRoleArn = serverless.service.provider.cfnRole; @@ -205,6 +191,7 @@ function addCustomResourceToService(awsProvider, resourceName, iamRoleStatements }); } }); + }); } module.exports = { diff --git a/lib/plugins/aws/customResources/index.test.js b/lib/plugins/aws/customResources/index.test.js index 5ecfffd40c4..9b5faed0cc9 100644 --- a/lib/plugins/aws/customResources/index.test.js +++ b/lib/plugins/aws/customResources/index.test.js @@ -2,7 +2,6 @@ /* eslint-disable no-unused-expressions */ -const path = require('path'); const fs = require('fs'); const chai = require('chai'); const sinon = require('sinon'); @@ -13,6 +12,7 @@ const CLI = require('../../../classes/CLI'); const childProcess = BbPromise.promisifyAll(require('child_process')); const { createTmpDir } = require('../../../../tests/utils/fs'); const { addCustomResourceToService } = require('./index.js'); +const customResourcesZipFilePath = require('./generateZip').cachedZipFilePath; const expect = chai.expect; chai.use(require('sinon-chai')); @@ -103,13 +103,8 @@ describe('#addCustomResourceToService()', () => { ]) ).to.be.fulfilled.then(() => { const { Resources } = serverless.service.provider.compiledCloudFormationTemplate; - const customResourcesZipFilePath = path.join( - tmpDirPath, - '.serverless', - 'custom-resources.zip' - ); - expect(execAsyncStub).to.have.callCount(3); + expect(execAsyncStub).to.have.callCount(1); expect(fs.existsSync(customResourcesZipFilePath)).to.equal(true); // S3 Lambda Function expect(Resources.CustomDashresourceDashexistingDashs3LambdaFunction).to.deep.equal({ @@ -270,13 +265,6 @@ describe('#addCustomResourceToService()', () => { ]) ).to.be.fulfilled.then(() => { const { Resources } = serverless.service.provider.compiledCloudFormationTemplate; - const customResourcesZipFilePath = path.join( - tmpDirPath, - '.serverless', - 'custom-resources.zip' - ); - - expect(execAsyncStub).to.have.callCount(3); expect(fs.existsSync(customResourcesZipFilePath)).to.equal(true); // S3 Lambda Function expect(Resources.CustomDashresourceDashexistingDashs3LambdaFunction).to.deep.equal({ diff --git a/lib/plugins/aws/deploy/lib/existsDeploymentBucket.test.js b/lib/plugins/aws/deploy/lib/existsDeploymentBucket.test.js index ce7a00c9ee5..1b8e5c46b8e 100644 --- a/lib/plugins/aws/deploy/lib/existsDeploymentBucket.test.js +++ b/lib/plugins/aws/deploy/lib/existsDeploymentBucket.test.js @@ -68,24 +68,25 @@ describe('#existsDeploymentBucket()', () => { }); }); - [{ region: 'eu-west-1', response: 'EU' }, { region: 'us-east-1', response: '' }].forEach( - value => { - it(`should handle inconsistent getBucketLocation responses for ${value.region} region`, () => { - const bucketName = 'com.serverless.deploys'; - - awsPlugin.provider.options.region = value.region; - - sinon.stub(awsPlugin.provider, 'request').resolves({ - LocationConstraint: value.response, - }); - - awsPlugin.serverless.service.provider.deploymentBucket = bucketName; - return expect(awsPlugin.existsDeploymentBucket(bucketName)).to.be.fulfilled.then(() => { - expect(awsPluginStub.args[0][0]).to.equal('S3'); - expect(awsPluginStub.args[0][1]).to.equal('getBucketLocation'); - expect(awsPluginStub.args[0][2].Bucket).to.equal(bucketName); - }); + [ + { region: 'eu-west-1', response: 'EU' }, + { region: 'us-east-1', response: '' }, + ].forEach(value => { + it(`should handle inconsistent getBucketLocation responses for ${value.region} region`, () => { + const bucketName = 'com.serverless.deploys'; + + awsPlugin.provider.options.region = value.region; + + sinon.stub(awsPlugin.provider, 'request').resolves({ + LocationConstraint: value.response, + }); + + awsPlugin.serverless.service.provider.deploymentBucket = bucketName; + return expect(awsPlugin.existsDeploymentBucket(bucketName)).to.be.fulfilled.then(() => { + expect(awsPluginStub.args[0][0]).to.equal('S3'); + expect(awsPluginStub.args[0][1]).to.equal('getBucketLocation'); + expect(awsPluginStub.args[0][2].Bucket).to.equal(bucketName); }); - } - ); + }); + }); }); diff --git a/lib/plugins/aws/deploy/lib/uploadArtifacts.js b/lib/plugins/aws/deploy/lib/uploadArtifacts.js index 29628b91524..b3baa9156f1 100644 --- a/lib/plugins/aws/deploy/lib/uploadArtifacts.js +++ b/lib/plugins/aws/deploy/lib/uploadArtifacts.js @@ -134,7 +134,7 @@ module.exports = { const artifactFilePath = path.join( this.serverless.config.servicePath, '.serverless', - `${this.provider.naming.getCustomResourcesArtifactDirectoryName()}.zip` + this.provider.naming.getCustomResourcesArtifactName() ); if (this.serverless.utils.fileExistsSync(artifactFilePath)) { diff --git a/lib/plugins/aws/invokeLocal/index.js b/lib/plugins/aws/invokeLocal/index.js index 1410aad07fe..02b7341aeee 100644 --- a/lib/plugins/aws/invokeLocal/index.js +++ b/lib/plugins/aws/invokeLocal/index.js @@ -243,7 +243,11 @@ class AwsInvokeLocal { this.serverless.cli.log('Downloading base Docker image...'); return new BbPromise((resolve, reject) => { - const docker = spawn('docker', ['pull', `lambci/lambda:${runtime}`]); + const docker = spawn('docker', [ + 'pull', + '--disable-content-trust=false', + `lambci/lambda:${runtime}`, + ]); docker.on('exit', error => { return error ? reject(error) : resolve(); }); diff --git a/lib/plugins/aws/invokeLocal/index.test.js b/lib/plugins/aws/invokeLocal/index.test.js index afab56b5091..1cffadbcf41 100644 --- a/lib/plugins/aws/invokeLocal/index.test.js +++ b/lib/plugins/aws/invokeLocal/index.test.js @@ -1219,7 +1219,7 @@ describe('AwsInvokeLocal', () => { ]); expect(spawnStub.getCall(2).args).to.deep.equal([ 'docker', - ['pull', 'lambci/lambda:nodejs12.x'], + ['pull', '--disable-content-trust=false', 'lambci/lambda:nodejs12.x'], ]); expect(spawnStub.getCall(3).args).to.deep.equal([ 'docker', diff --git a/lib/plugins/aws/lib/monitorStack.js b/lib/plugins/aws/lib/monitorStack.js index 1d50c3f933d..168d5267c3d 100644 --- a/lib/plugins/aws/lib/monitorStack.js +++ b/lib/plugins/aws/lib/monitorStack.js @@ -29,6 +29,9 @@ module.exports = { async.whilst( () => validStatuses.indexOf(stackStatus) === -1, callback => { + const cfStackStatusCheckFrequency = + options.frequency || process.env.SLS_AWS_MONITORING_FREQUENCY || 5000; + setTimeout(() => { const params = { StackName: cfData.StackId, @@ -134,7 +137,7 @@ module.exports = { reject(new this.serverless.classes.Error(e.message)); } }); - }, options.frequency || 5000); + }, cfStackStatusCheckFrequency); }, () => { // empty console.log for a prettier output diff --git a/lib/plugins/aws/lib/naming.js b/lib/plugins/aws/lib/naming.js index cbfd6d7d831..39f200b8dd5 100644 --- a/lib/plugins/aws/lib/naming.js +++ b/lib/plugins/aws/lib/naming.js @@ -453,8 +453,8 @@ module.exports = { }, // Custom Resources - getCustomResourcesArtifactDirectoryName() { - return 'custom-resources'; + getCustomResourcesArtifactName() { + return 'custom-resources.zip'; }, getCustomResourcesRoleLogicalId() { return 'IamRoleCustomResourcesLambdaExecution'; diff --git a/lib/plugins/aws/lib/naming.test.js b/lib/plugins/aws/lib/naming.test.js index 67865aebb69..b86c6771337 100644 --- a/lib/plugins/aws/lib/naming.test.js +++ b/lib/plugins/aws/lib/naming.test.js @@ -722,9 +722,9 @@ describe('#naming()', () => { }); }); - describe('#getCustomResourcesArtifactDirectoryName()', () => { + describe('#getCustomResourcesArtifactName()', () => { it('should return the custom resources artifact directory name', () => { - expect(sdk.naming.getCustomResourcesArtifactDirectoryName()).to.equal('custom-resources'); + expect(sdk.naming.getCustomResourcesArtifactName()).to.equal('custom-resources.zip'); }); }); diff --git a/lib/plugins/aws/logs/index.js b/lib/plugins/aws/logs/index.js index 6e24332ff43..eefb6b17bb7 100644 --- a/lib/plugins/aws/logs/index.js +++ b/lib/plugins/aws/logs/index.js @@ -2,10 +2,13 @@ const BbPromise = require('bluebird'); const _ = require('lodash'); -const moment = require('moment'); +const dayjs = require('dayjs'); +const utc = require('dayjs/plugin/utc'); const validate = require('../lib/validate'); const formatLambdaLogEvent = require('../utils/formatLambdaLogEvent'); +dayjs.extend(utc); + class AwsLogs { constructor(serverless, options) { this.serverless = serverless; @@ -77,21 +80,21 @@ class AwsLogs { const since = ['m', 'h', 'd'].indexOf(this.options.startTime[this.options.startTime.length - 1]) !== -1; if (since) { - params.startTime = moment() + params.startTime = dayjs() .subtract( this.options.startTime.replace(/\D/g, ''), this.options.startTime.replace(/\d/g, '') ) .valueOf(); } else { - params.startTime = moment.utc(this.options.startTime).valueOf(); + params.startTime = dayjs.utc(this.options.startTime).valueOf(); } } else { - params.startTime = moment() + params.startTime = dayjs() .subtract(10, 'm') .valueOf(); if (this.options.tail) { - params.startTime = moment() + params.startTime = dayjs() .subtract(10, 's') .valueOf(); } diff --git a/lib/plugins/aws/metrics/awsMetrics.js b/lib/plugins/aws/metrics/awsMetrics.js index e28c9bd11c6..c633288a092 100644 --- a/lib/plugins/aws/metrics/awsMetrics.js +++ b/lib/plugins/aws/metrics/awsMetrics.js @@ -3,9 +3,13 @@ const BbPromise = require('bluebird'); const chalk = require('chalk'); const _ = require('lodash'); -const moment = require('moment'); +const dayjs = require('dayjs'); const validate = require('../lib/validate'); +const LocalizedFormat = require('dayjs/plugin/localizedFormat'); + +dayjs.extend(LocalizedFormat); + class AwsMetrics { constructor(serverless, options) { this.serverless = serverless; @@ -27,14 +31,14 @@ class AwsMetrics { this.validate(); const today = new Date(); - const yesterday = moment() + const yesterday = dayjs() .subtract(1, 'day') .toDate(); if (this.options.startTime) { const sinceDateMatch = this.options.startTime.match(/(\d+)(m|h|d)/); if (sinceDateMatch) { - this.options.startTime = moment() + this.options.startTime = dayjs() .subtract(sinceDateMatch[1], sinceDateMatch[2]) .valueOf(); } @@ -107,8 +111,8 @@ class AwsMetrics { message += `${chalk.yellow.underline('Service wide metrics')}\n`; } - const formattedStartTime = moment(this.options.startTime).format('LLL'); - const formattedEndTime = moment(this.options.endTime).format('LLL'); + const formattedStartTime = dayjs(this.options.startTime).format('LLL'); + const formattedEndTime = dayjs(this.options.endTime).format('LLL'); message += `${formattedStartTime} - ${formattedEndTime}\n\n`; if (metrics && metrics.length > 0) { diff --git a/lib/plugins/aws/metrics/awsMetrics.test.js b/lib/plugins/aws/metrics/awsMetrics.test.js index 49fcbd2060b..f4221298bc5 100644 --- a/lib/plugins/aws/metrics/awsMetrics.test.js +++ b/lib/plugins/aws/metrics/awsMetrics.test.js @@ -7,7 +7,11 @@ const AwsMetrics = require('./awsMetrics'); const Serverless = require('../../../Serverless'); const CLI = require('../../../classes/CLI'); const chalk = require('chalk'); -const moment = require('moment'); +const dayjs = require('dayjs'); + +const LocalizedFormat = require('dayjs/plugin/localizedFormat'); + +dayjs.extend(LocalizedFormat); describe('AwsMetrics', () => { let awsMetrics; @@ -91,7 +95,7 @@ describe('AwsMetrics', () => { const yesterdaysDate = `${yesterdaysYear}-${yesterdaysMonth}-${yesterdaysDay}`; return awsMetrics.extendedValidate().then(() => { - const defaultsStartTime = moment(awsMetrics.options.startTime); + const defaultsStartTime = dayjs(awsMetrics.options.startTime); const defaultsDate = defaultsStartTime.format('YYYY-M-D'); expect(defaultsDate).to.equal(yesterdaysDate); @@ -121,7 +125,7 @@ describe('AwsMetrics', () => { const yesterdaysDate = `${yesterdaysYear}-${yesterdaysMonth}-${yesterdaysDay}`; return awsMetrics.extendedValidate().then(() => { - const translatedStartTime = moment(awsMetrics.options.startTime); + const translatedStartTime = dayjs(awsMetrics.options.startTime); const translatedDate = translatedStartTime.format('YYYY-M-D'); expect(translatedDate).to.equal(yesterdaysDate); @@ -138,7 +142,7 @@ describe('AwsMetrics', () => { const todaysDate = `${todaysYear}-${todaysMonth}-${todaysDay}`; return awsMetrics.extendedValidate().then(() => { - const defaultsStartTime = moment(awsMetrics.options.endTime); + const defaultsStartTime = dayjs(awsMetrics.options.endTime); const defaultsDate = defaultsStartTime.format('YYYY-M-D'); expect(defaultsDate).to.equal(todaysDate); diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.js index 7f90c3134a9..b45dee786e2 100644 --- a/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.js +++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.js @@ -30,9 +30,13 @@ module.exports = { 'Access-Control-Allow-Origin': `'${origin}'`, 'Access-Control-Allow-Headers': `'${config.headers.join(',')}'`, 'Access-Control-Allow-Methods': `'${config.methods.join(',')}'`, - 'Access-Control-Allow-Credentials': `'${config.allowCredentials}'`, }; + // Only set Access-Control-Allow-Credentials when explicitly allowed (omit if false) + if (config.allowCredentials) { + preflightHeaders['Access-Control-Allow-Credentials'] = `'${config.allowCredentials}'`; + } + // Enable CORS Max Age usage if set if (_.has(config, 'maxAge')) { if (_.isInteger(config.maxAge) && config.maxAge > 0) { diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.test.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.test.js index 4415a817e84..c81e2f107fc 100644 --- a/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.test.js +++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/cors.test.js @@ -157,7 +157,7 @@ describe('#compileCors()', () => { awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources .ApiGatewayMethodUsersUpdateOptions.Properties.Integration.IntegrationResponses[0] .ResponseParameters['method.response.header.Access-Control-Allow-Credentials'] - ).to.equal("'false'"); + ).to.be.undefined; expect( awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources @@ -194,7 +194,7 @@ describe('#compileCors()', () => { awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources .ApiGatewayMethodUsersDeleteOptions.Properties.Integration.IntegrationResponses[0] .ResponseParameters['method.response.header.Access-Control-Allow-Credentials'] - ).to.equal("'false'"); + ).to.be.undefined; expect( awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources @@ -233,7 +233,7 @@ describe('#compileCors()', () => { awsCompileApigEvents.serverless.service.provider.compiledCloudFormationTemplate.Resources .ApiGatewayMethodUsersAnyOptions.Properties.Integration.IntegrationResponses[0] .ResponseParameters['method.response.header.Access-Control-Allow-Credentials'] - ).to.equal("'false'"); + ).to.be.undefined; }); }); diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js index 34d24e83731..5f0a07c39c5 100644 --- a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js +++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js @@ -4,7 +4,6 @@ const _ = require('lodash'); const BbPromise = require('bluebird'); const ServerlessError = require('../../../../../../../../classes/Error').ServerlessError; -const isRestApiId = RegExp.prototype.test.bind(/^[a-z0-9]{3,}$/); const defaultApiGatewayLogFormat = [ 'requestId: $context.requestId', 'ip: $context.identity.sourceIp', @@ -29,54 +28,63 @@ module.exports = { defaultApiGatewayLogLevel, apiGatewayValidLogLevels, updateStage() { - // this array is used to gather all the patch operations we need to - // perform on the stage - this.apiGatewayStagePatchOperations = []; - this.apiGatewayTagResourceParams = []; - this.apiGatewayUntagResourceParams = []; - this.apiGatewayStageState = {}; - this.apiGatewayDeploymentId = null; - this.apiGatewayRestApiId = null; - - return resolveAccountInfo - .call(this) - .then(resolveRestApiId.bind(this)) - .then(() => { - if (!this.apiGatewayRestApiId) { - // Could not resolve REST API id automatically - - const provider = this.state.service.provider; - const isTracingEnabled = provider.tracing && provider.tracing.apiGateway; - const areLogsEnabled = provider.logs && provider.logs.restApi; - - if (!isTracingEnabled && !areLogsEnabled) { - // Do crash if there are no API Gateway customizations to apply - return null; - } + return BbPromise.try(() => { + const provider = this.state.service.provider; + this.hasTracingConfigured = provider.tracing && provider.tracing.apiGateway != null; + this.hasLogsConfigured = provider.logs && provider.logs.restApi != null; + this.hasTagsConfigured = provider.tags != null || provider.stackTags != null; - const errorMessage = [ - 'Rest API id could not be resolved.\n', - 'This might be caused by a custom API Gateway configuration.\n\n', - 'In given setup stage specific options such as ', - '`tracing`, `logs` and `tags` are not supported.\n\n', - 'Please update your configuration (or open up an issue if you feel ', - "that there's a way to support your setup).", - ].join(''); + if (!this.hasTracingConfigured && !this.hasLogsConfigured && !this.hasTagsConfigured) { + return null; + } - throw new ServerlessError(errorMessage); - } - return resolveStage - .call(this) - .then(resolveDeploymentId.bind(this)) - .then(ensureStage.bind(this)) - .then(handleTracing.bind(this)) - .then(handleLogs.bind(this)) - .then(handleTags.bind(this)) - .then(applyUpdates.bind(this)) - .then(addTags.bind(this)) - .then(removeTags.bind(this)) - .then(removeAccessLoggingLogGroup.bind(this)); - }); + // this array is used to gather all the patch operations we need to + // perform on the stage + this.apiGatewayStagePatchOperations = []; + this.apiGatewayTagResourceParams = []; + this.apiGatewayUntagResourceParams = []; + this.apiGatewayStageState = {}; + this.apiGatewayDeploymentId = null; + this.apiGatewayRestApiId = null; + + return resolveAccountInfo + .call(this) + .then(resolveRestApiId.bind(this)) + .then(() => { + // Do not update APIGW-wide settings, in case external APIGW is referenced + if (this.isExternalRestApi) return null; + if (!this.apiGatewayRestApiId) { + // Could not resolve REST API id automatically + + if (!this.hasTracingConfigured && !this.hasLogsConfigured) { + // Do crash if there are no API Gateway customizations to apply + return null; + } + + const errorMessage = [ + 'Rest API id could not be resolved.\n', + 'This might be caused by a custom API Gateway configuration.\n\n', + 'In given setup stage specific options such as ', + '`tracing`, `logs` and `tags` are not supported.\n\n', + 'Please update your configuration (or open up an issue if you feel ', + "that there's a way to support your setup).", + ].join(''); + + throw new ServerlessError(errorMessage); + } + return resolveStage + .call(this) + .then(resolveDeploymentId.bind(this)) + .then(ensureStage.bind(this)) + .then(handleTracing.bind(this)) + .then(handleLogs.bind(this)) + .then(handleTags.bind(this)) + .then(applyUpdates.bind(this)) + .then(addTags.bind(this)) + .then(removeTags.bind(this)) + .then(removeAccessLoggingLogGroup.bind(this)); + }); + }); }, }; @@ -91,7 +99,7 @@ function resolveAccountInfo() { function resolveApiGatewayResource(resources) { if (resources.ApiGatewayRestApi) return resources.ApiGatewayRestApi; const apiGatewayResources = _.values(resources).filter( - resource => resource.Type === 'AWS::ApiGateway::Resource' + resource => resource.Type === 'AWS::ApiGateway::RestApi' ); if (apiGatewayResources.length === 1) return apiGatewayResources[0]; // None, or more than one API Gateway found (currently there's no support for multiple APIGW's) @@ -103,7 +111,8 @@ function resolveRestApiId() { const provider = this.state.service.provider; const externalRestApiId = provider.apiGateway && provider.apiGateway.restApiId; if (externalRestApiId) { - resolve(isRestApiId(externalRestApiId) ? externalRestApiId : null); + this.isExternalRestApi = true; + resolve(null); return; } const apiGatewayResource = resolveApiGatewayResource( @@ -182,8 +191,8 @@ function ensureStage() { } function handleTracing() { - const tracing = this.state.service.provider.tracing; - const tracingEnabled = tracing && tracing.apiGateway; + if (!this.hasTracingConfigured) return; + const tracingEnabled = this.state.service.provider.tracing.apiGateway; let operation = { op: 'replace', path: '/tracingEnabled', value: 'false' }; if (tracingEnabled) { @@ -193,8 +202,8 @@ function handleTracing() { } function handleLogs() { - const provider = this.state.service.provider; - const logs = provider.logs && provider.logs.restApi; + if (!this.hasLogsConfigured) return; + const logs = this.state.service.provider.logs.restApi; const ops = this.apiGatewayStagePatchOperations; let operations = [ @@ -270,6 +279,7 @@ function handleLogs() { } function handleTags() { + if (!this.hasTagsConfigured) return; const provider = this.state.service.provider; const tagsMerged = _.mapValues(Object.assign({}, provider.stackTags, provider.tags), v => String(v) diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js index db646154e22..dd6a51c9ebf 100644 --- a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js +++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js @@ -41,7 +41,7 @@ describe('#updateStage()', () => { serverless.service.provider.compiledCloudFormationTemplate = { Resources: { ApiGatewayRestApi: { - Type: 'AWS::ApiGateway::Resource', + Type: 'AWS::ApiGateway::RestApi', Properties: { Name: 'dev-my-service', }, @@ -65,6 +65,7 @@ describe('#updateStage()', () => { items: [ { name: 'dev-my-service', id: 'devRestApiId' }, { name: 'prod-my-service', id: 'prodRestApiId' }, + { name: 'custom-rest-api-name', id: 'customRestApiId' }, ], }); providerRequestStub @@ -87,6 +88,16 @@ describe('#updateStage()', () => { old: 'tag', }, }); + providerRequestStub + .withArgs('APIGateway', 'getStage', { + restApiId: 'customRestApiId', + stageName: 'dev', + }) + .resolves({ + tags: { + old: 'tag', + }, + }); providerRequestStub .withArgs('CloudWatchLogs', 'deleteLogGroup', { @@ -340,19 +351,13 @@ describe('#updateStage()', () => { }); }); - it('should perform default actions if settings are not configure', () => { + it('should not perform any actions if settings are not configure', () => { context.state.service.provider.tags = { old: 'tag', }; return updateStage.call(context).then(() => { - const patchOperations = [ - { op: 'replace', path: '/tracingEnabled', value: 'false' }, - { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, - { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, - ]; - expect(providerGetAccountInfoStub).to.be.calledOnce; - expect(providerRequestStub.args).to.have.length(4); + expect(providerRequestStub.args).to.have.length(3); expect(providerRequestStub.args[0][0]).to.equal('APIGateway'); expect(providerRequestStub.args[0][1]).to.equal('getRestApis'); expect(providerRequestStub.args[0][2]).to.deep.equal({ @@ -365,22 +370,16 @@ describe('#updateStage()', () => { restApiId: 'devRestApiId', stageName: 'dev', }); - expect(providerRequestStub.args[2][0]).to.equal('APIGateway'); - expect(providerRequestStub.args[2][1]).to.equal('updateStage'); + expect(providerRequestStub.args[2][0]).to.equal('CloudWatchLogs'); + expect(providerRequestStub.args[2][1]).to.equal('deleteLogGroup'); expect(providerRequestStub.args[2][2]).to.deep.equal({ - restApiId: 'devRestApiId', - stageName: 'dev', - patchOperations, - }); - expect(providerRequestStub.args[3][0]).to.equal('CloudWatchLogs'); - expect(providerRequestStub.args[3][1]).to.equal('deleteLogGroup'); - expect(providerRequestStub.args[3][2]).to.deep.equal({ logGroupName: '/aws/api-gateway/my-service-dev', }); }); }); it('should create a new stage and proceed as usual if none can be found', () => { + context.state.service.provider.tracing = { apiGateway: false }; providerRequestStub .withArgs('APIGateway', 'getStage', { restApiId: 'devRestApiId', @@ -406,11 +405,7 @@ describe('#updateStage()', () => { .resolves(); return updateStage.call(context).then(() => { - const patchOperations = [ - { op: 'replace', path: '/tracingEnabled', value: 'false' }, - { op: 'replace', path: '/*/*/logging/dataTrace', value: 'false' }, - { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' }, - ]; + const patchOperations = [{ op: 'replace', path: '/tracingEnabled', value: 'false' }]; expect(providerGetAccountInfoStub).to.be.calledOnce; expect(providerRequestStub.args).to.have.length(6); @@ -454,24 +449,17 @@ describe('#updateStage()', () => { }); }); - it('should resolve custom restApiId', () => { - providerRequestStub - .withArgs('APIGateway', 'getStage', { - restApiId: 'foobarfoo1', - stageName: 'dev', - }) - .resolves({ - variables: { - old: 'tag', - }, - }); + it('should ignore external api gateway', () => { context.state.service.provider.apiGateway = { restApiId: 'foobarfoo1' }; + context.state.service.provider.tracing = { apiGateway: false }; return updateStage.call(context).then(() => { - expect(context.apiGatewayRestApiId).to.equal('foobarfoo1'); + expect(context.isExternalRestApi).to.equal(true); + expect(context.apiGatewayRestApiId).to.equal(null); }); }); it('should resolve custom APIGateway name', () => { + context.state.service.provider.tracing = { apiGateway: false }; providerRequestStub .withArgs('APIGateway', 'getRestApis', { limit: 500, @@ -499,15 +487,18 @@ describe('#updateStage()', () => { }); it('should resolve custom APIGateway resource', () => { + context.state.service.provider.tracing = { apiGateway: false }; const resources = context.serverless.service.provider.compiledCloudFormationTemplate.Resources; resources.CustomApiGatewayRestApi = resources.ApiGatewayRestApi; delete resources.ApiGatewayRestApi; + resources.CustomApiGatewayRestApi.Properties.Name = 'custom-rest-api-name'; return updateStage.call(context).then(() => { - expect(context.apiGatewayRestApiId).to.equal('devRestApiId'); + expect(context.apiGatewayRestApiId).to.equal('customRestApiId'); }); }); it('should resolve with a default api name if the AWS::ApiGateway::Resource is not present', () => { + context.state.service.provider.tracing = { apiGateway: false }; const resources = context.serverless.service.provider.compiledCloudFormationTemplate.Resources; delete resources.ApiGatewayRestApi; options.stage = 'prod'; @@ -517,6 +508,7 @@ describe('#updateStage()', () => { }); it('should resolve expected restApiId when beyond 500 APIs are deployed', () => { + context.state.service.provider.tracing = { apiGateway: false }; providerRequestStub .withArgs('APIGateway', 'getRestApis', { limit: 500, @@ -562,7 +554,6 @@ describe('#updateStage()', () => { return updateStage.call(context).then(() => { const patchOperations = [ - { op: 'replace', path: '/tracingEnabled', value: 'false' }, { op: 'replace', path: '/accessLogSettings/destinationArn', @@ -578,7 +569,7 @@ describe('#updateStage()', () => { ]; expect(providerGetAccountInfoStub).to.be.calledOnce; - expect(providerRequestStub.args).to.have.length(4); + expect(providerRequestStub.args).to.have.length(3); expect(providerRequestStub.args[0][0]).to.equal('APIGateway'); expect(providerRequestStub.args[0][1]).to.equal('getRestApis'); expect(providerRequestStub.args[0][2]).to.deep.equal({ @@ -598,12 +589,6 @@ describe('#updateStage()', () => { stageName: 'dev', patchOperations, }); - expect(providerRequestStub.args[3][0]).to.equal('APIGateway'); - expect(providerRequestStub.args[3][1]).to.equal('untagResource'); - expect(providerRequestStub.args[3][2]).to.deep.equal({ - resourceArn: 'arn:aws:apigateway:us-east-1::/restapis/devRestApiId/stages/dev', - tagKeys: ['old'], - }); }); }); @@ -684,9 +669,9 @@ describe('#updateStage()', () => { }; return updateStage.call(context).then(() => { - expect(providerRequestStub.args[4][0]).to.equal('CloudWatchLogs'); - expect(providerRequestStub.args[4][1]).to.equal('deleteLogGroup'); - expect(providerRequestStub.args[4][2]).to.deep.equal({ + expect(providerRequestStub.args[3][0]).to.equal('CloudWatchLogs'); + expect(providerRequestStub.args[3][1]).to.equal('deleteLogGroup'); + expect(providerRequestStub.args[3][2]).to.deep.equal({ logGroupName: '/aws/api-gateway/my-service-dev', }); }); diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js index 85c27573b65..c8ca295e565 100644 --- a/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js +++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/stage/index.test.js @@ -8,6 +8,7 @@ const childProcess = BbPromise.promisifyAll(require('child_process')); const AwsCompileApigEvents = require('../..'); const Serverless = require('../../../../../../../../Serverless'); const AwsProvider = require('../../../../../../provider/awsProvider'); +const generateCustomResourceZip = require('../../../../../../customResources/generateZip'); describe('#compileStage()', () => { let serverless; @@ -17,6 +18,11 @@ describe('#compileStage()', () => { let stageLogicalId; let logGroupLogicalId; + // Ensure to clear stored custom resource zip path + // As if it was created with previous test file run, it'll be not existent + // due to temporary home directory being removed in a meantime + before(() => generateCustomResourceZip.cache.clear()); + beforeEach(() => { const options = { stage: 'dev', diff --git a/lib/plugins/aws/package/compile/events/stream/index.js b/lib/plugins/aws/package/compile/events/stream/index.js index 8ef01ecf796..6ae838eda85 100644 --- a/lib/plugins/aws/package/compile/events/stream/index.js +++ b/lib/plugins/aws/package/compile/events/stream/index.js @@ -42,6 +42,7 @@ class AwsCompileStreamEvents { if (event.stream) { let EventSourceArn; let BatchSize = 10; + let ParallelizationFactor = 1; let StartingPosition = 'TRIM_HORIZON'; let Enabled = 'True'; @@ -88,6 +89,7 @@ class AwsCompileStreamEvents { } EventSourceArn = event.stream.arn; BatchSize = event.stream.batchSize || BatchSize; + ParallelizationFactor = event.stream.parallelizationFactor || ParallelizationFactor; StartingPosition = event.stream.startingPosition || StartingPosition; if (typeof event.stream.enabled !== 'undefined') { Enabled = event.stream.enabled ? 'True' : 'False'; @@ -166,6 +168,7 @@ class AwsCompileStreamEvents { "DependsOn": ${dependsOn}, "Properties": { "BatchSize": ${BatchSize}, + "ParallelizationFactor": ${ParallelizationFactor}, "EventSourceArn": ${JSON.stringify(EventSourceArn)}, "FunctionName": { "Fn::GetAtt": [ diff --git a/lib/plugins/aws/package/compile/events/stream/index.test.js b/lib/plugins/aws/package/compile/events/stream/index.test.js index 7701870c117..f3c129491dd 100644 --- a/lib/plugins/aws/package/compile/events/stream/index.test.js +++ b/lib/plugins/aws/package/compile/events/stream/index.test.js @@ -731,6 +731,7 @@ describe('AwsCompileStreamEvents', () => { batchSize: 1, startingPosition: 'STARTING_POSITION_ONE', enabled: false, + parallelizationFactor: 10, }, }, { @@ -774,6 +775,13 @@ describe('AwsCompileStreamEvents', () => { awsCompileStreamEvents.serverless.service.functions.first.events[0].stream .startingPosition ); + expect( + awsCompileStreamEvents.serverless.service.provider.compiledCloudFormationTemplate + .Resources.FirstEventSourceMappingKinesisFoo.Properties.ParallelizationFactor + ).to.equal( + awsCompileStreamEvents.serverless.service.functions.first.events[0].stream + .parallelizationFactor + ); expect( awsCompileStreamEvents.serverless.service.provider.compiledCloudFormationTemplate .Resources.FirstEventSourceMappingKinesisFoo.Properties.Enabled @@ -796,6 +804,10 @@ describe('AwsCompileStreamEvents', () => { awsCompileStreamEvents.serverless.service.provider.compiledCloudFormationTemplate .Resources.FirstEventSourceMappingKinesisBar.Properties.BatchSize ).to.equal(10); + expect( + awsCompileStreamEvents.serverless.service.provider.compiledCloudFormationTemplate + .Resources.FirstEventSourceMappingKinesisBar.Properties.ParallelizationFactor + ).to.equal(1); expect( awsCompileStreamEvents.serverless.service.provider.compiledCloudFormationTemplate .Resources.FirstEventSourceMappingKinesisBar.Properties.StartingPosition diff --git a/lib/plugins/aws/package/compile/events/websockets/lib/stage.js b/lib/plugins/aws/package/compile/events/websockets/lib/stage.js index c959d6b3099..17a97dafd64 100644 --- a/lib/plugins/aws/package/compile/events/websockets/lib/stage.js +++ b/lib/plugins/aws/package/compile/events/websockets/lib/stage.js @@ -2,6 +2,10 @@ const BbPromise = require('bluebird'); const ensureApiGatewayCloudWatchRole = require('../../lib/ensureApiGatewayCloudWatchRole'); +const ServerlessError = require('../../../../../../../classes/Error').ServerlessError; + +const defaultLogLevel = 'INFO'; +const validLogLevels = new Set(['INFO', 'ERROR']); module.exports = { compileStage() { @@ -15,7 +19,7 @@ module.exports = { if (provider.apiGateway && provider.apiGateway.websocketApiId) return null; // logs - const logsEnabled = provider.logs && provider.logs.websocket; + const logs = provider.logs && provider.logs.websocket; const stageLogicalId = this.provider.naming.getWebsocketsStageLogicalId(); const logGroupLogicalId = this.provider.naming.getWebsocketsLogGroupLogicalId(); @@ -35,7 +39,19 @@ module.exports = { Object.assign(cfTemplate.Resources, { [stageLogicalId]: stageResource }); - if (!logsEnabled) return null; + if (!logs) return null; + + let level = defaultLogLevel; + if (logs.level) { + level = logs.level; + if (!validLogLevels.has(level)) { + throw new ServerlessError( + `provider.logs.websocket.level is set to an invalid value. Support values are ${Array.from( + validLogLevels + ).join(', ')}, got ${level}.` + ); + } + } // create log-specific resources Object.assign(stageResource.Properties, { @@ -49,13 +65,12 @@ module.exports = { '$context.identity.user', '[$context.requestTime]', '"$context.eventType $context.routeKey $context.connectionId"', - '$context.status', '$context.requestId', ].join(' '), }, DefaultRouteSettings: { DataTraceEnabled: true, - LoggingLevel: 'INFO', + LoggingLevel: level, }, }); diff --git a/lib/plugins/aws/package/compile/events/websockets/lib/stage.test.js b/lib/plugins/aws/package/compile/events/websockets/lib/stage.test.js index 9afc94c923e..5cafa4a1d5a 100644 --- a/lib/plugins/aws/package/compile/events/websockets/lib/stage.test.js +++ b/lib/plugins/aws/package/compile/events/websockets/lib/stage.test.js @@ -1,6 +1,8 @@ 'use strict'; -const expect = require('chai').expect; +const chai = require('chai'); + +const expect = chai.expect; const sinon = require('sinon'); const BbPromise = require('bluebird'); const _ = require('lodash'); @@ -8,12 +10,20 @@ const childProcess = BbPromise.promisifyAll(require('child_process')); const AwsCompileWebsocketsEvents = require('../index'); const Serverless = require('../../../../../../../Serverless'); const AwsProvider = require('../../../../../provider/awsProvider'); +const generateCustomResourceZip = require('../../../../../customResources/generateZip'); + +chai.use(require('chai-as-promised')); describe('#compileStage()', () => { let awsCompileWebsocketsEvents; let stageLogicalId; let logGroupLogicalId; + // Ensure to clear stored custom resource zip path + // As if it was created with previous test file run, it'll be not existent + // due to temporary home directory being removed in a meantime + before(() => generateCustomResourceZip.cache.clear()); + beforeEach(() => { const options = { stage: 'dev', @@ -108,7 +118,6 @@ describe('#compileStage()', () => { '$context.identity.user', '[$context.requestTime]', '"$context.eventType $context.routeKey $context.connectionId"', - '$context.status', '$context.requestId', ].join(' '), }, @@ -152,6 +161,34 @@ describe('#compileStage()', () => { }); }); + it('should use valid logging level', () => { + awsCompileWebsocketsEvents.serverless.service.provider.logs = { + websocket: { + level: 'ERROR', + }, + }; + + return awsCompileWebsocketsEvents.compileStage().then(() => { + const resources = + awsCompileWebsocketsEvents.serverless.service.provider.compiledCloudFormationTemplate + .Resources; + + expect(resources[stageLogicalId].Properties.DefaultRouteSettings.LoggingLevel).equal( + 'ERROR' + ); + }); + }); + + it('should reject invalid logging level', () => { + awsCompileWebsocketsEvents.serverless.service.provider.logs = { + websocket: { + level: 'FOOBAR', + }, + }; + + expect(awsCompileWebsocketsEvents.compileStage()).to.be.rejectedWith('invalid value'); + }); + it('should ensure ClousWatch role custom resource', () => { return awsCompileWebsocketsEvents.compileStage().then(() => { const resources = diff --git a/lib/plugins/aws/package/lib/generateCoreTemplate.test.js b/lib/plugins/aws/package/lib/generateCoreTemplate.test.js index 2a2b8e1df63..65b141f3c46 100644 --- a/lib/plugins/aws/package/lib/generateCoreTemplate.test.js +++ b/lib/plugins/aws/package/lib/generateCoreTemplate.test.js @@ -153,7 +153,10 @@ describe('#generateCoreTemplate()', () => { }, ], }, - Tags: [{ Key: 'FOO', Value: 'bar' }, { Key: 'BAZ', Value: 'qux' }], + Tags: [ + { Key: 'FOO', Value: 'bar' }, + { Key: 'BAZ', Value: 'qux' }, + ], }, }); })); diff --git a/lib/plugins/aws/package/lib/mergeIamTemplates.js b/lib/plugins/aws/package/lib/mergeIamTemplates.js index c302e496f42..cd9f1cd5900 100644 --- a/lib/plugins/aws/package/lib/mergeIamTemplates.js +++ b/lib/plugins/aws/package/lib/mergeIamTemplates.js @@ -192,9 +192,11 @@ module.exports = { violationsFound = 'it is not an array'; } else { const descriptions = statements.map((statement, i) => { - const missing = [['Effect'], ['Action', 'NotAction'], ['Resource', 'NotResource']].filter( - props => props.every(prop => statement[prop] === undefined) - ); + const missing = [ + ['Effect'], + ['Action', 'NotAction'], + ['Resource', 'NotResource'], + ].filter(props => props.every(prop => statement[prop] === undefined)); return missing.length === 0 ? null : `statement ${i} is missing the following properties: ${missing diff --git a/lib/plugins/aws/provider/awsProvider.js b/lib/plugins/aws/provider/awsProvider.js index dce68f18bfb..928042ea8bf 100644 --- a/lib/plugins/aws/provider/awsProvider.js +++ b/lib/plugins/aws/provider/awsProvider.js @@ -232,6 +232,7 @@ class AwsProvider { }); const paramsHash = objectHash.sha1(paramsWithRegion); const MAX_TRIES = 4; + const BASE_BACKOFF = 5000; const persistentRequest = f => new BbPromise((resolve, reject) => { const doCall = numTry => { @@ -243,16 +244,23 @@ class AwsProvider { e.statusCode !== 403 && // Invalid credentials ((e.providerError && e.providerError.retryable) || e.statusCode === 429) ) { + const nextTryNum = numTry + 1; + const jitter = Math.random() * 3000 - 1000; + // backoff is between 4 and 7 seconds + const backOff = BASE_BACKOFF + jitter; + that.serverless.cli.log( _.join( [ - `Recoverable error occurred (${e.message}), sleeping for 5 seconds.`, - `Try ${numTry + 1} of ${MAX_TRIES}`, + `Recoverable error occurred (${e.message}), sleeping for ~${Math.round( + backOff / 1000 + )} seconds.`, + `Try ${nextTryNum} of ${MAX_TRIES}`, ], ' ' ) ); - setTimeout(doCall, 5000, numTry + 1); + setTimeout(doCall, backOff, nextTryNum); } else { reject(e); } @@ -284,7 +292,8 @@ class AwsProvider { if (options && !_.isUndefined(options.region)) { credentials.region = options.region; } - const awsService = new that.sdk[service](credentials); + const Service = _.get(that.sdk, service); + const awsService = new Service(credentials); const req = awsService[method](params); // TODO: Add listeners, put Debug statements here... diff --git a/lib/plugins/aws/provider/awsProvider.test.js b/lib/plugins/aws/provider/awsProvider.test.js index 401adfb03d9..24d25c5a94a 100644 --- a/lib/plugins/aws/provider/awsProvider.test.js +++ b/lib/plugins/aws/provider/awsProvider.test.js @@ -283,6 +283,41 @@ describe('AwsProvider', () => { }); }); + it('should handle subclasses', () => { + class DocumentClient { + constructor(credentials) { + this.credentials = credentials; + } + + put() { + return { + send: cb => cb(null, { called: true }), + }; + } + } + + awsProvider.sdk = { + DynamoDB: { + DocumentClient, + }, + }; + awsProvider.serverless.service.environment = { + vars: {}, + stages: { + dev: { + vars: { + profile: 'default', + }, + regions: {}, + }, + }, + }; + + return awsProvider.request('DynamoDB.DocumentClient', 'put', {}).then(data => { + expect(data.called).to.equal(true); + }); + }); + it('should call correct aws method with a promise', () => { // mocking API Gateway for testing class FakeAPIGateway { diff --git a/lib/plugins/aws/utils/formatLambdaLogEvent.js b/lib/plugins/aws/utils/formatLambdaLogEvent.js index 1359ddb0b39..a0f3469dbeb 100644 --- a/lib/plugins/aws/utils/formatLambdaLogEvent.js +++ b/lib/plugins/aws/utils/formatLambdaLogEvent.js @@ -1,6 +1,6 @@ 'use strict'; -const moment = require('moment'); +const dayjs = require('dayjs'); const chalk = require('chalk'); const os = require('os'); @@ -38,7 +38,7 @@ module.exports = msgParam => { return msg; } const text = msg.split(`${reqId}\t`)[1]; - const time = chalk.green(moment(date).format(dateFormat)); + const time = chalk.green(dayjs(date).format(dateFormat)); return `${time}\t${chalk.yellow(reqId)}\t${level}${text}`; }; diff --git a/lib/plugins/aws/utils/formatLambdaLogEvent.test.js b/lib/plugins/aws/utils/formatLambdaLogEvent.test.js index dacc6cc0f36..3c6260d7d61 100644 --- a/lib/plugins/aws/utils/formatLambdaLogEvent.test.js +++ b/lib/plugins/aws/utils/formatLambdaLogEvent.test.js @@ -1,7 +1,7 @@ 'use strict'; const expect = require('chai').expect; -const moment = require('moment'); +const dayjs = require('dayjs'); const chalk = require('chalk'); const os = require('os'); const formatLambdaLogEvent = require('./formatLambdaLogEvent'); @@ -23,8 +23,8 @@ describe('#formatLambdaLogEvent()', () => { const nodeLogLine = '2016-01-01T12:00:00Z\t99c30000-b01a-11e5-93f7-b8e85631a00e\ttest'; let expectedLogMessage = ''; - const momentDate = moment('2016-01-01T12:00:00Z').format('YYYY-MM-DD HH:mm:ss.SSS (Z)'); - expectedLogMessage += `${chalk.green(momentDate)}\t`; + const date = dayjs('2016-01-01T12:00:00Z').format('YYYY-MM-DD HH:mm:ss.SSS (Z)'); + expectedLogMessage += `${chalk.green(date)}\t`; expectedLogMessage += `${chalk.yellow('99c30000-b01a-11e5-93f7-b8e85631a00e')}\t`; expectedLogMessage += 'test'; @@ -36,8 +36,8 @@ describe('#formatLambdaLogEvent()', () => { '[INFO]\t2016-01-01T12:00:00Z\t99c30000-b01a-11e5-93f7-b8e85631a00e\ttest'; // eslint-disable-line let expectedLogMessage = ''; - const momentDate = moment('2016-01-01T12:00:00Z').format('YYYY-MM-DD HH:mm:ss.SSS (Z)'); - expectedLogMessage += `${chalk.green(momentDate)}\t`; + const date = dayjs('2016-01-01T12:00:00Z').format('YYYY-MM-DD HH:mm:ss.SSS (Z)'); + expectedLogMessage += `${chalk.green(date)}\t`; expectedLogMessage += `${chalk.yellow('99c30000-b01a-11e5-93f7-b8e85631a00e')}\t`; expectedLogMessage += `${'[INFO]'}\t`; expectedLogMessage += 'test'; diff --git a/lib/plugins/create/templates/aws-clojurescript-gradle/build.gradle b/lib/plugins/create/templates/aws-clojurescript-gradle/build.gradle index d10182eddfc..d35977ee41f 100644 --- a/lib/plugins/create/templates/aws-clojurescript-gradle/build.gradle +++ b/lib/plugins/create/templates/aws-clojurescript-gradle/build.gradle @@ -26,7 +26,7 @@ clojurescript { outputDir = 'js/out' target = 'nodejs' optimizations = 'simple' - npmDeps = ['moment': '2.22.2'] + npmDeps = ['dayjs': '1.8.17'] installDeps = true verbose = true } diff --git a/lib/plugins/package/lib/packageService.test.js b/lib/plugins/package/lib/packageService.test.js index 9c591075ecb..4d1a12ed391 100644 --- a/lib/plugins/package/lib/packageService.test.js +++ b/lib/plugins/package/lib/packageService.test.js @@ -627,9 +627,9 @@ describe('#packageService()', () => { }; serverless.config.servicePath = servicePath; - return expect(packagePlugin.resolveFilePathsFromPatterns(params)).to.be.fulfilled.then( - actual => expect(actual).to.deep.equal([handlerFile]) - ); + return expect( + packagePlugin.resolveFilePathsFromPatterns(params) + ).to.be.fulfilled.then(actual => expect(actual).to.deep.equal([handlerFile])); }); it('should include file specified with `!` in exclude params', () => { @@ -639,9 +639,9 @@ describe('#packageService()', () => { }; serverless.config.servicePath = servicePath; - return expect(packagePlugin.resolveFilePathsFromPatterns(params)).to.be.fulfilled.then( - actual => expect(actual).to.deep.equal([handlerFile, utilsFile]) - ); + return expect( + packagePlugin.resolveFilePathsFromPatterns(params) + ).to.be.fulfilled.then(actual => expect(actual).to.deep.equal([handlerFile, utilsFile])); }); it('should exclude file specified with `!` in include params', () => { @@ -652,9 +652,9 @@ describe('#packageService()', () => { const expected = [handlerFile]; serverless.config.servicePath = servicePath; - return expect(packagePlugin.resolveFilePathsFromPatterns(params)).to.be.fulfilled.then( - actual => expect(actual).to.deep.equal(expected) - ); + return expect( + packagePlugin.resolveFilePathsFromPatterns(params) + ).to.be.fulfilled.then(actual => expect(actual).to.deep.equal(expected)); }); }); }); diff --git a/package.json b/package.json index 9679923f80a..d191e0132e0 100644 --- a/package.json +++ b/package.json @@ -138,6 +138,7 @@ "cachedir": "^2.3.0", "chalk": "^2.4.2", "ci-info": "^1.6.0", + "dayjs": "^1.8.17", "download": "^5.0.3", "fast-levenshtein": "^2.0.6", "filesize": "^3.6.1", @@ -157,7 +158,6 @@ "lodash": "^4.17.15", "minimist": "^1.2.0", "mkdirp": "^0.5.1", - "moment": "^2.24.0", "nanomatch": "^1.2.13", "ncjsm": "^4.0.1", "node-fetch": "^1.7.3", diff --git a/tests/integration-all/api-gateway/tests.js b/tests/integration-all/api-gateway/tests.js index c519823219c..adf302225b6 100644 --- a/tests/integration-all/api-gateway/tests.js +++ b/tests/integration-all/api-gateway/tests.js @@ -117,7 +117,7 @@ describe('AWS - API Gateway Integration Test', function() { ].join(','); expect(headers.get('access-control-allow-headers')).to.equal(allowHeaders); expect(headers.get('access-control-allow-methods')).to.equal('OPTIONS,GET'); - expect(headers.get('access-control-allow-credentials')).to.equal('false'); + expect(headers.get('access-control-allow-credentials')).to.equal(null); // TODO: for some reason this test fails for now... // expect(headers.get('access-control-allow-origin')).to.equal('*'); });