Optimize custom resources generation

[medikoo]

This should partially address custom resource installation efficiency issues, as discussed recently on slack.

It ensures that custom resource package is prepared once for installed Serverless versions, and in further runs existing package is re-used.

[pmuens]

Thanks for working on this @medikoo 👍

I like the idea of having a cache. However I'm concerned about npm versioning. What if AWS fixed a bug in their SDK and published a new version? In that case we would also need to publish a new Serverless Framework version to "invalidate" the cache.

I know that this might be an edge case we don't focus on. Just wanted to point this out...

[medikoo]

What if AWS fixed a bug in their SDK and published a new version? In that case we would also need to publish a new Serverless Framework version to "invalidate" the cache.

It won't be the case, as when SLS version changes, it'll start caching to different path (note version token)

[codecov-io]

Codecov Report

Merging #7032 into master will decrease coverage by <.01%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #7032      +/-   ##
==========================================
- Coverage    88.5%   88.49%   -0.01%     
==========================================
  Files         229      230       +1     
  Lines        8428     8447      +19     
==========================================
+ Hits         7459     7475      +16     
- Misses        969      972       +3

Impacted Files	Coverage Δ
lib/plugins/aws/deploy/lib/uploadArtifacts.js	90.47% <ø> (ø)	⬆️
lib/plugins/aws/customResources/index.js	98.52% <100%> (-0.14%)	⬇️
lib/plugins/aws/lib/naming.js	99.21% <100%> (ø)	⬆️
lib/plugins/aws/customResources/generateZip.js	88.46% <88.46%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd93739...3a93697. Read the comment docs.

[pmuens]

It won't be the case, as when SLS version changes, it'll start caching to different path (note version token)

Yes, exactly. However what about the following (artificially made-up) example:

1. We ship v1.X which causes the CustomResources cache to be built.
2. There's a bug in the AWS SDK which causes problems with our CustomResoruces code
3. AWS fixes the bug and publishes a new version
4. We need to release a v1.Y so that the cache is re-built because users of v1.X will still use the old, buggy aws-sdk from the cache

This might be a very rare circumstance we can ignore (or I'm missing something here). As the saying goes:

There are only two hard things in Computer Science ...

[pmuens]

I tried it with the following serverless.yml file and it works as expected!

service: test-${self:custom.idx}

provider:
  name: aws
  runtime: nodejs10.x
  versionFunctions: false
  region: eu-central-1
  stage: dev

custom:
  idx: 0

functions:
  hello:
    handler: functions/handler.handler
    events:
      - eventBridge:
          pattern:
            source:
              - aws.cloudformation
            detail-type:
              - AWS API Call via CloudTrail
            detail:
              eventSource:
                - cloudformation.amazonaws.com

[medikoo]

Yes, exactly. However what about the following (artificially made-up) example:

1. We ship v1.X which causes the CustomResources cache to be built.
2. There's a bug in the AWS SDK which causes problems with our CustomResoruces code
3. AWS fixes the bug and publishes a new version
4. We need to release a v1.Y so that the cache is re-built because users of v1.X will still use the old, buggy aws-sdk from the cache

This might be a very rare circumstance we can ignore (or I'm missing something here). As the saying goes:

Technically we're already immune to that issue. As currently also without bumping a framework version, it's very hard for users to upgrade aws-sdk, as both npm install serverless and npm update serverless won't update dependencies if the already installed versions matches version. range in package.json.

So I guess, either with this caching or not, it's only the new patch release that makes a solid fix for users.

[medikoo]

I tried it with the following serverless.yml file and it works as expected!

I've also confirmed by running all integration tests (and picked one issue, which is already fixed in PR)