Provisioned Concurrency Does Not Work As Described

[AndrewBarba]

Bug Report

Provisioned Concurrency was recently added in Serverless v1.59 and described in detail in this blog post: https://serverless.com/blog/aws-lambda-provisioned-concurrency/

The blog post makes the claim that enabling provisioned concurrency is as easy as:

functions:
  hello:
    handler: handler.hello
      events:
        - http:
            path: /hello
            method: get
    provisionedConcurrency: 5

This is not true.

What this configuration will do is assign provisioned concurrency to the version number that was deployed, but API Gateway will continue invoking the Lambda with the $LATEST alias (no version). What effectively ends up happening is every deploy keeps assigning provisioned concurrency to each new version and never actually utilizes it. If a developer does not use any version pruning they will quickly rack up an expensive bill paying for new provisioned capacity on each version, and potentially worse, eventually exhaust their reserved (or unreserved) concurrency on their AWS account.

I think this is a critical issue and needs to, at the very least, be addressed in the blog post. Without some form of version cleanup or aliasing I don't think there's much Serverless can do to make this work in a simple configuration like that.

[medikoo]

@AndrewBarba great thanks for reporting that issue

We will shortly (hopefully tomorrow) publish an update that fix the behavior in a framework (to make blog post accurate).

[AndrewBarba]

Thank you really appreciate it

[AndrewBarba]

@medikoo Would the fix you guys are making need to be applied/supported on all event types? Example, the API for specifying an alias/version for APIG is different than the api for ALB

[medikoo]

Current idea is to create an alias latest (or similar) onto which provisioned concurrency will be setup, and ensure that APIGW endpoints will run against that.

If you have any better proposal, we'll definitely open to consider, any help is appreciated.

[AndrewBarba]

Definitely sounds right to me.

[shaheer-k]

The issue still seems to be there with the new version (1.59.3). Every deployment is creating additional provisioned concurrenies. I have set versionFunctions: false.

[medikoo]

Every deployment is creating additional provisioned concurrenies. I have set versionFunctions: false.

@shaheer-k This definitely should not be the case (and it isn't as I was testing, e.g. on example of this simple service: https://github.com/medikoo/test-serverless/tree/lambda-provisioned-concurrency). If it happens for you, can share the config with which I can reproduce that?

[shaheer-k]

@medikoo My obervations are following (severless version: 1.59.3)

1. It is not working on existing stack as its creating new concurrency for each deployment.
2. On a new stack:
   a. It works for the first time
   b. If I modify the provisioned concurrency (from 5 to 6) then cloud formation update failing
   (UPDATE_FAILED: Internal Failure)

These issues making it really hard to use the provisioned concurrency in serverless

[medikoo]

1. It is not working on existing stack as its creating new concurrency for each deployment.

• v1.59.3 won't delete provisioned concurrency configuration for versions published with v1.59.0-2
• v1.59.3 on first deploy will introduce a new special and only version on which provisioned configuration is configured (it's definitely not the case that with each following deploy you'd have new versions created)

b. If I modify the provisioned concurrency (form 5 to 6) then cloud formation update failing
(UPDATE_FAILED: Internal Failure)

This is the issue on AWS side (note Internal Failure comes from CloudFormation stack update, you can also see it in AWS console). We've communicated that issue to AWS, still we have no response on that so far.

I'm clarifying with AWS, the best way to handle that, if going through CloudFormation will be communicated as not stable (for prolonged period), we'll probably reconfigure our support to go through custom resources.

[dkobia]

Definitely some issues. Subsequent deploys leave the provisioned concurrency configuration attached to the special version that is not $LATEST as @medikoo says above.

Also to get this to work with an existing lambda, I had to delete all the versions then set versionFunctions to false. You can use this handy script if you're having issues with that: https://gist.github.com/tobywf/6eb494f4b46cef367540074512161334

[medikoo]

Subsequent deploys leave the provisioned concurrency configuration attached to the special version that is not $LATEST

Yes, that's the other issue, we're looking into. Unfortunately this makes it not reliable for setup that involves API Gateway.

[RogerVFbr]

Problem seems to persist. Any ideas if a fix is underway? This is very important for our project. Provisioned concurrency still seems to be pointing to a function version instead of $LATEST. "versionFunctions" has been set to "false" and alternative versions have been cleaned via script. However running "sls deploy" with some "provisionedConcurrency" value attached to the function declaration creates an alternative function version and points the provisioned concurrency to it. Redeploying the project does not seem to affect the situation. Is it possible that the version stated is equivalent to $LATEST, therefore it would be working? Can anybody please show a way to test if the invocation of latest is making use of the provisioned concurrency with this configuration?

[jplock]

@RogerVFbr i went back to using https://github.com/AntonBazhal/serverless-plugin-lambda-warmup until the provisioned currency issues have been resolved

[medikoo]

I'll reopen until we have all issues mentioned here solved.

[RogerVFbr]

@jplock Hi, thank you for your recommendation. This and other similar solutions seem to assure one constantly warmed up instance. Provisioned concurrency seems to be a more advanced solution, though. According to my research AWS SAM already implements an automated IaC solution on it’s configuration YAML. This feature totally brings Lambda usage to a next level, it would be nice to see Serverless provide a proper solution asap.

@RogerVFbr i went back to using https://github.com/AntonBazhal/serverless-plugin-lambda-warmup until the provisioned currency issues have been resolved

[vavasilva]

Now worked!

version sls:

[medikoo]

All issues (on framework side) should be fixed now. Published as v1.60.2

[pgill]

Framework Core: 1.62.0
Plugin: 3.3.0
SDK: 2.3.0
Components Core: 1.1.2
Components CLI: 1.4.0

Following this: https://serverless.com/blog/aws-lambda-provisioned-concurrency/ and not seeing any provisioned concurrency being set on my function after I deploy.

auth_test:
    handler: src/services/auth.test
    events:
      - http:
          path: v1/auth/test
          method: get
    provisionedConcurrency: 2

I have versioning turned off: versionFunctions: false

It seems like no alias is being created after I deploy: https://www.dropbox.com/s/9iq2xgbh0yv3ukf/Screenshot%202020-01-29%2017.58.26.png?dl=0

Any ideas on what the problem may be here?

[medikoo]

@pgill it's hard to say anything from what you provide.

Can you prepare a minimal case (which doesn't involve any plugins), post full content of its serverless.yml and command that you're using for deployment.

[peterlundberg]

In AWS Provisioned Concurrency can be enabled for a specific Lambda function version (or alias pointing to a version), so without versionFunctions it will likely not work as expected.

(Or is there some way to explicitly create a FunctionVersion when versionFunctions: false and reference it for provisionedConcurrency?)

[medikoo]

@peterlundberg it works with and without versionFunctions. In both cases version is created (as it's needed for provisioned concurrency). Difference is that when versionFunctions option is on, stale versions are not removed

[karocksjoelee]

hey , I am still facing this issue :
I updated the global package to latest , but it still not showing provisioned concurrency config in my Lambda console.

Framework Core: 1.68.0
Plugin: 3.6.8
SDK: 2.3.0
Components: 2.30.2

[medikoo]

I updated the global package to latest , but it still not showing provisioned concurrency config in my Lambda console.

Are you sure you're inspecting provisioned alias?

[karocksjoelee]

@medikoo I am using alias to separate my build environment . So I didn't add --alias provisioned while deploying , do you mean this affects ?

Is there anyway that I can using my own alias and provisioned at same time ?
sls deploy --stage dev --alias dev This is my currently deploy cmd.

sls deploy --alias prod as Production deployment.

[medikoo]

So I didn't add --alias provisioned while deploying , do you mean this affects ?

provisioned alias is configured by Framework automatically, note that provisioned concurrency can only be set on alias (and it won't work for $LATEST)

Therefore to inspect provisioned concurrency in AWS console, you need to visit a provisioned alias for your lambda

[karocksjoelee]

@medikoo umm .. I see .

[niketh90]

So there is no way to set provisioned concurrency on $LATEST . Right?

[shaheer-k]

The issue which I am facing is that when I enable both Provisioned concurrency and Canary, the API invocation is failing due to the alias issue.

[medikoo]

So there is no way to set provisioned concurrency on $LATEST . Right?

Yes, it needs to be on alias that's not $LATEST

[jackmuskopf]

I just ran into an issue with this. The CloudFormation update initiated by Serverless failed to reconfigure the provisioned concurrency for the new version (ie remove it from 23 and apply it to 24). This left the CloudFormation stack in an UPDATE_ROLLBACK_FAILED state.

I ended up deleting and recreating the stack. If I can reproduce this issue, I will update with more details.

Note: that was with version:
Framework Core: 1.72.0
Plugin: 3.6.13
SDK: 2.3.1
Components: 2.30.14

I've since updated, so the problem may be fixed.

[medikoo]

I've since updated, so the problem may be fixed.

We had many fixes related to provisioned concurrency support, since that version. Always ensure to use latest version of a Framework

[jackmuskopf]

I've since updated, so the problem may be fixed.

We had many fixes related to provisioned concurrency support, since that version. Always ensure to use latest version of a Framework

Yes, apologies for not checking the version before posting. Thanks!

[tiivik]

Hi,

I'm experiencing issues with provisionedConcurrency.

serverless --version

Framework Core: 2.29.0
Plugin: 4.5.0
SDK: 4.2.0
Components: 3.7.3

serverless.yml

service: my-lambda
provider:
  name: aws
  lambdaHashingVersion: 20201221
  ecr:
    images:
      appimage:
        path: ./
functions:
  myFunction:
    image:
      name: appimage
    memorySize: 5120
    timeout: 30
    maximumRetryAttempts: 0
    events:
      - http:
          path: bake
          method: POST
          cors: true
    provisionedConcurrency: 3

Running
sls deploy -s dev

Lambda console -> Aliases -> Provisioned concurrency displays the 3 functions ready.

However, invoking the lambda (through API gateway) results in cold starts. Even the first request after deploy results in a cold start.

Any ideas? Thanks!

[pgrzesik]

Hello @tiivik - do the invocation always result in cold starts or only after deployment? I'm not sure how exactly it's taken care of for provisioned concurrency + image on AWS side, but generally, Docker Images has to be optimized before the first invocation which is done automatically by AWS.

[tiivik]

Hello @tiivik - do the invocation always result in cold starts or only after deployment? I'm not sure how exactly it's taken care of for provisioned concurrency + image on AWS side, but generally, Docker Images has to be optimized before the first invocation which is done automatically by AWS.

Thanks for the reply @pgrzesik ! Second request immediately after first results in a warm start as expected, however after waiting ~15 minutes the request results in cold start again. Ran an overnight test invoking function every 30 minutes and resulted 16 cold starts out of 26 requests, while provisioned concurrency was set to 3.

Furthermore, invoking the function in Lambda console

• Test the function without specified alias -> Cold start
• Pick Aliases -> :provisioned -> Test -> Cold start

Similar issues
https://stackoverflow.com/questions/65181821/provision-concurrency-not-keeping-aws-lambda-function-warm-and-it-still-has-init
https://stackoverflow.com/questions/66174999/provisioned-concurrency-has-minor-impact-on-response-time-of-lambda-function
https://www.reddit.com/r/aws/comments/eq5zg9/lambda_provisioned_concurrency_questions/?utm_source=share&utm_medium=web2x&context=3

Edit:
In order to rule out something potentially getting initialised in the function handler I tested it with a minimal function where response is immediately returned (warm execution time of 1ms).

On the first request or requests coming in ~15minutes later, they are cold starts although concurrency is enabled:

See the 700ms init time – isn't the whole point of provisioned concurrency to eliminate this?

[pgrzesik]

That is indeed quite surprising @tiivik. However, looking at your report, it seems like it's an issue on AWS side as if I see correctly, the provisioned concurrency seems to be setup properly. Do you run into the same problem when not using Images to package your functions?