README
Serverless Step Functions
This is the Serverless Framework plugin for AWS Step Functions.
Requirement
Serverless Framework v2.32.0 or later is required.
TOC
- Install
- Setup
- Current Gotcha
- Events
- API Gateway
- Simple HTTP endpoint
- Custom Step Functions Action
- HTTP Endpoint with custom IAM Role
- Share API Gateway and API Resources
- Enabling CORS
- HTTP Endpoints with AWS_IAM Authorizers
- HTTP Endpoints with Custom Authorizers
- Shared Authorizer
- LAMBDA_PROXY request template
- Customizing request body mapping templates
- Customizing response headers and templates
- Send request to an API
- Setting API keys for your Rest API
- Schedule
- CloudWatch Event
- API Gateway
- Tags
- Commands
- IAM Role
- Tips
- Sample statemachines setting in serverless.yml
Install
Run npm install
in your Serverless project.
$ npm install --save-dev serverless-step-functions
Add the plugin to your serverless.yml file
plugins:
- serverless-step-functions
Setup
Specify your state machine definition using Amazon States Language in a definition
statement in serverless.yml. You can use CloudFormation intrinsic functions such as Ref
and Fn::GetAtt
to reference Lambda functions, SNS topics, SQS queues and DynamoDB tables declared in the same serverless.yml
. Since Ref
returns different things (ARN, ID, resource name, etc.) depending on the type of CloudFormation resource, please refer to this page to see whether you need to use Ref
or Fn::GetAtt
.
Alternatively, you can also provide the raw ARN, or SQS queue URL, or DynamoDB table name as a string. If you need to construct the ARN by hand, then we recommend to use the serverless-pseudo-parameters plugin together to make your life easier.
In addition, if you want to reference a DynamoDB table managed by an external CloudFormation Stack, as long as that table name is exported as an output from that stack, it can be referenced by importing it using Fn::ImportValue
. See the ddbtablestepfunc
Step Function definition below for an example.
functions:
hello:
handler: handler.hello
stepFunctions:
stateMachines:
hellostepfunc1:
events:
- http:
path: gofunction
method: GET
- schedule:
rate: rate(10 minutes)
enabled: true
input:
key1: value1
key2: value2
stageParams:
stage: dev
name: myStateMachine
definition:
Comment: "A Hello World example of the Amazon States Language using an AWS Lambda Function"
StartAt: HelloWorld1
States:
HelloWorld1:
Type: Task
Resource:
Fn::GetAtt: [hello, Arn]
End: true
dependsOn: CustomIamRole
tags:
Team: Atlantis
alarms:
topics:
ok: arn:aws:sns:us-east-1:1234567890:NotifyMe
alarm: arn:aws:sns:us-east-1:1234567890:NotifyMe
insufficientData: arn:aws:sns:us-east-1:1234567890:NotifyMe
metrics:
- executionsTimedOut
- executionsFailed
- executionsAborted
- metric: executionThrottled
treatMissingData: breaching # overrides below default
- executionsSucceeded
treatMissingData: ignore # optional
hellostepfunc2:
definition:
StartAt: HelloWorld2
States:
HelloWorld2:
Type: Task
Resource:
Fn::GetAtt: [hello, Arn]
End: true
ddbtablestepfunc:
definition:
Comment: Demonstrates how to reference a DynamoDB Table Name exported from an external CloudFormation Stack
StartAt: ImportDDBTableName
States:
ImportDDBTableName:
Type: Task
Resource: "arn:aws:states:::dynamodb:updateItem"
Parameters:
TableName:
Fn::ImportValue: MyExternalStack:ToDoTable:Name # imports a table name from an external stack
Key:
id:
S.$: "$.todoId"
UpdateExpression: "SET #status = :updatedStatus"
ExpressionAttributeNames:
"#status": status
ExpressionAttributeValues:
":updatedStatus":
S: DONE
End: true
dependsOn:
- DynamoDBTable
- KinesisStream
- CustomIamRole
tags:
Team: Atlantis
activities:
- myTask
- yourTask
validate: true # enable pre-deployment definition validation (disabled by default)
plugins:
- serverless-step-functions
- serverless-pseudo-parameters
In the example above, notice that we used Fn::GetAtt: [hello, Arn]
to get the ARN for the hello
function defined earlier. This means you don't have to know how the Serverless
framework converts these local names to CloudFormation logical IDs (e.g. hello-world
becomes HelloDashworldLambdaFunction
).
However, if you prefer to work with logical IDs, you can. You can also express the above Fn::GetAtt
function as Fn::GetAtt: [HelloLambdaFunction, Arn]
. If you're unfamiliar with the convention the Serverless
framework uses, then the easiest thing to do is to first run sls package
then look in the .serverless
folder for the generated CloudFormation template. Here you can find the logical resource names for the functions you want to reference.
Adding a custom name for a stateMachine
In case you need to interpolate a specific stage or service layer variable as the
stateMachines name you can add a name
property to your yaml.
service: messager
functions:
sendMessage:
handler: handler.sendMessage
stepFunctions:
stateMachines:
sendMessageFunc:
name: sendMessageFunc-${self:custom.service}-${opt:stage}
definition:
<your definition>
plugins:
- serverless-step-functions
Adding a custom logical id for a stateMachine
You can use a custom logical id that is only unique within the stack as opposed to the name that needs to be unique globally. This can make referencing the state machine easier/simpler because you don't have to duplicate the interpolation logic everywhere you reference the state machine.
service: messager
functions:
sendMessage:
handler: handler.sendMessage
stepFunctions:
stateMachines:
sendMessageFunc:
id: SendMessageStateMachine
name: sendMessageFunc-${self:custom.service}-${opt:stage}
definition:
<your definition>
plugins:
- serverless-step-functions
You can then Ref: SendMessageStateMachine
in various parts of CloudFormation or serverless.yml
Depending on another logical id
If your state machine depends on another resource defined in your serverless.yml
then you can add a dependsOn
field to the state machine definition
. This would add the DependsOn
clause to the generated CloudFormation template.
This dependsOn
field can be either a string, or an array of strings.
stepFunctions:
stateMachines:
myStateMachine:
dependsOn: myDB
myOtherStateMachine:
dependsOn:
- myOtherDB
- myStream
Adding retain property for a stateMachine
There are some practical cases when you would like to prevent state machine from deletion on stack delete or update. This can be achieved by adding retain
property to the state machine section.
stepFunctions:
stateMachines:
myStateMachine:
retain: true
Configuring in such way adds "DeletionPolicy" : "Retain"
to the state machine within CloudFormation template.
CloudWatch Alarms
It's common practice to want to monitor the health of your state machines and be alerted when something goes wrong. You can either:
- do this using the serverless-plugin-aws-alerts, which lets you configure custom CloudWatch Alarms against the various metrics that Step Functions publishes.
- or, you can use the built-in
alarms
configuration from this plugin, which gives you an opinionated set of default alarms (see below)
stepFunctions:
stateMachines:
myStateMachine:
alarms:
topics:
ok: arn:aws:sns:us-east-1:1234567890:NotifyMe
alarm: arn:aws:sns:us-east-1:1234567890:NotifyMe
insufficientData: arn:aws:sns:us-east-1:1234567890:NotifyMe
metrics:
- executionsTimedOut
- executionsFailed
- executionsAborted
- executionThrottled
- executionsSucceeded
treatMissingData: missing
Both topics
and metrics
are required properties. There are 4 supported metrics, each map to the CloudWatch Metrics that Step Functions publishes for your executions.
You can configure how the CloudWatch Alarms should treat missing data:
missing
(AWS default): The alarm does not consider missing data points when evaluating whether to change state.ignore
: The current alarm state is maintained.breaching
: Missing data points are treated as breaching the threshold.notBreaching
: Missing data points are treated as being within the threshold.
For more information, please refer to the official documentation.
The generated CloudWatch alarms would have the following configurations:
namespace: 'AWS/States'
metric: <ExecutionsTimedOut | ExecutionsFailed | ExecutionsAborted | ExecutionThrottled>
threshold: 1
period: 60
evaluationPeriods: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
Statistic: Sum
treatMissingData: <missing (default) | ignore | breaching | notBreaching>
Dimensions:
- Name: StateMachineArn
Value: <ArnOfTheStateMachine>
You can also override the default treatMissingData
setting for a particular alarm by specifying an override:
alarms:
topics:
ok: arn:aws:sns:us-east-1:1234567890:NotifyMe
alarm: arn:aws:sns:us-east-1:1234567890:NotifyMe
insufficientData: arn:aws:sns:us-east-1:1234567890:NotifyMe
metrics:
- executionsTimedOut
- executionsFailed
- executionsAborted
- metric: executionThrottled
treatMissingData: breaching # override
- executionsSucceeded
treatMissingData: ignore # default
CloudWatch Notifications
You can monitor the execution state of your state machines via CloudWatch Events. It allows you to be alerted when the status of your state machine changes to ABORTED
, FAILED
, RUNNING
, SUCCEEDED
or TIMED_OUT
.
You can configure CloudWatch Events to send notification to a number of targets. Currently this plugin supports sns
, sqs
, kinesis
, firehose
, lambda
and stepFunctions
.
To configure status change notifications to your state machine, you can add a notifications
like below:
stepFunctions:
stateMachines:
hellostepfunc1:
name: test
definition:
...
notifications:
ABORTED:
- sns: SNS_TOPIC_ARN
- sqs: SQS_TOPIC_ARN
- sqs: # for FIFO queues, which requires you to configure the message group ID
arn: SQS_TOPIC_ARN
messageGroupId: 12345
- lambda: LAMBDA_FUNCTION_ARN
- kinesis: KINESIS_STREAM_ARN
- kinesis:
arn: KINESIS_STREAM_ARN
partitionKeyPath: $.id # used to choose the parition key from payload
- firehose: FIREHOSE_STREAM_ARN
- stepFunctions: STATE_MACHINE_ARN
FAILED:
... # same as above
... # other status
As you can see from the above example, you can configure different notification targets for each type of status change. If you want to configure the same targets for multiple status changes, then consider using YML anchors to keep your YML succinct.
CloudFormation intrinsic functions such as Ref
and Fn::GetAtt
are supported.
When setting up a notification target against a FIFO SQS queue, the queue must enable the content-based deduplication option and you must configure the messageGroupId
.
Blue green deployment
To implement a blue-green deployment with Step Functions you need to reference the exact versions of the functions.
To do this, you can specify useExactVersion: true
in the state machine.
stepFunctions:
stateMachines:
hellostepfunc1:
useExactVersion: true
definition:
...
Pre-deployment validation
By default, your state machine definition will be validated during deployment by StepFunctions. This can be cumbersome when developing because you have to upload your service for every typo in your definition. In order to go faster, you can enable pre-deployment validation using asl-validator which should detect most of the issues (like a missing state property).
stepFunctions:
validate: true
Disable Output Cloudformation Outputs section
Disables the generation of outputs in the CloudFormation Outputs section. If you define many state machines in serverless.yml you may reach the CloudFormation limit of 60 outputs. If you define noOutput: true
then this plugin will not generate outputs automatically.
stepFunctions:
noOutput: true
Express Workflow
At re:invent 2019, AWS introduced Express Workflows as a cheaper, more scalable alternative (but with a cut-down set of features). See this page for differences between standard and express workflows.
To declare an express workflow, specify type
as EXPRESS
and you can specify the logging configuration:
stepFunctions:
stateMachines:
hellostepfunc1:
type: EXPRESS
loggingConfig:
level: ERROR
includeExecutionData: true
destinations:
- Fn::GetAtt: [MyLogGroup, Arn]
CloudWatch Logs
You can enable CloudWatch Logs for standard Step Functions, the syntax is exactly like with Express Workflows.
stepFunctions:
stateMachines:
hellostepfunc1:
loggingConfig:
level: ERROR
includeExecutionData: true
destinations:
- Fn::GetAtt: [MyLogGroup, Arn]
X-Ray
You can enable X-Ray for your state machine, specify tracingConfig
as shown below.
stepFunctions:
stateMachines:
hellostepfunc1:
tracingConfig:
enabled: true
Current Gotcha
Please keep this gotcha in mind if you want to reference the name
from the resources
section. To generate Logical ID for CloudFormation, the plugin transforms the specified name in serverless.yml based on the following scheme.
- Transform a leading character into uppercase
- Transform
-
into Dash - Transform
_
into Underscore
If you want to use variables system in name statement, you can't put the variables as a prefix like this:${self:service}-${opt:stage}-myStateMachine
since the variables are transformed within Output section, as a result, the reference will be broken.
The correct sample is here.
stepFunctions:
stateMachines:
myStateMachine:
name: myStateMachine-${self:service}-${opt:stage}
...
resources:
Outputs:
myStateMachine:
Value:
Ref: MyStateMachineDash${self:service}Dash${opt:stage}
Events
API Gateway
To create HTTP endpoints as Event sources for your StepFunctions statemachine
Simple HTTP Endpoint
This setup specifies that the hello state machine should be run when someone accesses the API gateway at hello via a GET request.
Here's an example:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: hello
method: GET
definition:
Here You can define an POST endpoint for the path posts/create.
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
definition:
Custom Step Functions Action
Step Functions have custom actions like DescribeExecution or StopExecution to fetch and control them. You can use custom actions like this:
stepFunctions:
stateMachines:
start:
events:
- http:
path: action/start
method: POST
definition:
...
status:
events:
- http:
path: action/status
method: POST
action: DescribeExecution
definition:
...
stop:
events:
- http:
path: action/stop
method: POST
action: StopExecution
definition:
...
Request template is not used when action is set because there're a bunch of actions. However if you want to use request template you can use Customizing request body mapping templates.
HTTP Endpoint with custom IAM Role
The plugin would generate an IAM Role for you by default. However, if you wish to use an IAM role that you have provisioned separately, then you can override the IAM Role like this:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
iamRole: arn:aws:iam::<accountId>:role/<roleName>
definition:
Share API Gateway and API Resources
You can share the same API Gateway between multiple projects by referencing its REST API ID and Root Resource ID in serverless.yml as follows:
service: service-name
provider:
name: aws
apiGateway:
# REST API resource ID. Default is generated by the framework
restApiId: xxxxxxxxxx
# Root resource, represent as / path
restApiRootResourceId: xxxxxxxxxx
functions:
...
If your application has many nested paths, you might also want to break them out into smaller services.
However, Cloudformation will throw an error if we try to generate an existing path resource. To avoid that, we reference the resource ID:
service: service-a
provider:
apiGateway:
restApiId: xxxxxxxxxx
restApiRootResourceId: xxxxxxxxxx
# List of existing resources that were created in the REST API. This is required or the stack will be conflicted
restApiResources:
/users: xxxxxxxxxx
functions:
...
Now we can define endpoints using existing API Gateway ressources
stepFunctions:
stateMachines:
hello:
events:
- http:
path: users/create
method: POST
Enabling CORS
To set CORS configurations for your HTTP endpoints, simply modify your event configurations as follows:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
cors: true
definition:
Setting cors to true assumes a default configuration which is equivalent to:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
cors:
origin: '*'
headers:
- Content-Type
- X-Amz-Date
- Authorization
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-Agent
allowCredentials: false
definition:
Configuring the cors property sets Access-Control-Allow-Origin, Access-Control-Allow-Headers, Access-Control-Allow-Methods,Access-Control-Allow-Credentials headers in the CORS preflight response. To enable the Access-Control-Max-Age preflight response header, set the maxAge property in the cors object:
stepFunctions:
stateMachines:
SfnApiGateway:
events:
- http:
path: /playground/start
method: post
cors:
origin: '*'
maxAge: 86400
HTTP Endpoints with AWS_IAM Authorizers
If you want to require that the caller submit the IAM user's access keys in order to be authenticated to invoke your Lambda Function, set the authorizer to AWS_IAM as shown in the following example:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
authorizer: aws_iam
definition:
HTTP Endpoints with Custom Authorizers
Custom Authorizers allow you to run an AWS Lambda Function before your targeted AWS Lambda Function. This is useful for Microservice Architectures or when you simply want to do some Authorization before running your business logic.
You can enable Custom Authorizers for your HTTP endpoint by setting the Authorizer in your http event to another function in the same service, as shown in the following example:
stepFunctions:
stateMachines:
hello:
- http:
path: posts/create
method: post
authorizer: authorizerFunc
definition:
If the Authorizer function does not exist in your service but exists in AWS, you can provide the ARN of the Lambda function instead of the function name, as shown in the following example:
stepFunctions:
stateMachines:
hello:
- http:
path: posts/create
method: post
authorizer: xxx:xxx:Lambda-Name
definition:
Shared Authorizer
Auto-created Authorizer is convenient for conventional setup. However, when you need to define your custom Authorizer, or use COGNITO_USER_POOLS authorizer with shared API Gateway, it is painful because of AWS limitation. Sharing Authorizer is a better way to do.
stepFunctions:
stateMachines:
createUser:
...
events:
- http:
path: /users
...
authorizer:
# Provide both type and authorizerId
type: COGNITO_USER_POOLS # TOKEN, CUSTOM or COGNITO_USER_POOLS, same as AWS Cloudformation documentation
authorizerId:
Ref: ApiGatewayAuthorizer # or hard-code Authorizer ID
# [Optional] you can also specify the OAuth scopes for Cognito
scopes:
- scope1
...
LAMBDA_PROXY request template
The plugin generates default body mapping templates for application/json
and application/x-www-form-urlencoded
content types. The default template would pass the request body as input to the state machine. If you need access to other contextual information about the HTTP request such as headers, path parameters, etc. then you can also use the lambda_proxy
request template like this:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
request:
template: lambda_proxy
This would generate the normal LAMBDA_PROXY template used for API Gateway integration with Lambda functions.
Customizing request body mapping templates
If you'd like to add content types or customize the default templates, you can do so by including your custom API Gateway request mapping template in serverless.yml
like so:
stepFunctions:
stateMachines:
hello:
events:
- http:
path: posts/create
method: POST
request:
template:
application/json: |
#set( $body = $util.escapeJavaScript($input.json('