You must have read a lot of about Serverless Architechture these days and there are some great solutions provided by AWS, Google cloud, IBM cloud and Microsoft and one of those solutions is FaaS, AWS provides this solution through AWS Lambda, google provides Cloud Functions, openwhisk is from IBM and Microsoft provides Azure Functions.
AWS provides AWS Step Functions to create a workflow out of these independent stateless functions which can pass on information.
AWS Step functions is part of AWS Serverless platform, which allows developers to quickly build and run workflow application services without worrying about servers and managing infrastructure which are highly scalable.
“AWS Step Functions makes it easy to coordinate the components of distributed applications as a series of steps in a visual workflow”, as mentioned in AWS website.
With Step functions you can execute workflow tasks as sequential, parallel or as a workflow task that run depending on conditional logic (choices). Each step in a step function is a task that performs a specific task which can take an input from previous task and pass on output to the next. Each step can be implemented using Amazon Lambda and can be monitored for success or failures with CloudWatch logs. Step Functions are executed synchronously.
Think of AWS Step Functions as a state machine, implementing a state machine with AWS is very simple using a JSON format template as defined in Amazon States Language. A Hello World state machine would be like,
Left pane shows the state machine code and right pane is dynamically displays modifications made to state code on refresh.
A State Machine takes in tasks list where each State is a step in Step Functions of Type Task, In the above image HelloWorld is a Type of Task which is the only step in this step function, we will discuss rest of state machine fields in brief later.
States are of following types,
- Pass – This state does not perform any action but instead just passes its input to output.
- Task – Task state executes a unit of work specified in Resource field, this could mostly be a ARN(Amazon Resource Name) of AWS Lambda function. A task may execute for a maximum time length of 1 Year but this optionally takes in 2 more fields TimeoutSeconds and HeartbeatSeconds, which specifies the max Timeout for the designated task to complete and HeartbeatSeconds specifies interval with in this time frame to check for the heartbeat of the Task running.
- Choice – Choice state is for conditional logic or choices, This state takes in an additional field Choices, an array of Choices Rules.
- Wait – Wait state gives you the flexibility to delay the execution for specified duration, it can take in both number of seconds to delay also the exact TimeStamp until which you can delay.
- Parallel – This task lets you execute tasks in parallel, which takes in additional field called Branches which is an array of tasks again.
- Succeed – Next two States are terminal states and its sole purpose is to terminate State machine successfully.
- Fail – also a terminal state which marks State machine as failure but it takes in additional two fields Error and Cause for further diagnosis of the failure.
Except for Choice, Succeed and Fail all States take Next or End field which specifies the next task/step to be executed.
This briefly sums up all the States, for their template types please refer Amazon States Language.
Step functions also provide the flexibility to deal with runtime errors, task failures due to exceptions in lambda functions, you can specify a specific type of error to be handled in template or a default case for handling any other errors, you can either Retry for x attempts or execute a specific step on Failure like below.
(click to enlarge)
Now that we have understood the basic State types of Step Functions let us look at amazon console to create a Step function that takes in a json input and passes it onto next step function to execute it.
We will use AWS Management Console for creating Step Functions, if you have a AWS account or a AWS free tier account, go to AWS Management Console and select Step Functions, we can create a State Machine either to author from scratch or we can select one from the templates provided by AWS for different use cases like a simple sequential state machine to a complex choices one or a state machine that executes in parallel.
After choosing one template, choose an name followed by an IAM role for your State machine, design your step function with your each step representing a specific task and click Create State Machine, in this case we will be calling two AWS Lambda functions one which takes in input, processes it and passes on to next step, notice that in each step we have specified a Resource key which is ARN of the lambda function that we will be invoking, if an invalid ARN is specified, step function will fail with a ResourceNotFoundException.
"cause": "Function not found: arn:arn-address-key:function:service1-lambda (Service: AWSLambda; Status Code: 404; Error Code: ResourceNotFoundException; Request ID: ff7d74a3-2167-11e8-9189-cd3d2434dfd9)"
Once created you can test your Step Function with Sample input and check it’s execution details in AWS Management Console, image below shows all the execution states, input and output of each state.
You can also create State Machines in your favourite IDE using AWS Step Functions Fluent Java API, but I recommend you use AWS console, it is much simpler to create one with the editor and visual workflow.
Now that we have created a State Machine we will look at how to use AWS Lambda to create Task that can be used as a step in Step Function.
AWS Lambda works on push and pull event model which executes a unit of code in response to events in AWS ecosystem. AWS Lambda functions are stateless, and it manages all the resources that are needed for the lambda function to execute.
Consider following in the image below where user has created a file object in S3 and S3 triggers an lambda function according to its event source mapping, AWS Lambda verifies permissions of the invoker in this case S3 and after verification it proceeds with executing the function that notifies user with an email or which makes a log entry which ever operation your lambda function is supposed to do.
AWS Lambda supports function code written in Node.js, Python, and Java and C#. AWS Lambda each trigger runs its own copy of lambda code parallelly if more than one event has triggered lambda, there by scaling the lambda function, default concurrency limit is 1000.
When creating a Lambda function you need to give it an IAM role which has right permissions for it to execute and use AWS resources, therefore you need to grant correct permissions for your function to execute as expected.
You can test your lambda functions locally using AWS SAM Local, your Lambda function can be triggered from the following AWS Service events, you can also trigger a Lambda function upon a Custom application events.
- Amazon S3
- Amazon DynamoDB
- Amazon Kinesis Data Streams
- Amazon Simple Notification Service
- Amazon Simple Email Service
- Amazon Cognito
- AWS CloudFormation
- Amazon CloudWatch Logs
- Amazon CloudWatch Events
- AWS CodeCommit
- Scheduled Events (powered by Amazon CloudWatch Events)
- AWS Config
- Amazon Alexa
- Amazon Lex
- Amazon API Gateway
- AWS IoT Button
- Amazon CloudFront
- Amazon Kinesis Data Firehose
Since we are looking at an example where a lambda function is triggered on S3 event, here is a sample S3 event payload, you can see the bucket and object information along with user information and time of trigger.
AWS Lambda has few Environmental variables that are available for your function and there are few that the you can set like PATH, NODE_PATH, PYTHON_PATH and TZ.
When creating a Lambda function you can create Tags for it which helps you organize and categorize your Lambda functions, you can also create alias for your Lamda functions. AWS execution and its logs and metrics can be seen AWS Cloud Watch and AWS X-Ray console.
Open your AWS Console and create a AWS Lambda function, there is a pretty good document explaining how to create a Lambda function.
Head over to you S3 console and create a event trigger for File upload or object create.
(click to enlarge)
You can see the Lambda function ARN (Amazon Resource Name) here, which is name of the lambda function that you have just created, which processes the file uploaded and sends out and email.
Here is the sample Lambda function written in node that handles the file upload in S3,
When creating Lambda function you can specify the handler function name this is the same as the folder name and handler is inside index.js, you can specify your own handler function name, by default it is exports.hanlder.
You can also create a test event and AWS provides a good list of event templates to choose from in our case we choose S3 Event type.
Head over to Cloud watch to check the execution logs for success or failure.
If you remember the States we created as part of Step Functions of Type:Task, this Lambda function can be used as of those Steps in AWS step functions by specifying Lambda function ARN in Resource field of Step Functions.
We have looked at what AWS Step functions and their use cases and types are, also how we can use AWS Lambda as part of Step functions to achieve desired workflows.
Happy Coding 🙂