Get Started with AWS Lambda and Kafka
This tutorial shows how to produce Kafka messages in AWS Lambda. If you want to consume Kafka messages in AWS Lambda then check this one
Create Kafka
First, create an Upstash Kafka cluster and topic following those steps. You will need the endpoint, username and password in the following steps.
Create Project
We will use Serverless Framework to create the application.
Then we will initialize a node project and install axios dependency.
Implement the Lambda Function
Open the handler.js and update as below:
You need to replace the endpoint, username and password above with the values that you copy from the Upstash Console.
The above code simply creates a producer and sends the message to Kafka.
Deploy the Lambda Function
You can deploy your function to AWS by running:
This command will output your URL. The output should be something like this:
Test the Function
Now let’s validate that the messages are pushed to Kafka. We can consume the Kafka topic using the REST API. You can copy the curl code to consume from the Upstash Console.
REST vs Kafka Client
We can also use a native Kafka client (e.g. KafkaJS) to access our Kafka cluster. See the repo for both examples. But there is a latency overhead if connecting (and disconnecting) to the Kafka with each function invocation. In our tests, the latency of the function with REST is about 10ms where it is about 50ms when KafkaJS is used. Kafka client's performance could be improved by caching the client outside the function but it can cause other problems as explained here.
Troubleshooting: If Lambda function outputs internal error
, check the
cloudwatch log (Lambda > Monitor > View logs in CloudWatch).
Was this page helpful?