Introduction

Upstash offers a REST API alongside TCP-based Kafka clients, enabling access to Kafka topics over HTTP. The REST API is particularly valuable in restricted environments, such as mobile or edge devices, as it provides a lightweight alternative to native Kafka clients. By utilizing the REST API, you can eliminate the need for manual management of Kafka clients and connections. It offers convenience and simplicity for interacting with Kafka topics without the complexities associated with native client implementations.

​

Get Started

If you do not have a Kafka cluster and/or topic already, follow these steps to create one.

In the cluster details section of the Upstash Console, scroll down the REST API section. You will see two basic REST API snippets there; the first one is to produce a message to a topic and the second one is to consume messages from a topic using Kafka consumer group mechanism.

• Producer

curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/produce/$TOPIC/$MESSAGE \
  -u {{ UPSTASH_KAFKA_REST_USERNAME }}:{{ UPSTASH_KAFKA_REST_PASSWORD }}

• Consumer

curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/consume/$GROUP_NAME/$GROUP_INSTANCE_NAME/$TOPIC \
  -u {{ UPSTASH_KAFKA_REST_USERNAME }}:{{ UPSTASH_KAFKA_REST_PASSWORD }}

Upstash Kafka REST API uses HTTP Basic Authentication scheme. You should copy the UPSTASH_KAFKA_REST_USERNAME and UPSTASH_KAFKA_REST_PASSWORD from the console and replace then in the code snippets shown above.

​

Produce

To produce a message just replace the $TOPIC variable with a topic name which you’ve created before and replace the $MESSAGE with the message you want to send to the Kafka topic.

curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/produce/mytopic/hello_kafka -u myuser:mypass

This will send the message to the mytopic Kafka topic and return the metadata related to the message as a JSON, like:

{
  "topic": "mytopic",
  "partition": 0,
  "offset": 0,
  "timestamp": 1637743323016
}

​

Consume

To consume messages from the topic, replace the $TOPIC variable with a topic name which you’ve created before, replace the $GROUP_NAME with a meaningful name to be used as the Kafka consumer group id, and $GROUP_INSTANCE_NAME with a name for Kafka consumer instance id.

curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/consume/mygroup/mygroup_instance0/mytopic -u myuser:mypass

This will consume some messages from the mytopic using Kafka consumer group mechanism and return the messages as a JSON array, like:

[
  {
    "topic": "mytopic",
    "partition": 0,
    "offset": 5,
    "timestamp": 1637745824883,
    "key": "",
    "value": "hello-world",
    "headers": []
  },
  {
    "topic": "mytopic",
    "partition": 0,
    "offset": 6,
    "timestamp": 1637745829327,
    "key": "",
    "value": "hello-kafka",
    "headers": []
  },
  {
    "topic": "mytopic",
    "partition": 0,
    "offset": 7,
    "timestamp": 1637745834756,
    "key": "",
    "value": "hello-upstash",
    "headers": []
  }
]

​

Responses

Each API returns a JSON response and they have their own specific structures. When the API call fails for a reason (illegal argument, unauthorized access, invalid API etc), a common error JSON message returned. Its structure is as following:

{error: String, status: Int}

error field contains the error message which explains the cause and status field shows the HTTP status code for the error.

​

Next

Apart from the basic usages explained in this section, there are three categories for the REST API:

• Producer API

• Consumer APIs

• Metadata APIs