QStash integrates smoothly with Anthropic’s API, allowing you to send LLM requests and leverage QStash features like retries, callbacks, and batching. This is especially useful when working in serverless environments where LLM response times vary and traditional timeouts may be limiting. QStash provides an HTTP timeout of up to 2 hours, which is ideal for most LLM cases.
Example: Publishing and Enqueueing Requests
Specify the api as llm with the provider set to anthropic() when publishing requests. Use the Upstash-Callback header to handle responses asynchronously, as streaming completions aren’t supported for this integration.
Publishing a Request
import { anthropic, Client } from "@upstash/qstash";
const client = new Client({ token: "<QSTASH_TOKEN>" });
await client.publishJSON({
api: { name: "llm", provider: anthropic({ token: "<ANTHROPIC_TOKEN>" }) },
body: {
model: "claude-3-5-sonnet-20241022",
messages: [{ role: "user", content: "Summarize recent tech trends." }],
},
callback: "https://example.com/callback",
});
Enqueueing a Chat Completion Request
Use enqueueJSON with Anthropic as the provider to enqueue requests for asynchronous processing.
import { anthropic, Client } from "@upstash/qstash";
const client = new Client({ token: "<QSTASH_TOKEN>" });
const result = await client.queue({ queueName: "your-queue-name" }).enqueueJSON({
api: { name: "llm", provider: anthropic({ token: "<ANTHROPIC_TOKEN>" }) },
body: {
model: "claude-3-5-sonnet-20241022",
messages: [
{
role: "user",
content: "Generate ideas for a marketing campaign.",
},
],
},
callback: "https://example.com/callback",
});
console.log(result);
Sending Chat Completion Requests in Batches
Use batchJSON to send multiple requests at once. Each request in the batch specifies the same Anthropic provider and includes a callback URL.
import { anthropic, Client } from "@upstash/qstash";
const client = new Client({ token: "<QSTASH_TOKEN>" });
const result = await client.batchJSON([
{
api: { name: "llm", provider: anthropic({ token: "<ANTHROPIC_TOKEN>" }) },
body: {
model: "claude-3-5-sonnet-20241022",
messages: [
{
role: "user",
content: "Describe the latest in AI research.",
},
],
},
callback: "https://example.com/callback1",
},
{
api: { name: "llm", provider: anthropic({ token: "<ANTHROPIC_TOKEN>" }) },
body: {
model: "claude-3-5-sonnet-20241022",
messages: [
{
role: "user",
content: "Outline the future of remote work.",
},
],
},
callback: "https://example.com/callback2",
},
]);
console.log(result);
Analytics with Helicone
To monitor usage, include Helicone analytics by passing your Helicone API key under analytics:
await client.publishJSON({
api: {
name: "llm",
provider: anthropic({ token: "<ANTHROPIC_TOKEN>" }),
analytics: { name: "helicone", token: process.env.HELICONE_API_KEY! },
},
body: { model: "claude-3-5-sonnet-20241022", messages: [{ role: "user", content: "Hello!" }] },
callback: "https://example.com/callback",
});
With this setup, Anthropic can be used seamlessly in any LLM workflows in QStash.
