I’ve tried a bunch of swarm frameworks and none of them work that well
We want to build a good swarm framework into our SDK at @LiteLLM
To do this, we want to work with devs that are actually building swarms in prod
If this is you please reply or dm me, would love to chat
LiteLLM can now stream text-to-speech audio on /v1/audio/speech. Your app can play the clip while the provider is still sending chunks.
Here is how to get started with streamed speech:
- Send the request: call POST /v1/audio/speech with "stream": true, or use stream=True in the SDK
- Play the audio: LiteLLM sends each chunk as the provider makes it. A long @ElevenLabs clip starts to play after 0.72 seconds.
- Track the request: when the audio ends, LiteLLM writes the spend row and releases the key’s max_parallel_requests slot.
Merged PR:
github.com/BerriAI/litellm/p…
We’re launching agent evaluations in the @LiteLLM SDK for developers who want to compare models and harnesses on their own tasks.
Before you had to write the eval loop, create temporary workspaces and collect the results in a spreadsheet.
Now you can reuse your agent, define typed tasks with a success check, and call litellm.agent_eval(). You get a scorecard with pass rate, cost, tokens and duration.
We're launching agent evals in the LiteLLM SDK.
You can now compare models and harnesses on your own tasks with litellm.agent_eval().
Read more below ↓
GPT-6.2 Astra and Sol from @OpenAI supported on LiteLLM.
GPT-6.2 Astra is priced at $8 input/ $40 output per 1M tokens, 20% below GPT-6 Astra.
GPT-6.2 Sol is priced at $2 input / $10 output per per 1M tokens.
Details on both models and how to use them with LiteLLM:
models.litellm.ai/models/gpt…models.litellm.ai/models/gpt…
LiteLLM now supports currencies other than USD for prices and spend.
Here is how to use your currency in 3 simple steps:
1 - Set prices in your currency: send pricing_currency: "EUR" with input_cost_per_token and output_cost_per_token on a chat request. LiteLLM converts the prices to USD for spend tracking.
2- Use daily ECB rates: LiteLLM gets the reference rates from the European Central Bank and caches them. You do not set exchange rates yourself.
3- See spend in your currency: on the Usage dashboard, select a currency. The card shows an estimate next to the USD total, with the date of the rate.
More details in the PRs:
github.com/BerriAI/litellm/p…github.com/BerriAI/litellm/p…github.com/BerriAI/litellm/p…
@LiteLLM Global Control Plane API lets you manage gateways across hybrid clouds. This helps with fault tolerance & data residency.
Read more: litellm.ai/blog?post=global-…
Founders building agents with usage-based pricing: how do you handle cost-tracking, spend limits, and charging users? Is it all custom code? Would you be open to having a gateway handle the complexities here?
For context, we're working on a customer portal for people using LiteLLM in prod, and would love to hear from people who are living and breathing these problems
What shipped in LiteLLM this week:
- Customer Portal (beta): serve your own customers through LiteLLM and attribute traffic and inference spend to each customer, team and member.
- /v1/messages is fully in Rust.
- Day 0 model support: Claude Fable 5.2, Claude Mythos 5.2, Gemini 4 Carbon, Gemini 4 Hydrogen and Gemini Decision 1, across Anthropic, Vertex AI, Bedrock, Azure and GitHub Copilot.
- @replicate : video, image and speech models on /v1/videos, /v1/images/generations and /v1/audio/speech.
Join us live for office hours Fridays at noon Pacific to learn more.
Generate videos on @fal through the LiteLLM Gateway, with the cost tracked per clip.
Two simple steps to generate a video on Fal via LiteLLM:
1 - Add the model: put the @fal video model in config.yaml or add in the Admin UI
2 - Send the request: call POST /v1/videos with a prompt, a size and a length in seconds. LiteLLM records the price of the clip based on the length and resolution.
Learn about the models supported: models.litellm.ai/providers
If your AI product is usage based, you’ve probably struggled with setting budgets on your customers, capping them accurately, and load balancing between them
The new @LiteLLM Customer Portal handles it all for you
Out now, read more below
Great walkthrough from Amazon Web Services (AWS) team on Building a self-service LLM gateway with Amazon Bedrock and LiteLLM
Blog: aws.amazon.com/blogs/machine…
S/o to Sitaraman Vijay Krishna, Eric Christensen, Saurav Bhattacharya for the deep dive!
LiteLLM Private Registry is now available to customers with an existing LiteLLM confidentiality agreement.
It provides early fixes for the most severe vulnerabilities. All security fixes still go public. OSS access policy is in development.
litellm.ai/blog?post=litellm…
Gemini 4 Hydrogen from @GoogleDeepMind: $1.50 In/ $7.50 Out per 1M tokens, and it surpasses Argon on the AA Intelligence Index.
1M context, 2.5x faster than 3.8 Flash.
Supported on LiteLLM.
models.litellm.ai/models/gem…