Baseten publishes a Rust and CUDA engine for DeepSeek V4.1 Flash, citing 2,060 tokens a second

Baseten has published DeepSeek Ultrafast, an inference engine for DeepSeek-V4.1-Flash written from scratch in Rust and CUDA, and says it decodes 2,060 tokens a second on eight NVIDIA B300 GPUs. Its README also reports 1,740 tokens a second on four GB300 GPUs and a prefill of 1,000,000 tokens in 11.7 seconds on the B300s.

Inference is the step where a trained model answers a request. Prefill is the model reading your prompt, and decode is it writing the reply, one token (a chunk of text) at a time. Baseten’s Phil Howes wrote that “the core engine is just rust + cuda” with “no pytorch, no sglang, no dsls”, and the company’s own X post announced it on 13 November (UTC).

We built an inference engine for DeepSeek V4.1-Flash from scratch in Rust and CUDA.

4xB300: ~1,740 TPS decode
8xB300: ~2,060 TPS decode
1M-token prefill in ~12s

It’s open-source.

— Baseten (@baseten) November 13, 2026

What the numbers measure

The decode figures come from one prompt, “implement the tetris game. output HTML code directly – no explanation.”, with the model’s thinking mode switched off. The README says each decode step drafts a block of tokens and verifies it, and that a step emits 5.6 tokens on average. Baseten says 30 runs landed at 1,700 to 1,770 tokens a second on the GB300s and 1,980 to 2,170 on the B300s.

Setup	Decode	Step time	Prefill, 1M tokens
4 x GB300 (TP4)	1,740 tokens/s	3.30 ms	13.2 s (76,000 tokens/s)
8 x B300 (TP8)	2,060 tokens/s	2.71 ms	11.7 s (85,000 tokens/s)

The reproduction script in the README streams a single request. The README doesn’t compare the engine with SGLang or vLLM, and it publishes no figures for many users at once.

What it’s built from

The engine serves DeepSeek-V4.1-Flash, which Baseten’s model page describes as a 552 billion-parameter mixture-of-experts model that activates 8 billion parameters per token in prefill and 16 billion in decode, with a 1 million-token context window. A Rust server exposes an OpenAI-compatible API, and the README lists cuDNN and cuBLAS for the vision encoder, OpenCV for image decoding, Hugging Face’s tokenizers and the Velo library for moving cache between servers.

The README says the kernels, the GPU routines that do the maths, were optimised by coding agents that could read the source of FlashInfer and SGLang while they worked. It needs four GB300s, four or eight B300s or four or eight B200s, the CUDA 13 toolkit, the DeepSeek weights and a Redis instance.

Sharing the cache between servers

Each server keeps recent conversations’ KV caches, the model’s stored working memory of a conversation so far, in host memory. When a conversation’s next turn lands on a different server, that server fetches the cache directly from the one that holds it, memory to memory over RDMA (a direct network link between machines’ memory), instead of recomputing the prompt. Redis records which server owns each cache.

Baseten says DeepSeek’s heavy KV cache compression makes the transfers fast enough to run on every request, which removes the need for prefix-aware routing and allows better load balancing. The repo includes an example Kubernetes deployment for several servers.

The licence

Baseten’s X post calls the release open-source. The repo’s licence file is the Elastic License 2.0, which lets you use, copy, distribute and modify the software, bars you from providing it to third parties as a hosted or managed service, and bars removing its licence notices.

Baseten sells the same model as a hosted product: the README says DeepSeek-V4.1-Flash-Fast is available on Baseten Model APIs. In an earlier post, Baseten said it had tuned more than 80 kernels for that model and that, in SGLang, the Python scheduler that queues GPU work limits throughput on flash models. It said it planned to release open-source work on kernel fusion, Programmatic Dependent Launch and host loop minimisation soon, and the README says “more performance improvements and open source releases are coming.”