GitHub rebuilt Git storage for thousands of agents pushing at once

GitHub’s new storage architecture is designed for a repository workload that looks very different from a team of humans making occasional pushes. The company’s target is thousands of concurrent agents writing to the same repository while CI, deployment systems and other agents amplify the resulting read traffic.

In internal testing, GitHub reports supporting 5,000 concurrent agents and 7,200 pushes per second with consistently low read latency. The design, called Project Rubicon, builds on two principles: minimise coordination and decouple storage from compute.

Only the reference update needs agreement

Git objects are immutable and content-addressed, so multiple writers can upload packs without coordinating their contents. The small part that needs agreement is the reference update: which commit a branch or other ref points to.

Rubicon stores packs, indexes and read accelerators in Azure Blob Storage. A separate replicated log service, Ref-CAS, stores references and the pack manifest. A write uploads its pack and then commits one Ref-CAS transaction that updates the manifest and affected refs.

Compute becomes disposable

Requests still arrive through Spokes, GitHub’s routing and load-balancing layer. Spokes assigns repository traffic to a small group of virtual replicas whose local disks are caches rather than authoritative state. If a replica disappears, others continue serving and a replacement rebuilds its cache as needed.

Background compaction runs on separate nodes directly against Blob Storage and Ref-CAS. That keeps maintenance work from consuming the CPU and memory needed for incoming reads and writes.

The hard trade-off is hidden in the refs

GitHub found that a single conditional-write blob capped throughput in the mid-hundreds per second. Sharding refs across segment blobs raised the write ceiling but made reads wait on every segment. Ref-CAS instead uses quorum-committed compare-and-swap transactions that lock only touched references.

Most agents can push to separate branches without contention. Unchanged reads cost a version comparison, while a lagging replica receives current refs and pointers to the packs it needs.

This is a substantial architectural answer to agent-scale development: keep Git’s interface intact, but move durability, coordination and serving into components that can scale independently.

The editorial lens

Software changes become consequential when they meet existing code, deployment habits and user expectations. Compatibility, observability and maintenance capacity often determine the real cost of adopting a new architecture or framework. That is particularly true for foundational libraries and infrastructure, where a small behavioural change can propagate through many downstream systems. In infrastructure, the crucial follow-through is boring but decisive: reproducible benchmarks, clear failure modes, rollback plans and predictable behaviour under load.

Before adopting a technical change, reproduce it on a representative codebase and document the migration path. Compare behaviour, resource use and failure modes with the existing implementation. Check whether tests cover edge cases and whether logs make failures diagnosable. Review release cadence, maintainer responsiveness and the effort required to roll back. A feature that saves time in a demo can create years of maintenance if its abstractions are difficult to inspect or its compatibility story is vague.

A compressed source summary cannot carry every operational detail. It may omit rollout timing, benchmark construction, eligibility, dependencies or conflicting results. That does not make the source wrong; it defines the limit of what the summary alone proves. Treat the initial report as a claim to be tested rather than a universal outcome. Check availability and version details for releases, reproducibility for research, and delivered usage for business claims. Where possible, run a small reversible evaluation and keep its measurements. Corrections and follow-up reporting can materially change the interpretation, especially when the underlying technology, pricing or policy is moving quickly. This distinction between a publisher’s evidence and its broader applicability is essential to responsible technical coverage.

Read GitHub’s full Rubicon engineering write-up.