Google’s TPU 8tx and 8i Max target different bottlenecks

Source: Google.

Google’s 9 November announcement adds TPU 8tx and TPU 8i Max to its eighth-generation family. Both reuse existing silicon in new systems: one prioritises confidential computing, the other memory-intensive inference.

The distinction matters because a useful AI computer includes much more than the accelerator. A processor waiting for data, another chip or an external tool can leave expensive arithmetic capacity idle. Google is changing the parts around its chips to address different sources of that waiting.

8tx extends the security boundary

AMD’s technical account places fifth-generation EPYC processors at the host side of TPU 8tx. They prepare data, coordinate workloads and communicate with the accelerators. AMD’s Secure Encrypted Virtualization technology helps extend protection from the CPU to the attached TPUs.

The system verifies an accelerator’s identity and security configuration before allowing access to protected memory. AMD describes encryption in CPU memory and during transfers between host and accelerator, with hardware isolation while the accelerator executes. That is a more precise description than saying the TPU performs its calculations on encrypted data.

Google’s confidential-computing documentation explains the underlying distinction: hardware isolation and attestation can reduce how much trust a customer must place in the cloud infrastructure. Applications and models still sit inside the trust boundary. A protected machine does not, by itself, establish that an application is correct or that an authorised agent will handle information appropriately.

8tx also brings the low-latency Boardfly network to 8t silicon. Google claims up to twofold lower latency than 8t, with near-zero security overhead. The launch does not supply enough workload detail to treat that figure as a universal speed-up.

An agent that makes several sequential model calls may benefit repeatedly from shorter inference waits. An agent waiting for a remote database, a test suite or a person has another constraint. As Sulat’s examination of inference speed explained, decoding rate and whole-task completion time measure different things.

8i Max doubles bandwidth, not memory capacity

8i Max combines two 8i chips in one package. Google says it doubles compute, on-chip SRAM and high-bandwidth-memory bandwidth relative to 8i. Those are three specific claims, not a claim that every memory specification doubles.

Network World’s account of Google’s briefing gives the capacity as 384GB of HBM, a 33% increase. It reports 17.2TB/s of memory bandwidth and 768MB of SRAM. These detailed figures come from the briefing report rather than a specification table in Google’s launch post.

Capacity determines how much can be kept close to the processor. Bandwidth determines how quickly it can be moved. Google’s architecture documentation describes model parameters moving from HBM into matrix-processing units. Adding arithmetic units without feeding them fast enough can simply move the bottleneck.

For long-context inference, model weights compete with the stored attention state commonly called the KV cache. More capacity can accommodate larger workloads; more bandwidth can help serve them faster. Neither number alone establishes a supported context length or the latency a particular model will achieve at a given level of concurrency.

The software has to keep the hardware busy

The briefing also describes TPU Sync, software that moves model state between accelerator memory, host memory and storage. When an agent pauses, keeping all of its state in scarce accelerator memory can prevent that memory serving another job. Moving it elsewhere and restoring it on demand is a utilisation strategy, with transfer cost becoming part of the trade-off.

This attention to the division of work also runs through the NVIDIA–Rebellions system planned for SK Telecom, which assigns input processing and output generation to different processors. The proposals are different, but both make interconnects and workload placement central to the design.

Google has not specified hardware pricing, regions or a general-availability date in the reviewed announcement. Its general-availability statements concern Cloud Pathways on GKE and GKE dynamic slicing, not the new TPU hardware.