Google claims that the cost of AI server memory has exceeded 75%, promoting a dual-track strategy for software and hardware
The SEMICON Taiwan 2026 Memory Summit took place on the 1st, where Nikhil Cherian, Senior Director of Supply Chain Infrastructure at Google Cloud under Alphabet, pointed out that with the popularity of multimodal and mixed expert architectures, AI computation has shifted from being power-limited to memory-limited, with high-performance memory accounting for over 75% of the cost of AI server hardware bill of materials. In the face of capacity, bandwidth, and power consumption bottlenecks, Google is breaking through the AI memory bottleneck through a dual-track strategy of hardware offloading for inference and training, and lossless quantization software algorithms.Google adopts an offloading strategy in hardware architecture, launching TPU 8i for low-latency inference and TPU 8t specialized for large-scale training. The TPU 8i is equipped with 288 GB of high-bandwidth memory, with SRAM capacity on the chip increased threefold to 384 MiB, placing dynamic conversation states and key-value caches on the chip itself to achieve zero chip-off latency. The TPU 8t forms a super-large computing cluster with 9600 chips, achieving a shared pool of HBM at a scale of 2 PB, eliminating chip-off data transfer bottlenecks, along with TPU Direct Storage technology.Google has developed the training-free TurboQuant lossless quantization algorithm, compressing the key-value cache of large models from 32 bits to 3 bits, reducing memory usage by six times without loss of accuracy, resulting in an eightfold acceleration in attention computation, and integrating old-generation DRAM technology to extend the lifecycle of components.
