Google Cloud TPU v6e: Benchmarking Gemma 3 on LLM Workloads

Moving Large Language Models from prototypes into production reveals a hard truth: infrastructure choices define both performance ceilings and unit economics. Standard benchmarks often overlook a critical fact — not all LLM requests stress silicon the same way. Google Cloud engineers benchmarked Gemma 3 12B and 27B on TPU v6e to answer how the platform behaves under structurally distinct workloads at scale.

What was announced

Google Cloud published a benchmarking study comparing Gemma 3 12B and Gemma 3 27B on TPU v6e, running through vLLM on a GKE Autopilot cluster with a 2x2 chip topology. The tests covered two workload profiles across 16, 32, 64 and 128 concurrent users: classification (ISL ~4000 tokens, OSL ~10 tokens) and generation (ISL 500 tokens, OSL ~1000 tokens).

Key results include a generation performance wall for the 27B model, which plateaus at a 4.12x normalized throughput multiplier at 128 users, while the 12B model scales up to 8.19x. For classification, both models reached comparable peaks of 6.04x to 6.37x, indicating that parameter size matters far less for prefill-heavy tasks.

Why this matters

For CIOs, CTOs and infrastructure architects, the study translates directly into cost and reliability decisions. Choosing the wrong model for the wrong workload can silently inflate GPU/TPU spend or trigger request dropouts under peak load. The benchmarks give data-driven guidance for right-sizing model deployments and configuring autoscaling triggers based on End-to-End latency rather than blunt CPU or memory signals.

Technical details

• Hardware: Google Cloud TPU v6e, single-host node pool, 2x2 chip topology on GKE Autopilot.
• Serving stack: vLLM via vllm-project/tpu-inference plugin.
• Models: Gemma 3 12B and Gemma 3 27B (open weights, via HuggingFace).
• Global serving config: max-model-len=128000, max-num-batched-tokens=8192, max-num-seqs=512.
• Generation scaling: Gemma 3 12B reaches 8.19x at 128 users; Gemma 3 27B plateaus at 4.12x past 64 users.
• Classification scaling: both models reach 6.04x-6.37x normalized throughput at 128 users without saturating TPUs.
• Recommendation: scale on E2E latency and apply VLLM_TPU_BUCKET_PADDING_GAP optimizations to conserve memory.

Cost-effective AI infrastructure must strictly align model selection and serving configurations to the unique input and output profiles of production workloads

Softprom and Google

Softprom is the official partner of Google. Our team helps enterprises design, deploy and optimize Google Cloud workloads, including GKE and TPU-based AI inference platforms for production LLM services.

This content was prepared as part of the Softprom DistriFlow project — an automated system for monitoring and adapting vendor news. Original source: original article.