In the rapidly evolving domain of artificial intelligence, deploying large language models (LLMs) efficiently and cost-effectively is paramount for businesses seeking a competitive edge. As a leading web development agency, Voronkin Web Development frequently navigates the complexities of cloud infrastructure to deliver optimal performance for our clients' AI-driven applications. This deep dive explores a critical, yet often invisible, performance bottleneck encountered when serving LLMs on different AWS GPU instance families. Our analysis reveals a staggering 3.7x difference in throughput, not due to obvious code changes or architectural disparities, but to a subtle, underlying hardware-software interaction that remains hidden from standard monitoring tools. Understanding such nuances is crucial for any organization building dependable and scalable AI solutions, especially when considering the significant implications for operational costs and user experience.

The Unseen Chasm: AWS Instance Architectures Under the Microscope

Our investigation centered on two distinct AWS GPU instance types: the g5g.2xlarge and the g6.2xlarge. While both are designed for GPU-accelerated workloads, their underlying architectures present a compelling study in performance divergence. The g5g.2xlarge instance pairs an AWS Graviton2 processor, an ARM-based aarch64 host, with an NVIDIA T4G GPU. The T4G belongs to NVIDIA's Turing architecture, characterized by its Streaming Multiprocessor (SM) version 7.5. In contrast, the newer g6.2xlarge harnesss an x86_64 host alongside an NVIDIA L4 GPU, part of the more advanced Ada Lovelace architecture with SM 8.9. These architectural differences, particularly concerning their GPU capabilities, proved to be the linchpin of our findings.

Beyond the fundamental CPU and GPU distinctions, there are notable differences in memory capacity. The T4G on the g5g instance offers 15,360 MiB of GPU memory, whereas the L4 on the g6 boasts a more generous 23,034 MiB. For our specific workload, serving the google/gemma-4-E2B-it dense reference checkpoint, we employed a custom, pure-JAX port. This approach bypassed common frameworks like PyTorch or vLLM, ensuring that any observed performance variations were attributable directly to the underlying hardware and JAX's interaction with it, rather than framework-specific optimizations or overheads. Crucially, the LLM payload — including its build ID, configuration, and weight bytes — was absolutely byte-identical across both instances. This meticulous control over the software stack and model state allowed us to isolate hardware as the sole variable influencing the dramatic performance disparity.

Setting the Stage for LLM Performance Benchmarking

Accurate performance benchmarking for large language models demands a rigorously controlled environment. Our setup leveraged AWS spot instances for both g5g.2xlarge and g6.2xlarge, a common strategy for cost optimization in non-critical or interruptible workloads. Access to the google/gemma-4-E2B-it model required a valid Hugging Face token. A fundamental requirement was the use of a Deep Learning base AMI tailored to the specific host architecture, as aarch64 and x86_64 images are not interchangeable. This ensured the operating system and core dependencies were correctly aligned with the Graviton2 and traditional x86 processors, respectively.

The installation process itself was streamlined, relying solely on a pip install jax[cuda13] command. This is a significant advantage, as JAX's distribution includes pre-built wheels that bundle the necessary CUDA components, eliminating the need for complex manual CUDA toolkit installations or Rust toolchains. This simplifies the deployment pipeline, a key consideration for rapid development and continuous integration in modern web development and AI projects. Beyond that, to enhance efficiency and reduce cold-start times, we implemented XLA's persistent compilation cache. This cache, pushed to S3 and restored upon instance boot, dramatically cut down setup times. For instance, on a g5g rig, restoring 805 files totaling 12 MB took a mere six seconds, even on a freshly launched instance, demonstrating the power of smart caching for cloud-native AI applications.

A critical aspect of any GPU-accelerated workload measurement is the warm-up phase. XLA performs ahead-of-time (AOT) compilation for specific execution shapes. Since max_new_tokens is a static_argnames entry in our JAX model, each unique combination of (bucket, max_tokens) triggers a compilation. The very first request to a newly initialized engine This means incurs this XLA compilation overhead, leading to significantly longer response times. Our tests showed that a first request on the g5g instance took 18.06 seconds compared to 4.50 seconds when warm – a 4.0x difference in whole-request time. Neglecting this warm-up period can lead to highly misleading performance metrics, misrepresenting an instance's true steady-state capabilities by a factor of four or more. This highlights the importance of meticulous benchmarking methodologies in AI system design and deployment.

Decoding the Throughput Discrepancy

With a robust benchmarking methodology in place, our throughput sweep revealed a stark contrast between the two AWS instance families. We measured performance for generating 64 output tokens, maintaining a concurrency of 1, and performing three repeats per cell, with the median reported. The results unequivocally demonstrated the superior performance of the g6.2xlarge instance. Across various input token lengths – from 41 to 3,593 – the g6 consistently delivered approximately 3.7 times higher decode throughput compared to the g5g. For instance, with 41 input tokens, the g5g achieved 12.9 tokens/second (tok/s) on its internal gauge, while the g6 soared to 48.5 tok/s.

This substantial difference persisted even as input context length increased. With 2,057 input tokens, the g5g's gauge remained at 12.9 tok/s, while the g6 maintained its lead at 48.3 tok/s. It's crucial to distinguish between the \"gauge\" (the engine's steady-state decode counter) and \"end-to-end\" (wall time over the entire request, including prefill). While end-to-end performance naturally decreases with longer input contexts due to the linear cost of prefill, the decode gauge – which specifically measures token generation speed – remained remarkably stable across context ranges on both instances (0.8% on g5g, 0.4% on g6). This distinction is vital for accurate performance analysis; conflating prefill time with decode degradation can lead to erroneous conclusions about an LLM's true generation capabilities.

The consistent decode performance, regardless of context length, indicates that the KV cache was not the limiting factor for decode speed on either instance. Instead, the performance bottleneck was tied to a cost proportional to the model's weights, rather than the input context itself. This observation further narrowed down the potential causes of the performance gap, pointing towards fundamental differences in how the GPUs process the core model operations. The maximum usable context on the g5g was determined to be 4,096 tokens; attempting to serve 5,120 prompt tokens resulted in a prefill transient failure, highlighting practical limitations of the older hardware for larger context windows in real-world LLM applications.

Unmasking the Culprit: The Hidden Cost of Data Type Conversion

To precisely pinpoint the source of the g5g's underperformance, we employed xprof for detailed profiling of 20 decode steps, with the service temporarily halted. The results were illuminating, revealing the hidden inefficiencies that plagued the older instance. On the g5g, a staggering 54.1% of the total kernel time during decode was consumed by data type conversion. Adding to this, another 32.8% was attributed to an fp32 gemvx path, which was entirely absent on the g6. This means that a shocking 87% of the g5g's decode time was not spent on actual mathematical computations essential for token generation, but on overheads related to data manipulation and inefficient floating-point operations.

In stark contrast, the g6 instance exhibited 0.0% time spent on dtype conversion and had no discernable fp32 gemvx path. Its total kernel time for the same decode steps was a mere 362.8 ms, compared to the g5g's 1,466.0 ms. This difference directly translates to the observed 3.7x higher decode throughput on the g6. The root cause lies in the GPU architectures: NVIDIA's Turing (SM 7.5) architecture, found in the T4G, lacks native support for bfloat16. Its primary 16-bit datapath is float16. When a bfloat16 operation is requested on a pre-Ampere GPU like Turing, the XLA compiler doesn't fail; instead, it silently emulates the operation by routing it through fp32 and performing necessary data type conversions. This emulation, while ensuring functionality, comes at a severe performance cost, effectively turning a supposedly efficient 16-bit operation into a much slower 32-bit one with additional conversion overheads.

Further analysis of memory bandwidth utilization corroborated these findings. The g5g instance operated at only 26% of its peak HBM (High Bandwidth Memory) bandwidth roofline, despite its theoretical capacity of 298.083 GiB/s. This low utilization is a direct consequence of the GPU spending most of its time on data conversion and inefficient fp32 operations, rather than feeding data to its compute units at maximum speed. Conversely, the g6, with its L4 GPU, ran at approximately 100% of its HBM bandwidth roofline (279.441 GiB/s), indicating that its compute units were efficiently saturated with useful work. This stark difference in memory bandwidth utilization powerfully illustrates the profound impact of hardware-specific data type support on overall LLM serving performance and resource efficiency.

The Silence of the Logs: Why This Bottleneck Remains Undetected

One of the most insidious aspects of this performance bottleneck is its invisibility. When an incorrect compute data type is encountered by XLA on an incompatible GPU, it does not typically raise an error or issue a warning. Instead, it silently emulates the requested operation. This means that if a bfloat16 operation is specified for a pre-Ampere GPU that lacks native bfloat16 support, XLA will transparently route it through an fp32 path, complete with the aforementioned data type conversions. From a developer's perspective, the code simply runs, albeit much slower, with no explicit indication in standard logs, metrics, or health checks that a significant performance degradation is occurring due to architectural mismatch.

Related Reading

Voronkin Studio specialises in AI and automation services — reach out to discuss your next project.