Latency-Aware Optimization Strategies for Efficient LLMs Serving on Heterogeneous Accelerators

Large Language Models (LLMs) are increasingly deployed in Latency-critical applications, where user experience is strongly influenced by how quickly the model produces its first token and maintains consistent token generation. In practical deployments, these latency requirements are often formalized as Service-Level Objectives (SLO) that set strict bounds on response times. Meeting such objectives requires careful control of both time-to-first-token (TTFT) and token-by-token latency (TBT). In this study, we evaluate three open-source LLMs on GPU, AWS Inferentia2, and Google Cloud TPU v5e. We profile latency across varying batch sizes and input prompt lengths, and leverage these profiles to identify optimal device configurations that maximize RPS while satisfying SLO requirements. We further explore disaggregated configurations that assign prefill and decode stages to different devices, leveraging device-specific strengths for improved efficiency. Our experiments show that disaggregated optimization consistently improves Requests Per Second (RPS) over GPU-only baselines, achieving up to nearly 2× improvement depending in certain workloads. These findings highlight the importance of latency-aware optimization in heterogeneous environments and establish design principles that clarify how prefill and decode stages should be distributed across devices to maximize throughput while satisfying SLOs in large-scale LLM serving systems.