Customers don’t just look at the cost of a single server when it comes to their AI platform expenditures. Minimizing total cost of ownership (TCO) and energy consumption in the data center are key goals for customers adopting AI, and LLMs in particular given their explosive increase in computational requirements. Text summarization, variable I/O length, CNN / DailyMail dataset | A100 FP16 PyTorch eager mode| H100 FP8 | H100 FP8, in-flight batching, TensorRT-LLM TCO and Energy Efficiency Improvements Llama 2 70B, A100 compared to H100 with and without TensorRT-LLM On Llama 2-a popular language model released recently by Meta and used widely by organizations looking to incorporate generative AI-TensorRT-LLM can accelerate inference performance by 4.6x compared to A100 GPUs.

Text summarization, variable I/O length, CNN / DailyMail dataset | A100 FP16 PyTorch eager mode | H100 FP8 | H100 FP8, in-flight batching, TensorRT-LLM GPT-J-6B A100 compared to H100 with and without TensorRT-LLM Adding TensorRT-LLM and its benefits, including in-flight batching, result in an 8X total increase to deliver the highest throughput. In Figure 1, H100 alone is 4x faster than A100. The following figures reflect article summarization using an NVIDIA A100 and NVIDIA H100 with CNN/Daily Mail, a well-known dataset for evaluating summarization performance. The following benchmarks show performance improvements brought by TensorRT-LLM on the latest NVIDIA Hopper architecture. Summarizing articles is just one of the many applications of LLMs. “It delivers state-of-the-art performance for LLM serving using NVIDIA GPUs and allows us to pass on the cost savings to our customers.” Performance comparison “TensorRT-LLM is easy to use, feature-packed with streaming of tokens, in-flight batching, paged-attention, quantization, and more, and is efficient,” Rao said. Naveen Rao, vice president of engineering at Databricks notes that “it has been an absolute breeze.” TensorRT-LLM improves ease of use and extensibility through an open-source modular Python API for defining, optimizing, and executing new architectures and enhancements as LLMs evolve, and can be customized easily.įor example, MosaicML has added specific features that it needs on top of TensorRT-LLM seamlessly and integrated them into their inference serving. It enables developers to experiment with new LLMs, offering peak performance and quick customization capabilities, without requiring deep knowledge of C++ or NVIDIA CUDA. TensorRT-LLM consists of the TensorRT deep learning compiler and includes optimized kernels, pre- and post-processing steps, and multi-GPU/multi-node communication primitives for groundbreaking performance on NVIDIA GPUs. Those innovations have been integrated into the open-source NVIDIA TensorRT-LLM software, available for Ampere, Lovelace, and Hopper GPUs and set for release in the coming weeks. NVIDIA has been working closely with leading companies, including Meta, Anyscale, Cohere, Deci, Grammarly, Mistral AI, MosaicML, now a part of Databricks, OctoML, Tabnine, and Together AI, to accelerate and optimize LLM inference. But their large size and unique execution characteristics can make them difficult to use in cost-effective ways. Large language models offer incredible new capabilities, expanding the frontier of what is possible with AI.

Benchmark cpu speed