Actual-world benchmarks for Llama-2 13B
Introduction
Within the realm of enormous language fashions (LLMs), integrating these superior programs into real-world enterprise purposes is a urgent want. Nevertheless, the tempo at which generative AI is evolving is so fast that almost all can’t sustain with the developments.
One answer is to make use of managed providers like those supplied by OpenAI. These managed providers supply a streamlined answer, but for individuals who both lack entry to such providers or prioritize elements like safety and privateness, an alternate avenue emerges: open-source instruments.
Open-source generative AI instruments are extraordinarily fashionable proper now and corporations are scrambling to get their AI-powered apps out the door. Whereas attempting to construct shortly, firms oftentimes overlook that with the intention to actually achieve worth from generative AI they should construct “manufacturing”-ready apps, not simply prototypes.
On this article, I need to present you the efficiency distinction for Llama 2 utilizing two totally different inference strategies. The primary technique of inference will probably be a containerized Llama 2 mannequin served through Quick API, a preferred alternative amongst builders for serving fashions as REST API endpoints. The second technique would be the identical containerized mannequin served through Text Generation Inference, an open-source library developed by hugging face to simply deploy LLMs.
Each strategies we’re are supposed to work effectively for real-world use, like in companies or apps. But it surely’s necessary to comprehend that they don’t scale the identical means. We’ll dive into this comparability to see how they every carry out and perceive the variations higher.
What powers LLM inference at OpenAI and Cohere
Have you ever ever questioned why ChatGPT is so quick?
Giant language fashions require a ton of computing energy and as a result of their sheer measurement, they oftentimes want a number of GPUs. When working with massive GPU clusters, firms need to be very aware of how their computing is being utilized.
LLM suppliers like OpenAI run massive GPU clusters to energy inferencing for his or her fashions. With the intention to squeeze as a lot…
