Doing more with less: LLM quantization (part 2)

Observation: The task to summarize the abstract of Attention is all you need paper, the responses are accurate and pretty similar. INT8 has the most lengthy response that’s two sentences long vs. one for FP32 and INT4.

Prompt #7:  A farmer has 17 chickens and cows in total. If the total number of legs is 44, how many cows are there, given that each chicken has 2 legs and each cow has 4 legs, and there are no other animals present?

Note: Correct answer is 12 chickens and 5 cows (12*2 + 5*4 =44)

Observation: All versions of LLM are pretty bad at the arithmetic problem! The only difference being INT8 (71 words) and INT4 (91 words) answers are a little bit more concise than FP32 (272 words).

Savings with a quantized LLM

We can see from the above examples that quantized LLMs are reasonably accurate in their predictions. Now, let’s look at the GPU resource consumption during inference for the various versions of LLMs.

Now let’s look at comparison of resource consumption across different versions of the same LLM.

The GPU consumption during inference reduces slightly as move from FP32 to INT4 (28% to 24%) but the GPU memory consumption reduces dramatically (37% to 8%).

What about enterprise use cases?

For an enterprise looking to benefit from generative AI, LLM needs to be benchmarked for specific use cases rather than encyclopedic knowledge or mathematical skills. These use cases could encompass tasks like customer service chatbot or text summarization that’s focused on a specific domain or company data.

New hire orientation for LLM

An off-the shelf LLM is like a new college graduate, both have a good general knowledge and natural language capabilities. When a new college graduate joins a company, the employee needs to be trained to become familiar with the company (and industry) specific knowledge or tasks before they can be productive. For example, when someone hears the word “cloud”, one would most likely think of other related words (white, dark, sky, rain, storm…). The context is a key factor in determining the priority of these related words. For a LLM, this priority is denoted by its parameters (e.g. weights).

For an IT enterprise, the term “cloud" will have different weight than an agriculture business. For an employee (or LLM) in an IT company, the word “cloud” will have higher weights for related words like computing, AWS and Azure. While an employee (or LLM) for an agricultural business, the word “cloud” will have higher weights for related words like rain, precipitation and storm.

The in-job training for a new employee is similar to fine tuning a LLM. In both instances, the goal is to get better at specific tasks within a given domain. For an employee, this could entail in-house training or shadowing other experienced colleagues. For an LLM, this means fine-tuning the model with a specific set of data or tasks that will adjust the weights resulting in improved performance.

Note: To be fair, it’s worth mentioning that it’s possible to add context or specific knowledge when querying the LLM through retrieval augmented generation (RAG). But providing a context with every query is cumbersome and doesn’t support fast response applications like a support chatbot. This means fine-tuning an LLM with company specific knowledge or tasks becomes a necessity for enterprise use cases.

There are several approaches to fine tuning an LLM, including reinforcement learning from human feedback (RLHF), knowledge distillation with student/teacher models, or parameter efficient fine tuning with LoRA. How can an enterprise fine-tune an off-the shelf LLM without incurring the massive costs (data science skills, infrastructure resources) involved with retraining a model?

Fine-tuning a LLM for common people

InstructLab is an open source project that allows every day users (as opposed to data scientists) to add new knowledge and skills to AI models using a simple question and answer format. This could be used by enterprises who want to customize an LLM with company-specific knowledge or tasks. Let’s put it to practice with sample data from this repository.

We’ll assume this is an insurance company customer service chatbot being asked about information that it doesn’t already have (in this case, a fictional car from the movie, “Back to the Future”). Once again, we’ll use the Mistral 7B Instruct model that we had used earlier.

Prompt (pre fine-tuning):  How much does the flux capacitor cost for DMC-12

Observation: The LLM accurately points out the lack of information for this fictional item.

Now let’s fine-tune this LLM by adding the missing knowledge (i.e. the cost of the flux capacitor ($10,000,000)). In reality, this could be any proprietary data an enterprise may want to add e.g. a product catalog, actuarial table or medical insurance coverage rates.

Now let’s see the results after fine tuning the LLM.

Prompt (post fine-tuning): How much does the flux capacitor cost for DMC-12

Observation: The LLM accurately provides the information provided during the fine tuning of the model. Enterprises can affect the behavior of LLM by tuning it with their proprietary data or customizing its responses tailored to specific tasks.

If you’re interested in learning more about how to use InstructLab to add new knowledge and skills to a LLM of your choice, check out the interactive step-by-step InstructLab demo here.