Conformal Prediction, LLMs and HuggingFace — Half 1
Giant Language Fashions (LLM) are all the thrill proper now. They’re used for quite a lot of duties, together with textual content classification, query answering, and textual content era. On this tutorial, we are going to present the right way to conformalize a transformer language mannequin for textual content classification utilizing ConformalPrediction.jl.
Particularly, we have an interest within the activity of intent classification as illustrated within the sketch under. Firstly, we feed a buyer question into an LLM to generate embeddings. Subsequent, we prepare a classifier to match these embeddings to potential intents. After all, for this supervised studying downside we want coaching information consisting of inputs — queries — and outputs — labels indicating the true intent. Lastly, we apply Conformal Predition to quantify the predictive uncertainty of our classifier.
Conformal Prediction (CP) is a quickly rising methodology for Predictive Uncertainty Quantification. In the event you’re unfamiliar with CP, you could need to first take a look at my 3-part introductory sequence on the subject beginning with this post.
We’ll use the Banking77 dataset (Casanueva et al., 2020), which consists of 13,083 queries from 77 intents associated to banking. On the mannequin aspect, we are going to use the DistilRoBERTa mannequin, which is a distilled model of RoBERTa (Liu et al., 2019) fine-tuned on the Banking77 dataset.
The mannequin could be loaded from HF straight into our working Julia session utilizing the Transformers.jl package deal.
This package deal makes working with HF fashions remarkably simple in Julia. Kudos to the devs! 🙏
Under we load the tokenizer tkr and the mannequin mod. The tokenizer is used to transform the textual content right into a sequence of integers, which is then fed into the mannequin. The mannequin outputs a hidden state, which is then fed right into a classifier to get the logits for every class. Lastly, the logits are then handed by a softmax perform to get the corresponding predicted chances. Under we run just a few queries by the mannequin to see the way it performs.
# Load mannequin from HF 🤗:
tkr =…
