What is the logit distribution of each token in T5?

In the T5 (Text-to-Text Transfer Transformer) model, the logit distribution for each token represents the unnormalized probabilities assigned to every token in the model's vocabulary before the application of a softmax function. Let's break this down:

Logits Explained:

- Logits are the raw output scores generated by the model's final linear layer before the probability conversion. They indicate how well the model "thinks" each token fits as the next token in the sequence.

- Each logit corresponds to a token in the model's vocabulary. For example, if T5's vocabulary size is 32,000, then for each position in the sequence, the model produces a vector of 32,000 logits.

Distribution of Logits:

- The collection of logits is not a probability distribution as they don't sum to one and can be negative or positive values. These are also known as 'unnormalized log probabilities'

- They represent the model's pre-softmax scoring of each potential token. These logit values are what the model uses to determine the relative likelihood of each token.

Conversion to Probabilities:

- These logits are then converted to probabilities using the softmax function. Softmax takes the raw logit values and exponentiates them to ensure positive values, then normalizes the resulting numbers so that they sum to 1, creating a valid probability distribution.

- Mathematically, given logits \( z_i \) for each token \( i \), the corresponding probability \( p_i \) is calculated as follows:

\( p_i = \frac{e^{z_i}}{\sum_j e^{z_j}} \)

How T5 Uses Logits:

- During the decoding phase (e.g., when generating text), T5 uses these logit distributions to select which token to output next. The token with the highest probability (derived from the logits) is typically selected or sampled from.

Practical Implications:

- Understanding logits can be useful for debugging model behavior, implementing custom sampling strategies, or analyzing the model's confidence in each prediction.

- Logits are the raw, intermediate representation of the model's beliefs, and while probabilities are easier to interpret, working with logits can often provide better control and allow for more specialized analysis of model behavior.

In summary, the logit distribution in T5 represents the model's unnormalized raw scores for each token in the vocabulary, which are subsequently transformed into probabilities via softmax for token selection.

Learn more about this