#lllmtext #generatingphilosophy The primary mechanism for establishing relationships between these position-aware token representations within each layer of the transformer is self-attention, often implemented as multi-head self-attention. For each token, its vector representation is linearly projected to create three vectors: a Query (Q), a Key (K), and a Value (V). The Query vector can be thought of as representing the token's request for information relevant to its context. The Key vector acts as a label describing the kind of information the token offers. [[The Value]] vector contains the actual content or features of the token to be potentially shared. To determine relevance, the Query vector of a token ((qi)) forms dot products with the Key vectors of all other tokens ((k_j)) within its accessible context. These dot products measure the similarity or compatibility between the information requested ((q_i)) and the information offered ((k_j)). The resulting scores are typically scaled by the square root of the key vector dimensionality ((1/\sqrt{d_k})) to stabilise training. These scaled scores are then passed through a softmax function, which normalises them into a probability distribution ((\alpha{ij})) across all source tokens (j). These (\alpha{ij}) values are the attention weights, summing to 1 for each query token (i). Each attention weight (\alpha{ij}) determines how much of the corresponding Value vector (vj) should contribute to the output representation of token (i). The final output for token (i) from the self-attention mechanism is a weighted sum of all Value vectors in the context: (\sum_j \alpha{ij} v_j). This process allows each token to dynamically construct its updated representation by selectively drawing information from other tokens based on learned notions of relevance for the current context. Multi-head attention performs this QKV calculation multiple times in parallel within a layer, using different learned projection matrices for each 'head'. This allows different heads to specialise in capturing different types of relationships (e.g., syntactic vs. semantic) simultaneously, leading to a richer overall representation after their outputs are concatenated and linearly projected.