Example 1
Arguments: [
[
1
],
2,
3
]
Expected result: [
[
[
-1,
0,
1
],
[
-2,
-1,
0
]
]
]Query positions are 1 and 2; with slope 1 their bias rows are [-1,0,1] and [-2,-1,0].
Tokenization and attention · Medium
ALiBi Attention Bias
Problem 18 of 50 · Python and NumPy
Problem
Return the additive ALiBi bias for cached attention, shape H × query_length × key_length. For head h, query i has absolute position key_length−query_length+i and bias[h,i,j] = slopes[h] * (j−query_position).
Return finite bias values for every entry, including future keys. Causal masking is a separate step and is not requested here. An empty query axis returns one empty list per head.
def alibi_bias(slopes, query_length, key_length):Constraints
- slopes has 1..8 nonnegative finite values.
- 0 ≤ query_length ≤ key_length ≤ 16.
- Inputs satisfy these constraints; invalid-input handling is not graded. Use only Python standard library and NumPy.
Examples
Example 2
Arguments: [
[
0.5,
2
],
1,
3
]
Expected result: [
[
[
-1,
-0.5,
0
]
],
[
[
-4,
-2,
0
]
]
]The query is at position 2. Each head scales distances [-2,-1,0] by its own slope.