Example 1
Arguments: [
"Hi",
256
]
Expected result: [
72,
105,
256
]H and i encode to bytes 72 and 105; EOS 256 follows.
Tokenization and attention · Easy
Byte Tokenization with EOS
Problem 11 of 50 · Python and NumPy
Problem
Encode text as UTF-8 bytes and return their integer values followed by exactly one EOS token. EOS lies outside the byte vocabulary. Preserve whitespace and Unicode exactly; do not split on words or append EOS per character.
def byte_tokenize(text, eos_id):Constraints
- text is a Unicode string of at most 128 characters, without unpaired surrogates.
- 256 ≤ eos_id ≤ 1000. An empty string still returns EOS.
- Inputs satisfy these constraints; invalid-input handling is not graded. Use only Python standard library and NumPy.
Examples
Example 2
Arguments: [
"é",
257
]
Expected result: [
195,
169,
257
]é occupies two UTF-8 bytes, 195 and 169, then EOS 257.