Tokenization and Information Theory
FoundationNew
0 answered1 foundation4 intermediate1 advancedAdapts to your performance
Question 1 of 6
120sfoundation (3/10)conceptual
In language modeling, "bits per byte" (bpb) measures the average bits assigned per byte of text. Why is bpb used instead of perplexity per token?