tokenizer / README.md
julien-c's picture
julien-c HF Staff
Add BLOOM license to tokenizer (#4)
d43158e
metadata
license: bigscience-bloom-rail-1.0

Tokenizer used for all BLOOM models

Tokenizer information are provided at https://huggingface.co./bigscience/bloom#preprocessing

TODO: point to paper once it comes out with extra details on the tokenizer