add eos token and the end of assistant content
#8
by
rbao2018
- opened
Repository Enhancement and Fixes
tokenizer_config.json
: Added theeos_token
to improve tokenization by clearly defining the end-of-sequence marker, which is beneficial for tasks such as text generation.config.json
: Added support for theTokenClassificationModel
to be compatible with the PPO algorithm in repositories likeverl
, providing configuration support for token classification tasks.modeling_bailing_moe.py
lm_head
Normalization: Fixed the logic so that the normalization operation is carried out regardless of whether it is in training or inference mode, ensuring stable and consistent output.load_banlancing_loss
: Added support and implementation to optimize the distribution of computational load in multi-expert models, improving training efficiency and performance.
I appreciate your contribution; it looks good to me.
m1ngcheng
changed pull request status to
merged