Spaces:
Running
on
Zero
Running
on
Zero
docs: update README.md (#1)
Browse files
README.md
CHANGED
@@ -53,7 +53,7 @@ This command will download the `fineweb_edu` and prepare it for training in the
|
|
53 |
python setup/download_prepare_hf_data.py fineweb_edu <MEMORY> --data_dir ./data --seed 42 --nchunks <NCHUNKS>
|
54 |
```
|
55 |
|
56 |
-
to download tokenizer (here llama3), use the
|
57 |
|
58 |
```bash
|
59 |
python setup/download_tokenizer.py llama3 <SAVE_PATH> --api_key <HUGGINGFACE_TOKEN>
|
|
|
53 |
python setup/download_prepare_hf_data.py fineweb_edu <MEMORY> --data_dir ./data --seed 42 --nchunks <NCHUNKS>
|
54 |
```
|
55 |
|
56 |
+
to download tokenizer (here llama3), use the following script:
|
57 |
|
58 |
```bash
|
59 |
python setup/download_tokenizer.py llama3 <SAVE_PATH> --api_key <HUGGINGFACE_TOKEN>
|