On: Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows

[venusenvy47]

Where do you find this information for each model?

[adrian_b → venusenvy47]

When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".

In that file you can see all tokens and their corresponding numeric codes.

[ComputerGuru → venusenvy47]

The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.