Hi! Model creator here. I happen to be a cultural historian and that's a main use case that I see. It's not complicated to learn about past events but having a general idea of the culture of the time (and its alieness from our perspective) is way harder. Monad is just prototype but there is a real potential for immersing experience through historical LLMs.
Thanks for doing this. We should talk! I am a historian and have been exploring using LLMs in teaching (https://resobscura.substack.com/p/simulating-history-with-ch...). I love this idea. I’m fairly sure at this point that history education will be one of the fields most positively impacted by LLMs, just because there is so much potential for redeploying primary sources in creative ways like this.
Certainly. In fact I see we already follow each other on Twitter :D
And yes totally. The other massive impact could be in source analysis. I have started using Mistral-Hermes for text annotation and it is both impressive and very fast.
Would be interesting to check with a suitably trained LLM if our understand of historical wording is actually correct. Whole parts of history might be written down but lost in translation if you see what I'm saying...
Great project! Do you have a list of the training/fine tuning data that went into it?
A great use would be to enable one to have conversations with Pascal or Leibnitz, etc.
For instance, I published online the complete text of the Mémoires de Saint-Simon (written in 1745-1755, but describing the second part of the reign of Louis XIV and the Régence, 1695-1721).
Saint-Simon was described by his contemporaries to be one of the greats conversationalists of his time. It would be so cool to chat with him.
Classic French transcription seems to be lacking. In particular, "s" used to be printed in a manner very similar to "f", but they're really s.
For example this:
> ce qui augmentoit ſes craintesc'eſt que certe innocente Vierge ne parloit iamais d'autre choſe aux Domeſtiques que du lcge d'Orl'cans donnant à connoitre à la façon dont elle en difcouroit que fon inclination eſtoit toute aux armes
should be spelled like this:
> ce qui augmentoit ses craintes c'est que cette innocente Vierge ne parloit jamais d'autre chose aux Domestiques que du ?? d'Orléans donnant à connoître à la façon dont elle en discouroit que son inclination étoit (or estoit) toute aux armes
Maybe there should be some kind of dictionary step before fine-tuning?
Ah it's completely volontary on my part: I want to keep the historical spelling as much a possible. That's why I used the google books OCR which does a better work at it than Gallica. That's still a bit erased in the current model (I don't think the tokenizer likes this so much).
Ok -- "avoit" instead of "avait" is indeed a different spelling -- but "f" in original text is not a different spelling, it's a different way of writing the same letter s (a different shape, but the same letter).
What is possible is to use a larger learning rate but this will be a hard trade-off with conversational capacities. Fine tuning is currently based on original texts with a synthetic prompt. The issues that people have noticed (repetitions, not remembering what was in the prompt) will be more significant if the learning rate is higher.
Maybe a solution will be to provide two different variant of the same model, one less immersive and more workable, and the other more immersive and buggy.
Yes it requires an extremely large diverse training set for the first unsupervised stage (pre-training). Then fine tune it on the smaller data set. But we may need to wait for the next generation of LLMs that incorporate planning algorithms so that it can better stay focused on its goal for whatever tasks we are asking it to do for research purposes. Otherwise we end up with this https://news.ycombinator.com/item?id=38418974
What did you use to train this? I'm thinking about making a fine tune of my own and if you have some tips to train it easier that would be much appreciated!
This seems like a fine way to steel man a controversy by going directly to the sources. I'd like to be able to, e.g., dial up a debate between Karl Marx and Milton Friedman. Even better if it's possible for Karl to convince Milton or visa versa.
