See my response to /u/nextworddev (https://news.ycombinator.com/item?id=38399702). In short, GPT-4 outperforms my LLM simply because my model is quite small (its smallest version of GPT2). At the moment, I'm trying to figure out if I can train a larger model, say Phi-1.5, and get results comparable to or even better than GPT-4. With enough computing power, I'm sure that would be possible given that:
1. GPT-4 lacks a sufficently advanced enough understanding of Australian law (likely owing to the composition of its training data).
2. Australian legal English can be considered a very small subset of broader Australian English and then English in general, which should make it possible to generate a model with a deep understanding of the domain without as many parameters as one might need to understand English in general, or even a popular dialect of English.
I might be wrong but I think they’re suggesting you start with GPT-4, which as you’ve said is already somewhat knowledgeable, and then either fine tune with or otherwise integrate your law db.
