Mike from Humanloop here - if you're interested in active learning we'll be around on this thread, also we're looking for fullstack SW engineers and ML engineers - https://news.ycombinator.com/item?id=25992607
Hey Mike - I did some work on an industry active learning system a few years ago. The high level finding was that transfer & nonparametric methods were huge wins, but online uncertainty and "real active learning" hardly worked at all and were super complicated (at least in caffe1 anyway lol).
Can you point to any big breakthroughs that have helped in recent years? Linear Hypermodels (https://arxiv.org/abs/2006.07464) seem promising, but that original experience has left me with some healthy skepticism.
Adding to what Raza as said - to your point on "real active learning" hardly working. I would be interested to hear what approaches you took?
We've found that the quality of the uncertainty estimate for your model is quite important for active learning to work well in practise. So applying good approximations for the model uncertainty for modern sized transformer models (like BERT) is an important consideration
Irrc we were using a Bayesian classification model on top of of fixed pretrained features from transfer, something along the lines of refitting a GP every time the number of classes changed. This was images as opposed to text, and after an epoch classification was ~ok but during training (eg the active bit) we didn't see much benefit.
Hi NMCA,
I'm Raza one of the founders of Humanloop. I totally agree that transfer learning is one of the best strategies for data efficiency and it's pretty common to see people start from large pre-trained models like BERT. Active learning then provides an additional benefit, especially when labels are expensive. For example we've worked with teams where lawyers have to annotate.
I think that you're right that it used to be much to hard to get active learning to work. Part of what we're trying to do is make it easy enough that its worth the benefits.
I'd also point out that people always focus on just the labelling savings from active learning but there are other benefits in practice too:
1) faster feedback on model performance during the annotation process
and 2) Better engagement from the annotators as they can see the benefit of their work.
I'd add that there is a deep connection between active learning and understanding the "domain of expertise" of a model, for example what inputs are ambiguous or low confidence, and which are out of distribution. E.g. BALD is a form of out of distribution detection - a point with high disagreement it not only useful to add to the training pool, it is a point for which the current model has no business making a prediction.
Hi Mike, let me know if I'm getting too into specifics for casual conversation. You mentioned a smaller dataset for training and I'm curious as to how much smaller. Like, for static image recognition, what type of dataset trade off are we looking at. Also, randomly, is there a compute cost trade off or is it just a smarter process?
I read the site, and no one likes cleaning data (no one), and answering questions from a "toddler machine" (for lack of a better term) doesn't sound as bad, but I was curious what potential trade offs there might be.
We've found you can get an order of magnitude improvement in the amount of labelled data you need - but there is some variance based on the difficulty of the problem. Because you are retraining the model in tandem with the data labelling process, there is additional compute associated to an active learning powered labelling process versus just selecting the data at random to label next. But this additional compute consideration is almost always outweighed by the saving of human time spent labelling.
I have a question on the compute aspect regarding your business model, hope I’m not being to nosy..
I tried HL, the experience was stellar (well done!) and it made me think...
To get AL working with a great user experience you need quite a bit of compute. How are you thinking about your margins, e.g the cost to produce what you’re offering versus what customers will pay for it?
Thanks for the feedback! It's a good question re compute. There are some fun engineering and ML research challenges that we are constantly iterating on that are related to this. A few examples
- how to most efficiently share compute resources in a JIT manner (e.g. GPU memory) during model serving for both training and inference (where the use case and privacy requirements permit)
- how to construct model training algorithms that operate in a more online manner effectively (so you don't have to retrain on the whole dataset when you see new examples)
- how to significantly reduce the model footprint (in terms of memory and flops) of modern deep transformer models given they are highly over-parameterised and can contain a lot of redundancy.
