I am an author on the paper, and just saw this thread. Thanks for all of your comments. Note this work focuses on errors in test sets, not training labels. Test sets are foundation of benchmarking for machine learning. We study test sets (versus training sets), because training sets are already studied in-depth in confident learning (https://arxiv.org/pdf/1911.00068.pdf) and because before this paper, no one had studied (at scale, for lots of datasets) if these test sets had errors in the first place (beyond just looking at the well-known erroneous ImageNet dataset, for example) and do these test sets affect benchmarks. The answer is, yes, if the noise prevalence is fairly high (above 10%). The takeaway, is you must benchmark models on clean test sets, regardless of noise in your training set, if you want to know how your model will perform in the real world.
One of the other comments mentioned that it may be better to treat Imagenet as a multi-label problem. Do you see value in treating the eval set as multi-label instead of single-label? Specifically, it may not be an erroneous label if there's actually multiple things present in the image.
Hi, friends. With NeurIPS behind us and ICML ahead, maybe you want to do some deep learning. Inspired by Justin Johnson's original work benchmarking the older GTX GPUs, I extended this work to the new RTX GPUs with benchmarks for most ResNet architectures on ImageNet and CIFAR. Along the way, I discovered a dramatic difference in performance based on how you position your GPUs. Enjoy :)
Hi, Hackers. I'm excited to share confident learning for characterizing, finding, and learning with label errors in datasets. To promote and standardize future research in learning with noisy labels and weak supervision, I've also open-sourced the cleanlab Python package: https://pypi.org/project/cleanlab/
Abstract: Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) has emerged as an approach for characterizing, identifying, and learning with noisy labels in datasets, based on the principles of pruning noisy data, counting to estimate noise, and ranking examples to train with confidence. Here, we generalize CL, building on the assumption of a classification noise process, to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This generalized CL, open-sourced as cleanlab, is provably consistent under reasonable conditions, and experimentally performant on ImageNet and CIFAR, outperforming recent approaches, e.g. MentorNet, by 30% or more, when label noise is non-uniform. cleanlab also quantifies ontological class overlap, and can increase model accuracy (e.g. ResNet) by providing clean data for training.
This is a follow-up to the previous post "I built Lambda's $12,500 deep learning rig for $6200" which had around 480 upvotes on Reddit. That previous build had only 3-GPUs and took some shortcuts. In response to the hundreds of comments on that post, including comments by the CEO of Lambda Labs, I built and tested multiple 4-GPU rigs. I'm back to share a near-perfect 4-GPU deep learning rig with the highest performance and reliability, no thermal throttling, and lowest cost. This build is nearly identical to Lambda's 4-GPU workstation, but costs around $4000 cheaper. Happy building!
Very nice build indeed! I was comparing it with Lambdas build and it's pretty good on comparison, any thoughts on the Threadripper vs Intel? Also regarding the PSU, I've seen some pretty nasty reviews on that one (https://www.amazon.com/Rosewill-HERCULES-1600-Continuous-Has...). How did yours perform so far?
Hi hackers! I built a deep learning workstation comparable to Lambda's 4-GPU ( RTX 2080 ti ) rig for about half the price. So you can do the same, I'm sharing a time-lapse of the build, the parts list, the receipt, and benchmarking versus Google Compute Engine (GCE) on ImageNet. You save $1200 (the cost of an EVGA RTX 2080 ti GPU) per ImageNet training to use your own build instead of GCE. The training time is reduced by over half. In the post, I include only 3 GPUs in my build, but the build (increase PSU wattage) will support a 4th RTX 2080 TI GPU for $1200 more ($7400 total). Happy hacking!
