It took about an hour of Googling OpenCV questions. Also, I've never actually used OpenCV before, and don't use Python regularly.
I haven't tested it against the full set of images. Even if it's not perfect you could at least use this to narrow down ones that are within some threshold, then manually verify they were tagged correctly.
It's much quicker to dump the bag of Skittles out and snap a photo than to arrange them and count them.
My program isn't perfect, but I only spent a couple hours on it. I think it could be improved, and better lightning when taking the photos would help too.
However, I was curious how well it did so I compared it to the author's results. My program misclassifies approximately 80 out of 27,740 Skittles.
I searched for matches and it did indeed find the same match the author found:
(first row is identical, subsequent rows are one different)
Since there are some errors, I did get lucky. However, if it's good enough, you could also manually verify ones that are close.
Anyway, personally I'd rather write code than manually count 27,740 Skittles, but I'm grateful the author did this because it's an interesting dataset to work with.
Author of the article here; I suppose I should confess that I did consider automating the counting, which did seem like an interesting problem... but I rejected the idea, for what I think is good reason: the sorting was the time-consuming part. If we skip the sorting, and depend on the code to get the counting right, and we assume that we are at worst off by one-- or more realistically, off by two, to account for the questionably-identifiable chunks of paste and such-- then how often do we need to go back for manual verification?
With hindsight (although up-front simulation bore this out as well), we see that we would have to go back and manually verify anywhere from 8% (off-by-one) to 25% (off-by-two); that's every fourth pack. Now consider how much more unpleasant that manual counting is, since the Skittles are haphazardly un-arranged in the image. In short, I'm unconvinced that an automated-- while similarly accurate-- accounting would be that much more efficient.
I'm not sure I understand why sorting is necessary.
> Now consider how much more unpleasant that manual counting is, since the Skittles are haphazardly un-arranged in the image.
It's actually very easy: my program outputs images annotated with each recognized Skittle circled with the guessed color: https://imgur.com/a/jlPWXRf You don't need to manually count, just make sure all the circles are the correct colors.
I'm also pretty confident you could improve the accuracy quite a bit by improving the lighting when taking the photos, and/or incorporating something more sophisticated like a neural network (probably trained with Skittles from several bags, separated by color+rejects, with many photos in different random arrangements)
> You don't need to manually count, just make sure all the circles are the correct colors.
I guess this takes me a while, and seems significantly more error-prone to me than the corresponding re-count when they are sorted. Granted, it's pretty easy when the Skittles are sorted as they are in these images. But how quickly can you scan this image: https://imgur.com/a/KpddGdH and check whether there are any errors? And how confident are you in that visual check?
You are right that this approach may be good enough to find a duplicate, which was the primary objective of the experiment. But I had hoped that this might also serve as a useful dataset for future student exercises, in probability, or even in just this sort of computer vision project... but I wanted to have accurate ground truth, so to speak. Inspecting your spreadsheet, it looks like this algorithm is still less than 95% accurate, even if we only evaluate the "clean" images with Uncounted=0.
That also demonstrates the difference between having the possibility to optimize input until the output matches the known values and having to process "real life" images made under different conditions.
The later is certainly more interesting: e.g. I dream of having an OCR that is "always right" no matter how bad the image is.
Controlled lighting is extremely helpful in computer vision and key to the operation of a lot of industrial uses of it. From a system design point of view it makes a lot of sense, but obviously it's not possible in many applications (and researchers obviously want to go after the toughest problems, and would rather mostly solve a hard problem than entirely solve an easier one).
What would it take to determine which brand (skittles, m&ms etc) and which sub-brand (tropical / peanut, etc) and how many calories are on the table?
In my head, you would determine the relative color difference of each piece to determine the brand, but I'm not 100% clear on how that kind of ai works.
Thanks. I have no idea if it's the "right" way to do this. It's the first computer vision problem I've done in a long time.
With better lighting you could probably distinguish between colors of different types of candy, assuming they're different enough. If you can determine the type of candy and know how many calories are in each piece it would be easy to count the calories, of course.
FYI all of your new comments are being marked as [dead] for some reason.
https://imgur.com/t7ivg0V
It took about an hour of Googling OpenCV questions. Also, I've never actually used OpenCV before, and don't use Python regularly.
I haven't tested it against the full set of images. Even if it's not perfect you could at least use this to narrow down ones that are within some threshold, then manually verify they were tagged correctly.
I'll post the code shortly.