Computer vision has been deeply integrated in manufacturing for 20+ years. if you've brushed your teeth or drank a sports drink in the last 10 years, your toothbrush or bottle has probably gone through a vision system that I write the software for.
Agreed, I worked on manufacturing quality assurance software that controlled vision to detect particles in vials of medicine 20 years ago. The main thing that has changed is the quality of the camera has greatly increased and the price of the camera has greatly decreased.
Yeah, when I started, we were using RS-170 cameras connected to $30k Cognex acquisition boards (all analog). The switch over to USB and then GigE has been fantastic.
I miss it in a way. USB cameras were just coming online and we're very good yet when I switched to a different industry. It looks like the company was acquired into an automation machinery parent company.
God, the first generations of both USB and Ethernet cameras positively SUCKED. Flaky, buggy, and expensive. Our first foray into Ethernet cameras was from a company called Opteon. They had taken a stock Intel ethernet card and flashed custom firmware onto it to support their custom framing. If the cards and the cameras didn't match exact versions, you could end up bricking one or both of them. They had to be sent back to the vendor to be fixed.
edit: Oh, hey! Opteon still exists! I'm sure their products are much better than those first generations.
oops, meant to say "were NOT very good yet when I switched to a different industry". Apparently I failed at typing and paying attention to a meeting at the same time. :)
Same here, RS-170 into Cognex MVS8100 ca. 20 years ago. Their pattern matching was gold standard at the time. We also used Matrox but I was a Cognex man at that time. Some of those systems are still running today but I do not miss the days of analog video one bit.
Aye. Back at university, 19-18 years ago now, I had a "mandatory"[0] year in "industry"[1], and some of the job advertisements were for adding computer vision to some process or other. Given how… out of date the teacher introductory module in the final year course was, there was no way this was students being asked to actually create those vision systems.
[0] scare quotes because I had the completely free choice between two otherwise identical degrees, one of which had a mandatory year in industry and the other did not, because the UK council tax system demands a different rate if you're a student on a course with a mandatory year in industry
[1] also a cheat, I worked for an academic research lab
I think it's funny that by you providing a major employer where you don't work nerd sniped a couple of us into guessing the other players in the space.
My best advice if you want to break into manufacturing is to pair Machine Learning / Computer Vision skills with an Engineering degree (Chemical, Mechanical or similar).
I work in heavy industry (Steel Mill) most people that get hired around here have Engineering degrees. Mining, Oil and Gas is quite similar.
We do engage with data science people etc (Usually as consultants) but there can be a lot of friction as oftentimes the conversation gets lost in translation. If you can fluently speak 'engineer' and 'data scientist' you would make yourself a very attractive hire.
Lots of materials science interest in this area. Applications are pretty diverse, but there are definitely opportunities at the microscope OEMs, as well as in manufacturing - steel and metals processing like sibling mentioned, also aerospace (I know some aerospace R&D folks that are doing a bunch of semantic segmentation for microstructure analysis - for like engine disk material design and that sort of thing) and other sectors. You might look into the additive manufacturing OEMs if that catches your interest, process monitoring (video and acoustic) is a big deal there
You could also look into the lab automation sector, that is taking off in a big way recently for materials and chemistry (round 2 for chemistry i guess). Chemspeed is the big name but there’s a bunch of smaller companies doing cool vision and robotics stuff in this area
Learn to get into machine vision? Start doing projects in the area and also learn about the mechanical, controls and electrical components of the equipment.
Just about anything you can think of, we've provided inspection equipment for it...
Toothbrushes, drink bottles, oil bottles, more labels than you can imagine, pharmaceuticals, credit cards, fast food packaging, retail packaging, wine bottles, liquor bottles, beer cans, injection molded plastics, IBM bottles, insert molded bottles, cosmetics, raw foods, countless barcodes, and many, many more.
Hijacking to post another example. I know from a previous work experience that automated dye-penetrant inspection (https://en.wikipedia.org/wiki/Fluorescent_penetrant_inspecti...), using CV for the automatic detection of indications, has been used in aerospace for at the very least 15 years (and quite likely much more).
Yeah we use them to do a bunch of checks on our lines, building out a warehouse will be ready in 2024 so using their latest computer vision tech integrated with all equipment controlled via SCADA
Honestly though, the suits i worked with, were all very dated and used hand constructed feature filters etc. to detect flaws. Usually, it was easier to adapt the environment (exclude external light etc.) instead of lengthy tuning sessions for the installer.
Usually the industrial cameras were also designed, so that local maintainers could readjust them, which excluded complex programming and happened in simple wizards or excel like programming surfaces. There was no time planned in to "retrain" further once the line was running. And it was cheap and good enough that way.
Thus the "cutting" edge tech seemed to be eternally 20 years behind the cutting edge in other sectors relying on machine vision.
We use "smart" cameras from Keyence and Cognex but the really interesting work tends to still be in PC-based, hand-coded vision systems. Usually hand-crafted C++ or C# code but increasingly using neural networks for some, usually non-quantitative (e.g. locating but not measuring), solutions.
I interviewed at Cognex 15 years ago. They eliminated their EE department down to one H-1B who broke down in tears as I tried to figure out why I was being interviewed by people with no knowledge about the job. They were solely interested in the ability to reverse engineer something without any documentation. It was clear they were just repackaging cheap SZ camera modules in overpriced yellow boxes. Everyone was glowing about their "legacy" product line from before they canned their engineers. Nothing but crickets when asked about the new stuff. The founders had constructed this weird personality cult around themselves. Glad I dodged that bullet.
As an integrator of vision solutions, smart cameras firmly occupy the space of "Nobody Ever Got Fired For Buying IBM"
Though with recent developments in machine learning, the case for PC-based solutions is a lot easier now than before. Behind all the fluff and shiny marketing, the incumbents are very stagnant.
15 years ago, I used to claim that there were more smart cameras sitting in engineers desk drawers than there were running in production. I think that was true until about 7-8 years ago.
Definitely not the case at my old job - we deployed a bit over 200 Cognex In-Sight cameras over my five-year stint there, almost all for bespoke inspection applications for customers.
They gave me plenty of swag, but if I wanted to play with one of their cameras I'd have to go out on the production floor.
Sounds like you worked for an integrator, so it stands to reason that you had a high success rate. Cognex, Keyance, DVT, et al sold a lot of smart cameras in batches of 1 and 2 to non-vision experienced engineers based on the lie that they could bolt it up to a conveyor and in an afternoon of programming on their game controller they could be up and running and miraculously improving their quality by 30% by next Monday. I think the vast majority of these cameras never saw production.
As an OEM the ones we deployed definitely saw (and are probably still seeing) production. I can personally confirm off the top of my head that Proctor & Gamble, Revlon, Duracell, Mars Candy, Bausch & Lomb, Pfizer, Gilead Sciences, Boehringer Ingelheim (and more) use Cognex cameras in their product packaging lines.
The promise is - as you say yourself - in systems that are easily maintained by non-vision experienced engineers. As I noted in another post, these are usually controls engineers that overwhelmingly prefer ladder logic on their PLCs and have little exposure to modern software engineering practices such as source control. Obviously it's not "up in an afternoon" - any sales rep that promised that got sent away (Keyence, I'm looking at you) - debugging consists of a lot of product test runs and more mechanical/controls work and definitely takes more than a day.
I tried on more than one occasion to put forward PC-based systems, but the customers wanted the smart cameras. Though I did frequently use OpenCV for batch image analysis in-house, I ought to write an article or two about that bit.
>I can personally confirm off the top of my head that Proctor & Gamble, Revlon, Duracell, Mars Candy, Bausch & Lomb, Pfizer, Gilead Sciences, Boehringer Ingelheim (and more) use Cognex cameras in their product packaging lines.
>I tried on more than one occasion to put forward PC-based systems, but the customers wanted the smart cameras.
Oh, all those that you listed also use PC-based systems. I know because all of them are also customers of ours.
Your end-users, do they mess around with the spreadsheet programming interface, via some kind of remote PC interface or not at all? We deployed quite a few Cognex InSights in the early days but version control and distribution of updates was a major headache.
Sidenote, I feel partly responsible because we bought a ton of systems from McGarry's previous company Acumen, then I guess he took off and created the InSight. Colorful guy, I remember him showing up with his fancy Porsche around that time...
> version control and distribution of updates was a major headache
I wound up writing an in-house tool that pulls the program files from each camera on the machine LAN over FTP and commit them to a local Git repo. There was also some futzing around with XML to get the backup metadata to work seamlessly, but it's not too hard to figure out.
Now getting co-workers to use Git and not various combinations of "Copy of (1) Copy of visionproject (FINAL) 3-2-16 2a.zip" was a different challenge.
Apologies, I forgot the rest of the question. The end customers generally only interacted with the Cognex cameras via HMI applications (VisionView on panels running various versions of FactoryTalk View), rarely would we hear about their plant engineers actually re-programming the cameras.
I would agree, many of those cameras are mouldering in our reclaim area now. But the newer generations are powerful enough that we can turn normal manufacturing engineers loos on simpler vision tasks and leave the challenging applications to more traditional systems.
Doesn't even mention National Instruments. This article is clearly cheerleading for a bunch of startups, and wants us to be ignorant of the larger picture. Robots have been picking up and stacking junk on assembly lines since the 1990s at the latest.
That demo of real-time blob detection and sorting by color filtering was doable in 1998. Earlier than that, even. I've found about 90% of the work in vision applications in industrial packaging is in the product handling and scene setup - focal length, lens selection, exposure time, etc. - all things familiar to a photographer. The last 10% is almost always handled by bog simple algorithms that can be more or less cobbled together from OpenCV's examples and boilerplate, the most complicated usually being OCR.
The value-add of these dedicated industrial vision systems is in integration. Fanuc's iRVision is good at sending spatial data back to the robot controller, but the interface itself is a horrid kludge that specifically requires Internet Explorer and in-person training at their own (admittedly very nice) facilities and promises of litigation if you so much as think about sharing documentation with co-workers.
Recording images during trial runs with their native tooling was impossible, as their under-powered processor couldn't handle saving 640x480 images at 10fps while also running the vision application. So we resorted to recording test runs by feeding the live view OBS, and everyone thought I was some kind of wizard for even considering that.
At least Cognex's In-Sight has the ability to simulate their weird spreadsheet-based vision programs without a camera. With Fanuc you need the whole $30,000+ robot+controller+camera setup and with real-time applications the only way to debug it is to run it in situ.
Now my most recent industrial vision experience is from 2019, so maybe some things have changed. But these are folks that often don't even know what source control is and will run screaming for the hills at the first sign of anything that's not Excel or ladder logic, and balk at the idea of paying an experienced engineer more than $100k all the while wondering why they aren't finding any talent.
I’m hugely enthusiastic hobbyist that would love to chat more about robotics, in particular how a hobbyist could get started with it (a robot arm + camera maybe?). I’d love to buy you virtual coffee, get in touch if you’re up to it!
I think it's structural. Salaries between EE and CS sharply diverged decades ago, and I don't think they'll ever meet again.
The finances on pure software are just so much better. Better margins, better scale, better return on equity. Since ROE is always going to be better (because you're not touching atoms) you'll always have better valuation on the stock market, and be able to pay programmers better.
It's less a "gap" in the market and more "the market functioning correctly". There's no law of the universe that says programming a robot has to pay as well as programming a SAAS webapp.
Think about scale. If you teach programming at a middle school, you have maybe 100 customers at a time. If you work for a hardware company, you have 1,000,000 customers. If you work for facebook, you have 3,000,000,000 customers. Which one of these will pay the most?
If the jobs require similar skills/aptitude/talent and a similar level of "investment" in job-specific training and experience, then the market force expectation is that as the information about the pay gap becomes clear, potential engineers would avoid EE jobs in favor on CS jobs, and EE training in favor of web development training, until the shortage of employees forces hardware companies to pay robot programmers as good as SAAS programmers or be unable to hire robot programmers.
There are many gaps in the robotics market. But there are 2 main show stoppers:
1. Starting robotics venture is very expensive. You need at least 3 engineers(hardware+software+electronics) for at least 3 years with tons of expensive hardware to reach MVP.
2. The clients will not buy from a company that might be gone in 2 years when the whole installation is planned for a decade. The client is mostly integrator choosing familiar system components.
The 3rd show stopper is that the product must work 100% or the time. 99,5% is not enough. Automation is here to replace people instead of having a robot with maintenance crew nearby.
>...detection and sorting by color filtering was doable in 1998...
I did a project with Lego sorting marbles in 2009 doing this in High School. By that point, anyone with $100 and a few hours of spare time could put together a rudimentary sorting system.
In the corruagted box industry, we use camera systems to elimate missed glue lines on tabs, skewed folds, and misaligned print in real time as the material goes through the manufacturing machines.
Smaller neural nets are commonly used in character recognition, but typical smart cameras or embedded robot controllers don't have anywhere near enough compute to run deep networks in real time. The Fanuc I was ranting about in this thread had something like 64MB of RAM in 2018. Maybe some systems are out there using Coral TPUs with custom TF Lite models.
As for PC-based systems, I would be very surprised if deep learning models weren't being used in production somewhere. But in a factory environment you can go a very long way with primitive feature recognition and good control over the scene and lighting, and the customer just cares that whatever you're doing just works and any new method will have to be enough of an improvement to be worth the cost of development time.
> As for PC-based systems, I would be very surprised if deep learning models weren't being used in production somewhere.
They definitely are. ~5 years ago I built a PC-based system that detected grain direction of wooden boards (looking at the end of the board).
Initially I resisted the ML approaches and my first attempt was basically hand-crafted image analysis pipeline- split the image in segments, apply Gabor filter with kernels of various angles and try to fit a curve to results. It kind-of-worked but I wasn't entirely happy with it's performance on the test data.
Even the simple classifier models that could execute on a fanless PC without a GPU outperformed my solution, and after a few more training runs the handcrafted code was replaced by #include <tensorflow.h>.
This year I'll have to extend the system with on-site training mode, where an operator has a pushbutton to label the images and re-train the model.
The 900lb gorilla in the deep learning room that everyone likes to ignore is that machine learning is horrible at providing corrective action data. Traditional machine vision is well adapted to providing statistical data such as "the diameter of the pizza is out of tolerance by 8mm" or "there are supposed to be 22 pepperonis on the pizza, but only 19 were found". Machine learning leans towards "it's not a good pizza" and doesn't provide a lot of additional data.
As the other person said, this is largely a strawman. It would be possible to build some kind of black box system, but it's not mandatory. An OK/NOK pizza classifier could do so without telling you why, and in some measures may be more robust than some kind of filter, threshold, morphological rules, pepperoni counter, but the latter will not tell you if something is wrong with the crust. A pepperoni object detector would be trivial and way more robust than whatever classical pepperoni finder you could build.
> machine learning is horrible at providing corrective action data
I don’t recognise the truth in what you are writing.
> there are supposed to be 22 pepperonis on the pizza, but only 19 were found
Instance segmentation is a solved problem. A properly constructed and trained neural network can tell you exactly how many pepperonies it sees and exactly where. Telling if that is the right number is a trivial problem from there.
> the diameter of the pizza is out of tolerance by 8mm
Here too, the neural network can recognise the edges of the pizza and then you can fit a shape to it. You can do this second step either with classical algorithms or with a machine learning one. (I would use a classical algorithm if the pizza is meant to be circular or rectangular shaped, and a machine learning algorithm if they are aiming for something weird, like an Italy shaped pizza or something.)
> Machine learning leans towards "it's not a good pizza"
Sounds like you have only heard of simple classifier models.
I will accept your opinion as I have never implemented a complete ML based solution. All of my opinion is based on promises and demonstrations for ML products such as Cognex VIDI. If those systems have capabilities like you describe, they have not been well presented during their sales pitches.
> All of my opinion is based on promises and demonstrations for ML products such as Cognex VIDI.
I see! Thank you for the explanation. That now makes sense.
Basically you were talking about what is available on the market as a product, and I was talking about what the state of the art is in machine learning. Now obviously if you actually want to put a factory line together you care about the available products, not what is possible in theory.
It is kind of like asking someone if it is possible to travel to the moon. If you are asking a physicist they will do some calculations with the rocket equation and will tell you that it is perfectly possible. If you ask the same question from a travel agent they will tell you it is not possible because they can’t sell you a moon holiday right there and then. They are both right, just in different contexts.
> If those systems have capabilities like you describe, they have not been well presented during their sales pitches.
All I can tell to those companies is that they should “git gud”. :)
Thank you for your explanation about the context you were talking about!
I've observed that the ML (or otherwise) model quality is not really the bottleneck in most computer vision systems so it gets less attention. The state of the art is way beyond what you would see in an implemented solution (like cognex), but some combination of market immaturity and that not being the biggest problem means there's not much industry demand for really good models
Useds few Basler GigE and USB3 camerasfor a robotics competition at the university, was fun, cameras were easy to use.. only later I saw how they are used in the industry.
Unfortunately, until we really need to push high scale manufacturing back to the states, it’s not changing anything for US manufacturing. I worked for a company that took almost a decade just to change from devicenet to ethercat, predictive analytics took 5. Any sort of “smart” system just doesn’t have a huge momentum unless we’re producing items at China rate and need to maintain cost low.
Sort of agree but a couple counterexamples from my machine vision career:
- Agriculture and food processing, which cannot be offshored as easily, requires very challenging machine vision solutions. Dirty environment, unpredictable lighting, unpredictable object appearance.
- Proto and small scale high tech manufacturing, pre-offshoring or sensitive IP, requires machine vision solutions that are both sophisticated and quickly adaptable
How is this wishlist related to anything that is being talked about in this thread? I'm very confused by what you're adding to the conversation.
I also wish robots would do the menial labor that I do not enjoy and would take care of all of my basic needs.
But this is an article about basic computer vision beginning to impact basic manufacturing. What you're talking about is decades in the future if ever. I'm very confused.
Edit: The OP originally talked about an agricultural robot that could charge itself, do all the home chores, and fix things around the house. Now it's just one sentence.
A general-purpose robotic handyman for consumers is many many decades away (at least). And if such a thing did exist it would have massive massive implications for the labor market--both on its own and because of the implications for all the other things that AI could do were such a robot possible.
Computer vision in a very constrained environment is much much different and often isn't even suitable for many "simple" tasks that aren't constrained quite enough.
What if there was some sort of LLM like breakthrough? I could see someone doing a technique where they track all movements of a person going throug their day with a fine grained bodysuit. They could then use that as tokens for generative input that could give a humanoid robot intuition about how to move around and perform tasks that would match an LLM's ability to respond to arbitrary questions.
You'd walk face first right into Moravec's Paradox [1] which observes that higher order intelligence is far easier than basic locomotion & cognition. Probably because the former has only been evolving for millions of years in humans while the latter has been evolving in all animals for hundreds of millions of years.
We can't produce an electromechanical device that is capable of the kind of fine motor control 99% of animals are capable of, let alone doing it on an industrial scale. We're not even at the "promising proof of concept" stage and what use is more advanced software when we're not even close with the hardware.
I suspect this is related to a serious upcoming bottleneck in AI. A skilled human can do all manner of physical handyman-style tasks, and I can imagine AI tackling a lot of the cognitive parts: reading the manual, figuring out what generally needs to be done, reading industry standards, etc. But a human has also trained themselves about how a screwdriver and a saw actually behave, how to get all the pieces to stay together while doing a job, and a bunch of other physical things work, and can real the manual, etc with this context. AI doesn’t have access to the physical part, at least for the time being.
This runs right into a lot of cognitive science research and debates. Which have not really resolved a lot of things related to AI historically and have been pretty much on the sidelines as GPU-driven machine learning has done some impressive things (and a lot of parlor tricks).
One of the ideas around cognition is that a lot of what we regard as intelligence in the physical domain, including intelligence below human levels, involves creating physical models about how the world works, which AIs are literally not able to do at all. You can instruct them in various ways, e.g. Boston Dynamics, but they have no way to internalize and actually understand novel physical world situations.
Some very smart people suspect that ML is a very powerful technique but that "better ML" only gets you so far.
> it’s not changing anything for US manufacturing.
I don't think this is true. I work for a US company producing industrial equipment based heavily on machine vision. Our products (along with those of our competitors) have changed the entire industry we support, for the better.
Ours is only one specific part of the manufacturing space, but I fully expect the impact to spread to other parts as well.
Does the industrial equipment enable manufacturers to eliminate human labor in the production process or is it more of a way to replace existing machines with a more reliable, performant etc. solution? If you want to share this info.
It increases the efficiency (that is, reduces waste) of the use of raw materials in production. In some circumstances, it probably does replace a small number of human workers, but that's not its main effect and isn't the source of the savings our customers get from it. It's mostly doing a job that wasn't being done before.
From what I understand, there are increases in efficiency due (in part) to the ability of machines to run at all hours, helping manufacturers to work with delayed and unpredictable supply chains. Also due to reducing tiresome, back-breaking manual labor.
High valued assemblies tend to be manufactured in the USA. Defect detection early and often prevents waiting until completion to scrap them (much better to scrap when you only have $10,000 in subassemblies and labor into it than then when you have $50,000). If that assembly is a bottleneck you are also reducing the impact to your larger production schedule (scraping it one week into assembly versus 4). Computer vision is hugely important for this. Computer vision capturing each stage also greatly helps to quickly isolates what is introducing the situation resulting in scrap. Instead of having scrap meetings trying to determine why a completed assembly is failing you identify the failure point as close to when it occurs as possible.
Being able to identify molds reaching end of life prior to parts failing QA for being out of tolerance is also huge for American manufacturing.
Where it's way less important is when you are spitting out eraser tips or other 'high scale' manufacturing.
Is this actually true, or is it just hysteria from someone unfamiliar with manufacturing? Because I live in New Zealand, which is on a similar deindustrialization path to America, but with obviously much less manufacturing in the first place. But we nonetheless have a burgeoning manufacturing robotics industry for what we still have - wood processing, pulp and paper, agriculture, etc.
In what sense do you mean deindustrialization? It appears from various reports that NZ manufacturing output it at a record high. Do you mean as a % of GDP?
In the U.S. the number of people employed in manufacturing is lower than ever but the value of manufacturing has been steady at 12% of GDP since World War 2 ended.
I can imagine that China also has massive infrastructure and a manufacturing environment built up over the last years that may become increasingly hard to replicate in the US. I bet there some "critical mass" for high-volume manufacturing that's needed, if you don't count subsidies. Even if it's all robots, you still need suppliers etc.
(Not Cognex)