Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens.
> Another problem is that the job while rewarding is not very lucrative.
Do you mean it's lower paying than typical bigcorp software jobs outside of FAANG, or just that there aren't a lot of startups with astronomical valuations in the media FTL space?
Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer.
Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazing.
I left the embedded space and have never looked back.
So $SALT_MINE, a highly profitable privately held Fortune 500 just decided to revamp their pay scales. They've now decided that they want to be at the 50 percentile remuneration-wise in the durable goods sector. That is the white goods sector, washing machines and such.
I predict we will lose all of our engineers - embedded dudes included.
May I ask how did you manage to leave the embedded world and where did you go after? I'm asking since after investing 6 years into this field, which I love, and jumping a couple of companies I realized the market(Europe) is really bad for this gig. Not only is our work highly challenging it's also poorly paid while at the same time our CEO is crying to the local press they can't find devs(to work for peanuts) and is forced to look for them in Asia.
I just went to do something else. I'm definitely a generalist, and now work mostly with "big" data.
I had zero embedded experience when I started doing embedded dev, and then I had zero data experience -- but a surprising amount of general experience is applicable!
"Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. "
Heaven for generalists that always love doing new kinds of things. Once I learned about it, I knew I probably should've done embedded instead of security research. Of course, now there's significant interest in overlap. Might not be too late to learn all that stuff after all. :)
As someone that identified a bug in a Drobo firmware once and was offered a job on the spot I think the problem with attracting talent is two fold.
The first problem is really two parts, not only is it rare to find people who have passion for storage related technologies but very few will gain exposure to these technologies to develop that passion.
Kids don't routinely grow up with a SAN in the house. They do tend to grow up with lots of internet connected consumer caliber devices and can easily gain exposure to working with these technologies.
I was fortunately able to explore this type of technology in depth because a family owned business let me tinker with their server equipment in high school.
After college I then co-founded a startup back before the cloud became big. That meant we needed to make use of old hardware to provide service to our customers at a price point that made our service profitable. Old drives were not a reliable way to do that. New drives were extremely expensive for old servers back in the day when SCSI was the interface that you expected for a server. We had to get creative and play with JBOD devices. ZFS was an amazing tool for us in those days, and it still is for anyone who wants to tinker.
The other aspect is that while these skills are valuable for creating a "job" they do not have potential for creating "massive wealth". Why learn about storage if you aren't going to be part of the first 10 employees at a company that has a $10B exit? Let Amazon and the other cloud vendors worry about that stuff.
Knowledge is power though. I recently came across an AI startup that I'm now helping. They were spending significant money using GPU computational power to provide artificial intelligence training through a cloud provider. They blew through about $300k in credits within the first year to give you an idea of how much money that type of power can cost.
I am now helping them cut over to their own co-location facility. The first year alone they will save so much money it will pay for the next three years.
Reading that helps reinforce the idea that no matter what path you are on in the field there is potential that some random thing you learned about an SSD firmware helps you optimize some growth stage companies product and ultimately that helps you build wealth.
>very few will gain exposure to these technologies to develop that passion.
This, exactly.
SSD firmware is opaque, hard to learn from outside. On the other hand, trending web-based framework has all the source code opened, with great documents, and ready to use tools. No wonder young people of today find passion on other things rather than SSD.
I'm mostly curious because I work in a storage-adjacent field (NAS) for a BigCorp and the pay is pretty good, if not quite FAANG level. It's not a startup by any means, but I will easily become a multi-millionaire in a handful of years. I was curious about the other side of the fence.
Huh? If you're going to "easily become a multi-millionaire in a handful of years", then your pay is more than pretty good, and certainly not worse than FAANG level.
Sorry, handful of years from today. I've been working for 7 years now. FAANG comp would probably be 20%-25% higher; I'm mostly good at keeping my expenses down and saving a high proportion of my income. I've also had the good fortune of the bull market working in my favor for the entire time I've been employed.
Even at that pedestrian level, if you pack them tightly enough I could imagine fitting a few billion years at least into the volume of an average-sized handful!
Sorry, handful of years from today. I've been working for 7 years now. FAANG comp would probably be 20%-25% higher; I'm mostly good at keeping my expenses down and saving a high proportion of my income. I've also had the good fortune of the bull market working in my favor for the entire time I've been employed.
It's been about 5+ years at this point so I don't recall all of the details. I can recall that they offered me a job after I pointed out the bug. From what I recall, I politely declined but I helped them test a beta firmware for a while.
>The other aspect is that while these skills are valuable for creating a "job" they do not have potential for creating "massive wealth". Why learn about storage if you aren't going to be part of the first 10 employees at a company that has a $10B exit? Let Amazon and the other cloud vendors worry about that stuff.
That seems like a completely ridiculous way to try to organize your life. Almost no companies have $10B exits.
> Users and administrators almost certainly prefer a 20 minute IO latency over data corruption.
If the drive part of a RAID setup I would actually prefer it just reports itself failed and doesn't slow down access to the array by scanning itself for 20 minutes.
As far as I know, that is one of the main differences when buying enterprise or nas drives compared to consumer drives. With nas drives, the firmware gives up very quickly since the drive is assumed to be part of an array with redundancy. Consumer drives will retry reads for a very long time before reporting i/o error.
> Another problem is that the job while rewarding is not very lucrative.
Do you mean it's lower paying than typical bigcorp software jobs outside of FAANG, or just that there aren't a lot of startups with astronomical valuations in the media FTL space?