Hacker News new | past | comments | ask | show | jobs | submit login
Taking over a Dead IoT Company (kchung.co)
1114 points by pulisse on Jan 10, 2023 | hide | past | favorite | 191 comments



Hello, author here. Happy to answer any questions!

My apologies for the downtime, I wasn't expecting much traffic today since I submitted the post to HN yesterday but I've started scaling my server now!


Love this story. I did something similar with a website. Previous owner took it down, bought the domain from him and it organically grew to over 1m users per month with the same exact functionality.

I wouldn't discount your ability to bring this back to market. Would consult a lawyer to see what implications taking over the branding and APIs have. But clearly this has a big market.


I would really like to read more of this story. E.g. what did you do differently, did the original owner ever get in touch with you after it grew, stuff like that. Only if you can, ofc!


I just want to say that I love that you did this. You have great skills and you used them for good. I've never seen such a clearly white-hat hack :)

People's stuff will work again thanks to you and you should feel great about it!


Thank you for the kind words! Made my morning!


I'll ask the same question as asked by @js2 below:

Once the new server was up, did any signs from the field connect to it?

Do you know how many signs are out there actively looking for the mothership?


Here's my reply to him from https://news.ycombinator.com/item?id=34328461

> Good question! No signs connected to the server until I reached out to some other sign owners to try out my instructions.

I do not know how many signs are out there. I imagine most people would have just unplugged their sign after the company's API vanished since any data would be stale and useless.


Love the story! At one point, you are asking about the purpose of the giant hole in the backside of the display: The most plausible explanation is that, it is to allow air circulation within the device to prevent overheating.


Another explanation that I saw [0] was that it was for people to pop the back panel out. I think this is the most likely explanation but it didn't occur to me while working with the sign. I feel like a little notch would have been more appropriate for an actual product.

[0] https://hackaday.com/2023/01/09/iot-archaeology-leads-to-api...


That particle board material would crumble too much if they used a notch for removing the back. I've seen other items in similar cases with the exact same drilled hole in the back cover. Many used staples to hold the back cover on as wood screws tend to damage the (other) particle board used to make the sides. The "designers" probably used it because they had some other item at home (stereo, TV, loudspeakers or clock-radio).


In a retail or home setting, you can mount the sign on a wall and route electrical wiring through the hole for a flush fit.


Why would you call it a good idea?

Even your calculations are way too optimistic.

Giving it little thought, I can say that NYC would have at best 10k sales of this item. In total... including any knockoffs.

The product may have scaled to be able to sell to London, Paris, Tokyo, Soul, Beijing, Shanghai, Hong Kong and Singapore. But this is the total addressable market for this item.

Let's say you have 2 engineers in US working on this full time - that's 500k in salaries(that includes all associated staff).

In NYC you would be able to charge $300, but no other major metro area has that kind of disposable income. In London the max price would be $170, Paris and others - $150... And boom, you can't even make a product to cover development costs.(not to mention any operating costs)

Cool art project and commercial viability clash perfectly here. This item is at best - an art piece, that should cost as such($3000+).


> in NYC you could charge $300

That's 2 days of median disposable income.

For comparison:

San Francisco, U.S. - Disposable income after rent: $4,710

Zurich, Switzerland - Disposable income after rent: $4,626

Chicago, U.S. - Disposable income after rent: $3,298

Boston, U.S. - Disposable income after rent: $3,188

New York City, U.S. - Disposable income after rent: $3,157

Sydney, Australia - Disposable income after rent: $2,615

Melbourne, Australia - Disposable income after rent: $2,485

Oslo, Norway - Disposable income after rent: $2,342

Copenhagen, Denmark - Disposable income after rent: $2,285

Wellington, New Zealand - Disposable income after rent: $2,075

Depending on your list and method, New York could be 44th on global purchasing power index:

https://www.numbeo.com/quality-of-life/rankings.jsp?title=20...

Or 15th:

https://defence.pk/pdf/attachments/screen-shot-2019-11-27-at...

Anyway, "no other major metro area has that kind of disposable income" seems inaccurate.


> Let's say you have 2 engineers in US working on this full time - that's 500k in salaries(that includes all associated staff).

USA, the land of start ups!


Sharp mind and entertaining writing!

I've had a few projects like this where I list out the BOM then go for 'reasonable business success profit' and scratch my head at exactly this line:

> At even just 60k per founder, with the profit per sign sold being maybe $400, you would probably need to sell ~600 signs at full price per year to create enough revenue to run payroll.

Hardware is an exponent to the value of the internal reward, but the root of all business profit. We need a better funding mechanism for this style of development - more like the hardware lifecycle - wasn't Bolt doing something like that?


Just curious what made you come up with at least $30 per LED panel, as these should run you about $10 in bulk.


I had found some tweets by the company where they talked about using an Adafruit panel that was $40. The price on Amazon was about $30 so I figured I would go with the lower price. They may have switched to a lower cost panel but my guess is that didn't happen.

EDIT: Here is the specific tweet: https://twitter.com/NYCTRAINSIGN/status/926106932573810688


The generic name for these panels are "HUB75", based on the interface they use for driving them. I can confirm, these are $10-15/ea in small quantities on sites like Aliexpress. But you're likely right, they were probably buying retail from Adafruit.


"HUB75" refers to the 16 pin connector and specific structure of the square wave signals it's expecting (row order, pixel order, etc). This particular sign appears to use a 4mm pitch (space between each R/G/B led), indoor-level brightness SMD leds, and modules that are 32x64 pixels, like this one on AliExpress: https://www.aliexpress.us/item/2251832064290423.html?spm=a2g... ( ~$8 + ~$9 shipping to the US for single qty).

There are various "HUB75" panels of different pitch, brightness, LED type, matrix size, etc. Also, there's variations like "HUB75E" with 5 address lines instead of 4. Address lines are typically A/B/C, A/B/C/D, or A/B/C/D/E. And there's also "HUB08", "HUB12" and other standards.


Why would you buy retail if you are a company buolding and selling hardware to begin with?


Everything in the article screams that they did not know what they were doing.

Why would you think that a company using a $25 Adafruit HAT that could be found for a magnitude less elsewhere would be using cheap LED matrices?

Put another way, what from the article gives you enough confidence to say that they would have used cheaper parts?


I suspect the intention was to get an MVP out the door and get some initial "growth & engagement" so they can attract VCs - cost-effectiveness wasn't a priority at this stage.

It's a shame because there is a small but sustainable business here.


Have any signs in the field connected to your new API and updated?


Yes I reached out to another owner who was able to connect the sign to the API. I've reached out to more people but haven't gotten too many responses. It's been 5 years after all. If you know anyone with one of these signs send them this post!


I think the other mark on the back was for the reset button. Thanks for sharing.


Hello, author! That was interesting!


An interesting read. Thank you.


super interesting read, thank you for that.


Awesome work!


Very interesting read. Based on our experience developing and manufactoring air quality monitors, making a customized PCB should be done as early as possible in the developing process.

With easy to use editors (e.g. easyEDA) and cheap and fast PCB production/assembly (even for small quantities), there is no reason to use overized MCUs/mini computers like the Pi and expensive modules e.g. from Adafruit.

Also, BOM components selection is key. For the same functionality, e.g. a multiplexer there are many different chips available -often at very different price points. Availability in times of chip shortages are also key to keep an eye on.

Technically this product is very simple and the BOM costs they had were a magnitude too high. Too bad because it seemed that the market was ready and they could have become a very successful business.


> there is no reason to use overized MCUs/mini computers like the Pi and expensive modules

There is if you don't have the expertise. Of course, arguably if that's the case, you shouldn't be developing such a product anyway. But as the post detailed, by switching to a Pi Zero and removing the Adafruit HAT, the BOM costs would already drop significantly. I think if you're smart about it, you absolutely can build a viable product this way. Certainly not as profitable, but possible.


I’ve been working on a hobby project for about 6 years, and early on I got very excited about productizing it. It led me down some interesting paths and really made me think about overall BOM cost, designing for assembly, ease of use, etc. It also revealed some weaknesses for me, as I kept making decisions based on something I was sure I could figure out, rather than what I actually knew how to do.

Eventually I switched some things up and it got much easier.


I'd read this article!


As would I!


All of this presumes that the person making these costs $0 per hour. Turning hobby into a business is rarely a successful scenario.

This screeches to a halt, the moment that you put into perspective. My typing this is equivalent to a few dollars, because most people here have at least $50 per hour engineers. Designing PCBs for existing modules(ex. ESP32) takes a very long time(and a lot of knowledge) and it's not just "put this chip here and have software figure it out".


"there is no reason to use overized MCUs/mini computers like the Pi and expensive modules e.g. from Adafruit."

Definitely not for scaling, but for deploying prototypes this approach is awesome! Eg use the Pi to create some prototypes, get user feedback, launch a kickstarter, do what you said. I worked at robotics startup that did this, worked super well for our first robots. Podcast about it: podcasts.apple.com/us/podcast/erik-schluntz-cobalt-robotics/id1649845905?i=1000584727151


How difficult would it be to run a Linux operating system on this custom PCB, allow for Wifi connectivity, etc to provide for the features the product provided. Like the wifi-setup, it having internet connectivity, ability to poll an API, etc.

Genuinely curious, as I've typically used things like the Pi for projects like this but have been genuinely curious what it would take to strip it down to something less powerful and achieve the same result.


A Raspberry Pi Pico W is a microcontroller (no operating system) that incorporates Wifi. The way it works is you write a program that runs in a loop and talks to the GPIO ports or whatever. It's incredibly simple and it all runs from flash, so no operating system or anything else needed. They cost less than AUD 10 (about 7 US-bucks)

https://raspberry.piaustralia.com.au/products/raspberry-pi-p...

Since I am a Go developer, I like using TinyGo with these type of boards, which means I get concurrency (goroutines) as well, although sadly Tinygo doesn't yet support the Wifi module. However, I've used other boards with Wifi, and once you're connected to WiFi you can poll APIs and do anything else you do on the internet. No OS required!

As an example of a complete, stand-alone program, I wrote a digital clock in TinyGo. (https://github.com/doctor-eval/clocky) - you plug the Pico into a 7-segment display and it operates just like an old school digital clock.

Sadly it can't (yet) connect to the internet due to missing wifi support in Tinygo, and it's running on a pre-W Pico, but you get the idea.


If you are OK with FreeRTOS instead, then it's even cheaper.

The cheapest you can get is about $5(Espressif assembled board), with full LED matrix controller.(that's the cost of manufacturing, though)

a full blown Linux stack is $10+


I’m making a project with a pi pico and it will have to do all of that. It already sends an update to the server per second with wifi. A full raspberry pi is over kill for most of these projects.


This further hits home the lack of business acumen of the founders. Let's assume BOM is an order of magnitude off -- 10x what it should be. That doesn't mean fixing the problem would 10x profits. It would merely reduce BOM by 10x. Thus, if you can sell the thing for $600, you actually 20x Profit!

Missed opportunity by this startup.


> With full control of the domain, we can create a new API based on what the sign is expecting and revive all of the signs that are out in the field.

Once the new server was up, did any signs from the field connect to it?


Good question! No signs connected to the server until I reached out to some other sign owners to try out my instructions.


Great post!


This was really interesting - a well deserved spot at the top of the front page. It's rare to see an article that combines the technical and business analysis so well.

It's amazing to me how quickly the company loaded itself up with staff. Being kinda familiar with ESP32 development, I thought at the beginning that the company would have 4-6 people - 1-2 technical, 1-2 production, 1 business/sales/marketing, 1 admin. It seems like they opted to target the luxury nostalgia market with the $600 price point rather than a more value-driven $300, and wrapped themselves in an aura of success rather than growing it naturally. The descriptions on the resumes of the former marketing staff juxtaposed with the business reality was...quite a contrast.

It's especially sad as the basic idea was good and seems sustainable at a lower price point - with a metal/plastic housing it might have been an easy sell to businesses near subway stations, for example, and developing cosmetically different versions for other large cities would be quite feasible. I wonder what made them take the 'growth corporation' approach rather than crowdfunding the prototype > product route, which was fairly well established even 5 years ago. That offers a fairly clear roadmap for new ventures and is sufficiently familiar to consumers that innovators can do some market segmentation and have a cushion of patience to get them though the design to manufacturing transition.

A great case study for anyone thinking about launching a product for fun or profit.


I think the 600$ price point was inflated with the Pi and the Arduino being used. Simply changing some parts for cheaper ones would have allowed to reduce the BOM cost and thus reduce the price. The author mentions this in his post.

Also seemingly not a lot of people paid 600$ for the sign, with a lot going for a lot cheaper than that.


That's a part of what the article dives into. For example the LED matrix hat they were using from Adafruit was probably the most expensive option they could have used at $25/unit, and the article gave examples of much cheaper (i.e. 10% of the cost) parts that would have done the exact same job with a little extra engineering time/effort. Replacing the Pi with an ESP32 would have saved money, but like the article mentions using a Pi wasn't entirely a bad idea considering the free marketing they got from the foundation.


> It seems like they opted to target the luxury nostalgia market with the $600 price point rather than a more value-driven $300

But they were not aiming at the $600 price-point. That's just the estimated price-point that OP estimated from the BOM. They were actually selling at much cheaper than that, which probably contributed to their collapse.


Towards the end of the article the author mentions not wanting to pursue this as a business as there are so many cheap programmable signs out there already.

As someone who would love to buy one of these and customize it for myself does anyone have experience or recommendations with any of these?


Some experience. The cheapest and easiest route is to buy a pre-made sign from AliExpress and just use the serial/usb interface to send data to the sign. That's somewhat limited, as you can't really control each pixel in a real time way, and you're limited to sending a series of "screens" as either text (in their limited choices of fonts) or images, defining some linger time, transitions, etc. With some visual disruption as you send new data to redefine what to display. But it's easy. There's also variations on this theme. Some signs, for example, have had people who reverse engineered the protocol so that you can send stuff from your own scripts instead of the supplied (usually crappy) software.

The other path is to buy plain led matrix modules and use some SBC to directly control what's displayed. More fun, but more work. This particular sign uses qty 2, 4mm pitch, 32x64 pixel panels, side by side. Like this one: https://www.aliexpress.us/item/2251832064290423.html?spm=a2g... Lots of choices in pitch, single/three/full color, indoor or outdoor brightness, etc. Though you need to ensure your choice of driver (see paragraph below) supports it. They have different refresh rates, data sequence, and timing requirements.

Then, you pick what to drive it with. Lots of choices there. There are libraries and hardware level-shifter adapters for a variety of things like a Parallax Propeller, Raspberry Pi, BeagleBone Black, etc. With various advantages and disadvantages. Fwiw, the Rpi is usually trickier as you're bitbanging from a non-real-time OS. The Parallax and BeagleBone have better ways to be quasi real time.


Thank you for the very detailed answer. Though I think you may have just convinced me to stay with my current solution of Raspberry Pi's + 4k TVs for custom display signage in my office (but it's not nearly as cool).


You can use a colorlight card to control the panels, they can be had for $15 and connect to the nerwork using gigabit ethernet ( must be gigabit ). You then send then data using the windows software or you can use a Pi/computer running FPP software to send whatever you want.

This would reduce cost to:

2x panels

1x colorlight card

1x power supply

Misc cables / case

Edit: The card can handle an 8x8 matrix of panels ( IIRC ) and the cards can be chained together to make larger displays. I dont recall the max size. The biggest I’ve made is 5x5 for a xmas display.


Interesting. Can you interact in a sort of live/api way, or is it limited to writing configuration and "launching" it?

Edit: Ahh, that's a great option...assuming that reverse engineered way to send frames is stable, etc.


So the configuration is a one time thing to set the card parameters, you shouldn’t need to touch it once its setup for your panels and layout. As far as I know it can only be done using their widows app.

The display is done by sending the card network frames ( someone did a reverse engineering of the protocol, i dont know the details ). You can send the display anything you want.


I just got home and was able to look at some of the RE efforts, here is one that looked interesting:

https://hkubota.wordpress.com/2022/01/31/winter-project-colo...


Did you intentionally leave out Arduino from the "what to drive with" options?


Assuming largish, multi-color matrixes, and/or chained ones, yes. A ~10Mhz+ pixel clock, double-buffering, or just the size of the pixel map would strain most of those. This sign is 32x128 pixels, with each pixel taking data for RGB (on/off), with the PWM rate controlling brightness for each of the three colors. Though there are examples of using something beefier/faster like an ESP32. There's also in this case, the assumed requirement of fetching train data and transforming text into pixels.

The panels themselves don't take data and hold it. You have to refresh the display constantly, typically on a 1/8 or 1/4 duty cycle.

You can drive smallish, single color matrix displays with a cheap MCU. Or there's also the approach of using addressable LEDs instead of these more "dumb" matrix modules, like arranging the smart strips into a matrix. Moves the refresh logic down into each LED.


The ESP32 can handle a 64x64 HUB75 display pretty handily. Several of them, even. There’s a solid library out there [0] that just pumps pixels right from memory via the ESP32’s built-in DMA engine. Adafruit_GFX is the supported library for drawing primitives/text/bitmaps/etc., so you can re-use a lot of code that’s out there.

But let’s start with a simple, quick hit of dopamine: There’s a web-based installer [1] (it uses web serial, so Chrome is required) for a couple varieties of clock designed for a 64x64 HUB75 panel and an ESP32. $20 in parts from AliExpress if you already have a suitable 5V power supply.

To use that site, all you need is that aforementioned 5V power supply—4A or more recommended, 10A if you’re planning to drive a full panel of pixels on an outdoor panel at full brightness—a 64x64 HUB75 matrix panel, and an ESP32. Everything except power is run directly to the ESP32. Many panels even come with a 16-pin female IDC connector for the panel, with individual DuPont-style female connectors at the other end that you can just slip over the pins on the ESP32.

The source code for each clock (repo is linked from the web page) is a great starting point for making your own versions.

The panels, as others have mentioned, are shockingly cheap from the usual sources like eBay [2] or AliExpress (search for “HUB75 panel” or “p3 64x64”), but even those in a hurry can get a 64x64 panel for under $40 from Amazon [3].

[0]: https://github.com/mrfaptastic/ESP32-HUB75-MatrixPanel-DMA [1]: https://clockwise.page [2]: https://www.ebay.com/sch/i.html?_nkw=HUB75%20Panel [3]: https://www.amazon.com/dp/B0B3F7WKJ1/


That is impressive, though it's using a few tricks to keep up, like relatively low-duty cycle panels, and BCM in place of PWM.


Yeah. I tried a project driving 32x16 pixels with a Arduino (an Uno, ATMega328) and it was just barely on the edge of usable for mild animation. Furthermore, I could really only use it at full brightness. These panels are extremely bright, and my project was meant to be desktop display. For the next version, I moved to an ESP8266 which provides plenty of compute speed to drive these panels at a brightness that’s comfortable.


I see now.


There are a lot of different sorts of programmable signs out there: https://tidbyt.com/ https://www.vestaboard.com/


Thank you! The Tidbyt looks like the more approachable of the two. Less than $200 vs $3k for the Vestaboard and a it has a purposefully simplified Python like dev lang to pull data and shove it into widgets.


I'm a happy Tidbyt user. The device is really well made, it has a bunch of widgets that you can load from their app, and you can create your own and push to it (although I haven't spent much time trying it).


adafruit sells most of the parts you need if you go the DIY route https://www.adafruit.com/product/5362

great tutorials too https://learn.adafruit.com/rgb-led-matrix-cube-for-pi


Second this. The 32x64 version is what I used to make this digital tide clock as a project with my nephew https://www.filepicker.io/api/file/UzFNawTTWq4h5h6YF9Im. It fetches tide data from NOAA to tell you next high and low tides.

We also made an arcade game with the LED screen.

Not the cheapest route but it’s relatively easy and fun to get it working


If you're willing to go for a programmable wrist-sized sign (aka a programmable smartwatch) I'd go with Bangle.js.


Kudos to the author for a cool reverse-engineering job. Nice hack.

The sign wasn't the most problematic "mininum viable product" I've seen. If it were a rapidly developed first prototype, I'd even be impressed.

I would bet they spent significantly more than $3 on the packaging and miscellaneous BOM items. I was going to say the wiring harness alone would be much more than that, but it looks like most of it is included in the Adafruit panel kit.


This was a really good read. It's part IOT security, part story about startups gone wrong.

The WayBack Machine has it https://web.archive.org/web/20230109144459/https://blog.kchu...

As does Google

https://webcache.googleusercontent.com/search?q=cache:71rSDb...


"As we discussed earlier, even at $300 the product is too cheap. The sign should have likely been selling at $600 from the very beginning."

Even at $600 - there's a screenshot saying the company had three engineers and a marketer.

If your 4 employees earn $25k per annum, with the $150 parts cost you'd have to sell 222 signs at $600. And if the employees were on $100k, that's 888 signs a year.

That's a heck of a lot of signs, at the price. And you'd have to keep hitting those sales targets so long as you wanted the API to stay up.

It would make much more sense if the business was a one-engineer passion project - someone who had a day job and did a few hours a week maintaining this as a side project would have much more achievable sales targets!


Why does a project like this need three engineers? Two, I can kind of see (a hardware guy and a software guy, though it doesnt look like the hardware guy was doing much here), but three?


I wish I could get contracting houses on board with the notion that there are many tasks that require 30-50% of a person but greatly increase the quality of the work. I almost convinced some people at the last place but convincing the customers is something else entirely. Code reviews by more expensive staff for instance. Architectural reviews. Security audits.

One engineer makes a mess because there is nobody to push back. Two engineers either fight or achieve full echo chamber. Three balances some power dynamics, but you definitely don't need 3 FTEs. In fact for this project you probably find 3 of your friends' friends to sit together for 4 hours twice a week and slam out some code or designs, in exchange for a little money and some equity. You even get to write off food and beer as a business expense.


>but three?

The third one designed the wooden case and the manufacturing process for it ... /s


People have to go on vacation sometimes


So you let the thing run and brief the other on what to do if SHTF. We're talking hundreds of units out in the field, not millions. Every aspect of that can be provisioned for ahead-of-time, and cheaply at that, and then the SWE can walk away for weeks at a time.

If the HWE leaves for a bit, same thing. If he's making units production stops, but maybe the SWE can fill in since these dont look like difficult builds.


This makes me wonder how Tidbyt's business is doing. They seem to have a fair amount of employees.

I actually built something very similar and all in my spare time. I've been wondering how realistic making a side-business out of it is, but if you are a single engineer that has to code the firmware, server, build the displays and market it all... it's pretty tough (especially in your spare time).


We actually only have 4 FTE's today and our business is both sustainable and growing. Our scale is also a couple orders of magnitude larger than the numbers from the article, and we have much better control over our BOM.

Hardware is a tough business and managing cash flow & inventory requires you to be a lot more deliberate than a pure SaaS business. Fortunately we had a lot of examples going in of how not to do things.


Have you considered making a larger version that is something like double the width, and would sell for approximately double (maybe $300?).

I just took a look around the tidbyt website and it looks like a fairly well polished product.

https://tidbyt.com/products/tidbyt


Glad to hear! Happy Tidbyt user here. The device is incredibly well made, and love how you've been iterating. Wish the company all the success!


What a great reply. Thanks!


Or charge less and get a recurring fee for the API.


This. It is fatal to charge a one-time price for services that have perpetual operating expenses.


Perhaps, but still tough to make the business plan work for a $100k wage bill.

833 subscribers at $10/month? 8,333 subscribers at $1/month?


dont invest in non-viable things. if you can't cover your expenses, it's not a viable business.


Articles like this are part of why I love Hacker News so much. I've done a little investigation into producing hardware myself and concluded it was just too big a task for someone without the background who didn't want to put up a ton of capital or deal with overseas manufacturing. I loved reading this.


I am new to hn, and really liked this article. If you know about other interesting articles like this, please mention them. I would love to read.


I'm semi-frequently submitting electronics/hardware hacking articles here. Maybe browse my history of submitted articles to see if there is something among them. Here are some examples:

- "I hacked a Joy-Con controller to have a Capacitive Trackpad", https://news.ycombinator.com/item?id=34329927

- "Smartknob – an open-source input device", https://news.ycombinator.com/item?id=30646371

- "DVD Laser Scanner Microscope", https://news.ycombinator.com/item?id=26012652

- "A simple 11.2 GHz radio telescope", https://news.ycombinator.com/item?id=26078761

- "Home-Built Scanning Tunneling Microscope", https://news.ycombinator.com/item?id=26740968


Sounds like I need to partner with the author.... as I wrote my own predictive Caltrain late train model/alert site:

https://caltrain.live

(twitter @bettercaltrain)


Nice analysis. I think the author really hit on the key problem: expensive BOM due to poor engineering choices.


The entire project looks like the result of a high school student's first Instructables. It's hilariously under-engineered and it's clear that these folks had absolutely no idea of how to build a product, IoT or otherwise.

Hint: if your product plans involve a raspberry pi, you probably fucked up.


I think dunking on the rpi is harsh - if it allows you to prototype and get your small-volume product out the door quickly, there's lots of other areas you could optimize the BOM first. Better to make a product with 75% gross margins as a single dev in 6 months than a product with 80% gross margins with a team of 6 in a year. Of course, they also overhired and wasted money on other BOM components


The problem with rPi is storage and filesystem. You need to issue a shutdown command and wait for it to complete before removing power in order to help ensure that it will come up without problems. This will literally never happen with a consumer IoT device. Also, SD media is extremely prone to failure.

It's fine for prototype at a very early stage, absolutely not fine for something you ship to a user.


You would typically use a read-only filesystem in an embedded system, not the default raspbian style distro. This allows you to a) not require a clean shutdown and b) saves your physical media from writes.


This. Check out https://balena.io as one example.


Ironically today, for embedded hardware projects, it would be a perfectly fine idea to use a Raspberry Pi... Pico microcontroller.


On the other hand, see https://news.ycombinator.com/item?id=33892009, https://www.raspberrypi.com/success-stories/korg-synthesizer... for example on how some companies use Raspberry Pi compute modules in their products.

Am assuming here that they're adopting strategies to minimize/work around the problems you mention.


The computer module instead of the standard Pi is the first step of those strategies. It doesn't use the SD card (in fact the data sheet gives details on how to configure the OS if you won't be shutting it down before removing power) and the device as a whole requires smaller electrical current that the standard device.


If your product needs to output HDMI, I can easily see an argument for using a Pi. If your product needs a couple GPIOs to drive an LED matrix? Using an RPi is pretty difficult to justify. You're likely going to spend more on the SD card for the Pi than you would on more suitable microcontroller (which can still be easily programmed and, in my experience, has a better [or at least no worse] story for GPIOs than the Pi)


It's difficult to drive these LED matrix devices from a typical MCU, especially with chained panels, lots of LEDS, etc. Either due to the ~10Mhz+ output needed, or the memory to hold all the pixel data. Consider, for example, that each of the two panels in this sign has 32x64X3 (6144) leds. You have to send on/off for all of those, and then control brightness for each color by using PWM...meaning half-red brightness would be changing that pixel so it's on/off/on/off fast enough to beat persistence of vision.

It's more common in the pre-built devices to have a controller that's using a main CPU plus a FPGA or CPLD. The better hobbyist drivers use things like a beaglebone black and it's onboard "PRU" that can drive real time from memory it shares with the ARM cpu and Linux.

Also, since it's pulling train data, you need more smarts or cycles than a typical MCU to pull via WiFi/https, transform text into pixmaps, etc. Though there are examples of beefy MCUs doing all this well, like an ESP32. Or examples of small MCUs driving a small single color matrix.


RP2040 is a pretty fitting MCU for this use case thanks to its PIOs coupled with DMA. It got me some impressive refresh rates on a 64x32 HUB75 display – over 2 kHz in 24 bit color mode. The lack of networking capabilities out of the box is a bit of shame though.


ESP32 is perfect for this application. RPI is far too beefy, unless you really want to do everything locally (without relying on an external server to package up the train data for you)


Sure, though 240Mhz, 32bit, optional gobs of PSRAM, etc, aren't usually what comes to mind when you hear MCU.

Edit: Yes, you could even use an ESP8266, though 32x128x3 plus X bits of PWM brightness would mean being limited to drawing from storage rather than in-memory manipulation. I'd jump right to the WROOM type devices with PSRAM.


If you need a microcontroller with networking (such as the train sign would), I think the ESP32 (or older/cheaper/less capable ESP8266) is one of the first that comes to mind for most engineers, especially those who aren't full-time firmware engineers.


You are quite right that the ESP is way over-specified for many tasks, but it's not like the uC is gonna quit and take a more fulfilling job elsewhere. You can get tiny boards for close to $5/ea, or under $5 if you don't need the USB connector. - and that's in single-digit quantities. The hardware configuration is very flexible, eg if power consumption matters you can run it a lot slower, switch off subsystems you don't need, use a watchdog timer to simplify exception handling etc.. There's a rich and constantly expanding SW ecosystem so you can get to a working prototype very fast.


How would you get a quick MVP of PCB + needed interfaces? I'd love to know if there is a PCB design as a service + manufacturer that allows small size of order (as a 3rd party integration). I think they definitely exist .. I just have to figure out who to talk to first.


I recently hired and working with someone for PCB design/manufacture low volume of IoT devices. As other threads mentions, you can always find folks on fiver and upwork.

Hardware academy is also good, where you can ask questions and meet other folks who build electronics products.

https://predictabledesigns.com/academy/

Happy to talk and give pointers, email in my profile.

Edit: I've no affiliation with hardware academy


I don't know of any fully featured services, but something like Fiver (https://www.fiverr.com/) in combination with services like JLBPCB (https://jlcpcb.com/) or PCBWay (https://www.pcbway.com/) could act as an equivalent.

An integrated combination or even a board design offering from either of the two mentioned manufacturers would be awesome and really help bring down the barrier to entry for hobbyists that want to try moonlighting as entrepreneurs


I don't know of people who do PCB design, but I'm sure the usual suspects (Fiverr, Upwork, etc.,) will have them. I can do it, but I avoid that kind of work unless absolutely necessary.

However, for small-quantity, high-quality PCBs at a great price, once you have the design done, it's hard to beat OSHPark and OSHStencils for solder stencils. I highly recommend both.


You can contract the pcb work out on fiver and have the boards made at one of the Chinese board houses nowdays. I think I've seen people on Fiverr that provide turn key solutions you pay for it in turnaround spread and back and forth though.


There are existing PCBs out there for driving HUB75 displays (like the one used in this project).


I wonder whether this particular unit was representative of the production target build. Maybe it was an early prototype? Certainly after 250K revenue (if you believe that number) there was funding for a design rev.


I didn’t quite understand the need for an injection attack. Once you had bought the domain and could respond to the devices‘ API calls, weren’t you in the driving seat already?


I actually had written more about the exploit & vulnerability in my original drafts but I cut it out because it was a bit boring to read.

You are correct that with domain control I am able to serve content to any sign but the content will only be loaded once at boot time. Any future updates would have needed to come from their defunct AWS IoT connection (ignoring full restarts).

Using the exploit I remove the connection to AWS IoT and update some of the code to better connect it to the recreated API so users can update their signs in mostly real time.


Would you mind posting that content somewhere? I would find it very interesting!


Sure I took out the relevant section and put it here: https://docs.google.com/document/d/e/2PACX-1vTYSTUp3eTjfD-hG...


Thank you!


A guess, but...

"At boot time, the Config Server will pull the latest configuration from an HTTP server. In addition, the Config Server will connect to an AWS IoT Core endpoint to receive real time config updates from an MQTT server."

So, perhaps to bootstrap information they didn't have, like the current configuration? It sounds like they sold different types of signs with different resolutions, and also whatever train station setup config there was. Maybe a one-time exploit to upload that info from the sign itself.


Yes the exploit removes the AWS IoT connection so that updates can come from the recreated API.

There was only one type of sign but it did come in various different cases.


The author couldn't have accessed the original AWS account, which is what managed configuration updates.


I think RPi in this thing says it all. It screams it was built by somebody that does not know how to develop a physical product for mass market.

There are only three parts in this product that should have any significant cost to them:

* the display itself,

* the wooden enclosure,

* the packaging the product comes in,

If there are switches, potentiometers or encoders I am really partial to well working ones and these are not so cheap. It is nice to have but at least they make the product feel more expensive so it is not a total waste.

Everything else can be done dirt cheap. I don't think chinese are the best way to learn product design but they are pretty darn good at saving every last penny (or yuan in their case). I found it illuminating to study some of the devices of chinese origin (as opposed to ones that are only manufactured in China).


Even the wooden enclosure could be done for significantly less.

It's effectively a picture frame. There are plenty of standard box frame profiles they are very similar to what they have built. If it was me, I would have explored working with a picture frame company for final assembly, many are used to "assembling" products with extra components (think of stuff on Etsy). The frames are simply cut with a framing guillotine and stapled together.

My estimate would be frame (including custom cut back panel), and final assembly would be in the region of $10-$15 at a US/UK framers. (Excluding electrical BOM)


Picture frames are indeed perfect for this. I built something similar and the frame cost me around £37.50. My full BOM was around £130 (and that's without any economies of scale).


Used to work on an iptv set top box user interface (html+js on opera embedded) back in the days. BOM for the board designed in my country... 150$. Chinese manufacturer proposed a redesign 100% compatible with the same features... BOM at 15$ IIRC


How did you find a Chinese manufacturer to redesign it for you?


it was foxconn, they had some capacity :)


You would be surprised at how often people with no experience try to build a physical mass market product that nothing more than an RPi in a box. And they often go under because the BOM is too high and the software is garbage.


That's why when I started with design I decided to go for niche products for businesses. Kind of thing that when they have to buy it they will not be looking at the price too much.

Low volume (I was doing all soldering myself!), high sticker price, not especially profitable for me.

But makes it so much easier to design when you don't have to compromise so much and you don't have to be very inventive trying to save on BOM. And it is fun to be able to just do whatever you fancy.

I think starting mass market production for price conscious clients with no experience is just asking for problems.


Similarly, see https://tidbyt.com - RGB LED matrix desktop appliance to show weather, sports scores, etc.


In 2020 I set out to build something very similar[0]. This was before I even knew NYCTrainSign existed. I started with zero electronics knowledge and learned a lot. My prototype has been running solidly for way over a year and while I've always hoped I would be able to turn it into an actual product I could sell... doing so proved challenging.

Really what blocked me the most was CE/FCC conformance. I have done a lot of research into it and I couldn't figure out a way to get it without a massive investment upfront ($10k+). If anyone has experience in that area I would love to talk, my contact info is in my profile.

Who knows, maybe I will get it to a point where it can be sold eventually. I'll certainly write up about my learnings from the project at the very least.

0 - https://twitter.com/d0m96/status/1427055272980328460


https://www.instagram.com/p/BVAIWfGBARk/

my problem was squeezing the MTA API parser into the ESP32 flash...

as long as you use off-the-shelf / already conforming modules, like the esp wroom etc, the cost for FCC compliance testing goes down into the thousands. it's still a line item, but over a run of 1000-2000 units, it turns into something that costs about a buck per board :)


> my problem was squeezing the MTA API parser into the ESP32 flash...

Ahh, I wouldn't do that. The way I've done it is by pushing as much of the API handling logic off the ESP32 and onto a standalone server. The ESP32 is effectively a dumb client which takes in instructions like "WRITE westminster 5 min" from the server and draws the appropriate text/pixels/etc.

> as long as you use off-the-shelf / already conforming modules, like the esp wroom etc, the cost for FCC compliance testing goes down into the thousands. it's still a line item, but over a run of 1000-2000 units, it turns into something that costs about a buck per board :)

yeah... it's $1 per board unless none of your boards sell :)


that's fair ;)

when kickstarter was first taking off, I loved using it to springboard only the projects that would at least break even: if there wasn't enough demand, nothing happens, but as long as the minimum is high enough to cover all the costs, it worked out pretty well


Crowdsupply is great for those things now. They're much better at helping get hardware projects off the ground, and they can handle S&H too.


Yah using the pre-canned esp32's make the cert process a lot easier.

@dom96 thats cool! Ping me on the Nim discord sometime if you want. I've not done FCC but did do some hardware certification stuff. For low volume products there possible ways around full FCC certs I believe. IIRC, you can sell things as "kits". I wonder if theres also exemptions for "prototype builds" too.


A few years ago I created something similar for the London Tube although it was much more crude, just a pi zero and a strip of RGB LED lights that represented the colours of the tube lines.

The store that made the LED strip caught wind of it some how and they cut out a wooden enclosure for me to put the thing in, with the tube line names etched into the wood, which completely blew me away.

My dad kept on telling me at the time to sell it as a product but even then I knew that not being a hardware guy and not really having any experience building products it would be a dumb, expensive move - and stories like this reaffirm that this was the right thinking.

Not saying it can't be done, it's just someone with more electronics expertise is way more qualified than a guy throwing together a python script on a Pi zero with retail components.


there is an in-between: a write-up and a post on hackaday.com :)

would love to see the Pi zero and hacked together code on there!


Very good read. The parts focusing on the cost of the BOM for the signs really shines a light on just the basic difficulties of hardware compared to software companies


Seems down, mirror here: https://archive.is/dIbYd


What a good story "IoT" about :) Absolutely useless piece of kindergarten "engineering" + insanely inflated prices + real sales for any (low) price cause nobody wants this sh... + crowd of idlers in management growing like bacteria in a Petri dish = "Internet of Things". Nice!


Did they really think it is ok to include a remote access without telling customers?!

I guess this more often the case than people realize (especially with such hobby level hard- and software), but it actually is a huge security risk. Because once you are on a Raspyberry Pi in a customers network, there are no limits..

Insane.


$600? That's way overpriced.

Get some digital picture frame out of Shenzhen, like this one, that wholesales for $30 - $50.[1] That thing has WiFi, a 4-core CPU, and runs Android. Reprogram it to just run your application. Sell for maybe $75-$100. The manufacturer will add your logo and do custom packaging if you order at least 50.

There are even cheaper discrete LED signs that talk WiFi, but those would be kind of intense for home use. Those you sell to shops near stations, so people can shop or eat while watching the train arrival info.

[1] https://www.alibaba.com/product-detail/IPS-Display-7-8-10-13...?


Wait, you are open sourcing code that you did not write but reverse engineered? It that legal?


Google vs Oracle seems to say that offering up a compatible api with different internals is legal. Exploiting the signs seems risky though.


Fujitsu vs IBM also says you can make compatible hardware and release it as well. And of course they lost other, similar cases during the PC Clone era.


In the absolute no this is not legal, but in practice for a company that has been dead for a while after scamming a lot of people I wouldn't be overly concerned of legal repercussions.


Yes, legal. But in a parallel universal where it weren't legal, there are no damages in this case.


Any idea what these LED matrix signs cost as the raw unit, ready to integrate with your own thing for a DIY project?

Does the cost scale somewhat linearly with number of LEDs or horizontal x vertical resolution in pixels?


They are standard P4 panels, you can buy them for about $20-30 depending on seller/volume. They are 64x32 pixels.

You can also buy P5, P10 and other sizes. They number tells you how many millimeters between each physical pixel.

I use P10s for my Christmas display, they look good from about 15feet away or more.


https://jlcpcb.com/partdetail/Worldsemi-WS2812BMini/C527089

$0.01 per LED

About $2 per 100x100 mm board if you order about 100 of them.

About $1 per above board for assembly service.

Total: $6 per 15x15 LED board of 100x100 mm.

Total: $12 per 30x30 LED board of 100x100 mm.

I have no idea about how many would you actually need.

As for the power supply, each LED draws 12 mA. So you are going to need quite a beefy supply.

15x15 grid = 2.7 A @ 5 V

30x30 grid = 10.8 A @ 5 V


I was thinking something more like assembled modules of 100x100 or similar LEDs in chunks all together


Thoroughly enjoyable read.

Maybe if i play devils advocate it will add some value: People have castigated the high RPi cost (among others) and they are probably right about it, but there is something remarkable about been able to ship even a small number of devices with a full blown computer and a unique UI.

Maybe the more fundamental problem was not so much the hardware cost but not shipping enticing enough software to get people excited about the device?


IMO that Raspberry Pi 3 was way overkill for the project even back then and must have massively inflated the BOM.

Nice article overall :)


Often when showing off something cool I built for fun, people say "you should make those and sell them!"

This company is a great example of why I don't.

Mad respect for the author though. It'd be cool enough to just get one sign working, but to take over the API and domain is pretty awesome.


The complete git history and the development .bash_history file are my favorite parts.


This made me think of Chumby


App access could have been $1/mo. They could have expanded to other cities.


Your newsletter isn’t working heads up


Thanks for letting me know. I will have to take a look. I setup Ghost a long time ago and just really use it for the blogging...


Not to take away anything from the article but what does this have anything to do with IoT?


it was an IoT product (internet-connected LED sign)


That's just a display not an IoT project. The display did nothing as a thing.


Looks like they were trying to sell early prototypes as final products?


Wow, amazing story and writing!


Woof. Adafruit + Raspberry Pi in a shipped product screams to me that they did not have a real electronics guy on their team.

https://ukdepartureboards.co.uk/ is the British equivalent of this, and it seems to do everything right (don’t own one, but have seen on Twitter)- optional subscription, and nice looking hardware (seems like they contracted out for that). Though at the minute I suppose there are barely any trains running here :)


This is all too common in my experience, "oh we could just use an arduino/pi and a hat and a peripheral" rather than "Design a system that does <x>". As an EE who spent their primary career programming (horrors I know) and in systems analysis, I see it as the logical extreme of "why use a 555 when I can program an ATTiny to be a timer and do other stuff too!"

The allure of having one be able to add features "with just a bit of code" is catnip to product managers. They seriously cannot seem to resist going there.


Oof. My electronics stopped before GCSEs, and even I know about 555s.

I can believe someone might prefer software for prototyping, but manufacturing? Well, I've seen prototypes enter production, so yes, but still, ugh.


A attiny might not make sense, but you can get something like the PMS150C for around ~5ct, where it makes more sense to use a single MCU than a 555 with all the extra needed BOM Items.


Well, you can always choose this: https://www.hwlibre.com/en/troll-duino/


Usually yes, but it sometimes makes sense in a very low volume product.


The Code Quality section in the article also listed software red flags that point to the product being Amateur Hour. From the archive.org About Us link OP provided[1], there seemed to be plenty of "coders" but maybe not much embedded SW experience? They couldn't even agree on what language to use, whether to use tabs or spaces, and couldn't manage to produce a pristine Micro SD image without bash_history and a git repository littered onto it.

Kind of reminds me of a small company I used to work for where I noticed shortly after joining that they 1. had no source control or reproducible builds, and 2. ship Debug builds with symbols and no optimizations to customers, because they couldn't get the Release configuration to run without crashing.

1: https://web.archive.org/web/20180107132644/https://www.devsh...


> and it seems to do everything right

Last time I looked they were missing CE conformance, which afaik is pretty illegal. Surprised they haven't been found out yet.


Additionally, the composition of certified products does not lead to a certified product. Even adding an enclosure around a certified board implies the recertification.


This still seems like its fallen into same BOM trap.

You could buy a phone (moto e20), same size as their small display at 90e vs 140e for the board above...

Sell a nice android app, subscription and be done with it.

Phone packs a lot more features.


That's an indescript rectangle that could just show anything and could be from anywhere. They are selling a specific shape of rectangle that shows stuff about London and aesthetically signals that it is from London.


Maybe it’s very North American of me but I cringe when the author says “bad team”. Objectively, they didn’t have the skills, experience, or advisors to do this thing. And if the money truly disappeared, then someone crossed over into “bad” territory.

But hardware startups are brutal. Kickstarted hardware is torture.

Even experienced and well capitalized tiny teams go through this. A friend did one, rapidly realized the only option was to somehow get Chinese manufacturers excited about a small run of a strange product. Then two years later was trying to QA injection molds from across the ocean and switched manufacturers two times.

They delivered, but only after years of what must have felt like crawling over broken glass.

Maybe the NYC Train Sign team realized they couldn’t profit from the sign itself, but, unwilling to let the viral moment pass, were going to use it to establish themselves in consulting. You know… sell pickaxes.


I agree completely about the difficulty of a hardware startup, but being "good" means knowing in advance what kind of challenges you'll face, or at least knowing who to consult about it, and when to listen to their advice. I guarantee that at several points in the development of their product, someone pointed out the high BOM cost and the math required to become profitable. At this point Kickstarter has been around for long enough that even people without direct experience in hardware development should have enough case studies to know what they're wading into.


Indeed, I enjoyed the article, but felt the snarkiness was bit too much.

Startups fail all the time, and hardware startups are a minefield.

If they had succeeded then we'd be singing their praises on how they started off with a scrappy product built from off the shelf parts and then managed to productionise it and outsource the manufacturing to china...


I build something similar https://sschueller.github.io/posts/vbz-fahrgastinformation/ but with way less BOM and I keep getting asked that I should sell them. The primary reason I don't is because I don't want to support something like that for the next 10 years...


heh, same. I built a little custom sign for my airbnb. Part of my home automation system:

https://imgur.com/a/8A5IKV6


Very cool, and well executed! Do you have build details anywhere?


I don't, but it's essentially an Adafruit LED matrix + Pi + Adafruit PiHat.

I modified the adafruit python library to read lines from Redis

then I have another script that makes the API calls, has a REST server, etc and publishes text to reds queues.


The way the hostname is set, allowing an incredibly simple shell injection, reminds me that the S in IoT stands for Security.


> It seems that the company could remotely connect to a terminal on every sign.

What a red flag... they basically had a backdoor to everyone's home network.


That BOM is crazy. It looks like (and is) a hobby project that never had any consideration given to manufacturerability.


Taking over their DNS isn't going to help if they did things correctly. If they did things right each device is going to be doing cert verification with AWS IOT, and that verification will fail. That's client and server cert validation, unless they turned that off.

On reset it should still verify the server cert when it tries to call home, but since you have access to the image you can replace that (if they didn't stash it somewhere securely, that is).


The key point here would be "did things correctly" :)

The sign did use AWS IoT for real time configuration updates however initial configuration was pulled from their HTTP server. Using the vulnerability I describe in the article I just remove the connection to AWS IoT.


I’m sorry, but did you actually read the article? The one which details the process and shows a video of it working?

This is just straight up gaslighting. “That thing you said you did isn’t going to work”.


> Taking over their DNS isn't going to help if they did things correctly.

If you had read the post, you would have seen that they did, in fact, not “do things correctly”.


There are other ways to accomplish that (asymmetric cryptography says hi). You don't need AWS IoT for everything. I cringe when people's default solution is to make their company dependent on a niche product of another company, especially when its easily avoidable.


Could you explain this further? How would ensure that when you connect to https://trainsignapi.com that it validates the response is coming from the expected server? Would you hardcode a certificate on the device? What if you needed to upgrade the certificate? I don't quite understand how AWS IoT fits into it either.


If you're doing it "properly", you should have your own CA, not using public CAs, because public CAs will give certificates to anyone who controls servers on your domain or controls the domain; not just you. All it takes is someone to buy your expired domain, and they can get a new, publicly valid cert.

If you needed to update the CA, you'd need a firmware upgrade, served with a certificate signed by something chained to the old CA. And then, more likely than not, you'd want to do further updates on a new hostname, because it's hard to do a single hostname and send the right certs to the right clients.

This is a giant pain, and I don't blame people for using public CA infrastructure instead. Especially if your company goes bust, who cares?




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: