Hacker News new | past | comments | ask | show | jobs | submit login

I was one of the architects working on Ford SYNC. Internally, Ford has a big problem with open source software from a legal stand point. You have a fortune-5 company here. Translation for you: They are a very large target for lawsuits. The open source software provides no indemnification.

You also have to remember when SYNC was in the planning stages long before the public knew about it. I was part of the original team responsible for taking the whole concept to reality.

I can tell you that internally, it was one of the most well run software architecture teams I was ever apart of. The middle level managers really know how to run software teams which is not something you might expect from a big old automotive company.

But in many ways, our hands were tied. Recall about the time Bill Gates and Bill Ford riding around in a model-T. A lot of us didn't want Windows for automotive and were trying to champion Linux. Also, the iMX3 which was in the first generation SYNC modules was at the time a slow processor.

And to top things off, a fair bit of the software running on the SYNC module is not written by Ford. Ford partnered up with a crappy company called B-Squared.

The Maps are not Ford's either. When I was on the team, Ford had partnered with INRIX for Maps, traffic and direction.

SYNC is just not a simple single board computer with some apps running on it. There is an entire eco-system build around the SYNC module because it's connected to the vehicle networks (CAN Bus). Microsoft & Ford being partners (Windows servers in data center), Microsoft wanted to handle some of the software development but when Ford asked Microsoft to sign off on some legal agreement asking MS to take any and all responsibility for things like inadvertently deploying an air bag, they backed off.

Also, Ford SYNC as I said has an eco system of roughly 20-30 backend applications that support it. Some of these have to do with 911 assist and the TREAD act. The TREAD act is the US Federal Government's oversight on safety claims.

Because via SYNC you can report problems and have your vehicle serviced, their is a lot internal logic that keeps track of what is called EOL (End of Line) data about a vehicle which needs to funnel into TREAD act reporting.

Think about the not so long ago Toyota problem where they tried to blame floor mats for gas peddles getting stuck.

EDIT: Also, the GUI is not Microsoft CE native. The WindowsCE is just the OS running on the SYNC module. The GUI was Adobe ActionScript. Trust me, many of us yelled very loud about how stupid an idea this was but because B-Squared own the implementation (not the architecture), they were aloud to chose whatever they wanted to meet Ford specs. The results are a history less: JD Powers gave it SYNC poor ratings and many of us (myself included) got the hell out of dodge.

Middle managers, managing the day to day software architecture and specifications, were caught between a rock and a hard place with senior management (Allan Mullaly, Mark Fields, Bill Ford, Marcy Klevorn) pushing these partnership relationships we were forced to work within.




"The GUI was Adobe ActionScript. Trust me, many of us yelled very loud about how stupid an idea this was but because B-Squared own the implementation (not the architecture), they were aloud to chose whatever they wanted to meet Ford specs."

Sadly, this is how GUI design is in a lot of larger companies. The corporation hires a design firm to flesh out the UI concepts and test them on customers, so they quickly storyboard stuff up in Illustrator and Adobe Flash/Shockwave/whatever.

The concepts get tested, everyone gives a thumbs up, and then the software team gets handed a bunch of precooked assets and wireframes and told "make it run."

Then it's up to the embedded guys to make the system fly, except either a) the design firm has created graphics and animations so complex the underpowered hardware (designed and certified years ago on the last h/w cycle) can't make it go or b) the design guys took way too long in testing and focus groups so now the software guys are under the gun and instead of a rewrite/port, they port a crappy FlashLite player to the target hardware and cross their fingers.


The original Apple TV does this, in a fashion. The whole UI is a glorious mess of Quartz Composer.

Granted, that's no ActionScript, but very much a prototype being pressed into duty.


> The open source software provides no indemnification.

Just as Google and Apple do, QNX contains multiple open source components from other parties that QNX does not own the copyright to including code licensed under APACHE, BSD-2C, BSD-3C, BSD-OPENSSL, BSD-V, ISC-V, GPL2-EX2, MIT, MIT-V, MPL, MPL10, PYTHON, UL, ZLIB (this is copy-pasted from QNX's own open source license compliance documentation). Anyone can sue Ford directly for using those components just as they can with the use of Google or Apple software.

Now, if QNX itself provides a complete indemnification from both copyright and patent lawsuits arising from the use of said code and Apple and Google do not, that's a different story. But unless that is the case, Ford is no safer with QNX.


I'm not a lawyer but sat in many of the internal meetings. From what I understand, anyone who signs up to work with Ford Motor also signs a legal agreement saying you take on legal responsibility for your actions....

As an example, Ford is also a big Linux shop but they do not use any old Linux distro. They have partnered with Novell (SUSE Linux) who indemnifies them from all open source code found in the SUSE distribution.

When I was on Ford SYNC, no Google code ran on the module. The feature they are talking about is called Send To Sync and it's a web service where the SYNC module can receive a map planned on Google Maps. The mapping software in Ford SYNC is simply given the data output in the form of lat/long info.


There is no way that Novell or QNX are big enough to indemnify Ford unless we're talking about some kind of huge business or E&O policy or something; in which case Ford is paying the bill for that.


It doesn't matter that it's open source, it matters that you're buying it from someone who is assuming the liability if it's bad. Bad when I worked for the government we bought a lot of Red Hat enterprise licenses not because we wanted the support (it was for a classified machine) but simply because we weren't allowed to use "freeware".


There are providers of indemnification for certain components. OpenLogic is a big provider.

I agree that Ford appears to consider the benefit of a QNX system high enough to accept the risk of patent lawsuit.


They do now that they know the "benefit" of a WinCE system!


Disclaimer: I'm also working in the industry.

My thoughts are that Adobe Actionscript (in conjuction with AIR) isn't that bad. It could do years ago everything that is now reinvented with HTML5 and MVC frameworks (like angular), has a decent performance (if the port of the AIR runtime to your plattform was well done) and the the language is halfway sane (at least better then Javascript). I also liked it better then the new trend towards QT, because you can do everything in a single language.

I actually know some high end cars with some decent user-interfaces that are also using Adobe technology.

However you can do crappy UIs and a crappy implementation in every technology. If JD power gave bad ratings it's most likely more due to a bad UI design and usability than due to the underlying technology.

I actually had expected that Ford uses Silverlight Embedded, as this was pushed by Microsoft at that time. Interesting that this is not case.

Good to hear that Ford has at least some decent understanding about software. Here in europe trying to do something Software-like isn't a pleasant experience at all.


Former ActionScript developer here. Can confirm, it was actually pretty great. It's a hell of a lot more of a cleaner language than Javascript, for instance.

Don't forget that most game UIs are still built in ActionScript, and they're just fine.


A lawyer can find a legal issue in pretty much anything you intend to do. I've seen teams getting railroaded by this. It usually starts with someone higher up wanting some legal advice and the lawyer seizes the opportunity and scares the hell out of execs. Sometime company had been burned by lawsuit already so they double down on even a remotely possible legal threats. Suddenly everything is required to go through the legal team who let you know that all your possible actions might cause getting sued. Before you know company needs a large legal team to review and craft complicated stupid agreements even with janitors.

The fact is that there are fairly easy workarounds for this kind of legal threats. The "I Agree" is quite powerful button. Most consumer protection legal threats are actually fairly difficult to execute - thanks to severely weakened consumer protection laws in most states. Even if they do get executed in few rare cases or even become class action, settlements are fairly easy to achieve with often tiny impact on balance sheets. I understand that risks in automotive industry is higher when safety is compromised that case cause billions of dollars in recall in extreme cases (for example, Toyota and Firestone cases) but this is like wearing bullet proof vest all the time because you fear you will get in to some physco's shooting spree. These lawyer driven companies don't get wake up call until competition starts doing all these supposidly legally risky things and grabs all market share. Companies that win usually are fire firstt, ask forgiveness later kind. This is also the reason why enterprisy companies like Ford will never come out first with self driving car even if they had tech.


> Ford partnered up with a crappy company called B-Squared.

I had to work with some of those folks after they left B-Squared. Ugh. I shouldn't have to explain why you shouldn't just dump everything in the master branch, how to write a unit test, or why it's a bad idea to pull builds for app store submission from a random dev's machine. But I did. Which went a long way in explaining my horrific experience with the SYNC-enabled Ford I rented once. To someone not versed in the basics of professional software development, choosing embedded Adobe Flash probably seems like a good idea.


OMG, some of the backroom conversations where utterly laugh out loud hilarious. I distinctly recall getting a test spec. from B-Squared that read and I quote, "the application will leak memory just not megs and megs." This was literally in one of their test specs., I kid you not. The test was design to exercise certain features while monitor memory for tell tale signs of a software bugs.


B-Squared had some very good talent also; a few of my UW CSE classmates worked there in the mid/late 90s. I think it kind of went downhill when none of their own products took off and they basically became a pure consulting company.


This explains a lot. Every time I use my car stereo to play music from my iPhone I wonder if any decision maker at Ford tried to use the entertainment system for its intended purpose.


The first generation of the Ford's SYNC computer was designed in cooperation with Continental AG and is built around a 400 MHz Freescale i.MX31L processor with an ARM 11 CPU core, uses 256MB of 133 MHz Mobile DDR SDRAM from Micron and 2GB of Samsung NAND flash memory, runs the Windows Embedded Automotive (CE) operating system, and uses speech technology by Nuance Communications.

http://en.wikipedia.org/wiki/Ford_Sync

The first generation SYNC has a monochrome display (blue or red on black) with a very basic GUI (like a 90s Nokia cell phone). Despite a 400MHz CPU and WinCE the menu is slow and looks ugly, but it is solid and works.

Later SYNC generations had a graphical GUI with touch, that's probably the slow Actionscript based one the other commentor mentioned.

With the MyTouch issues, several european Ford 2014 models still ship with the first generation SYNC. The SYNC menu features an appstore entry despite the appstore was never activated for non-US models. The Google Maps feature is not available, in the EU version. The MyFord website with additional diagnostic features about the car as well as software update downloads is only available for US car models. There are no updates e.g. for the european models, not from the car dealers nor from Ford website.


Interesting insights. Not sure if any of this also applies to MyFordTouch... ehh turd. I haven't seen any embedded system as slow, sluggish and buggy as MyFT. After bringing my gf's 2014 Ford Fusion for the 5th time to the dealer for MyFT issues we decided to file a lemon claim with Ford. Don't get me wrong, Ford isn't the only car manufacturer with crappy half-asked infotainment solutions. As a matter of fact, I can't say I've actually seen any implementation done right, except for possibly Audis MMI, which doesn't rely on touchscreens and touch buttons as much but has physical buttons for operating HVAC controls, navigation, etc. In my opinion, very big mistake by Ford to replace all of the physical buttons by stupid touchy buttons you constantly end up activating unintentionally or having to wait for the sluggish GUI/touchscreen to load up just to adjust HVAC controls. In my opinion a good example of rushing out a product and trying to solve problems that don't exist.


Ford MyTouch is just a marketing term for SYNC; they are the same things just called different things. Everyone inside Ford calls it SYNC but rest of the world and what you see in dealerships is MyTouch.

Ford had another venture with DWalt (the tool company) and they partnered with Magneti Marelli who did most of the work. I wasn't involved in that project, but that endeavor was a situation where Ford basically said here some dashboard space and a list of specs., make it work. This system was called Ford Works.


Thanks for the insight - having worked for long in the Enterprise trenches I can certainly understand the difficulties of moving one thing to other. However it mostly takes strong management to get over the inertia in the name of doing something greater.

But as far as the Open Source indemnification goes - for the Android Auto use case it would be completely irrelevant right? That would be like suing Vizio for showing a pirated movie on the screen as far as I can tell.

For using AOSP - I can see that there may be some reservations there. But it is not unprecedented - Honda I think is using Android 4.0.4 to run their in-car system. Besides the biggest threat to Android from a patent / legal standpoint is Microsoft. Worst case Ford would pay up the $5 or whatever and still win on not paying QNX and having a better solution.

As for the dependencies / ecosystem - they could still support the older released models and start working with vendors/suppliers to get what they need supported on the new system. It's doable because presumably with the new Architecture - most things are either built in or you don't need those and people already have experience building what's missing. For example I don't see why they can't build the CAN Bus connectivity into Android in a way that it supports the same existing functionality.

I know it's work, money and risk but I think it would be towards a better outcome and there are some potential saving in not rebuilding everything from scratch and maintaining it and paying less royalties etc.


I presume the older software is tied to a relationship with B-Squared that has deteriorated beyond what current economics will allow and B-Squared doesn't know QNX or Linux. B-Squared's claim to fame is they talk about how much they know WindowsCE very well. It's their tool of choice even though Adobe ActionScript is largely to blame.

Keep in mind to add SYNC to your vehicle, it's $300 (or it was when it first came out). Cheap right? Ford is in the business to sell cars and what sells more cars, they want.

I'm not a lawyer, I see your point and was somewhat vocal on my own, but in my tenure I learned that Ford is a very successful company and they pay smart people to keep them out of trouble. I have to give Ford props for not taking any bail out money but I digress...

One more tid-bit I will add is that Ford never deals directly with the public. You get your vehicle serviced through dealerships.

So when Ford SYNC came out, they were trying to find away for customers to buy additional services via the SYNC system. This of course meant Ford had to now deal with the whole concept of accepting credit cards or finding a partner who knew how to accept credit cards.

As an architect on the team, one of my tasks was to assess the whole PCI compliance, what it would do to us if we got hacked, etc... In the early days, Ford tried to partner with PayPal to handle the credit card stuff but because Ford has a products it would collect for that are also provided by 3rd parties, it just made things very challenging. I'll spare the details but one of the issues I worked on was single sign on technology. You have a mix of technologies like WS-Federation, SAML, etc... and trying to get Ford and all the partners to adopt and stand up this stuff takes a L O N G time. Consider also some of these smaller companies don't have same budget as a Fortune-5 or even know what SAML is so you have a lot of educating going on too.


I've always wondered how B-Squared got such big design wins. They also did the Coca-Cola Freestyle.

Best I can tell is that they're so tight with Microsoft (ex-employees, maybe?) so when a large F-100 class company wants an embedded GUI and goes to Microsoft first, MS sells them on WinCE and then points the client to B-Squared to finish the work.

Or maybe B-Squared is the only firm left still writing GUIs for WinCE.


Your comment hit me like a bolt of lightning. Every time I use a Freestyle machine I can't believe someone delivered a product that is so agonizingly slow. It's such a basic idea, why does it take what feels like 1-2 seconds to respond to Every. Single. Click.


Last time I ate at a restaurant that had a Freestyle machine, the thing was stuck in an error state and everyone had to stand around and wait for the cashier to go reboot it before they could fill the soda cups they had purchased. Disaster.


Freestyle has been a very successful marketing concept, mostly for Coca-Cola. Less so for the stores that adopt the machine. It isn't exactly driving traffic into the stores like Coke had promised. They've also had a slew of maintenance problems.

Pepsi has had time to catch up and has recently launched Spire, their multi-flavor machine. They chose a simpler approach and skipped the whole microfluidic flavor cartridge thing. It's simply a revamped fountain/flavor dispenser with a new interface. It works just as well as Freestyle IMO, and actually has an advantage of using less physical space up front (but still needs the whole syrup stack behind the scenes)

http://www.pepsispire.com/


I know of no cars in the US that are selling with Android-based head units (those Honda ones are Europe-only). There are plenty of cars with Linux-based head units though


Thanks emcrazyone - can you comment at all about whether the team looked at the large and mature history of learning and standardization of displays and controls in the air / flight industry? These displays and controls are designed to optimally communicate information and reduce distraction in a safety critical environment -- just like driving.


The short answer is no. Ford partnered with B-Squared because someone knew someone is the way I understood things. Around the time SYNC was getting to be announced, Bill Gates & Bill Ford were seen in local media (TV news, papers, etc...) riding around in a model-T at Henry Ford Village with the reports being that Microsoft wanted to get into automobiles. Out of this, somehow, B-Squared was chosen (pushed on guys like me) with the comments always being they have a lot of domain expertise in bringing up WindowsCE on embedded platforms.

If you interested in the flight industry, something I've been looking at too:

Check out Disti (www.disti.com). Disiti is a company with an HMI(Human Machine Interface) that is gaining a lot of attention and they came out of a military flight industry. Their displays are in the Apache helicopter and the Virgin Atlantic Space Ship as a couple of their selling points. Because of their military involvement, everything they do is ISO26262 compliant which is attractive to the automotive space.


That is sales and marketing in a nutshell: Someone knew someone.

Don't forget that after all your math and science and engineering efforts are over, it still comes down to who knows whom. I'm not trying to be cynical, I'm trying to say: Plan for your end game. Don't fool yourself about the actual nature of that end game; it is not technical.


> Microsoft wanted to handle some of the software development but when Ford asked Microsoft to sign off on some legal agreement asking MS to take any and all responsibility for things like inadvertently deploying an air bag, they backed off. ... Think about the not so long ago Toyota problem where they tried to blame floor mats for gas peddles getting stuck.

Are you saying that the navigation and entertainment system in Ford vehicles is not isolated from the critical safety systems? If that's the case I hardly think sluggish interaction is the biggest problem.


Normally they are seperated. Safety related features run on dedicated ECUs running a real-time OS like Autosar. They are communicating via bus systems (CAN, LIN, Flexray). Even the vehicle related features on infotainment systems normally run on a dedicated CPU in the headunit (running an RTOS) which communicates with the entertainment-related CPU (running WinCE, QNX, Linux...) by IPC.

Some bad designs might neglect those guidelines and build everything in one box to save costs. However I hope that those only happens in the cheapest offerings.

However I think the discussion between Ford and MS could be more due to political reasons. Automotive OEMs like to put all risks (whether likely or unlikely) to their suppliers and make them liable. Changes from standard automotive contracts are unlikely. And MS might not be not accept such a standard contract.


>If that's the case I hardly think sluggish interaction is the biggest problem.

It is crazy because it's not as though a CAN or LIN bus is an expensive thing. It has been discussed ad nauseam that the option systems need to be segregated from the critical systems, and as far as I was aware, the entertainment / option systems were segregated.


Ford's also the company that has critical systems disable if the key is bumped out of the ignition. And then "Fixed" it by closing the hole on the keyfob so people couldn't hook more stuff on it and thus reduce the chance of it falling out.

The fact that any safety critical systems just die when the key comes out of the ignition, while moving at high speeds, is even more damning, IMO.


Sure you're not thinking about GM? They're the ones who were in the news recently for those ignition issues.


Oh dear, you're right, but it's too late to edit. Please downvote to hide it.

My point though is that the fact cars would be that fragile at all, that simple mistakes could disable critical systems, is bizarre. The MP3 player system should be totally unable to affect other operations, full stop. The ignition system shouldn't be able to kill power to critical systems.

Airplane passengers shouldn't be able to access flight controls.


> You have a fortune-5 company here. Translation for you: They are a very large target for lawsuits. The open source software provides no indemnification.

I work for another fortune-5 company who uses linux in many products that we sell. Another well-regulated industry, too. I will concede that the hazards for operating automobiles are well-beyond many other industries' products.

I own a 2010 Ford Fusion with SYNC and I really enjoy the features. I suppose it could've been done better, but the speaker-independent voice recognition is stellar.


Yeah, B-Squared is not a company I would do business with again. Very crappy code, very frustrating to work with.


Sorry to threadjack, but since you're here...do you have information on where to find coding standards for the auto industry? I'm kind of curious how you manage these projects with weird and precise requirements and using languages like C. Any good stories?


A starting point for you is the so-called MISRA-C standard: http://en.wikipedia.org/wiki/MISRA_C


All the automotive companies are interested in ISO26262. ISO26262 has many levels. Although, SYNC had no safety connection so it wasn't required to meet any of the ISO26262 levels.



That's some great insight, thanks. I own a Ford and Sync is just a horrible, PITA to operate piece of software. Extremely slow and sluggish. Maps and directions had hardly any intelligence to it and most of the time I end up pulling GMaps on my phone - not something I envisioned doing when I purchased the car with the then-awesome (or so I thought) navigation system.

The one thing I do like though, is the voice commands. At least that actually made things easier. Speech recognition wasn't perfect, but it worked most of the time.

I wish there were more competition in the automobile in-vehicle tech platform space. I wish someone would get it right, given that most people are in their vehicles a good portion of the day.


Wow, this is an amazing comment. I'm actually quite happy with sync on the 2014 ford escape. It is easy for me to get to what I need to do. Over time, I've realized it's better than a lot of physical button implementations for certain things.


I have a 2014 Escape too and I am happy with Sync insofar as I can get most things I need done done without actually using Sync :-) HVAC has buttons below the screen, radio has buttons on the steering wheel. And the feature I use most often is probably the hands-free phone call integration over bluetooth, which seems to work well without any intervention once it is set up.

Agreed that the actual GUI is atrocious and clunky. The way it deals with text messages is hilarious. Really I just pretend it doesn't exist and I'm happy.


Great post, thanks for taking the time to share! And, interesting point about how well-run the architecture team was ... and how it wound up not mattering because of the relationships that senior management was pushing.


I have never seen team who only does architecture and no implementation ever succeed. You usually end up with lot of UML diagrams and PPTs that are unfortunately not executable. When "implementors" arrive at the scene, they realize all the things that had been overlooked from 30,000ft. Finally what gets implemented is either good with little resemblance to those pesky UML or terrible but in full compliance of "architecture". Avoid pure architecture teams at all cost.


Unfortunatly that happens in automotive too often :( OEMs have little knowledge about implementation, most actual work is done by contractors.


Thank you for the details. I always wondered what was going on over on the other side of the wall.




Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: