Hacker News new | past | comments | ask | show | jobs | submit login

The problems with Core Data iCloud sync are vastly understated in this article.

> "Yes, there are various problems"

Yes, chief among them that it doesn't work. I'm quite serious here. No one has gotten an iCloud-based Core Data store to withstand the actual multi-device usage it was designed for.

We're not talking about "works, but buggy". We're not even talking about "works but is a pain in the ass". We're talking "core features do not function". We're talking about "no known workarounds to bugs at the core of system that prevent it from working in any way that resembles the documentation".

And it hasn't worked since iOS 5.0, released almost two years ago. We have been through another major yearly update and several point updates, and no visible progress has been made on Core Data iCloud sync. The documentation has not improved, and with every point release developers go back and survey the situation only to find more bugs and more and different obscure error codes being emitted by the system.

Core Data iCloud sync would be bearable if it were just buggy. It would even be bearable if there was forward momentum on the vast amount of brokenness in it. But neither have demonstrated themselves.

But enough whining, let's address some points made by the article.

> "Or you can allow offline editing and build your own custom conflict-resolution solution, which I can guarantee will make YOU very unhappy... iCloud can resolve this problem by letting your users edit data offline and resolving any merge conflicts automatically"

Yes indeed, why build your own conflict-resolution layer, which you will probably get wrong in horrible ways, when you can rely on a working solution?

Poignant question if there was a working solution. Author is describing an iCloud that does not exist. Core Data iCloud Sync conflict-resolution does not work, and when it does not work, it has failure modes which result in the irrecoverable corruption of your entire data store, and in combination with other iCloud bugs results in the complete inability for your app to edit its iCloud bucket ever again.

Ever again.

I wish I didn't have to use so much italics here, but the point really must be made.

> "even if your app has been terminated, the iCloud daemon can keep running and upload your data whenever it gets an opportunity"

> "with CDIS, data is pushed into your app when it’s ready"

Indeed, these are the points for using iCloud over other competing cloud sync services on iOS.

But that's a little like saying the Ford Pinto has a really nice tape deck. It has this really neat, really useful feature when it isn't exploding. It's like saying dinner service on the Titanic is great - none of this matters when you can't even get the boat to float.

> "And that they have really seasoned, very smart people working on that Core Data team. I can’t imagine Apple releasing something as ambitious as a distributed, decentralized database system without “thinking it through”."

But they didn't think it through.

Author, you've been working with CDIS for the last two years. The technology behind it is neither mysterious nor is it secret. CDIS isn't broken because Apple can't write code. CDIS is broken because it is architecturally unsound.

Core Data iCloud Sync breaks down from a high level like this:

- You break each transaction into your database out as a diff. You store this diff in a directory that the iCloud daemon is syncing. Thus, all new diffs (called transaction logs by CDIS) are replicated to all devices. So far so good.

- It is trivially easy for a combination of devices (or really, just two) to generate a combination of diffs that do not playback to a consistent database state.

- In a normal world where CRUD operations are handled by a server with some understanding of the underlying data model, the server resolves conflicts as the authoritative data source, and thus consistency is maintained.

- In the world of CDIS, there is no authoritative server, nor is one device ever designated as the canonical state (this would be unrealistic, since devices get retired and lost, nor is there anything special about a device that makes it canonical). Instead, conflict resolution is left to each device, the implementation of which is broken. Which is to say, your CDIS-enabled database will work fine until the moment two devices make conflicting changes. At which point it is irrecoverably corrupted.

- The CDIS client's reaction to this is to revert your local copy of the database to its last known good state and cease communicating with iCloud entirely. This error occurs silently without either notification via UI (to the user) or API (to the developer). There is no way to query this state, and as of today the only visibility into this error is via console logging.

- Because the corruption has already occurred server-side, this means all connected devices using this store are all now disconnected from iCloud when using the app.

- As previously mentioned, there is no simple way to detect this error state via the API. Even if you were able to figure out that this has occurred, there is no way to destroy your database and start over. Due to bugs, lack of documentation, or architectural oversights, there is enough metadata about the database that the app cannot delete, that deletion of the store and its associated diffs do not put iCloud back to a clean state.

This is why people are giving CDIS a lot of shit on the internet. Nobody seriously believes Apple is incompetent at programming, but even a mild dive into the underlying technology on which CDIS is based shows that this entire architecture is unworkable. It's not that Apple can't get it right, it's that they've committed to an architecture that no one can get right.

As someone who has spent the last few years neck-deep in iOS, who makes a living writing iOS apps, I want iCloud to work right. It is in my interest to cheer on Apple's every move. But short of completely scrapping their existing architecture I do not see how it can be meaningfully improved (read: made to actually work).

> "CDIS is a really compelling technology for many iOS / Mac developers to adopt in their apps, and you should be seriously looking at iOS6 or iOS7 as the point to jump onto the CDIS train if it fits your needs"

NO. I for one would be ecstatic if CDIS works as advertised in iOS7, but under no circumstances should anyone consider using CDIS on an iOS6 device.

> "and it deserves less dismissiveness"

Dear author. We, the iOS developer community, have not acted with dismissiveness about CDIS. In fact we raced towards it with high hopes. We fought the scant documentation. We fought the bugs. We poured into the dev forums in droves. We helped each other and we shared our learnings.

We are angry now because we have tried. Nobody dismissed CDIS out of hand - the anger here comes from tens of thousands, if not hundreds of thousands, of collective man hours exhausted attacking this problem from every possible angle, and we are no further from where we were when CDIS first dropped in iOS5.

All of the points you make are valid in a universe where the technology you're talking about is functional. It would be insanely great to have a cloud-based data store that does conflict resolution for you, that updates in the background, that removes our responsibility for hosting it, that has no operational overhead on our part, and pushes instead of pulls. Indeed, those are all great reasons to use such a technology - if it existed.

But as it is no one can make a Core Data iCloud store stand up for more than an hour. And no one can recover a corrupt data iCloud database after it's fallen over. So unless you've figured out the solution to the above two problems, I'd kindly suggest that you stop treating the entire iOS developer base like ignorant, petulant children.

That was way longer and way angrier than I intended, but the article is really IMO nonsensical, and the author's tone that the iOS community has someone "dismissed" the technology or somehow not given it a fair shot, bothered me greatly.




I'm extremely unimpressed with the software engineering coming out of Apple recently. So many of their major new iOS APIs either don't work (Core Data/Sync), are just not well thought through (Storyboards) or seemed to have been designed with much real world testing at all (AutoLayout).

They have a great foundation in Obj-C/Cocoa Touch but I'm very quickly losing confidence in their ability to move the platform forward.

Android is much less mature in many ways but seems to be under much more expert guidance lately, both architecturally and at the UI level.


I'm curious about what you think is wrong with Storyboards or AutoLayout. Storyboards certainly lack features so not all apps can be done with them, but nonetheless I think they're pretty solid. And AutoLayout works like magic, haven't had any problems (the IB-adds-constraints-for-you thing is annoying, but necessary)


Storyboards:

1. Interact poorly with version control.

2. Scale badly with a lot of VCs.

3. Segues are untyped and are inadequate to express complex navigation schemes.

AutoLayout:

1. Excessively verbose normal syntax and excessively limited and cryptic and untyped shorthand syntax.

2. IB tools are woefully inadequate for editing.

3. Small adjustments to complex layouts in IB completely scramble the constraints.

4. Does nothing to make the 95% use case of nested grid layouts explicit and simple.


2. Is the fundamental deal breaker. They're impossible to use in a multi-developer environment, because of the absolutely guaranteed merge conflicts at every turn.


I now have a one-VC per storyboard rule, which helps. I wouldn't use them all if I could design custom table/collection cells in xibs.


Thanks for that feedback! I thought I had conveyed my own displeasure with CDIS at the end, and mentioned a few times that "only if it worked well". Maybe I should've highlighted this more to reflect my own frustrations that I've experienced so far. I just felt that the tone had gone to an extreme in the other direction (that it's completely unworkable; and even if it did, you shouldn't support it).

My point was mostly to discourage people from completely writing off the idea around CDIS, that you shouldn't work with it "even if it worked", and countering some of the incorrect assumptions that some people were making. I think there's great reasons to adopt it (theoretically), and (this might be personal opinion) I don't believe that they're too far from cracking it. I guess time will tell. I've figured out a few hacks to backup data, to recover from failed syncs, walk through users with cleaning up their iCloud containers and get devices to setup sync again. They're mostly hacks that shouldn't be required, but have gotten to the point where users mostly have a great experience with the app, and even if sync fails or they start seeing weird errors, I can recover their data and walk them through restoring their state. I've only seen 3-4 people who have lost access to their data completely, and that really sucked. But that's a risk anyone takes with any technology, home-made or provided by some other platform.

I am open to the idea that I'm maybe too optimistic about all this. But having spent a lot of time and energy on CDIS, and having my app in production supporting CDIS for a few months, I think it was valuable information to share. A lot of issues that devs might run into while testing don't actually pan out in real-world usage. People don't have their devices sitting open side-by-side, trying to edit data simulatenously. But in any case, my plan is to compile a list of all the issues I've experienced next, and maybe some of the hacks that I'm using to work around them, and do another write-up in the near future.


> "that you shouldn't work with it "even if it worked""

I simply have not seen this attitude at all on the internet. Everybody wants the promise of CDIS. All of the anger, frustration, and vitriol is around the broken promise of it, not the concept.

> "and (this might be personal opinion) but I don't believe that they're too far from cracking it."

That remains to be seen. There is a growing sentiment in the Apple dev community that Apple's QA quality is slipping, and slipping hard, and this has a large detrimental impact on developers.

I was recently at a conference where this "joke" was made multiple times. Whenever someone commented that "X might be fixed in iOS7" someone would pipe up "Great, then we can use it when iOS8 comes out."

The upgrade rate on iOS devices far outstrips Android, but even now, on the eve of iOS7, a good 15-25% of devices are still on iOS5. There is effectively a one-year gap between Apple coming out with new API, and being able to require the use of the API in your app (unless you feel like leaving up to a quarter of your userbase out).

With things like CDIS this one-year gap is effectively lengthened to TWO years. Apple comes out with something new that's broken enough to be unusable in production. We wait another full year for them to finish baking it. And then we wait another year for the install base to be wide enough to ship an app with it.

With CDIS this is coming up to be a FOUR YEAR GAP, assuming they make it production-stable in iOS7.

There is plenty of reasonable skepticism when it comes to doubting whether CDIS will be fixed at all, or be quietly swept under the carpet as a failed technology.

> "I've only seen 3-4 people who have lost access to their data completely"

In my experience data loss has never been an issue - loss of iCloud connectivity is. Which is to say, once a conflict resolution failure occurs, that device is now an island - no changes made there will be communicated to any other device, and no changes made elsewhere makes it to that device either. And there is no way to restore this connection that anyone is aware of it.

Data loss hasn't ever been a major problem here - when a breaking conflict resolution occurs Core Data's default behavior is to go all-local rather than delete data.

The inability to repair a broken iCloud link is 100x more aggravating than simply breaking, since it means having to tell a user that they are SOL. By far the greatest frustration I've encountered re: CDIS is exactly this - that once it breaks, it's permanently broken.

> "But in any case, my plan is to compile a list of all the issues I've experienced next, and maybe some of the hacks that I'm using to work around them, and do another write-up in the near future."

I'm looking forward to it, though I remain extremely skeptical that Core Data is production-ready.

I've now, collectively, been in rooms with literally thousands of angry CDIS developers. Very smart developers who have yet to be able to ship a single CDIS-enabled app because its stability is simply atrocious. I'm wondering how it is that your experience is so different from everyone else's - where you reportedly have CDIS working in a production app with few/no issues, while others can't even get their Core Data stores to stand up straight. That's a pretty huge gap in progress.


> I simply have not seen this attitude at all on the internet. Everybody wants the promise of CDIS. All of the anger, frustration, and vitriol is around the broken promise of it, not the concept.

- I referenced the Brent Simmons article and the Marco / Siracusa podcast early on in my post. They're all highly-visible and prominent members of the Apple dev community, and this was clearly their stance that I was challenging.

- Regarding upgrading, you can mark out certain features to be available on the latest version of iOS while maintaining general backward compatibility with the previous iOS version. My app is compatible with iOS5, but I don't expose the iCloud sync option to iOS5 users because I don't want them to use it. I've had 0 complaints about this policy.

> The inability to repair a broken iCloud link is 100x more aggravating than simply breaking, since it means having to tell a user that they are SOL. By far the greatest frustration I've encountered re: CDIS is exactly this - that once it breaks, it's permanently broken.

This isn't true. I encounter this problem fairly regularly with users, and the process I outline for them is this:

- turn off iCloud sync option from my app from all devices. This will make an object-by-object copy of 'ubiquity' sqlite db into their local sandbox

- nuke their container from Settings -> iCloud -> Storage and Backup -> Manage Storage -> Documents and Data -> Show All -> <my app> -> Edit -> Delete All. This cleans out their iCloud database and all metadata associated with it

- wait a few minutes for the delete to sync over to all devices

- choose one of the devices as the most recent 'truth' version, and use that to be the first device to sync up to iCloud

- connect second device to iCloud again, and this will fetch data from iCloud instead of uploading. Once done,from then on, sync will work again seamlessly across both devices.

The instructions have some nuance in them, but that's the gist of it. Resetting the iCloud container fixes almost all the problems that you would encounter with CDIS, and the hacks are around making backups and restoration of data as painless as possible, and trying to minimize data loss. Again, this adds customer service add-on, but it's far far from impossible.


> "- nuke their container from Settings"

This is the part that's news to me - I've observed myself and corroborated with many other devs that nuking a container prevents the container from being recreated by that app ID ever again.

This really is the one big sticking point - if nuking containers actually worked properly the scope of the problem is greatly reduced. Sure, potential data loss, but at least your devices start talking again.

I'd be interested in seeing code, but this is definitely something outside of your app. Very curious. I was very recently in a room full of CDIS devs who complain that the above is still broken in iOS 6.1.3, and I'm at a loss as to how you've been able to achieve successful recreation of an iCloud container that has been deleted via Settings.


Nuking a store works great; always has, even with iOS5 actually. And 'nuking' I mean specifically going into settings and doing a Delete All. If you delete the root ubiquity folder in Mobile Documents using Finder, then yes, you're never going to be able to use that ubiquity container again.

After nuking a store and waiting a few minutes (ideally restarting your device as well), the container is just as good as new. You can try it out with a basic CDIS app yourself and see.


That's the news part - I'm not talking about deleting the ubiquity container. Ever since iOS5, and even in iOS6, my experience - and the experience of every developer I've ever run into using CDIS - is that deleting a container via Settings -> iCloud -> Manage -> {appname} -> Delete All results in a permanent breakage.

I'm not calling you a liar - I'm quite confident it works fine for you, but I'm wondering how our experiences can be so different. This isn't just me, this is literally confirmed by entire room fulls of people who've been wrestling CDIS for months.

I'm baffled.


Just to chime in: I use this all the time in testing, and it's never failed on me. You have to do the right thing in your app to watch for these changes and gracefully reset and all, but I've nuked the container (from Settings->iCloud, on Mac and iOS) dozens of times now and never had permanent breakage of any kind.

Occasionally, after doing this, it'll appear to take much longer to provision a new ubiquity store (sometimes minutes, which is unacceptable), but never actually broken.


It's not just me. You can go through a lot of the threads on devforums.apple.com, in the iCloud section, and you'll hear the same thing. The solution to almost all problems is using the Delete All hammer. It's just an unpleasant solution, but definitely fixes the problem and lets you start fresh.


> No one has gotten an iCloud-based Core Data store to withstand the actual multi-device usage it was designed for.

Seems like the guys who build iawriter did or maybe we're talking about something different


iA Writer is document-based, and I believe these are working fine in iCloud. Things break when iCloud is used to sync Core Data based apps.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: