Hacker News new | past | comments | ask | show | jobs | submit login
Why iOS Devs Should Consider Using Core Data iCloud sync (zaal.tumblr.com)
32 points by zulfishah on March 31, 2013 | hide | past | favorite | 30 comments



  iCloud can resolve this problem by letting your users edit data offline and 
  resolving any merge conflicts automatically (and in the case of iCloud Document
  syncing, giving you some options to get feedback from the user to help resolve 
  the merge conflicts). If this worked seamlessly, it would add great value to
  your product. 
If it worked seamlessly, which it doesn't. Conflict resolution is very application specific, I'm not convinced that a one-size-fits-all automated solution will ever be able to handle the edge cases as well as the original developer, who has intimate knowledge of the problem domain and the structure of the data to be synced.

  App Store Features: you might get featured by Apple on the App Store for using
  their latest, greatest technologies.
Don't think this is relevant for iCloud anymore. Besides, Apple has featured tons of apps that do their own proprietary sync or use Dropbox for syncing. (e.g. Evernote, Omnifocus, 1Password). The quality of the app is much more significant than the APIs it uses.

  Future Plans to move to Android / Web: a legitimate concern. But why not wait
  till your iOS product has grown big enough to have substantial demand for
  Android and/or Web, and develop your custom back-end solution then?
Migrating your user's data off iCloud one year later is going to be a massive, massive pain to deal with. And database syncing is not like file syncing, it won't be trivial to integrate another syncing mechanism into the application. Things will start to get quite messy if you also want to interleave iCloud in there for backwards compatibility.


Good points, though I'll just say this:

- using Apple-sanctioned technologies might improve your chances of getting featured by Apple. Of course, well-known popular apps get featured regularly by Apple, but this is one of the factors that goes into their calculations of who to feature. This is not a reason per se for using iCloud, but an added incentive (as I mention, it really didn't pan out for me!)

- Exporting data out of iCloud shouldn't be any more complicated than importing data into iCloud is. You can easily walk through the Core Data object graph and do whatever you want with the data. And with NSIncrementalStore, you can easily keep the core of your application the same and utilize some other web service to power the data for your app.


There are a lot of alternatives to iCloud. Most of them are just backend-as-a-service, but some of them are really sync engines. Dropbox has been around a long time, and is really popular for iOS file sync.

My team at Couchbase has been building mobile NoSQL databases that sync[1], for a number of years now, and our tech is used by a few apps in the app store already.[2]

This preamble is to say that I've had plenty of conversations with developers who would be in iCloud's target market but decided not to use it. And the big reason they won't use it has very little to do with the reliability and user experience problems that keep coming up in the critiques of iCloud. The big reason they don't use it, is that they need to control their data.

It's one thing for a small game or journaling app to outsource data management to Apple, but an entirely different thing for medical apps, enterprise sales, or other "serious" apps to do that. Perhaps in the long term Apple will offer an enterprise version of iCloud that you can run on your own terms. But until that happens, most serious apps are going to want to run their own infrastructure.

BTW all the code we write is open source, so if you are interested in sync capable databases with offline and p2p support, join us. [3]

[1] https://github.com/couchbase/couchbase-lite-ios [2] https://github.com/couchbaselabs/TouchDB-iOS/wiki/TouchDB-In... [3] https://groups.google.com/forum/#!forum/mobile-couchbase


Thanks for this. When I looked at the couch offerings just a few months ago, I was turned off by the need to run what looked like a lot of heavy Couch code on my devices. The mobile offerings didn't look mature. Couchbase Lite looks promising, and makes me want to look at the Couch space again.


Not to mention that very few businesses can afford to be iOS-only. Apple's refusal to even consider supporting other platforms with their core technologies dooms them to an Apple-only niche, IMO. That game only works with an overwhelming majority market share.


The problems with Core Data iCloud sync are vastly understated in this article.

> "Yes, there are various problems"

Yes, chief among them that it doesn't work. I'm quite serious here. No one has gotten an iCloud-based Core Data store to withstand the actual multi-device usage it was designed for.

We're not talking about "works, but buggy". We're not even talking about "works but is a pain in the ass". We're talking "core features do not function". We're talking about "no known workarounds to bugs at the core of system that prevent it from working in any way that resembles the documentation".

And it hasn't worked since iOS 5.0, released almost two years ago. We have been through another major yearly update and several point updates, and no visible progress has been made on Core Data iCloud sync. The documentation has not improved, and with every point release developers go back and survey the situation only to find more bugs and more and different obscure error codes being emitted by the system.

Core Data iCloud sync would be bearable if it were just buggy. It would even be bearable if there was forward momentum on the vast amount of brokenness in it. But neither have demonstrated themselves.

But enough whining, let's address some points made by the article.

> "Or you can allow offline editing and build your own custom conflict-resolution solution, which I can guarantee will make YOU very unhappy... iCloud can resolve this problem by letting your users edit data offline and resolving any merge conflicts automatically"

Yes indeed, why build your own conflict-resolution layer, which you will probably get wrong in horrible ways, when you can rely on a working solution?

Poignant question if there was a working solution. Author is describing an iCloud that does not exist. Core Data iCloud Sync conflict-resolution does not work, and when it does not work, it has failure modes which result in the irrecoverable corruption of your entire data store, and in combination with other iCloud bugs results in the complete inability for your app to edit its iCloud bucket ever again.

Ever again.

I wish I didn't have to use so much italics here, but the point really must be made.

> "even if your app has been terminated, the iCloud daemon can keep running and upload your data whenever it gets an opportunity"

> "with CDIS, data is pushed into your app when it’s ready"

Indeed, these are the points for using iCloud over other competing cloud sync services on iOS.

But that's a little like saying the Ford Pinto has a really nice tape deck. It has this really neat, really useful feature when it isn't exploding. It's like saying dinner service on the Titanic is great - none of this matters when you can't even get the boat to float.

> "And that they have really seasoned, very smart people working on that Core Data team. I can’t imagine Apple releasing something as ambitious as a distributed, decentralized database system without “thinking it through”."

But they didn't think it through.

Author, you've been working with CDIS for the last two years. The technology behind it is neither mysterious nor is it secret. CDIS isn't broken because Apple can't write code. CDIS is broken because it is architecturally unsound.

Core Data iCloud Sync breaks down from a high level like this:

- You break each transaction into your database out as a diff. You store this diff in a directory that the iCloud daemon is syncing. Thus, all new diffs (called transaction logs by CDIS) are replicated to all devices. So far so good.

- It is trivially easy for a combination of devices (or really, just two) to generate a combination of diffs that do not playback to a consistent database state.

- In a normal world where CRUD operations are handled by a server with some understanding of the underlying data model, the server resolves conflicts as the authoritative data source, and thus consistency is maintained.

- In the world of CDIS, there is no authoritative server, nor is one device ever designated as the canonical state (this would be unrealistic, since devices get retired and lost, nor is there anything special about a device that makes it canonical). Instead, conflict resolution is left to each device, the implementation of which is broken. Which is to say, your CDIS-enabled database will work fine until the moment two devices make conflicting changes. At which point it is irrecoverably corrupted.

- The CDIS client's reaction to this is to revert your local copy of the database to its last known good state and cease communicating with iCloud entirely. This error occurs silently without either notification via UI (to the user) or API (to the developer). There is no way to query this state, and as of today the only visibility into this error is via console logging.

- Because the corruption has already occurred server-side, this means all connected devices using this store are all now disconnected from iCloud when using the app.

- As previously mentioned, there is no simple way to detect this error state via the API. Even if you were able to figure out that this has occurred, there is no way to destroy your database and start over. Due to bugs, lack of documentation, or architectural oversights, there is enough metadata about the database that the app cannot delete, that deletion of the store and its associated diffs do not put iCloud back to a clean state.

This is why people are giving CDIS a lot of shit on the internet. Nobody seriously believes Apple is incompetent at programming, but even a mild dive into the underlying technology on which CDIS is based shows that this entire architecture is unworkable. It's not that Apple can't get it right, it's that they've committed to an architecture that no one can get right.

As someone who has spent the last few years neck-deep in iOS, who makes a living writing iOS apps, I want iCloud to work right. It is in my interest to cheer on Apple's every move. But short of completely scrapping their existing architecture I do not see how it can be meaningfully improved (read: made to actually work).

> "CDIS is a really compelling technology for many iOS / Mac developers to adopt in their apps, and you should be seriously looking at iOS6 or iOS7 as the point to jump onto the CDIS train if it fits your needs"

NO. I for one would be ecstatic if CDIS works as advertised in iOS7, but under no circumstances should anyone consider using CDIS on an iOS6 device.

> "and it deserves less dismissiveness"

Dear author. We, the iOS developer community, have not acted with dismissiveness about CDIS. In fact we raced towards it with high hopes. We fought the scant documentation. We fought the bugs. We poured into the dev forums in droves. We helped each other and we shared our learnings.

We are angry now because we have tried. Nobody dismissed CDIS out of hand - the anger here comes from tens of thousands, if not hundreds of thousands, of collective man hours exhausted attacking this problem from every possible angle, and we are no further from where we were when CDIS first dropped in iOS5.

All of the points you make are valid in a universe where the technology you're talking about is functional. It would be insanely great to have a cloud-based data store that does conflict resolution for you, that updates in the background, that removes our responsibility for hosting it, that has no operational overhead on our part, and pushes instead of pulls. Indeed, those are all great reasons to use such a technology - if it existed.

But as it is no one can make a Core Data iCloud store stand up for more than an hour. And no one can recover a corrupt data iCloud database after it's fallen over. So unless you've figured out the solution to the above two problems, I'd kindly suggest that you stop treating the entire iOS developer base like ignorant, petulant children.

That was way longer and way angrier than I intended, but the article is really IMO nonsensical, and the author's tone that the iOS community has someone "dismissed" the technology or somehow not given it a fair shot, bothered me greatly.


I'm extremely unimpressed with the software engineering coming out of Apple recently. So many of their major new iOS APIs either don't work (Core Data/Sync), are just not well thought through (Storyboards) or seemed to have been designed with much real world testing at all (AutoLayout).

They have a great foundation in Obj-C/Cocoa Touch but I'm very quickly losing confidence in their ability to move the platform forward.

Android is much less mature in many ways but seems to be under much more expert guidance lately, both architecturally and at the UI level.


I'm curious about what you think is wrong with Storyboards or AutoLayout. Storyboards certainly lack features so not all apps can be done with them, but nonetheless I think they're pretty solid. And AutoLayout works like magic, haven't had any problems (the IB-adds-constraints-for-you thing is annoying, but necessary)


Storyboards:

1. Interact poorly with version control.

2. Scale badly with a lot of VCs.

3. Segues are untyped and are inadequate to express complex navigation schemes.

AutoLayout:

1. Excessively verbose normal syntax and excessively limited and cryptic and untyped shorthand syntax.

2. IB tools are woefully inadequate for editing.

3. Small adjustments to complex layouts in IB completely scramble the constraints.

4. Does nothing to make the 95% use case of nested grid layouts explicit and simple.


2. Is the fundamental deal breaker. They're impossible to use in a multi-developer environment, because of the absolutely guaranteed merge conflicts at every turn.


I now have a one-VC per storyboard rule, which helps. I wouldn't use them all if I could design custom table/collection cells in xibs.


Thanks for that feedback! I thought I had conveyed my own displeasure with CDIS at the end, and mentioned a few times that "only if it worked well". Maybe I should've highlighted this more to reflect my own frustrations that I've experienced so far. I just felt that the tone had gone to an extreme in the other direction (that it's completely unworkable; and even if it did, you shouldn't support it).

My point was mostly to discourage people from completely writing off the idea around CDIS, that you shouldn't work with it "even if it worked", and countering some of the incorrect assumptions that some people were making. I think there's great reasons to adopt it (theoretically), and (this might be personal opinion) I don't believe that they're too far from cracking it. I guess time will tell. I've figured out a few hacks to backup data, to recover from failed syncs, walk through users with cleaning up their iCloud containers and get devices to setup sync again. They're mostly hacks that shouldn't be required, but have gotten to the point where users mostly have a great experience with the app, and even if sync fails or they start seeing weird errors, I can recover their data and walk them through restoring their state. I've only seen 3-4 people who have lost access to their data completely, and that really sucked. But that's a risk anyone takes with any technology, home-made or provided by some other platform.

I am open to the idea that I'm maybe too optimistic about all this. But having spent a lot of time and energy on CDIS, and having my app in production supporting CDIS for a few months, I think it was valuable information to share. A lot of issues that devs might run into while testing don't actually pan out in real-world usage. People don't have their devices sitting open side-by-side, trying to edit data simulatenously. But in any case, my plan is to compile a list of all the issues I've experienced next, and maybe some of the hacks that I'm using to work around them, and do another write-up in the near future.


> "that you shouldn't work with it "even if it worked""

I simply have not seen this attitude at all on the internet. Everybody wants the promise of CDIS. All of the anger, frustration, and vitriol is around the broken promise of it, not the concept.

> "and (this might be personal opinion) but I don't believe that they're too far from cracking it."

That remains to be seen. There is a growing sentiment in the Apple dev community that Apple's QA quality is slipping, and slipping hard, and this has a large detrimental impact on developers.

I was recently at a conference where this "joke" was made multiple times. Whenever someone commented that "X might be fixed in iOS7" someone would pipe up "Great, then we can use it when iOS8 comes out."

The upgrade rate on iOS devices far outstrips Android, but even now, on the eve of iOS7, a good 15-25% of devices are still on iOS5. There is effectively a one-year gap between Apple coming out with new API, and being able to require the use of the API in your app (unless you feel like leaving up to a quarter of your userbase out).

With things like CDIS this one-year gap is effectively lengthened to TWO years. Apple comes out with something new that's broken enough to be unusable in production. We wait another full year for them to finish baking it. And then we wait another year for the install base to be wide enough to ship an app with it.

With CDIS this is coming up to be a FOUR YEAR GAP, assuming they make it production-stable in iOS7.

There is plenty of reasonable skepticism when it comes to doubting whether CDIS will be fixed at all, or be quietly swept under the carpet as a failed technology.

> "I've only seen 3-4 people who have lost access to their data completely"

In my experience data loss has never been an issue - loss of iCloud connectivity is. Which is to say, once a conflict resolution failure occurs, that device is now an island - no changes made there will be communicated to any other device, and no changes made elsewhere makes it to that device either. And there is no way to restore this connection that anyone is aware of it.

Data loss hasn't ever been a major problem here - when a breaking conflict resolution occurs Core Data's default behavior is to go all-local rather than delete data.

The inability to repair a broken iCloud link is 100x more aggravating than simply breaking, since it means having to tell a user that they are SOL. By far the greatest frustration I've encountered re: CDIS is exactly this - that once it breaks, it's permanently broken.

> "But in any case, my plan is to compile a list of all the issues I've experienced next, and maybe some of the hacks that I'm using to work around them, and do another write-up in the near future."

I'm looking forward to it, though I remain extremely skeptical that Core Data is production-ready.

I've now, collectively, been in rooms with literally thousands of angry CDIS developers. Very smart developers who have yet to be able to ship a single CDIS-enabled app because its stability is simply atrocious. I'm wondering how it is that your experience is so different from everyone else's - where you reportedly have CDIS working in a production app with few/no issues, while others can't even get their Core Data stores to stand up straight. That's a pretty huge gap in progress.


> I simply have not seen this attitude at all on the internet. Everybody wants the promise of CDIS. All of the anger, frustration, and vitriol is around the broken promise of it, not the concept.

- I referenced the Brent Simmons article and the Marco / Siracusa podcast early on in my post. They're all highly-visible and prominent members of the Apple dev community, and this was clearly their stance that I was challenging.

- Regarding upgrading, you can mark out certain features to be available on the latest version of iOS while maintaining general backward compatibility with the previous iOS version. My app is compatible with iOS5, but I don't expose the iCloud sync option to iOS5 users because I don't want them to use it. I've had 0 complaints about this policy.

> The inability to repair a broken iCloud link is 100x more aggravating than simply breaking, since it means having to tell a user that they are SOL. By far the greatest frustration I've encountered re: CDIS is exactly this - that once it breaks, it's permanently broken.

This isn't true. I encounter this problem fairly regularly with users, and the process I outline for them is this:

- turn off iCloud sync option from my app from all devices. This will make an object-by-object copy of 'ubiquity' sqlite db into their local sandbox

- nuke their container from Settings -> iCloud -> Storage and Backup -> Manage Storage -> Documents and Data -> Show All -> <my app> -> Edit -> Delete All. This cleans out their iCloud database and all metadata associated with it

- wait a few minutes for the delete to sync over to all devices

- choose one of the devices as the most recent 'truth' version, and use that to be the first device to sync up to iCloud

- connect second device to iCloud again, and this will fetch data from iCloud instead of uploading. Once done,from then on, sync will work again seamlessly across both devices.

The instructions have some nuance in them, but that's the gist of it. Resetting the iCloud container fixes almost all the problems that you would encounter with CDIS, and the hacks are around making backups and restoration of data as painless as possible, and trying to minimize data loss. Again, this adds customer service add-on, but it's far far from impossible.


> "- nuke their container from Settings"

This is the part that's news to me - I've observed myself and corroborated with many other devs that nuking a container prevents the container from being recreated by that app ID ever again.

This really is the one big sticking point - if nuking containers actually worked properly the scope of the problem is greatly reduced. Sure, potential data loss, but at least your devices start talking again.

I'd be interested in seeing code, but this is definitely something outside of your app. Very curious. I was very recently in a room full of CDIS devs who complain that the above is still broken in iOS 6.1.3, and I'm at a loss as to how you've been able to achieve successful recreation of an iCloud container that has been deleted via Settings.


Nuking a store works great; always has, even with iOS5 actually. And 'nuking' I mean specifically going into settings and doing a Delete All. If you delete the root ubiquity folder in Mobile Documents using Finder, then yes, you're never going to be able to use that ubiquity container again.

After nuking a store and waiting a few minutes (ideally restarting your device as well), the container is just as good as new. You can try it out with a basic CDIS app yourself and see.


That's the news part - I'm not talking about deleting the ubiquity container. Ever since iOS5, and even in iOS6, my experience - and the experience of every developer I've ever run into using CDIS - is that deleting a container via Settings -> iCloud -> Manage -> {appname} -> Delete All results in a permanent breakage.

I'm not calling you a liar - I'm quite confident it works fine for you, but I'm wondering how our experiences can be so different. This isn't just me, this is literally confirmed by entire room fulls of people who've been wrestling CDIS for months.

I'm baffled.


Just to chime in: I use this all the time in testing, and it's never failed on me. You have to do the right thing in your app to watch for these changes and gracefully reset and all, but I've nuked the container (from Settings->iCloud, on Mac and iOS) dozens of times now and never had permanent breakage of any kind.

Occasionally, after doing this, it'll appear to take much longer to provision a new ubiquity store (sometimes minutes, which is unacceptable), but never actually broken.


It's not just me. You can go through a lot of the threads on devforums.apple.com, in the iCloud section, and you'll hear the same thing. The solution to almost all problems is using the Delete All hammer. It's just an unpleasant solution, but definitely fixes the problem and lets you start fresh.


> No one has gotten an iCloud-based Core Data store to withstand the actual multi-device usage it was designed for.

Seems like the guys who build iawriter did or maybe we're talking about something different


iA Writer is document-based, and I believe these are working fine in iCloud. Things break when iCloud is used to sync Core Data based apps.


I like iCloud but it is only an add-on sync option for most games we do as they are android/ios/web/desktop. A non fully cross platform cloud sync option just doesn't help speed up development much but it s a great low barrier sync for iOS only customers.

For iOS it is pretty easy though, anything in Docs gets synced if logged in and the app is setup for iCloud. It has always been a little touchy when they added the Caches/Documents requirements. You have to setup a no backup flag to prevent it from syncing if storing anything there which can be a problem if you store too much.

    BOOL success = [URL setResourceValue:[NSNumber numberWithBool:YES] 
        forKey:NSURLIsExcludedFromBackupKey error:&error];


Hey Apple, why don't you get to studying on good old Lotus Notes? Multimaster, highly replicated, non-relational, conflict-reconciling, document-focused storage for what, twenty years?


I assume you know that Lotus Notes was basically just a sync engine. That was its one and only core feature [1]. Apple is overextended - they have to keep up with Android and maybe even WP8. To fix iCloud CDIS with application-specific arbitrary data models is a very difficult task. Especially with their dumb master approach. A true client/server solution requires a smart server to mitigate and handle sync conflicts. As users demand more and more functionality, including offline capabilities, iOS apps begin to morph into client/server ptograms. So much so that what most developers end up doing is writing their own client/server sync. iOS and Android budgets dive into the deep end once sync requirements get baked in.

[1] http://www.ibm.com/developerworks/lotus/library/ls-NDHistory...


Couldn't agree more than it's just badly implemented right now. It's at a "barely viable" stage right now. Article was meant to question the very assumption that it's a just a fundamentally bad idea to begin with, and all the negativity that's recently coming out in the press. Yes, it's not for every app or every situation, but it could be a great solution for a lot of apps (if only it worked a bit better!).


Again I hope people can separate Core Data and iCloud. Core Data works perfectly fine (though I suspect some will argue with me on that point, I've had no issues with it since 4)

But iCloud is just badly implemented for the moment. Will probably get better. Apple doesn't call iCloud beta, and people will probably be pissed since it doesn't work properly.

I suspect that if they did call it Beta, some might have had gone easy on it for all the faults out there.

I gave it a try when the API was accessible, I didn't like it, never looked back.

1)SSO can work with other form of social logins - and will also work from other platforms/clients

2) Privacy? As a developer you have access to those files

3) This works better if it's stored on a server that has a more public API since you probably want a web interface and etc.


Currently downloading the referenced app (Contacts Journal CRM) to see if the hype is real.


So far the app is great, but I'll need to give it a real test run over the next week or so to see if I run into any data issues. Right off the bat I'd like to mention both something worth of appraisal yet slightly worrisome. The iCloud sync in Contacts Journal CRM gives the user an "Identity" number associated with their account. I don't have a lot of experience with iCloud but I've never seen this done before. Is this provided by Apple in the iCloud implementation? It's awesome you guys have some redundancy to protect against Apple failing with your data, but it's also kind of an indication that theres some likeliness to there being some issues with iCloud.


Thanks! The ID number is only referenced in this file: ~/Library/Mobile Documents/BJ97GLR9R3~com~cjournal~icloud/Configuration.plist

It's just a reference to check if your current device is participating correctly with the rest of the iCloud container, so that you can tell if someone deleted the iCloud container behind you while the app wasn't active, the app would be able to handle that situation. It's only exposed to the user in case something is wrong and the user can quickly check this value from that plist file. In practice, it hasn't proven too useful to expose this to users, but something to build on for a future update.


Perhaps the Google Drive Realtime API could be a substitute. https://developers.google.com/drive/realtime/reference/gapi....




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: