
IOS5: There's a reason it's called 'beta' software - jasongullickson
http://mbarclay.net/?p=1317
======
martingordon
You can only get so far with a blog post. Have you filed a radar yet? I just
did (rdar://9598285). Here's the text you can copy and paste (make sure you
mention it's a dupe of 9598285).

 __*

Title: Users leave bad reviews for app that don't work on beta OS releases

Summary:

The point of the prereleases is to give developers time to modify their apps
to address any compatibility issues (as well as add new features) before the
public release of the operating system. Despite Apple's best intentions at
keeping OS betas limited to developers, the fact of the matter remains that
average users can and do get their hands on prerelease software. This leads
many non-savvy users running applications not meant for the new release and
often leave negative reviews on the App Store because the app is incompatible
with the new OS.

For example, see this blog post: <http://mbarclay.net/?p=1317>

One potential solution to this is to disallow app reviews for users running
unreleased OSes. Another is to tighten up the developer program so that each
subsequent beta release allows more and more devices to receive installs. With
the current 100 devices per account right off the bat, there is widespread
abuse of access to prerelease software.

~~~
masklinn
> One potential solution to this is to disallow app reviews for users running
> unreleased OSes.

That would only work for on-device reviews, but those users can always leave
reviews through iTunes.

~~~
martingordon
I think it would be a larger engineering effort to totally block it out, but
blocking it on the device would at least preventing "heat of the moment"
reviews.

Luckily, most of the larger OS updates also required a new version of iTunes,
so they could block it for beta releases of iTunes also.

~~~
mahyarm
For all itunes accounts who have a beta ios install associated with them, they
cannot post reviews until a certain expiry date. Developers would not really
mind the trade off and it provides a small disincentive for non developers to
not install pre release versions.

------
mortenjorck
After years of countless web apps and services hopping on the bandwagon of the
perpetual beta, the potency of the term has become diluted. Users don't think
of betas as experimental builds of software for testing purposes; they think
of Gmail, which only shed its beta status as a formality two years ago with
150 million users.

Give them any path, even a tricky or expensive one, to obtain a beta of a sexy
new mobile operating system and they will, expecting it to work as well as the
other "betas" they've used online.

------
jasonlotito
Unfortunately, beta has come to mean "Private release to a select few
individuals" to many people.

As for the bug reports in the review area, I imagine this is for a couple
reasons.

First, it's the easy way to give feedback to the developer. I don't know of a
way to provide feedback within the App Store architecture other than through
the review process. Any other method has a chance of being painful. Being
forced to sigh up to some developer site bug tracking system, and then getting
sent email after email notifying you of every status update. Of course, I also
have to search around for _how_ to contact this particular developer with a
bug. There is no easy method that I know of to do this.

Secondly, it's a sure way of getting the attention of the developer. Reviews
directly impact the developer, and an email directly to the developer might
not get a response as quickly as you want.

I'm not saying either reasoning is correct, or right. I'm not judging the
reasoning. I just imagine that's the two biggest reasons for using reviews for
bug reports.

~~~
jasongullickson
_Being forced to sigh up to some developer site bug tracking system, and then
getting sent email after email notifying you of every status update. Of
course, I also have to search around for how to contact this particular
developer with a bug. There is no easy method that I know of to do this._

Nobody ever said beta testing was easy, and implying that it should be demeans
both the testing professional and the developer.

~~~
jasonlotito
I was referring to not using the review section as a bug report area,
something that happens, beta or not. Sorry for the confusion.

------
teilo
This makes me wonder how many people ponied up the $99/yr. just to have access
to the beta. Actual developers should know better. I wish Apple could figure
out some way to prevent this - but if they were to prohibit reviews from
people running a beta, they would also prevent legitimate reviews at the same
time.

I also think there's a place for non-developers to be running the beta -- if
they are responsible citizens. I'm a dev, but not much of one yet. I'm on the
beta, and I have discovered plenty of broken apps. When I discover issues, I
communicate them directly to the app's developer. As a result I have also
ended up beta-testing a number of apps for said developers. This is just as
legitimate a use of beta-testing as developers testing their own apps.

~~~
masklinn
> This makes me wonder how many people ponied up the $99/yr.

Probably not that many, but each developer gets 100 UDID to activate, and I've
seen many adding random UDIDs on their accounts for people who wanted to
install iOS5

> they would also prevent legitimate reviews at the same time.

There is _no_ legitimate public review from a beta iOS device. Legitimate
reviews from iOS-beta testing will be done directly to the developers, not via
the AppStore.

~~~
tibbon
Right, if I remember properly when I downloaded it clearly said that the
contents of iOS5 are confidential. That to me doesn't mean "blog it all over".

~~~
masklinn
Yes, developers are supposedly bound by an NDA on matters of beta iOS
versions.

However, I do not know if that NDA has clauses about adding random third-party
UDIDs (though developer accounts _have_ been banned for that in the past), and
in any case it does not bind those getting their UDID added who do not own
developer accounts.

------
vaporstun
I'm going to have to agree with him. It is a bit ridiculous that someone using
the iOS 5 beta has their reviews go in along with the rest of them.

Apple should make it so reviews are only shown for the OS of the user's phone,
so if I am on iOS 5 I can see the beta reviews, but most average joes still on
iOS 4 would only see those relevant to iOS 4.

This also solves an issue of users still on iOS 3 with an app that may no
longer be compatible for iOS 3 clogging up the reviews for they aren't
relevant for most current iOS users.

~~~
napierzaza
No that doesn't make sense. As a developer you don't want to start at 0
reviews every time there is a OS update. Reviews are pretty important, and if
it periodically looks like you don't have them it would be very bad.

Currently they roll over the reviews ever version of the App the developer
releases. That makes sense because bugs get fixed and features are introduced.

You should not see any difference in the different versions of iOS. Though it
might perform differently on different hardware.

I feel as if Apple hasn't really done very much to help the developer note
what platforms they are targeting in an easy way. If you don't want iPhone 3g
users to use the app, they shouldn't be able to download it that that device
in the first place etc.

~~~
masklinn
> You should not see any difference in the different versions of iOS.

Wrong. Any OS feature available only in version N and above may be used
conditionally while the application still works on versions N-1 and under.

Case in point: Apple's Game Center is only available from iOS 4.2 (which
excludes 1st generation iPod Touch and original iPhones), and is not available
on iPhone 3G.

What if a game developer is OK with a 4.0 baseline, or even a 3.1 baseline so
iPhone and Touch 1 users can still play, but he still wants users with
compatible devices to have access to Game Center achievements and voice chat?

He conditionally enables the feature.

> If you don't want iPhone 3g users to use the app, they shouldn't be able to
> download it that that device in the first place etc.

Set UIRequiredDeviceCapabilities correctly and that's done for you. For
instance, the 3G iPhone uses ARM v6 and only provides OpenGL ES 1.1. Require
armv7 and opengles-2 and you've ensured your application will not be
installable on 3G iPhones.

For an example of that, see Infinity Blade [0] which clearly states it is not
compatible with the 3G (more precisely, it specifies that it's for 3GS and 4
in the "required configuration" section) and which will not install on a 3G.

[0] [http://itunes.apple.com/fr/app/infinity-
blade/id387428400?mt...](http://itunes.apple.com/fr/app/infinity-
blade/id387428400?mt=8)

------
Simucal
One fix to this problem would be to prevent beta iOS versions from posting
feeback at all.

Once they are out of beta the feedback feature can be turned back on.

------
tibbon
There was a thread on Reddit that people were swapping access to the iOS beta,
filled with people complaining about bugs, speed, etc.

These users weren't developers, were essentially pirating the software, didn't
file bug reports and clearly just thought that it was something cool to get
early and didn't understand the implications of it being a beta.

~~~
masklinn
> There was a thread on Reddit that people were swapping access to the iOS
> beta

There have been numerous threads on that, starting as soon as iOS5 became
available.

> These users weren't developers, were essentially pirating the software,
> didn't file bug reports and clearly just thought that it was something cool
> to get early and didn't understand the implications of it being a beta.

And those who pointed out these issues were generally mercilessly down-voted.

~~~
tibbon
Hivemind at its finest.

------
CRASCH
This is kind of closing the barn door after the horses get out and I realize
there is a delay in getting new versions out.

Check the OS version against a list of tested versions.

If the version isn't in your list of tested versions refuse to run, or explain
that the software may not work properly and please do not post reviews.

------
gaius
The problem is to marketing twerps "beta" means "OMG cutting edge unobtainium
LOL" so they not only plaster it everywhere, they don't even know what it
really means.

------
bonaldi
I thought iOS and OSX had versioned frameworks etc exactly to avoid issues
like this. Yet every beta seems to bring nasty crashers from core
functionality. How come?

~~~
davidwhodge
The majority of problems we've seen come in the form of existing API calls
that have some slightly new/different behavior.

They seem to do a decent job with versioning major changes. But our App breaks
on iOS5 because of subtle changes.

Point being...there are far too many moving parts to expect them to get
everything right in a beta release.

------
napierzaza
I think this should not come to any surprise to iOS developers. People use the
reviews page as a bug report page. For apps I've worked on we got way more
negative reviews for particular bugs than we ever got emails reporting said
bugs.

There might be a warning already, but if there isn't Apple should make it very
clear that every iOS developer has provided technical support contacts and
other helpful links that they can use to report bugs. Bug reports don't belong
in the review unless the developer never responded to your contacts or never
fixed your bug after contacting them.

I also think that the developer should be able to contest reviews that solely
create FUD and aren't really reviews at all.

