

Ubuntu Unity usability testing results and analysis - keyist
https://lists.ubuntu.com/archives/ubuntu-devel/2011-April/032988.html

======
andyking
I work for a small non-profit radio station where we've got a couple of Ubuntu
boxes (running 10.10, or whatever the latest version with standard GNOME 2
is). I've only done it because we recently expanded our volunteer base and
didn't have XP licences to cover computers for every desk!

I've actually been really surprised at how well the volunteers have taken to
using the machines, and how some even seem to prefer them to the tried-and-
tested XP installation we've got on other computers in the building.

It helped that we were already using Chrome, Thunderbird and OpenOffice as our
standard applications throughout the station - so when I installed the
machines, I just stuck big, bold icons for each one on the desktop to make it
obvious what to do. Audacity was a new one (we're using Adobe Audition on
other machines) but people took to it reasonably well.

Comments I've received have been along the lines of _"There's a lot less shit
on the desktop than on those other [XP] computers," "It's easier to find what
you're doing, and it seems a bit faster too,"_ and so on. For some reason, our
XP machines accumulate junk on the desktop, whereas the Ubuntu boxes stay
relatively clean!

It works for a lot of our older, less experienced members who find it
difficult to find, say, the Thunderbird icon among a pile of old documents and
folders on the Windows desktop, or get confused when Start -> Email opens up
an empty install of Outlook Express for some reason. It often feels like I
spend more time helping people out when something's disappeared, they can't
print, they can't find something on a Windows box than actually doing my job,
so the Ubuntu machines have been a boost - they seem to just keep trooping on.

Of course, our users are simply booting the machine and opening up some
standard applications that also run on Windows, and not going in and changing
settings, or anything - I suspect that's where it would fall down in
usability.

However, I've tried out recent pre-releases of Unity and GNOME 3 and found
them pretty confusing, and I expect a lot of the users who've made positive
comments to me about Ubuntu at work would too. It seems like a step backwards
to me, with too many unclear mysterious icons, and bits of the UI whizzing on
and off the screen while I try to work - and I'm sticking with the LTS at
home, and not updating the work machines, either.

~~~
JonnieCache
I have had similar experiences putting it on my friends laptops.

If you come to me asking for a reformat job twice in 6 months, you get ubuntu.
They never need to ask again after that.

~~~
usedtolurk
...or perhaps they daren't ask again ;)

------
btmorex
Overall, pretty interesting. A lot of those are definitely usability bugs. I
wish someone would do a comparison between mac os x, windows, ubuntu with the
same tasks to see how they stacked up against each other.

However:

 _5/11 participants (P2, P3, P5, P9, P10, P11) crashed Unity during their hour
of testing. And towards the end of her test, P11 opened a zombie quicklist
that stayed on top of everything and didn't respond to clicks._

This ubuntu release is shaping up to be pulseaudio 2.0. I know everyone will
just say "use LTS", but somehow debian testing manages to be both more up to
date and more stable than normal ubuntu releases. I'm not sure why canonical
can't at least match that.

~~~
fingerprinter
I have to disagree pretty vehemently.

Are you running 11.04 beta2? I've been running it for months now and I LOVE
Unity. I installed on my wife's machine, and while she doesn't love it (she
really doesn't care one way or another about computers etc etc) she
appreciates it in many ways. Though, the best way I can describe my wife's
interaction with Unity is she gets less annoyed with it than she does/did with
our iMac. True story.

Unity quickly became part of my workflow and improved it greatly. Super-W,
Super-# are awesome, Dash is good (not great) and I like that something is
built-in to the system that allows me to do basic searching etc (I hope this
improves, but initial pass is really quite good). My workflow feels more fluid
and solid w/ Unity, and I feel more comfortable, and hence, more productive
than when I was in OSX just a short while ago.

And stability is something that is hard for me to gauge as I've been running
since January on my test machine and beta1 on my main machine without too much
problem. I get some quirks here and there, but nothing drastic. I'm running it
on three machines: a mac mini, a lenovo t61p and a lenovo t60p. The t60p is
the least responsive (naturally) and that is mostly b/c the ATI drivers are
not as good as Nvidia's. The mini and t61p are nvidia and after installing
those drivers everything works great.

And to compare, I used a Gnome3 live CD for several days trying to decide if I
wanted to install something with Gnome3 (most likely Ubuntu w/ Gnome3 if that
was the case, tbh...Ubuntu packaging makes it worth it alone) and while I
think Gnome3 is a good release, it can't really compare to Unity. I like where
Gnome3 is going, but I don't see it as something that directly competes with
Unity at the moment.

I think Unity has a ways to go, but if you compare Unity in 11.04 to, say, OSX
when it first came out...by god man, Unity is lightyears ahead in usability an
consistency. People tend to forget how rough OSX was for a couple of releases.
Apple has their act together now, but it took some time. And I LOVE something
like Unity existing on Linux and being FOSS...pushing boundaries...

~~~
rufugee
This ^

I've been running Unity for 24 hours now, and I can honestly say it's been the
best desktop experience I've had in all my years. Notice I said best "desktop
experience", and not "best Linux desktop experience". I have been running
Windows since 3.1, Linux since Red Hat 5.2, and OS X since, well, Leopard.
Unity is by far the best and most cohesive experience I've had so far. It tops
OS X by a wide margin...it has many of the same sort of niceties with window
management that actually makes sense (yes, I think OS X window management is
horrid). Add to this the fact that it bakes some of my favorite compiz
features (grid plugin anyone?) in and I couldn't be happier.

I know there will be some naysayers...many, in all likelihood, but in this
daily Linux desktop user's eyes, Unity is a _VERY_ big win for Ubuntu.

~~~
viraptor
If you haven't tried it yet, give original Gnome 3 / Gnome shell a go. You
might like it even more than Unity. I get a feeling that by jumping straight
to Unity, they will cause many people to be excited about it, not knowing
they've lost something at least as good.

~~~
Bo102010
I tried both for the first time last week.

Unity is not bad by any stretch, but it feels unfinished to me when I use it.
I kept hitting the wrong thing and breaking the flow of whatever task I was
doing. I'm sure with some more experience with it I'll quit doing that, but it
kept violating my expectations and stealing my attention from more important
things.

Gnome 3 was a big change, but felt much tighter and polished. I didn't seem to
have to think too much about how to do stuff.

Based on that experience, I think I'd like to use Gnome 3 on Ubuntu long-term.

------
quadhome
Some of these are humbling. They reinforce how disconnected I-- a programmer--
am from the average person.

> P1 recovered amazingly well after trying to save "Letter to Mr Smith
> 08/04/11", and getting the vile response "Error stating file
> '/home/ubuntu/Documents/Letter to Mr Smith 08/04': No such file or
> directory"

That's something I'd never consider, what with various directory separators
baked into my subconscious.

~~~
realugglan
Yes, I thought the same thing! Really shows the value of running tests with
people with other backgrounds.

Just for fun I tried saving a document in Microsoft Office 2007 here with a
filename containing slashes. I got a slightly better response: "Ogiltigt
filnamn" (Illegal filename). Not as confusing, but still no help at all on
what was wrong with it or how to correct it.

~~~
hesselink
I just tried the same on the Mac (in TextEdit). If you start with a slash, you
get a 'go to directory' dialog. If you enter a slash later in the filename, it
saves with a slash in the filename. The 'magic' is that it actually uses
colons, which are shown as slashes in the UI. If you try to enter a colon in
the filename, it replaces it with a dash. Renaming a file in finder to use a
colon, gives a message similar to your 'Illegal filename'.

~~~
danielsoneg
Wow - I didn't know about that / on the Mac. You, sir, have just saved me
three key presses on probably 75% of both my save And open windows...

------
dkarl
_Nobody understood Ubuntu One._ _4/11 people (P7, P9, P11, P12) thought the Me
menu icon might be a close button_

With silly corporate-ad-campaign-flavored names like "One" and "Me" it's no
wonder.

~~~
andybak
Hear hear! Almost every company tries to inflict crap like this on users at
some stage. It's a prime case of marketing trumping all other factors.

~~~
jberryman
Imagine my delight trying to do some testing on VMware products for the first
time yesterday, and trying to navigate this maze of meaningless names:

<http://en.wikipedia.org/wiki/VMware#Products>

------
Legion
> _P7 and P11 thought that "LibreOffice Calc" was a calculator_

"Calc" is just not a smart name for a spreadsheet application.

> _and P7 and P9 thought Ubuntu Software Center was the Recycle Bin._

I understand that, because that _is_ what the icon looks like.

~~~
Bo102010
Is there a good name for a spreadsheet application besides "Spreadsheet?"

"VisiCalc" was the first one on a PC, "Lotus 1-2-3" became popular later,
"Excel" is now the dominant one...

~~~
heyitsnick
In all seriousness, what's the problem calling it "LibreOffice Spreadsheets"?
Along with "LibreOffice Presentations", "LibreOffce Documents", "LibreOffice
Databases"; are we so pressed for space we can't just say exactly what it is?
The "name" is libreoffice, I don't see why the individual components have to
be unique or catchy.

~~~
Egregore
May be they didn't want it to be confused with spreadshit?

------
vessenes
This was my favorite -- DESIGNERS, THIS IS WHAT AD BLINDNESS LOOKS LIKE IN AN
APP: Don't design stuff like this in, people don't notice it anymore.

" Only 2/6 noticed an XChat Gnome notification, despite (1) a notification
bubble appearing, (2) the Ubuntu button going blue, (3) the messaging menu
envelope going blue, and (4) an emblem appearing on XChat Gnome's launcher. "

~~~
travisp
As a non-typical user with two large monitors, I never notice gnome
notifications. They're small and often on the monitor I'm not looking at.

------
divtxt
How far we've come!

I remember about 10 years ago, with the rise of XP and the advent of OS X,
expecting Linux desktop to fall ever further behind on a usable & full-
featured desktop.

Here we are discussing the usability tests of big UI innovations.

I'd like to thank everyone who got us here! ( _cough_ including Redmond
_cough_ )

------
akavlie
I run with a laptop (display on, but rarely used) connected to an external
monitor. Linux assumes the laptop is the default, and changing the external to
primary is next to impossible.

Unity puts everything important on the laptop display in this configuration.
That alone is a deal breaker for me.

Note that Ubuntu Classic works perfectly in this setup. My panels go to the
external monitor when it's plugged in, and move back to the laptop immediately
when it's unplugged.

Unity (as it's currently designed) would pose some serious issues in a dual-
display setup even if I could move it to the external monitor. As the launcher
is stuck on the left, I'd be constantly overshooting it (laptop display is to
the left).

~~~
rufugee
I have it running on dual displays with no problems whatsoever. That said, I
have two 22-inch monitors and not a laptop screen. Unity is early days
now...I'm sure your issue will be fixed soon enough.

~~~
ericb
Ubuntu on my dell latitude, running 10.10, doesn't handle unplugging/switching
primary monitors. So I am far less confident that the parent post's issue will
be fixed anytime soon...

------
rubergly
I wonder: 11 people seems like a rather small sample size; does Canonical
typically do any testing with larger samples?

I'm surprised that only 5/10 people tried to open another Firefox window by
clicking the launcher icon. Personally, I think clicking the launcher icon not
opening a new window is silly, but that's because I'm used to Windows 7 and
not OSX.

~~~
kemayo
Jakob Nielsen does suggest [1] that only 5 users will get you about 80% of the
problems in your software, and 15 will get you damn-near-everything. So 11 is
pretty good.

[1] <http://www.useit.com/alertbox/20000319.html>

~~~
jerf
Also, if you do have the resources to do, say, 100 user tests, you actually
actively _don't_ want to do them all at the same time or it will be a waste
because they will all say the same thing.

If anything, this was _over_ tested. There's a lot of redundancy in those
results, just as Nielsen would predict. This is not a scientific experiment to
determine the effect of a drug, and you do not always need or even necessarily
_want_ a 95% confidence interval, as the costs would exceed the benefits here.

~~~
Terretta
_There's a lot of redundancy in those results_

Helps so devs can't reply, "Well, that guy was just stupid".

~~~
robrenaud
I am a dev (though not on frontend/UI stuff). But that's exactly the point.
Many users are for the most part pretty stupid, and that is reality. Unusable
software is unusable.

~~~
cgislason
Even if your users are not stupid, designing as if they are is usually
beneficial.

------
hasenj
Here's what I think should happen before final release:

\- Make the launcher always on by default

\- Remove the recycle bin icon (I really don't get the point of it)

\- Remove the applications icon (redundant with the Ubuntu button). Perhaps
add a "System" lense view instead (system settings and all that).

\- Add an "expose" icon (super-w). And perhaps hide the workspaces icon by
default.

~~~
pbhjpbhj
>" _\- Remove the recycle bin icon (I really don't get the point of it)_ "

Is it so you can find docs that you "deleted" by accident?

~~~
hasenj
I mean the point of it having a special place in the launcher.

------
ambiguity
It would have been nice to have a test group that used the classic Gnome 2
desktop. This would give the Unity scores a bit more context.

~~~
alagu
I guess that really wasn't the point. The intention was to find out the
usability bugs in Unity.

~~~
hristov
But that should be the point. I am still not convinced why Unity is necessary.

------
calpaterson
It's a shame they're doing this at so late a stage. I understand that 11.04 is
now in feature freeze and Unity will be released as is

~~~
SpookyAction
I agree, you'd think they'd conduct usability tests BEFORE making the choice
to switch to Unity. It seems that Ubuntu makes UI changes for the sake of
being different, not better. Like moving all the window controls to the left
side of the title bar in the current release of Ubuntu. Why?

------
acabal
If almost 50% of the users managed to crash the GUI with routine tasks _in the
span of a single hour_ , that's a sure sign that it isn't ready for prime-time
yet, regardless of what anybody's opinions on features or functionality are.

I personally think this is almost definitely going to end up as another
Pulseaudo-style debacle that'll jade even more Ubuntu users. This kind of
stuff is exactly why I never recommend Linux to friends, even though I
personally use it on my day-to-day machine: because it's just not (crashwise)
stable enough for Grandma, and Shuttleworth has a very bad habit of making his
end-users his beta-testers. Grandma isn't going to loyally log in to
Launchpad, report a bug and reboot; she's going to complain to me, and then
I'm going to reinstall Windows 7, which for all its faults at least doesn't
crash once every two hours.

~~~
fingerprinter
I know people hate this answer, but this is how I look at that: LTS.

Seriously, I install LTS for several things: servers and non-techies. Why? b/c
I view non-LTS releases as awesome for me, you, people on this site but not
for my grandma.

What I've seen Ubuntu do is they have given people who want to be on the
bleeding edge a way to be on the bleeding edge every 6 months, and that is
freakin' awesome. But if you don't want to be there, and frankly, most people
don't, they give you an LTS and 10.04 is a pretty freakin' good OS for people
who check their webmail and need a writer/spreadsheet with minimal fuss.

Using this model, it is the best of both worlds.

~~~
acabal
Maybe true, but as a web developer I can't in good conscience install an OS
for a user that won't upgrade to FF4 for more than 6 months. That's the
PPA/milestone release system's blessing and curse: easy to install software,
hard to get the latest version without terminal witchcraft or upgrading the
entire distro and eating whatever other changes (Unity) that upgrade brings.

~~~
fingerprinter
Just had another look over the policy:

<https://help.ubuntu.com/community/FirefoxNewVersion>

Seems perfectly reasonable, and more importantly, prudent given the LTS
nature.

To summarize:

* FF4 is available in a PPA right now - ppa:mozillateam/firefox-stable * 10.04 will continue use FF3.6 in main as long as Mozilla supports it (probably about 6 more months) * Once mozilla stops support for 3.6, 10.04 will switch to the latest stable (if that is FF5, that is what gets installed).

Seems very reasonable. Imagine if you are a huge corporation and you
standardized on Ubuntu 10.04. Would you want every package to get updated to
the latest and greatest each time it comes out or would you want security
updates to those packages? If you want latest and greatest, jump on the 6
month cycle. If you want to get security updates for a known system, stick to
the LTS. Even so, for those that WANT to get the latest of select packages,
there are ways.

So, if you feel that having the latest and greatest for your users is
important, stick to the non-LTS cycle and pick it and go. You can also install
10.10.

Have a look at the EOL dates...you can use 10.10 with full support until April
2012, getting you through 2 more Ubuntu releases and to the next LTS.

<https://wiki.ubuntu.com/Releases>

------
steevdave
I work with Ubuntu for work ( we make ARM devices ) so every so often I have
to run Natty to test if it is usable yet. If you have a "decent" _OpenGL_
graphics card, then depending on when you install it may or may not work. I
currently have an issue with working with the dock or launcher or whatever
name they are calling it. Right clicks are being passed through to the desktop
so I can't remove or add anything to it.

If you don't have an OpenGL based video card ( no ARM machine does, they all
use a subset of OpenGL called OpenGL ES ) then upon logging in, you get a long
dialog explaining that you need to logout and choose classic desktop. And it
has an exit button. When you click said exit button, after a bit it loads the
classic desktop, however the xsettings manager doesn't get started (or it
might, just depends on if it feels like running it seems) and gnome becomes
very ugly.

There is no mention of Unity-2D which is a version that is QT based. And it
suffers from the same issues, xsettings may or may not launch and then you
have a ton of dialogs about apps crashing in the background ( this is with
either Gnome "classic" or with Unity-2D ).

Ubuntu made great strides in making the Linux desktop accessible for everyone,
but this latest release shows just how much further they need to go.

Keep in mind that at the time the decision to write and use Unity was made,
the Gnome 3 desktop was in an absolute mess. A lot of work has happened since
then and it would be even better if Mark Shuttleworth could swallow his pride
and just scrap the Unity project and work with upstream like before.

I mention the ARM bit at the beginning of my statement if only out of
frustration due to the fact that our company provided Ubuntu developers with
over 50 machines that are in either a desktop configuration or netbook yet it
still doesn't run anywhere close to where it should on them.

~~~
aphexairlines
How well do the ARM desktops run if you log in with a basic gnome2 session, no
gnome-panel, plus avant-window-navigator?

------
pbhjpbhj
What this story is missing is a couple of screenshots from the particular
implementation used.

Anyone?

~~~
krakensden
If you google for "Ubuntu Unity screenshots" you should be fine. You can also
download the beta and run it in a VM if you'd like to see it in person.

~~~
pbhjpbhj
There are specific issues mentioned in the post that I wouldn't expect to be
present in just any version.

<http://chemicaloliver.net/linux/ubuntu-unity-on-eeepc/> and
[http://www.greenhughes.com/content/samsung-
nb30-touchscreen-...](http://www.greenhughes.com/content/samsung-
nb30-touchscreen-ubuntu039s-unity-alternative-tablets) look quite different.

------
PetrolMan
I installed 11.04 and when I was first setting everything up (because the
video drivers weren't installed yet) I played around with Gnome and actually
ended up with a setup that looked a lot like Unity. The difference was I had a
lot more control over the panels which I really liked.

Then, once the drivers were in place, I switched to Unity and was initially
really wowed by the way it looked. I really like the idea behind the side menu
but right now it is a bit finicky. Sometimes it will partially open when I
move my mouse to the edge and other times it works just fine. I absolutely
hate the launcher menu (might not be the right term - app drawer or whatever
else might be more fitting) simply because it makes it a chore to find
applications.

I've also found Unity relatively unstable. I've had it simply lock up and I
haven't found a way to get around it other than resetting my computer.

Even stuff like the way moving a window to the edge of the screen and it
taking up a portion works seems inconsistent. I can sometimes get 1/3,
sometimes 1/2 and sometimes 2/3 but I honestly haven't spent enough time
trying to figure out why it works in different ways. Nor do I feel I really
should have to... UI should be relatively intuitive.

Anyway, I think Unity is promising but it is really rough right now... really
rough.

------
mitko
On the other hand this testing doesn't include _any_ other ubuntu user. Of
course you'll have some learnability gap. I see also a lot of windows people
having trouble to figure out in their first few hours in OS X. While Unity
definitely needs some polish, it already makes it more _efficient_ to manage
my desktop experience on my netbook running Ubuntu. The guys at Cannonical are
trying to do major innovation and provide some consistency with the old UI.

------
steevdave
Where is P6? Was there one and their comments were so bad that they didn't add
it to the list? Numbers missing in a list like that really bug me. It also
causes me to wonder why it never comes up. I haven't read the thread since a
lot earlier and being as I know a few people around the Ubuntu camp I asked
them and no one seemed to have any idea why P6 was not on the list.

------
tcarnell
Great post! I started using Ubuntu Unity on my netbook, however, I had first
installed Ubuntu Desktop and added Unity later so I can switch between them at
loging time.

I think that Unity is great! Really encapsulates the spirit of a netbook - a
small, versatile and fun communications device. I have an asus 1005pe - which
I can also highly recommend.

------
nrbafna
I am really surprised one thing didn't come up. It's about the super+(?)
shortcut.

If you press them very quickly, say 'super+D', then it will take to desktop
and open dash as well. To be safe, you have press 'super', wait till numbers
appear on the icons in launcher and then press 'D'

~~~
nl
_I am really surprised one thing didn't come up. It's about the super+(?)
shortcut._

Really?? You are surprised that the test group didn't somehow magically
discover an edge case in a feature they didn't know existed and has no
discovery mechanism?

~~~
nrbafna
Well, they did use the shortcut.

It happens to me 90% of the time I use super+d.

------
dhruvbird
wow! this is how usability tests should really be done! :-)

------
omouse
Needs more participants who aren't students or teachers heh. Interesting
results though.

