"That is a problem if you’re a dishonest designer. After all, how do you tell your client that you’ve just reinvented the wheel? You can’t just use boring old buttons in your shiny new product."
The core problem is that people have trouble differentiating between the desire to improve something and the desire to change something.
Also the desire to copy vs innovate. How many "1 button + proprietary dialect of morse code" monstrosities happened because someone heard Johnny Ive brag about reducing the number of buttons on the apple remote?
i used an apple remote recently in an airbnb, i don't know what generation, but it was an atrocious experience. Why would buttons also be a touch interface that perform the same function as the underlying buttons? It was very confusing.
What about flipping it on its head? The problem with being a designer is you want people to notice your buttons. You can't just let them disappear into the background.
20 years ago, the complaints were that everything was dumbed down, and the "power users" were being hung out to dry.
Well, now everyone is a power user. The average iOS user has at least 5 years experience with the UI. People don't need as much hand-holding.
Ahem. I'm old enough to have a hands-on experience with DOS and Win3.1.
When someone hands me their iPhone (eg to show something some page in Safari or asking for help with something) I struggle to navigate anywhere. I have almost none iOS experience, because I never had iPhone. And new users (like kids born after iPhone 1, lol) have zero experience with iOS too.
I guess that dependends on your definition of a power user. Some might see someone spending multiple hours a day with a given software sufficient to be called a "power user".
If I had to make up a definition right now, I'd go for something like: Someone who invests a small(er) amount of time to e.g. configure the software to their needs or learns some keyboard shortcuts knowing it will save them a large(r) amount of time in the long run, by tiny savings spread over weeks (or decades). It's more about the mindset. With that definition it becomes clear why vi is often considered "power user only".
With the frequency of product redesigns the commercial software sees, I can understand why many people hesitante to invest upfront.
Yes! Please, give me big fat buttons. The thing I hate the most in the UI nowadays is to be randomly touching/clicking around to find what is touchable/clickable.
Yes, skeuomorphism went too far but the pendulum swigged way too far in the opposite direction. Metro, flat, ultra minimalistic design makes user interactions harder when they remove useful visual clues from interfaces.
I think there’s places for extreme skeuomorphism even if it’s not suitable everywhere. Nothing wrong with letting the designer have fun with absurdly detailed buttons for a game UI for example.
> The thing I hate the most in the UI nowadays is to be randomly touching/clicking around to find what is touchable/clickable.
Yeah, it reminds me of the old graphic text adventure games (e.g. Zak McKracken) where you had to hover the mouse over the entire screen to find the thing you are supposed to click on. If you're luck, the cursor might change shape. I generally prefer UIs to not be puzzles.
Yeah, if you think of terminal programs like old text adventure games that you have to keep typing stuff until something works, they're literally trying to make GUIs as unintuitive as terminals now.
Just swipe, shake, tap the corners, try scrolling even if there's scroll bar, until it does something!
To open settings, you're supposed to rotate the phone 180 degrees around the vertical axis, and then 180 degrees in the opposite direction. It's so obvious.
Did you read the first footnote I wrote in the piece? It’s actually about this exact thing you are talking about:
In many graphical interfaces that expect a mouse as input, the programmers will make the clickable area of the upper row buttons the same as the lower row ones, although this will be invisible at first. You can hover with your mouse over the buttons and then, magically, a button shape will appear. It’s not going to be a nice 3D-button shape however but just an outline. This serves no functional purpose. For instead of showing the user where all the buttons are from the outset, the whole affair becomes sort of a hide and seek game, where users have to guess which item they might be able to click.
What I don't get is the complaint about clickable text without outlines to make them look like buttons on a blog that has tons of hyperlinks for text not embedded into paragraphs. If an outline is required to understand those things can be clicked, why do they drop them on their own blog? Those are just as much buttons as the "Agree" example.
And then the top of their blog has an image that is clickable to go to the home. No information that the image is clickable, no information about where it would go.
The links at the bottom of their blog aren't inline in any other text. They're "clickable UI elements that stand on their own", "in a group with other related [links]". One could easily argue they're more button like, especially the email and call links at the very bottom of the page.
The author is complaining about nobody would understand to click on Accept in their example without having an outline around it and yet they expect people to understand to click on "eMail" and have it open their default email client.
Fair enough – however, links are still not buttons. Links should include an href and add a new entry to your browser history. Buttons should cause something to happen on the current page without changing your browser history.
If a link should add a new entry to the browser history, then their mailto and tel links at the bottom should really be buttons then. And yet we don't see people complaining about that.
I genuinely think that Windows 2000 was the perfect aesthetic.
It had just the right amount of skeuomorphism, without going overboard; probably because of hardware limitations.
What I think happened is that as computers and phones got more and more powerful, developers and designers wanted to use the UI to show off a bit. Culuminating in mad animations like a shredder destroying your old boarding pass in Apple Wallet.
Then there was a complete rejection of skeuomorphic design with iOS7, and it's just got more and more extreme since then.
Win2k is pretty good, fixing the one big problem with Win9x, which was its dreary darkish gray base color.
Personally I’m a bit more fond of the Platinum theme from Mac OS 8 up through 9.x, though. Similar color scheme and principles, but it uses rounded corners and soft shading in a few places Win9x/2k doesn’t which makes it feel a bit more warm and accommodating without negatively impacting usability.
It's a matter of taste and back then, you could change it to whatever color you liked. For me, it that darkish gray was too light and I switched it to an even darker gray.
I didn't delete old passbook entries enough in Passbook (now called Wallet) that I ever minded the animation, I kinda miss stuff like that!
Its nice to add whimsy to traditionally under-used features of a platform IMO.
Though I suppose if I was deleting passes all the time I might find the delay a little frustrating, but that could be solved by allowing multi select and delete right?
I fully agree. Even in the XP days and after 2k was EOL-ed I would regularly opt for 2000. XP started a trend of "shiny" UI, for lack of a better description.
I remember in elementary school having a class once where we went to the computer lab and learned about the Internet. I distinctly remember the teacher saying that links were always blue and underlined, and we even had a demo site with a grid of links and had to click the right one (some were different colors, some weren't underlined, etc). The funny thing is, I remember thinking at the time that clearly they didn't HAVE to be blue & underlined because otherwise how would the wrong links work on the page!
There was a time when I agreed with this idea 100%, and I do think there are still many antipatterns of interface and interaction design that come up quite commonly (tight spacing, undersized click targets, etc)… But I’ve come to think of these posts as harmful oversimplification.
The example of the touch feedback cuts both ways: those buttons exist in context, and one can certainly provide touch feedback without a border/shadows. Text-only buttons also exist in context and the language can do a lot to explain what happens.
A counter example is a physical button on manufacturing equipment. It’s huge, tactile, and labeled. I know I could press it, but that doesn’t necessarily help me understand what happens when I click it.
Everything on a screen /could/ be clicked or tapped, it’s inherent to the medium, making something “look clickable” doesn’t necessarily help clarify if/when I should click it, or what the result will be
> A counter example is a physical button on manufacturing equipment. It’s huge, tactile, and labeled. I know I could press it, but that doesn’t necessarily help me understand what happens when I click it.
On the other hand, you can be very sure you won't press it unintentionally if you stay away from it, and the difficulty in pressing it provides a hint of how big of a deal it is to press it. Membrane buttons, on the other hand, can very well implement flat or hidden buttons that one might press by accident.
in the accessibility settings of iOS one can enable “button shapes” which renders hyperlink-style underlines on buttons masquerading as text. very handy for me. also appreciate differentiate without color, on/off labels, and high contrast: it is much faster for my ADHD brain to navigate iOS with these affordances, relevant details are more salient and my brain has more chance to build a visual hierarchy.
There are some great options hidden there! There's a zoom (I use 3 finger double tap to trigger) and you can have it read the text on a page for you.
It's a robotic voice, but it's pretty great if you want someone to narrate eg daring fireball length posts to you (and you can speed it up to 1.5x+ speed)
my favorite is under audio you can play some basic background noises like white noise or water, macOS supports this too, surprisingly effective and builtin, no shady app store needed :)
For HTML, the main alternative to a button is a link, which does have some affordances (often underlined, changes color on hover.)
We are sometimes told not to use a button when a link will do [1]. Supposedly, links are better for accessibility or for user expectations. You can style links to look like buttons, but I have trouble making them exactly the same if they’re next to each other (for example, the same height) and it seems like doing it the hard way.
You may be misunderstanding the concern expressed on that page. A button communicates action when activated; a link communicates navigation (not necessarily "load new page", perhaps just "show new information on current screen"). Using a button for navigation or a link styled like a button for action has potential to thwart these expectations in various ways.
I'll admit, sometimes this makes some sense; you may be building an interaction that doesn't map cleanly onto either "trigger action" or "show information", and you wish to pick and choose the semantics that best fit your needs.
...But then there's the situation you allude to, where you have a button (action) and a link (navigation) and are just trying to make them look the same for consistency. Even if you pull it off visually, the chance that an end-user's client software will misinterpret what you're doing in some way and cause confusion is high.
The distinction between navigation and action seems rather blurry. Two pages often have often different views of the same entity. You’re not really “going somewhere,” particularly if it’s not a full page reload.
But I guess the main point is not to use buttons in your nav bar.
1) Can you return things to exactly the state they were before simply by hitting the browser's "back" button once? Then use a link. Otherwise use a button.
2) Will this navigation have side effects (eg, change values in the DB)? If so, then use a button. Otherwise, use a link.
Take it fairly seriously. Using the wrong elements and markup messes with screen readers, which makes the web harder to use for people with visual impairments and makes your websites vulnerable to ADA lawsuits. If you need CSS help, there's a lot of free resources for that. Some pitfalls you might run into when styling links and buttons to look alike are font faces, display properties, widths, borders/outlines, and paddings, so check those as your first culprits. You can use aria-roles in a pinch, but really you should use the right tool for the right job.
I like a lot of what they're saying, but I actually find all of their redesigns to look worse than the original. I don't prefer the typeface they've selected, and their boxy buttons look two decades old.
I agree that the terms agree/disagree aren't well differentiated, I don't agree with their solution
I find their photos redesign to be much worse than the original. Buttons are an affordance that you use to instruct users on new or rare behavior. If a user regularly does something or uses something you can strip it down like the back button.
> I like a lot of what they're saying, but I actually find all of their redesigns to look worse than the original. I don't prefer the typeface they've selected, and their boxy buttons look two decades old.
The actual design was not the point of this piece, if anything, it’s an invitation to re-explore the design of the button. By being hung up on the implementation details, you have missed the forest for the trees.
Back in the early 2010s when I was doing freelance web development, I learned very quickly to stop gathering feedback from the client on specific steps because, for example, if I showed them a greyscale wireframe for page to solicit feedback on the layout, they would say, “the text organization looks good, but I wanted it to be these different colors, also I don’t like the Latin text in there, where is the content that we agreed would be in there? Also the logo is missing, also none of the links in the footer work, I don’t like the font, etc etc.” I thought at this point (and especially on this forum) we were past this sort of ill-informed feedback.
There's the classic "when someone tells you there's a problem here, they're usually right. When they propose a fix, it's usually wrong"
To me, this post falls squarely in that bucket. It correctly identifies issues and does not find the correct solution. Unfortunately, that undermines my confidence in the author, and that makes me suspicious of their comments on the photos page or regarding the typeface.
The author says "Apple used a bad font" but I missed it if they explained why, and their replacement didn't feel better to me. That's not convincing! I don't think it's unfair for me to say "yes, you've identified a problem but not the solution" and I don't think that's a meaningless comment.
I like buttons, and since I'm not a designer, I often use obvious affordances or make simple product decisions (this is to my detriment: too many modals or toasts). This includes buttons. But without hearing from Apple's designers (or from any designer defending their decisions), I leave this post feeling unconvinced. Their argument wasn't good enough for me to think it would beat an argument in favor of the designs Apple employees made.
I was actually thinking of writing a piece about the legibility of typefaces in the context of UI. It would have gone far beyond for this post though. In the meantime, go to the website of Monotype and write the word “Iliad”, once in Neue Haas Unica (similar to the Apple font) and once in FF Unit. Notice how you can tell apart the capital “I” from the lowercase “l” and “i” much better. It’s things like this that make typefaces like FF Unit much better for UIs and also signage (in hospitals for example).
I would love to read your (future) piece about legibility of typefaces!
You're correctly describing facts (eg the font apple uses doesn't let me tell the difference between llama and Ilama), but that's mostly not needed when reading their blob of legalese text.
In the extremely subjective world, I find the monotype hard to read in that narrow format because it implies a verticality - like looking at an art deco building. Somehow my eyes are able to scan the font they've selected more easily. This is, of course, possibly because apple has made me read a lot of their typefaces.
The context matters a lot, and so you want the stop sign (or possibly even the simple marketing sign) to be unambiguous. You want good kerning that won't make words bleed together or break at incorrect points.
I tend to like the fonts that are trendy on the web these days though - I find them fairly easy to read (and I read them a lot!)
One thing that surprised me is that comic sans can be easier for someone who is dyslexic to read because b and d are not mirrors. So I would be sympathetic to a lot of typeface choices if I understand why.
Yeah, the apple ghost buttons are a dialog with the user. They say "you have seen this situation 1000s of times, and are tired and bored". They slip into the background as your finger autopilots to the same spot it always goes. When the action becomes rote, the button becomes tacky.
On, say, a kitchen stove, I find the same minimal design frustrating, because I don't use it enough to be familiar, and I want 100% certainty.
You're entitled to your opinion of course, but not everybody has the same sense of aesthetics as you do, and the whole point of the article is to prioritize usability over any one person's opinion about what looks pretty.
I actually have the opposite opinion. I think the Apple version looks 10 years old (which is about the age it actually is. IOS 7 came out around 2013 iirc.) The rounded rectangles looks 20 years old. But is more usable so is superior in my opinion. I agree that the font the author chose is less aesthetic but it is much easier for someone with dyslexia or adhd to read, making it more functional as well.
Yes! My counterargument will reference the rust language ergonomics initiative post (which imo is seminal)
In particular, they say
> One final point. "Implicitness" is often relative to where the language is today, something that seems radical at first—like type inference!—but then quickly disappears into the background, to the point that it no longer feels implicit at all (see Stroustrup's rule).
- do not provide optical lines to be aligned with other screen elements (look more chaotic on the screen)
- are visually more difficult to find on the screen (enhances stress, makes work mor tiresome)
- are more difficult to aim at with the mouse (requires more precise movement, error prone, leads to frustration)
- are more difficult to be identified a active element (violating rule of discoverability)
In many cases there is no good color choice either. Quite often you have dark grey on light grey or other combination low on contrast.
Things described in the article were part of fundamental knowledge in system ergonomics 20xrs ago. After "UI" or "UX" designers took over usability went downhill on many screen. Today the most elementary rule of good design is commonly ignored ("Form follows function").
Some kind of more button like UI element will inevitably come back. Wait for 5 or 10 years and new designers will look at flat UIs as something that their parents and old people used. They'll want something different as a statement of being new and hot, them and the products they'll be working on. It happens in fashion with colors, shapes and even with full clothes. It's going to happen with UIs too.
> On the left however, the button for going to your albums is just the word “Albums” and a little arrow shape. Why those two different concepts on the same screen?
Clicking on "< Albums" could change the page completely. You navigate to another page. The buttons "Select" and "..." stay on the same page. I think it makes sense of using two distinct styles.
If I had been a real stickler when writing my piece, I could have pointed to the “Select” button missing an ellipsis (…), making it “Select…”. Since the first Macintosh, when choosing a command from a drop down menu that would present you with a modal dialog, the command in the menu would have the ellipsis. This would tell you that you would get asked something after selecting the command. So “Save” but “Save as…”.
Since “Select” presents you with a mode to select things before you do anything with them, it would technically need the ellipsis. However, I suspect it isn’t there (amongst other things) because the round button next to it is already an ellipsis. That would have to be changes as well.
So you see, I could probably write a whole piece just about that… :-)
Did skeuomorphism go too far in the wrong direction? I think it's gone too far in the wrong direction. I don't mind my notes app resembling an actual notebook that I would use. Compare Notes.app to say, OneNote. Having floating text and iconography everywhere is disorienting. Bring back buttons (and in cars, too)!
I hate flat design so much and I can't believe even Gnome Shell switched to it only last year. With the added "benefit" of making it extra hard to change the theme.
I wonder who thought that removing button outlines from headerbar will somehow improve things? Now it looks like a strange title with icons mixed with text. Still hoping that GNOME accessibility initiative might result in some sane improvements to GNOME UI.
Skeuomorphism is a term most often used in graphical user interface design to describe interface objects that mimic their real-world counterparts in how they appear and/or how the user can interact with them.
Unfortunately, one of the many pieces, where the term affordance is confused with what should be the term signifier. When people talk about the former, they usually mean the latter.
I should preface this by saying that I am a fan of buttons in general. Mostly for aesthetic reasons. For most users and most cases, the usability impact is probably insignificant though. That said, this article doesn’t do a good job arguing for buttons.
> In the first row we see a series of icons that are supposed to be buttons. The only way you could potentially recognise them as such, however, would be if they were implemented in a user interface. So it is in fact only the context that lets you recognise them as buttons, not their appearance by themselves.
So what is the problem? If you can recognize they are buttons in context, that sounds like a successful design.
> In touch interfaces however, that is often not the case. There, the actual outline of the graphic – let’s say the minus – is often the only thing that can be touched.
If you implement your touchscreen buttons like this, I’m not sure what to say. It’s not a problem with the style of button, but a bug in your implementation. I can’t say I’ve ever come across it in the wild in anything that is remotely competently made, as it would basically be unusable.
> Apple is using lazy button shapes for “Select” and the ellipsis character (…) on the right. On the left however, the button for going to your albums is just the word “Albums” and a little arrow shape. Why those two different concepts on the same screen? In the redesign, everything that works like a button also looks like one.
There’s a perfectly good reason why they look different. The back arrow is a navigation pattern that is used across the entire OS. It should be consistent with the same pattern on other screens and in other apps. The design should be deferential because it is so ubiquitous. On the other hand, the two buttons in the right are specific to this screen. It would be a mistake to make them so similar when their scope and purpose is fundamentally different.
> So what is the problem? If you can recognize they are buttons in context, that sounds like a successful design.
Then you didn’t get my point about cognitive load or I wasn’t explicit enough. Sure you can recognize the icon-only buttons in context (mostly) but if they work worse in a sterile (or call it academic) example when compared to buttons with shapes, how are they better when in an UI? The UI will help but the cognitive load will still be higher than if they looked like buttons by themselves.
> If you implement your touchscreen buttons like this, I’m not sure what to say. It’s not a problem with the style of button, but a bug in your implementation. I can’t say I’ve ever come across it in the wild in anything that is remotely competently made, as it would basically be unusable.
I had this in several apps across the years. If it happens often enough, you develop an itch in the back of your head that makes you ever so slightly distrust buttons in that shape. Something that can be completely remedied by the solution I proposed.
> There’s a perfectly good reason why they look different. The back arrow is a navigation pattern that is used across the entire OS.
It’s a bad pattern, made by people who confuse website content with native GUIs. I can’t go as far as to explain all the reasoning behind this but just this much: The back arrow I proposed is something that – and believe me or not, I realised after I wrote the piece – had already existed in iOS before the flat nonsense.
Again, I love a good button, and I’m not arguing against them. But I also recognize that using a simple icon or word as a touch target has its benefits in many contexts. There are reasons beyond numb-headed fashion that lead to their use. The context thing is crucial - if context is already communicating something, that is usually way more powerful. After all, it’s not enough to understand that something is a button, you also have to understand what it does (or more specifically, people who have a use for it need to understand what it does, and people who don’t have a use for it need to recognize it is irrelevant to their situation). This can only be done with context.
I think the problem with your article is that you are not taking the style you’re arguing against seriously. If you think it’s always wrong and can’t see any good reason why it’s used, you will not be able to make a good argument for why to use more traditional buttons instead.
> But I also recognize that using a simple icon or word as a touch target has its benefits in many contexts.
Yes. Sometimes. I said so in the afterword of the piece. Maybe you missed it: “Does every virtual button – every button in a graphical user interface – have to look like a pressable, physical 3D button? Of course not. They also don’t need to look exactly like my redesigns either. On a case-to-case basis it might even be better to do something else entirely.
The whole idea is to reduce cognitive load. And since the brain works by recognising patterns and dividing the environment up into areas, this reduction is best done by making elements with different functions appear markedly distinct from one another. It is, in other words, a fallacy to believe, that the brain has an easier time if everything looks “simplified” in the way which happens with the flat design doctrine. The opposite is the case.”
> There are reasons beyond numb-headed fashion that lead to their use.
Certainly not where buttons that used to have button shapes were replaced with only icons or text.
> After all, it’s not enough to understand that something is a button, you also have to understand what it does (or more specifically, people who have a use for it need to understand what it does, and people who don’t have a use for it need to recognize it is irrelevant to their situation). This can only be done with context.
That is correct but it also wasn’t the scope of the article at all. It was simply about the importance of understanding which element on a bitmapped screen is a button in the first place.
How to make a button – no matter what it looks like – communicate what its function is would be more than enough scope for another article.
> I think the problem with your article is that you are not taking the style you’re arguing against seriously.
In what sense am I supposed to take seriously a UI decision that was detrimental to usability?
> If you think it’s always wrong and can’t see any good reason why it’s used, you will not be able to make a good argument for why to use more traditional buttons instead.
That is not a logical statement. I can, for example, say that National Socialism is always wrong and still make a good case against it.
All that being said, I’m actually a great fan of reduced aesthetics. The torn-off paper in the various notes app and leather stitching were horrible. But as I said in the piece, They are still to be preferred if the only other option is a reduction that makes the UI unusable. My exact words were: “Just because a user interface uses 3D-buttons and some shading doesn’t mean that it has to look tacky. In fact, if you have to make the choice between tacky-but-usable and minimalistic-but-hard-to-use, tacky is the way to go. You don’t have to make that choice though: It’s perfectly possible to create something that is both good-looking and easy to use.”
Can anyone give an example of a popular app where they've been consistently very confused by what's a clickable button/link?
I get that a chunky 3D button probably scores the highest in terms of advertising that it's clickable, but is e.g. the text "agree" and "disagree" in blue when all the text around it is black (from the article) so confusing you'd give it something like 1 out of 10 in comparison? Or is it more like 7 out of 10?
I've haven't found it a huge problem in the apps I use. HN has lots of links on the page that aren't underlined or even a different color to non-clickable text for example.
For what it's worth, WCAG for accessibility says buttons should have a visible border and hyperlinks should be visibly different to the body text (underlined, or a different color/brightness) so there are some standards and guidelines.
> Can anyone give an example of a popular app where they've been consistently very confused by what's a clickable button/link?
since I don't use mobile apps I can't give you an example except to say that I run into this a lot in todays design landscape.
My favorite is when it's not obvious where one window ends and the other begins on my windows box.
There's nothing wrong with form but form over function needs to die.
Just as many programmers ignore aesthetics in lieu of functionality (aka programmer interfaces), many designers ignore ux and functionality while chasing aesthetics.
I use a web app at work that is iconified links all over the place and drop down arrows with palettes of clickable bold text. If I go a week without using it, I have such a hard time finding things because there’s not a menu bar like desktop apps. It’s all a multi step process and discovering the clickable elements in each component.
Blackmagic Camera, Filmic Pro and Halide. All iOS. Their problems are not exactly the same across but the button shapes are a great part of it. The other problem is, that people invent abstract icons for abstract ideas and then expect people to remember every one of those. Whatever happened to text labels beneath buttons?
I think the point about visual feedback when a tap registers is really important. On touch devices it is pretty much standard that some % of your taps will fail to register, either registering on some other screen element (because mobile UI designers love to make REALLY TINY buttons right next to other touchable elements) or just not registering, perhaps because your finger shifted too much before you lifted it up, or you were off by a couple pixels.
So having clear visual feedback that a touch registered, and showing you where it registered, is huge.
I reserve a special curse for people who implement touch feedback separate from touch processing, though. I've used a couple different apps where it's possible to create a visual 'touch' that doesn't actually trigger a click on the button, and that's infuriating.
This problem exists in physical buttons too. My thermostat has buttons that can make the mechanical click without actually activating the button, and I've experienced them elsewhere. It is infuriating.
You are both correct. That’s one of the reasons I mentioned multi-sensory feedback. Every little bit helps. Now, sound can of course be annoying, but it’s up to the discretion (and taste) of the designer to make that call or not – as always, depending on the context.
Another reason to use buttons: if you wanted to automate the button pushing, it's easier to create a computer vision function to look for round rectangles vs going full-blown machine learning trying to identify things that could be buttons. (Source: I make software and robots for pushing buttons.)
Generally, I agree, if you can get access to the underlying structure/source, then you should use that. But I've been making UI test automation tools for 20 years. There are some cases where adding annotations is not possible or not feasible at low-cost, but the feature still needs to be automated.
Fluent Search for Windows has a feature where typing a letter combination can click any element on the screen. I appreciate it for the same reason people like shortcuts.
Fluent Search is a desktop app and this feature is meant to work regardless of which apps are on the screen (browser, Notepad, whatever else) so I think there's a legitimate point here although it might be a niche one not relevant to you.
(I don't know enough about Windows internals to answer with confidence if a better solution is available like examining the OS accessibility tree, but I guess an advantage of Fluent Search's approach is that UIs from toolkits not implementing accessibility standards aren't excluded. The current approach has drawbacks like false positives too though because the ML model isn't 100% accurate.)
Sounds like a reason why people would not use buttons as more often than not people actively have something against UI automation for mostly no good reason.
I'm assuming that if you're trying to automate with UI automation that there was no API provided? Automating UI interaction just seems like last resort for pro users and an act of desperation, or screen scrapers.
If you're testing said buttons with a custom machine, don't you already know where the buttons are beforehand? Why would one need to analyze single frames and look for buttons using machine vision?
- Device provisioning - Automating a device right after power-on to do initial set-up, add to Wi-Fi, install apps, etc. (Other methods, like ADB, have to be enabled first. Who does the enabling?)
- Security testing - Interacting with a device that may have malware on it. You use a robot so that it is 100% unmodified while you study it. For further safety, the device is run inside a Faraday box, and you use the robot to access the device from outside. (Otherwise, you do everything manually in a human-sized Faraday cage.)
- Performance testing - Getting timing data of app performance is more accurate when the device is unmodified and you use an external tool to interact with the UI.
- Testing medical devices - Required to be tested exactly the way a user would interact with the device.
- Point of Sale testing - Some real world end-to-end tests involves inserting or tapping a card and entering a PIN on the screen.
- Or you're interacting with a touchscreen that has no other programmable interface, but you still want to test automate it (like any touchscreen on any device that is not a phone or tablet, e.g. an HVAC or alarm system panel.)
(These are all things I've been paid to make robots for over the last decade.)
example, I have to enter my information on a form every day but the form has no accessible api. I am just copying content from when place to another. Perfect use for a robot.
OK. Now where do I learn all these functional aspect of UI/UX?
I was thinking of following Human Interface Guidelines. But reading from people here, it just means the UI will look more uniform and fancy. The functionality aspect is not guaranteed.
But the current premier existing design standards (he calls them out in the post) are examples of what not to do. More of those isn't going to fix anything
The core problem is that people have trouble differentiating between the desire to improve something and the desire to change something.