Hacker News new | past | comments | ask | show | jobs | submit login
AT&T's Rube Golbergian Web Form (joellehman.com)
146 points by jal278 on June 25, 2011 | hide | past | favorite | 62 comments

I work for AT&T. I have an obligation to inform you that what I say doesn't represent their official blahblah whatever. You should know this isn't an on-message marketing communique.

AT&T, the company that created Unix and C, could be said to no longer exist, or more correctly could be said to be a different company than the one now called by that name.

A little (abridged, simplified) history: AT&T, you may remember, sold its cell business to Cingular long ago. Bell Labs went to Lucent before that. Eventually, it even sold its residential phone business off. They backed out of a lot of markets and dropped the best R&D lab they could have had.

What is called AT&T now is actually SBC, a Baby Bell with a penchant for out-sourcing. SBC bought Cingular, AT&T, Pacific Bell, lots of other companies. Their AT&T purchase was motivated in part by the name: everyione has heard of AT&T.

SBC brought with them metric tons of bureaucracy, all running in IE. Disgusting. It's not just the external web interfaces. We have to deal with this BS internally, too. 1990s web interfaces that only work in IE (sometimes requiring 7, sometimes requiring 6) for every interaction with corporate. Taxes, mandatory training, time reporting, everything.

We have to grab a spare Windows machine or run a VM with XP in it. Most of the tech side of company knows and hates the whole thing. The impenatrable bureaucracy makes it impossible to find out who to complain to. There is no escape. The article is dead-on about what's wrong, and I know first-hand, because we have to eat that dog food weekly.

That said, it's a very cool place to work overall. For us, the consultant-generated tech is a dent in the Porsche. But for a customer trying to give you money? Even my mom doesn't use IE and she wouldn't understand the problem with ActiveX controls.

Some history from Colbert: http://www.youtube.com/watch?v=rsCp-1hgfxI

Thanks for clarifying, I'll put this explanation in the post to make it more accurate. Interesting how an intuitive guess at the cause was so far off; the true cause was more insidious and historical. Inherited cruft from an acquisition.

funny, we're talking about almost the same thing here http://news.ycombinator.com/item?id=2695521

Well at least they're dogfooding.

Over the last 7 years I've set up more than half a dozen DSL/cable/FIOS modems and routers (I've moved a lot). I've always used Linux or OS X (which was as problematic as Linux in 2004).

FWIW, every company I've worked with (AT&T, Comcast, Verizon, others) has a method of activating everything that does not require installing any software. If you have a technician on site installing something they can do it for you; if not you may have to call them to do it.

In my latest round with Verizon FIOS (4 months ago), the technician let me know that they have the option "in case you just moved in and your computer is packed away." Use that as an excuse, or say you haven't purchased your computer yet but your family is over visiting and need to use the WiFi.

I've activated without installing anything with every broadband provider I have had as well and most of the time it is quite easy.

Cable seems to be activated just by using their modem the last few times and qwest dsl can be activated by a simple login on a browser and waiting for them to verify.

Wow, that is truly horrible.

Although, think of it like this: AT&T is doing what every web developer in the world wishes they were able/allowed to do. It's getting people off IE6.

Granted, the process should be "If IE, then update, else, just show form"

>"In short: a simple web-form system shouldn’t end up installing Internet Explorer 8 for you. I think it’s interesting particularly because no lean startup would ever ever do this in a million years, yet we aren’t particularly surprised when a big company is flushing dollars down the toilet in this way."

I would hardly be surprised if doing it this way saves ATT millions of dollars. The default browser for Windows XP is IE6 and a large percentage of new ATT internet customers are highly likely to use whatever browser is on their system, and even a small fraction of ATT's customers is millions of people.

Also keep in mind that this default behavior was almost certainly implemented several years ago when the use of IE6 was even greater than it is today in order to avoid lengthy technical support phone conversations with people like your grandparents when they purchased their first computer.

I'll add that it costs ATT very little to only directly support IE - sure a few calls from Firefox users escalate but that is probably offset by the fact that as a market segment those users are more likely to be technically savvy.

ATT knows its market segment in a way that most startups would do well to emulate - and recognizing how their setup procedure goes to their bottom line is a valuable lesson about focusing on MVP rather than perfect code.

Well, and how these reasoning justifies the need of ActiveX (the solely reason of IE8 installation) in order to fill in a web form?

They probably hired a company like Accenture/Deloitte/etc who flew in a small army of 30-40 people to align the customer enrollment process with some enterprise infrastructure standards.

Since you don't need 40 people and a couple of $700/hr partners to build a web form, they broadened the scope of the process to pick low hanging fruit or synergize paradigms.

That may make no sense to you, but I'm sure it sounded good at the 16th hole (and 4th cocktail) at the country club when the $700/hr partner was pitching it.

I suspect that ActiveX enabled browsers allows them to further reduce support costs, e.g. installing a 32 bit version of IE8 allows the Flash driver to be installed thereby reducing calls from grandpa whose new system came with a 64 bit version of Windows 7 and therefor uses the x64 version of IE by default - not to mention that ActiveX also allows ATT to install more powerful software on the customers browser for support and tracking purposes.

64-bit Windows actually uses the 32-bit IE by default. You have to use a special Internet Explorer (64-bit) shortcut to run the 64-bit version. I don't think that's an issue.

And -

To (hopefully) install a shim that lets them install software that lets their help desk help their customer more effectively. If I were trying to support millions of 'ordinary users' and had them calling me every time anything didn't work the way they expected, I'd want something on their desktop that let me help them (a gotomypc type of agent, for example).

I don't think the parent is trying to justify the overall ridiculousness of the process, merely explaining what their rationale would be for even going down the IE-only path in the first place.

I'm not sure if anyone can justify the need for ActiveX in this case.

The assumption that ATT needs to do it, is like assuming that a website needs to use cookies - it doesn't follow the money. The article implies that this process costs ATT millions, whereas logic would make the continued existence of such a practice unlikely and analysis of ATT's cost and revenue structure would indicate that it probably has the opposite effect on their bottom line.

ATT's homepage for DSL customers is Yahoo, and it is heavily Flash dependent. The same logic that serves as part of the justification for Apple's iOS strategy underlies ATT's decision making - reducing options to reduce support costs and enhance user experience. It is better if Grandpa's introduction to the internet has everything working when he lands on your homepage for the first time.

Again, ATT knows their market segment.

I don't quite follow you. You seem to be saying that AT&T is smart enough not to do things that aren't cost effective, so there must be a reason for this.

I think that's overly generous. Aaron Patterson spoke at RailsConf. He said he works for AT&T, and to submit an expense report, he scans paper receipts to a PDF, emails that somewhere, which converts the PDF to a fax, which faxes it somewhere else to be converted back into a PDF.

That is clearly a stupid process. It's not there because AT&T has analyzed the business logic; it's there because a large, old business can accumulate layers of bureaucratic process more easily than a small, young one. There are more committees and more inertia. And maybe if you make billions, wasting thousands of dollars worth of employees' time doesn't make you sit up and take notice.

I'd guess this web form is a similar problem. If you have a 5-person company, everyone in the company hears customer feedback and maybe talks to customers personally. I doubt the CEO of AT&T even knows this form exists.

The experience of an employee such as Aaron Patterson is pretty much irrelevant to the bottom line given more than two orders of magnitude difference between the number of employees and customers (~260,000 employess vs ~100 million wireless customers without counting landline, internet, and TV customers). [http://www.att.com/gen/investor-relations?pid=19251]

The fact that they are leveraging legacy business processes rather than reengineering them for the sake of aesthetics (keep in mind thtat even with reengineering Aaron would still be scanning his receipt and the person in accounting would still be dealing with a PDF and it doesn't cost them anything to use idle bandwidth because they own the pipe).

Don't misunderstand me, ATT doesn't get any of my money because of my past dissatisfaction with them. My point is that it is a mistake to misinterpret their webpage in the way the article does.

[edit] In fact the process of converting PDF's into faxes and back into PDF's makes a lot of sense from a document management standing because it will convert Aarons 2400 dpi 32bit scan of a Chili's receipt to the 100 dpi 1 bit image which is sufficient to do the job. Thereby reducing long term storage costs, standardizing the stored format, and almost certainly integrating with the initial portions of the office automation system which no doubt were designed to convert faxes from the field into digital documents for use by accounting staff.

>The assumption that ATT needs to do it, is like assuming that a website needs to use cookies - it doesn't follow the money. The article implies that this process costs ATT millions, whereas logic would make the continued existence of such a practice unlikely and analysis of ATT's cost and revenue structure would indicate that it probably has the opposite effect on their bottom line.

Well, that's one way of interpreting it. Another way is to say that AT&T is a monopoly provider and therefore has no incentive to cut costs. The fact that the setup process is baroque doesn't cost AT&T anything - its not like customers have a plethora of alternatives for their broadband access. So, what AT&T optimizes for is cost of development, and I can assure you it was cheaper to develop this "solution" using half-trained consultants than it would have been to develop a well-designed setup page using experienced software engineers.

I don't quite follow you. You seem to be saying 1) AT&T is smart enough not to do things that aren't cost effective and 2) forcing people through the pain of getting a particular browser improves their experience later, or maybe their ad views on Yahoo. Is that right?

Remember it is easy to detect IE6 using on the client-side using JS and on the server-side using user-agent detection. Then if the ActiveX is really needed, you can do that using JavaScript too.

Now: Would DropBox do something this silly?

Yeah. DropBox installs Growl without the user's permission. http://growl.info/thirdpartyinstallations.php

And just for kicks, they accidentally open up your files to anyone who knows your e-mail address.

Rule #1 when calling AT&T DSL support: Always, ALWAYS say you're running Windows with IE. Lie. Fake it. Whatever.

If you say anything else you'll derail your support. Your goal is to get through the level 1 did-you-plug-everything-in-lets-restart-your-machine gauntlet.

That, or go to the forums at dslreports.com and hope you can contact a sympathetic AT&T engineer.

This seems to be a good rule for calling any cable/phone ISP.

The last time I mentioned using a Mac the support person ignored that I had already tried pinging certain gateways to check connectivity, and proceeded to condescendingly walk me through going to Applications->Utilities->Network Utility so that I could run ping from a tab there.

Of course I quit after 15 min of this, tried to access again in a couple hours, and it was just a network failure in my area, anyway. Geez.

I'll add Rule 1A: Never talk net-speak with tier-1 support.

"I've run some traceroutes and packets are getting lost at your router in Elmhurst. When I do get through, ping times are over 700 millseconds"

"Okay, sir, what I'm going to need you do to is close your browser and restart it"

I'll add Rule 1A: Never talk net-speak with tier-1 support.

If only "shibboleet" worked in real life...

Or the ultimate helpdesk answer: "Please reinstall Windows and try again."

Is there no "I'm an IT professional. Please transfer me to someone that knows that I'm talking about" option for support?

I had to setup AT&T DSL last week. The modem itself didn't have a connection (blinking DSL LED, no Internet LED), and I made the mistake of mentioning to the tech that I was using an Apple router instead of the Cisco router they sent me. He stopped investigating the problem right there and the only thing he would do for me was transfer me to Cisco support for the router even though I expressly stated that there was no problem with the router.

Could be worse. AT&T let my mother-in-law sign up for service. She gets the modem, we set it up, blinking DSL light for days.

We call, after two hours on the phone they agree to send a tech out. Tech takes one look at the pole and says "you're too far from the office. They never should have let you sign up"

The last time I called comcast (in the middle of the night) I actually spoke to someone who used linux at home.

Quite surprising, and she was quite knowledgeable and helpful too.

Comcast also has some complex IE only registration, but if you call them and say you use linux they'll say no problem ask for the MAC of the modem and activate you on the phone.

How many people give up before reaching the form, and end up calling someone at AT&T for help? The costs must be enourmous.

ATT has a pretty good infrastructure for handling phone calls from customers - being that they are a phone company with more than 120 years of experience in doing so (founded 1885).

So it is probably not as expensive as one might think at first blush.

From my experience with their phone support I'll agree that it probably isn't expensive, but only because it seems to be chronically understaffed. Multi hour wait times appear to be the norm when dealing with AT&T.

Something I heard from a senior developer at another well-established phone/cable/ISP: the company will do everything in its power to avoid having staff pick up the phone, including avoiding development of new products or features that might generate support calls. At the scale of a large telco/ISP, having just 10% of users calling up tech support and talking with a rep for a few minutes adds up.

So, you must have had IE6 or IE7? As much as I hate being forced to run any particular browser, one less machines running these old browsers is a good thing.

Typically, "upgrade your browser" techniques are welcomed in HN. Google drops support for IE6, then IE7 - smiles all around. Apps block off IE6 - "good for them".

AT&T does something similar and it's met with frustration?

Requiring IE* should be frowned upon. But anything that says "upgrade from IE6", in end-user land, is probably a good thing.

My frustration was because IE*, and thus some form of Windows was required. I use modern browsers, just never IE, and windows only when I must.

So I agree with your point about upgrading being a good thing, but the underlying absurdity is that something inherently platform independent (a webform) was perverted into something entirely specific to not only a particular OS, but a particular browser.

My ire with IE being updated was that I was never going to use IE ever again (by choice), whether it was 6 or 8; and so whether it was upgraded or not would not affect my browsing experience on sites other than this particular instance. So for me, it was just one additional hoop that I had to grudgingly jump through.

I just signed up for AT&T and their whole signup process is incredibly broken. Their customer section is divided into three sections: Wireless; Internet and Phone; and U-verse. Each has it's own login system. One use sequential IDs for logins, one usernames and the other email addresses.

I spent ages fighting the login and ID recovery system before I found out my DSL connection came under "U-verse" and not "Internet". Their login page even has a helpful picture of a TV to illustrate the U-verse link: http://www.att.com/accounts/?source=IC4425j4900s2000

I still have no idea what "U-verse" is and what differentiates it. I just assume that some crappy marketing manager thought the name was cute.

Four sections. The iPad plan has its own login hidden off to the side.

Why is the form asking for gender as a required field?

It's probably just a language thing, so the forms can say "him" or "her"?

Edit: or in support calls, they sometimes accidentally say "sir" to a woman etc., this might help with that.

Or they are selling this info to a third party... I've heard of a story of a guy that used his British girlfriend's identity to sign up to one of your American internet (or cable) providers and than got a lot of spam for her, even though that's the only place he used her name.

(Yes, I know this sounds like an urban myth, but I can dig up where I heard it straight from the guy, he has a podcast).

Of course, a lot of this is plain incompetence, but there is a security motive explaining why enterprises only support specific browser versions, even if it's an irrational one. Enterprises fear people will blame them even when the browser is at fault. Celebrity's AT&T account gets hacked? "ZOMG AT&T doesn't care about its users' data."

It's not entirely crazy either, when you consider how mainstream media and consumers perceive security incidents. The problem, though, is when enterprises tend to be overly conservative, not acknowledging "reasonable risk". And being out of touch with browser trends, ie assuming that only a particular version of IE is safe. (In this case, IE is probably required because the developers didn't know any better, but in other cases, an out-of-date understanding of browsers is the rationale for supporting only IE.)

There is (or was) a similar system profiler program that installs itself from Windstream's DSL service, and will not allow itself to be removed from a PC by any means I am aware of. Totally unacceptable on my machines that I purposefully kept perfectly clean!

It's not what you see, it's what you don't see.

Comcast installation was/is similar -- done in my case during the course of the Comcast technician's visit. I got rid of their crap when I reimaged the machine. Didn't affect service in any perceptible manner. (It did get rid of their branding from the IE window title, though -- another useless annoyance.)

If I have to go through similar, again, I'm going to do it in a disposable virtual machine image (reverting back to the last snapshot).


You can change the 'Microsoft Internet Explorer provided by Yoyodyne' string to whatever you want.

Yeah, I'm more bothered by the services their default process installs. They can monitor their modem, if they must. They can keep their nose out of my system. Though branding the IE windows title is just downright annoying, particularly for non-technical users. In addition to taking up space and being distracting, every screenshot they make carries it, like it or not.

People are reminded of you every month when they pay their bill, Comcast. Don't push it. /rhetoric

Should be 'Goldbergian'. There is a 'd' in there.

So what I want to know is whether the ActiveX control is actually doing anything necessary [1] to automate the install process. Is it telnetting to the modem or something like that?

Other commenters mentioned you can call them to activate your account. What info do you have to give them that's not collectible via a web form?

[1] I'm assuming the whole ActiveX thing is to push a download of IE8 because the code whatever tool they used to build the signup form only works in IE8. But it just boggles.

The landing page [...] plays some audio instructions [...] There is a five minute pause as AT&T’s website “checks my system.” [...] my browser is not supported — well, I use chrome on ubuntu [...] Firefox. Wrong. It requires an ActiveX control [...] Windows XP [...] IE [...] ActiveX control. I’m getting frustrated by this point.

Wow, this is one patient guy. I would have gotten frustrated by the second point already and would have thrown something against the wall by then!

I noticed a surge in traffic to EndlessForms.com and realized it was coming from this post. Thanks to Joel for mentioning our site[1]! For those of you that don't know what I am talking about, see Joel's comment on the original post where he recommends EndlessForms.com, where you can design objects with evolution and 3D print them. Thanks all for checking it out.

[1] http://endlessforms.com

I didn't run an ActiveX control. I did it from my Mac. Granted this isn't Chrome on Ubuntu, but I would be very curious to find out what would happen if you have Chrome on Ubuntu report it's user agent as Safari on Mac.

It would be even better if AT&T could somehow be convinced to run the same code that runs in normal browsers anytime the user agent isn't IE. Even better if they could somehow be convinced not to even run ActiveX in IE.

Hmmm.. Random un*x variant. Then the backup plan is an OS from 2001. Sounds like the plot from Woody Allen's Sleeper...

A faster approach would be to spoof your user agent -- announce yourself as IE 8 on Windows XP.

That would work if they hadn't used ActiveX

I wonder if the form would have worked anyways. People bring work laptop home that IT might have configured to not use ActiveX

Though you're probably right, probably would have failed.

Only where AT&T is involved could we rush to the defense of IE6.

Who said dinosaurs are extinct... They are hiding in plain sight...

spent two days trying to subscribe using linux after I found out they are the only ones around... I finally found a phone number that has a short wait and somewhat knowledgeable people


Cui Bono.

What if Microsoft pays AT&T on the side to force a large number of people to use their platform? Even if most people already do, it instills in people the believe that that Line-ux thing doesn't work with the Internet.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact