Hacker News new | comments | ask | show | jobs | submit login
Saying Goodbye to Louisville (googleblog.com)
207 points by seanclayton 12 days ago | hide | past | web | favorite | 207 comments

"For that, and many other reasons, we are deeply grateful to Mayor Greg Fischer, the City of Louisville and its residents for their partnership and spirit of innovation over the past two years."

That's why we can't trust Google with anything critical. They view the world as an experiment they can quit anytime. Definitely don't buy a Google self driving car or anything else you expect to work in a few years.

There is a good chance that businesses moved to Louisville because of Google Fiber and they are being shown the finger "Thanks for indulging us, we got what we wanted and you can go f... yourself".

You make a good point, but I think it's a corporate problem, not a google problem. Expecting loyalty from businesses is pointless. Look at NFL franchises. They'll move from cleveland to baltimore or Oakland to Las Vegas or from LA to St.Louis and back to LA without a second thought. Look at the rust belt where companies just left for "greener pastures". Or pharma companies merging and moving their headquarters to europe to exploit tax loopholes. Businesses are in it for themselves despite all the ads and PR talking about how they care about the community.

I don't think it's really about loyalty, it's more about managing expectations. Some brands use reliability and long term support as selling points, so you expect that from them. They have an incentive to keep offering long term support in order to preserve their reputation. It might not be important if you're buying, say, a smartphone that you expect to keep for a couple of years but it's critical if you're buying something that you expect to work with two decades from now, like some industrial machines for instance.

Google clearly is not in this category. They will drop basically anything with very short notice (two months here) if they don't feel like doing it anymore. That may be fine if you're buying a Google phone but that's definitely a problem for anything that can't just be replaced instantly and for (relatively) cheap.

Can you imagine if people who manufacture components for industrial designs behaved like Google does? "We have these subway trains with busted brakes, but it turns out that they were manufactured by Google and they dropped the project two years ago. I guess we have to buy new trains then..."

>Can you imagine if people who manufacture components for industrial designs behaved like Google does? "We have these subway trains with busted brakes, but it turns out that they were manufactured by Google and they dropped the project two years ago. I guess we have to buy new trains then..."

I can imagine it. It happens every day.

In an industrial setting it is not uncommon to need a part but the stock version of it is not available. It could be that the manufacturer switched from making cooling tower components to fly swatters. Or, it could be that the system you're maintaining is a hundred years old.

You might find a 3rd party replacement part from China. If not, you try to build it from generic components (everyone uses the same bearings, o-rings, springs, etc). If that fails you find a company who sells a similar product and give them your specifications. Finally, failing all other means, you pay a machine shop (or a foundry if you're making the cast iron part of the braking system) to build your part. Anything that has already been built can be built again.

If you want security in terms of products you are buying, you get a warranty. There are laws that enforce warranties. So you get a warranty, and when its going to expire you get an extension, and when you cant renew anymore you plan to replace that product.

The same is true of utility companies. If you are worried about fly by night fiber companies, you should require a lengthy advanced notice-- perhaps a year or two. And if Google blesses some other city instead of yours because you wanted the warranty, then you let it go.

" If you are worried about fly by night fiber companies, you should require a lengthy advanced notice--"

That's the thing. Google uses its big name and reputation to gain entry but they behave like a fly by night operation.

The only reputation that Google has left in my head right now is that the thing that you rely upon can be gone overnight. I'm not using it for anything remotely critical.

For most non-tech users Google has a stellar reputation unfortunately.

Heck, I can't even get some of my techy friends to drop hangouts even though google's retiring it. One of them even said "I'm going towait to see what google's next product is", you mean allo? Duo? What's possibly next that could be worth it? Why would you want to stick with that treatment?

> Can you imagine if people who manufacture components for industrial designs behaved like Google does? "We have these subway trains with busted brakes, but it turns out that they were manufactured by Google and they dropped the project two years ago. I guess we have to buy new trains then..."

Like in the NYC Subway?


> The old-fashioned system requires intensive care. When parts break, which at their age happens often, this busy repair shop springs into action. Like doctors, mechanics examine the patients. Some sound like they're on their last breath. Many of the companies that made these components are long gone. So workers here have to manufacture their own replacement parts.

You have to be aware of the potential for the obsolescence of any service you use. Sometimes they drop the project, sometimes they go out of business. The end result is the same. I don't get mad at PageNet for not supporting my pager any more.

The components referenced in that quote are literally from the 1940s. We're talking vacuum tubes.

Comparing 80 year old parts to a project that is barely two years is not really fair.

I have high confidence that Verizon will continue to offer fiber at my house for years to come. “Will Verizon decide to stop serving this place?” was not a question I had to ask when looking for a place to live. Expecting some stability isn’t about loyalty, it’s just good business. Google is going to start scaring people away from stuff they actually want to sell if they have a history of abandoning things.

Does Verizon offer fiber in Louisville?

Google Fiber was more about scaring ISPs into investing in fiber, than serving its own fiber. Once bribery and lawsuits failed to stop Google Fiber, ISPs started to compete.


Verizon doesn't really offer fiber in Louisville, it does have a presence but its not large at all (~3 neighborhoods). AT&T ramped up their fiber roll-out here over the last two years and have made great progress; I am currently on their 1GB fiber plan and am enjoying the speed and prices. However, with this latest Google announcement I'm fearful that the AT&T expansion will end and the prices will rise. Hell, Louisville is in the list of cities announced for AT&T 5G service, but you can't actually purchase it today and now I'm not sure if it will ever arrive. I'm really not loving Google today.

I was thinking about Google Project Fi but not anymore.

Worst case, Google Fi is discontinued. Then you find another provider and port your number.

Unless your Google Wallet is disabled, in which case your Google Fi account ends up in arrears and you lose your number. Google Fi can only be paid for with Google Wallet, nothing else. Google Wallet is a separate team from Google Fi.

Is that some hidden layer thing? Because I've had fi for a little over three years and never once messed with google wallet.

Moreover the legacy stuff is heavily regulated, in part, so that the telco can't suddenly decide to pack up and leave.

Stuff that isn’t heavily regulated still works this way.

I’m pretty confident that my pest control company will be around for a while. If there’s a problem with my HVAC system, I know I’ll be able to call the people who installed it. The company I bought my cars from will be selling cars for a long time. And I could go on.

Google has the right to do this, but they’re starting to gain a reputation for it and that’s not going to be good for them.

> Google has the right to do this, but they’re starting to gain a reputation for it.

Starting? It's been a running joke around certain communities (including this one) for years and years.

I don't use anything from Google I'd consider critical if there is a reasonable competitor.

Say what you want about Microsoft they generally don't just chop you off at the knee's (I mean they've done it but not in the way google does).

I would say Microsoft actually goes to far in the other direction; they have so much legacy cruft from supporting features that were last used during the Reagan administration that it adversely affects the quality of their products.

Windows 10 is abandoning cruft, but that hasn't seemed to help with product quality. (Maybe they shouldn't have eliminated the separation of test and Dev)

Windows 10 actually moved to the test in production model. That's why consumers get the releases first. That way if it brick's granny's laptop she won't be able to complain to anyone about it online.

Pest control and HVAC service may or may not up and leave, but it's not a big deal when they do, because there are many options to choose from and most HVAC servicers will service all kinds of equipment (even if they have a preference for what kinds they install).

Telco is regulated, so if Verizon abandons your market, they will need to have a plan that leaves you with service. Which is most likely to be selling it to Frontier.

That Google is willing to abandon markets for Fiber is really a testament to their strategy here: Google fiber was not a real attempt to be an ISP, they just wanted to show it was possible to inspire incumbents to get gigabit out to people. Once AT&T started rolling out gigabit fiber, Google was done.

> Stuff that isn’t heavily regulated still works this way.

No it doesn't. That pest control company may pack up and leave (although this is admittedly unlikely), but AT&T and Verizon are legally obligated to offer POTS service.

You say “no it doesn’t” and then immediately agree with me. The fact that it’s unlikely they’ll pack up and leave is what I’m talking about here.

And I’m not talking about POTS service. I don’t have it.

Unlikely and unable are two different things.

Ok? I said nothing about being unable to leave.

It's not a corporate problem in the general technology industry. Large enterprise vendors like Red Hat and Microsoft carefully manage their reputations by keeping their product lines stable and continuing to support certain products for many years even when in isolation it doesn't make economic sense.

I think Google is one step ahead of others in terms of unreliability. GE won't pull out of aircraft engines with 2 months notice.

Meh, people do this too. Individuals migrate from one location to another for economic opportunity. Some people even "merge" into aggregated family units to exploit tax loopholes. People are in it for themselves despite all the talk about how they care about the community.

"Ask Me About Amazon" -- City of Seattle

>> There is a good chance that businesses moved to Louisville because of Google Fiber

It would be astonishing if the total number of businesses for which that's true is more than two. I can't imagine what sort of company would base a decision to move to Louisville on the availability of a Google Fiber Internet connection, nor can I imagine how the loss of Google Fiber would be any kind of serious problem (there are other ways to connect to the Internet in Louisville).

Things like this are often used to attract businesses to a location.

businesses may not move to the location for the google fiber themselves, but it certainly makes Louisville a more attractive place to move for developers, and a good population of developers is certainly enough to get companies moving to an area.

>>it certainly makes Louisville a more attractive place to move for developers

I suppose it might be one small factor a developer might put on the "pros" side if her list when deciding where to live, but it's surely a very small factor. Would anyone seriously choose a city to live in based on a marginally faster or cheaper Internet connection? I find it difficult to believe, but maybe it's true for some.

This is why we did not consider GCP when we were looking at moving some of our services to the cloud. Never know when Google will discontinue or drastically change pricing on services.

We're a GCP customer and I definitely cringed in real life when I read the post. We're building cloud portability into our new deployment tooling and moving as many things as possible to kubernetes, so we can be somewhat prepared.

Oh no, Google is the main supporter of Kubernetes, yikes!

I assume you're trolling, and already know that kubernetes is open source, has many key contributors outside of Google, and that we can spin it up on bare hardware if for some unlikely reason we needed to do that.

There are also Amazon and Microsoft in the Kubernetes space, so it is assured just in case Google gets bored with the project.

There are now developers from hundreds of companies contributing to Kubernetes. Google remains the largest contributor, but is less than half and declining as a percentage of the whole, even as their total contributions actually increase.


Disclosure: CNCF develops DevStats

I'm not sure I can cosign the cynicism toward google in the particular instance. I am a Fiber customer in Austin and I've always been under the impression that this essentially an experiment and that it likely wont be around forever. Perhaps that is not a widely held enough believe however.

Cutting people off within two months is just nuts. One year may be acceptable.


As far as I know fiber isn't the only provider anywhere, and getting internet set up if you're already wired takes hours or less in most places (I left the Comcast store and was using my wifi an hour later last time I moved).

Yeah this is exactly my point. Fiber is a competing service in every market it serves. Customers do have a choice. Now unfortunately they do not but and they're just going back to the way things were.

No reasonable person will have difficulty making other arrangements in a nine week period.

You're being downvoted for this but I suspect you're right. The people we're talking about aren't literally random people. They're people who switched to Fiber in the first place.

On the other hand, it seriously forced other ISPs to step up their game from what I hear. So not all is lost.

> other ISPs to step up their game

Eh. Spectrum (formerly TWC, formerly Insight) upped their max speeds. AT&T did a very limited Gigapower rollout to selected areas.

The lasting impact, if anything, will likely be that Google got the metro government to pass a one touch make ready ordinance and defend it in court.

And why wouldn't they step back their game if the competition does?

It’s a bit harder to put the cat back in the bag. I’m assuming (and hoping) it’ll take a few years for that kind of degradation, and some new competition (or municipal service) could make its way in.

I doubt they'll roll back speeds. Instead they'll increase the cost of the higher speeds significantly.

Once (multiple) ISPs invested in infrastructure, they have lower marginal cost so can stay and compete. Sure, there are still antitrust risks, but better than nothing.

Even price increases can take some time to do, as you have people in 1-2 year contracts. I just hope that’s enough time for competition, or that it doesn’t happen.

> Even price increases can take some time to do, as you have people in 1-2 year contracts.

It takes a while to do to those customers.

It can happen as soon as "let's update the webpage" for new customers. Especially when the only high speed internet game in town goes from three providers to two.

This. "Our special deal for new subscribers has ended." And they'll sign up all the people who were on Google Fiber and want to continue having those speeds at the new higher price point, which, of course, is only valid for the first 6 mos or a year, after which it increases to the new "normal" price.

With wireless 5G coming, people will have a real alternative to something that uses a physical line to the house. Brace yourselves, competition is coming.

Due to the wireless industry's bandwidth cap (or severe service degradation after soft-cap) as a standard feature, I have little hope for 5G.

Color me unconvinced about 5G.

Hope I'm wrong.

any company that price matched Google will shortly begin raising rates.

In Louisville neither AT&T nor Spectrum bothered to price match because Google never covered enough neighborhoods for them to care. Spectrum raised their rates considerable, with no infrastructure investment at all. AT&T did a modicum of infrastructure investment just to make it appear that they were doing anything at all.

I live in Louisville and I've had AT&T's Gigabit fiber product for over two years, in two different residences in different parts of the city. I frequently see new signs that they're deploying to new neighborhoods, including some neighborhoods that made me scratch my head a little. They're definitely making more than a modicum of investment in their infrastructure here.

Much of my neighborhood only qualifies for AT&T's 90s ISDN-era speed DSL, which they've confusingly rebranded Uverse despite not being proper broadband in 2019, and worse priced it to match real broadband Uverse. So you may have to pardon me if I'm deeply skeptical of AT&T's infrastructure investments.

I also have AT&T Gigabit fiber (through IgLou) and have been pretty pleased with the service. However, I'm fearful that with Google's pull-out AT&T will lose all motivation to improve and will just start raising prices instead.

This. Literally overnight, you'd see a 100%, 200%, or even more, increase in speeds. Unfortunately with those market pressures gone, companies will go back to not caring after this...

Be that as it may, it doesn't mean its sustainable.

Those "100% speed increases" were ISP's willing to lose some money to retain existing customers in select markets, using their other profitable markets to subsidize them.

I'm 100% for more competition in the quasi-monopoly ISP arena, however googles entree into the ISP market was just freshman exuberance. They were flush with lots of shareholder money and had young idealistic founders who figured they could "revolutionize the space" because they were smart and cared.

Over the past few years, google has been shutting down all their naive money losing ventures and getting into the serious business of making money (ie. selling your personal info to the highest bidder).

Moving to Louisville for Google Fiber would be a pretty reckless decision for a business which wasn’t wholly dependent on it (like a reseller or Google Fiber)

Imagine your google self driving car pulling a Revolv.

So... you plan to put your trust in Comcast or CenturyLink?

I can't trust Google. Every time they build something good (Reader, Inbox) eventually they will kill it even despite users clearly opposing the decision. And those are just web pages, here they're denying of ISP whole town which was serviced before.

So for example how can I use their e-sim provider Fi if I know for sure in one of my travels I will get a notification 'we are closing Fi next week. Find new provider.'.

I use their Photos service extensively. But in a couple of years I know it will get closed too, once they fully used it's capacities for training internal ML models.

Inoreader just changed their plan basically nurfing their free plan... I'd really love google reader to make a comeback.

The also got rid of discussion and blog search which along with getting rid of google reader helped accelerate the siloization of the internet.

> Every time they build something good (Reader, Inbox) eventually they will kill it even despite users clearly opposing the decision.

To be fair, users aren’t customers. People don’t have any right regarding your product if you give them for free.

Those are by analogy, but in the case of Fiber, which we're talking about here, the users are customers. So what rights do they have now?

This is getting old.

There are survey companies out there paying people to offer their opinions on various stuff.

Google gets this for free.

Yes, you can see Google users aren't customers, they are more like unaware/unpaid workers, providing free tuning service to Google's always hungry data analytic engine.

Thank god we have GDPR in the EU so you absolutely can't use this argument anymore. It's both wrong and a failure to engage in a meaningful debate about one of the biggest companies of our time and how they treat and value their customers / users.

Nothing with a lot of users is ever going to be shut down (most features of inbox are ported back to Gmail so that one doesn't count imo). Therefore I wouldn't worry about photos. You can help dredging up Reader for decades if it makes you feel better, but that was clearly a fringe product and the fact that no clear alternative arose to replace it testifies to the fact that it represented a pretty small/unprofitable market.

To try and hijack the top post with some of the comments I've scattered across this thread:

Google did a remarkably poor job in Louisville, and it's absolutely no surprise to me that they're giving up.

Google's strategy was to build a FTTH network with microtrenching. In practice, this meant cutting a 2" deep groove in city streets, placing a fiber optic cable in it, and using an expanding rubber gasket material to cover the groove.

This failed in spectacular fashion. One of the best pictures was this: https://www.wdrb.com/news/belknap-neighborhood-residents-con...

... a fiber optic cable, barely buried under the surface of the road, exposed after a few freeze/thaw cycles.

If you click on no other link about Google Fiber in Louisville, click on this one. This scene is repeated all over the city.

There won't be much in the way of "dark fiber left behind" because most of the Google Fiber rollout was FTTH done ridiculously poorly.

And for the customers? The only company competing for residential gigabit service is AT&T, which (wisely?) rolled out their service to high density apartment areas and wealthy condos; for most residents, Spectrum or Uverse DSL are the only real competitors.

Some cites:

https://www.techrepublic.com/pictures/photos-how-google-fibe... -- Microtrenching will save Google!!

https://www.wdrb.com/news/google-fiber-announces-plan-to-fix... -- uh oh, things aren't so good...

https://9to5google.com/2019/02/07/google-fiber-closing-in-lo... -- we give up, 2" of trench won't do it.

Wow, thanks for linking all of this.

Key quote:

> We would need to essentially rebuild our entire network in Louisville to provide the great service that Google Fiber is known for, and that’s just not the right business decision for us.

This is absolutely incredible. Did they not weather test it before rolling this out across the city? Did city workers know what what these marks were so as to not to cut across these lines doing routine road work? I wonder if any of these removal/repair costs are going to fall on the city or if Google is taking care of it? This is a pretty amazing story.

I think this city was their weather test. Kind of like how Gmail was "beta" for over 5 years. In the Google model, everyone is a potential tester.

Looks like some really bad business practices all around, from planning to execution to ultimately telling the customers they're not worth the investment.

It seems, having gashes in the road, exposed to salt and wear, will increase rate the roads require maintenance. These cities need to monitor and ensure Google is paying those bills.

Is outline.com GDPR compliant or do they just not care?

No one really knows what's compliant

People downvoted this but apparently the local news provider is throwing HTTP 451s for Europeans, so don't.

Yes- the whole thing was an experiment, and it failed. Now they are moving on.

"That’s because we were trialing a lot of things in Louisville, including a different type of construction method — namely, placing fiber in much shallower trenches than we’ve done elsewhere."

Read as: We messed around with your street infrastructure using untested construction methods, and it didn't work very well so WE OUT PEACE. You know what they say, fail fast byeee!

Move fast and break stuff, up to and including your sidewalks and roads.

If it isn't three feet underground it's going to get cut somewhere along the line.

Did google just waltz in and start tearing up the roads? No. Someone with authorization power in Louisville OKd the project. Do they deserve no blame?

I can't even picture a scenario where another ISP just says "Hey, we're packing up shop, you've got two months to find a new Internet provider". I guess I suppose it's happened somewhere, sometime, but it certainly feels one of a kind.

It's not a scenario I would have ever planned for, to say the least, but anyone who has service with Google is probably used to it.

A lot of dial-up ISPs (at all levels) went bankrupt and disappeared back around the turn of the millennium.

I don't know where this expectation comes from that you should be able to pick a service provider and then use them until the end of time. Hell, it's harder to change your primary bank than to change your ISP. Businesses disappear all the time. Every time I move I go through the same hassle of scheduling an installation of new Internet service.

> A lot of dial-up ISPs (at all levels) went bankrupt and disappeared back around the turn of the millennium.

This strikes me as a particularly poor counterexample, as those businesses were effectively fungible: outside of your local ISPs email address, there was nothing keeping you from stopping your service at 718-867-5309 and switching to 718-387-6962. The underlying infrastructure for either of those ISPs was handled by someone else.

Eh, unless 718-387-6962 was long distance. My family went 1.5 years w/o internet after the local ISP went bankrupt, since nobody offered access that wasnt long-distance. Yes, this was a rural area, but those businesses weren't quite as fungible as you think.

> The underlying infrastructure for either of those ISPs was handled by someone else.

Last mile, yes. But there was plenty of ISP specific infrastructure.

> Hell, it's harder to change your primary bank than to change your ISP.

Is it though..?

I just changed my primary bank. The most difficult part of it was listening to the account manager person tell me about what a terrible mistake I'm making and how the other bank is just using me.

In some of my previous residences there is only one choice for ISP.

I think that's because rather than shutting down, a lot of smaller cable providers are simply acquired, or at least their customers are. Fiber is a different ball game than cable though, as I understand it.

Is it that big of a deal? Most places have Comcast or similar. It takes maybe a week or two to set it up.

It can be. A cursory search showed that the other fiber competitor in the area is AT&T. Google Fiber was $70/mo and AT&T is $80/mo and that was for the highest speed offered. With Google exiting the market, the fiber competition is gone and it allows AT&T to raise prices, if they want as there aren't any other options. It puts AT&T back into the fiber monopoly

$70 wasn't a sustainable price. The competition can use Google's withdrawal as a fairly reasonable justification to raise rates to a sustainable level.

Centurylink is offering gigabit for $65 flat (no taxes or fees) across its footprint, and 100Mbps for $55, 40Mbps for $45. At $70 a month Google was significantly above the average price people paid for internet, but they were offering a premium product (gigabit). Had they offered slower, more affordable tiers to residences that didn't initially jump, they likely could have broken even.

The problem with overbuilding is uptake rates, its not good to only capture 28% of the market or similar, you really need 40+% to break even. The City of Tacoma ran into this with their cable network, and so has Google. Webpass is Google's fix for that, only targetting larger buildings where a 10% uptake rate gets them in the green.

> only targetting larger buildings where a 10% uptake rate gets them in the green

That's not really an acceptable way to operate what is essentially a public utility at this point. We clearly need better regulation of these companies.

> Centurylink is offering gigabit for $65 flat (no taxes or fees) across its footprint

Oh word! They are in Louisville? Where do I sign up?

Hint: "average prices" for high speed internet mean very little in the USA as those prices vary wildly. I pay $30 less for 300Mbps service than a remote coworker does for 75Mbps - from the same company.

Centurylink isn't in Louisville, was just trying to highlight how Google Fiber has encouraged average speed per dollar spent to rise, and how other large providers are getting financially solvent wen running fiber.

Most of Centurylink's territory has only halfhearted competition from the local cable company (Comcast, Charter, Wave), and they just recently lost a lawsuit where the state AGs from Oregon, Washington, Minnesota and a few other states ganged up and prosecuted them for hidden fees and malicious billing practices, thus why they are on the straight and narrow now, with the price you see being exactly what you will pay.

$70 is typical for Google's Fiber 1000 plan. The reason it isn't sustainable in that location is they did a shitty job installing the fiber and they would have to redo the infrastructure. That doesn't mean that the price wasn't sustainable.

A sustainable price is one that has margins large enough that problems like that aren't fatal.

Infrastructure is far different than selling widgets. To make a profit, you want to get as many customers as possible even if you have to mix price points. The goal is to maximize revenue per home passed, not get a handful of homes at $20 a month higher price point.

It can take AT&T 90 days to turn up a fiber circuit. With this volume of customers (assuming there are 10,000 Google fiber customers in Louisville) who knows how much longer that will be

AT&T simply does not maintain enough fiber coverage in Louisville to absorb all displaced Google Fiber customers. These customers will primarily be going back to Uverse (DSL) or Spectrum (cable).

If the other ISPs are qualitatively worse (you mention Comcast here...), it's bound to make a lot of people very angry.

I read this as the Google fiber was terrible- many have probably already switched, and entrenched (pardon that) ISPs probably are better. Now, price-- that's another story.

Well, that's the issue - even with google fiber around, it was the corporate version of plutocracy. Functional triopoly? I'm not sure what to call it.

I expect Comcast rates will shoot up now that another player has left the field.

Funny story, I recently had a broadband provider cancel a 2-day old 2-year contract we had signed with them because they discovered the cable hookup was broken and didn't want to be responsible for repairing it. Never heard of that happening before, but apparently those contracts are only binding for the customer.

IANAL, but my understanding is everybody has 3 days to back out of of a contract. Maybe it's a local thing, though.

A 3 day cancellation for contracts is absolutely not standard.

that is a really weird and really wrong thing to think about contracts.

I think it happens a lot more often than you think, though often it's more directly presented as a particular market being sold to another ISP. However, it really is just an ISP closing up shop in that market and selling off its network/customers to someone else.

Most recently Frontier Communications comes to mind.

It sounds like the ones that went with Google haven't had a good provider since. Maybe those for whom and ISP is critical have already left?

Am I the only one going to say it?

Kudos to Google Fiber for actually trying to innovate in a stagnant industry. I'm no insider here, but I suspect the micro-trenching did go through internal testing which looked promising before they approved it for real city use.

If the micro trenching had been successful we'd be cheering for Google for finding an innovative way to bring fiber to more people. Turns out it didn't work in their one trial city and since they discovered this at a time when Google Fiber is no longer expanding it is no surprise that they don't "re-open" Louisville and retrench the entire city.

For a community that celebrates failure as a natural consequence of innovation, I'm surprised at all the negativity here. Not a mention of the 2 months of free gigabit internet they're giving to customers. Yea, Google Fiber is just a bunch of soulless profiteering corporate jackasses /eyeroll/.

I have family in Louisville and visit on a regular basis and have seen the issues firsthand. The microtrenching has caused and will continue to cause damage to the streets. The rubber sealant is commonly free from the trench and potholes form quickly. These are fixed by tax dollars and there is not a quick fix short of repaving every street.

The community here isn’t digging on Google trying to innovate. The anger comes from a highly successful company taking a risk, seeing it fail, and just peacing out leaving others to clean up the mess.

Did Google Fiber agree to do the road maintenance going forward? Sounds like the city council was probably the group that decided to shoulder the risk of damaged roads.

If Google Fiber misrepresented their microtrenching then that would certainly be fraud and they would be liable for damages.

I celebrate failure of a project as a natural consequence of innovation. I don't see how failing a community is worthy of celebration.

When I try a crazy new idea at work, and it fails, I don't just leave a mess and walk away. Come to think of it, I don't do that if it succeeds, either. Supporting your mess is part of professionalism.

This is the key difference between doing research and providing a service. Are you serving the people or just the technology? Google is full of smart people who love technology and want to do research all day, but they're not very service-oriented.

Should they be? Eric Schmidt said of Google Fiber, "It’s actually not an experiment; we’re actually running it as a business". I'm not seeing that here. They learned something valuable about a technology they're using, and they're leaving customers in the lurch.

> just leave a mess and walk away

You make it sound like the people of Louisville are now in a worse position than before Google Fiber came, and that they've somehow left the community in a lurch. All that's really going to happen is the subscribers go back to whatever ISP they had previously.

Now if there's evidence that Google Fiber is abandoning a liability for torn up roads, or misrepresented their micro-trenching to the city council that made the deal, that would definitely qualify as "leaving a mess", but I haven't seen evidence of that.

Can post-2010 Google execute on anything "all the way" through?

Is there some sort of institutional ADD?

This is what happens when you build two decades of company culture of rewarding employees for launching new products over boring old maintenance and iterative development.

I think part of it's because they almost exclusively hire people who will soon leave if they're told to do too much "boring" work and aren't allowed to half-ass it to get it done fast and move on to something more fun.

> Is there some sort of institutional ADD?

I work at Google, opinions are my own.

Obviously this is my personal take, but I don't think that's what it is.

To me the issue is the performance/rewards system. Imagine you're someone high up and in charge of 10 projects, all asking for headcount/resources, while you have a limited amount to give. Presumably you're going to give more headcount/resources to the projects that are growing faster. If the differences in growth are substantial, maybe you'll decide to actually just let some projects die so their teams can be absorbed into the other projects with better prospects.

Google's performance system is an attempt at being really objective, and it's hard to put a precise value on goodwill lost by discontinuing a product. Not only that, but the effects of discontinuing a product are felt company-wide as opposed to by an individual. Nobody is going to look up who decided to discontinue a product after all.

I guess my question is: Is management there not realizing this is a huge crisis point for them? Clearly, you can make a mistake in performance rewards, but the question is, why hasn't it been fixed? Surely people at the C-level have to have noticed Google is getting a reputation literally defined by sudden product termination, and that the whole company is harmed every time any given product is shut down.

I would think this would be a top TGIF question almost every week at this point (because it seems that's how often Google product shutdowns are in the news).

> I would think this would be a top TGIF question almost every week at this point (because it seems that's how often Google product shutdowns are in the news).

Actually this week (!) was the first time my question was upvoted enough to be asked and it was exactly about this.

My impression is that they are aware but willing to trade our reputation for the ability to experiment or be more aggressive in funding more successful products

> Presumably you're going to give more headcount/resources to the projects that are growing faster.

Absolutely not. Your presumptions are misplaced.

Assigning resources to growth is usually the most stupid thing to do. By definition the projects that can grow the most are the projects with no customers or revenues, especially new projects that are nowhere, they can only grow. https://xkcd.com/1102/ . Investing in these is to the detriment of established projects that actually bring revenues and keep the lights on, and companies that neglect to keep the lights on tend to die.

Assign resources to the projects where they have the most impact. And business impact should be measured by return on investment (or loss if not invested), this could be almost anything in practice, growing revenues/customers, maintaining existing customers/revenues, not getting fines and lawsuits, reputation control, assets/inventory/cashflow optimizations to save money, etc...

> Assigning resources to growth is usually the most stupid thing to do. By definition the projects that can grow the most are the projects with no customers or revenues, especially new projects that are nowhere, they can only grow. https://xkcd.com/1102/ . Investing in these is to the detriment of established projects that actually bring revenues and keep the lights on, and companies that neglect to keep the lights on tend to die.

Well you can consider growth in terms of total users and not the percentage.

> Assign resources to the projects where they have the most impact. And business impact should be measured by return on investment (or loss if not invested), this could be almost anything in practice, growing revenues/customers, maintaining existing customers/revenues, not getting fines and lawsuits, reputation control, assets/inventory/cashflow optimizations to save money, etc...

We don't disagree. You are talking about what makes sense for a business while I am referring to what makes sense for an individual in an environment where growth is what determines your performance review. That is a bit simplified obviously since of course revenue does matter.

My point is that it's exactly this misalignment of what's good for the company vs what's good for individuals working there that is the problem.

Move fast and oooh shiny!

Can the Hacker News community ever forgive Google for shutting down Reader?

Has Google given the Hacker News community enough reasons to forgive them for shutting down Reader, yet?

And this is why public sector or PPP are better than private sector for necessities. You expect access to Internet to be stable like water or electricity, but clearly that wasn't the case with Google.

It's quite a leap from having an ISP (unusually) shut down its service in an area to concluding that internet connectivity should be a public utility. Surely there must be some middle ground.

It's not that big of a leap: Silicon Valley mindset believed it get dive in to physical infrastructure, "disrupt it", and make a difference. Turns out it couldn't, because of the natural monopolies that are already controlled by incumbents with little incentive to do better, and because it turned out physical infrastructure is "harder" than they wanted to deal with.

You can't "disrupt" the power company, and you can't "disrupt" the water company. Not without a giant warchest to rebuild pipes and electric lines. Not without causing a lot of accidental incidental damage. No one actually wants that, they want to pay their utility bills and move on with their lives. It's past time we admitted that internet service is in the same category.

The middle ground, or the average of two positions, isn't necessarily any more correct than either of the two original positions. It's the argument to moderation fallacy.

Cooperatives could be a good middle ground: they have to turn a profit, but the members have their say in how things are run.

It would be like concluding that water is best handled by the private sector because Flint, MI.

It's funny because the best arguments for internet being a utility have nothing to do with government reliability (or lack there of) but rather economic arguments like natural monopolies. It seems like OP just has an anti-market agenda.

Both perspectives support each other. Natural monopolies are one thing, but if you want people to have stable and reliable Internet - as most tech companies do - then it's best when Internet access is not something companies are allowed to differentiate on.

Has this happened anywhere else where the ISP wasn't Google?

It sounds like what they're saying is that they botched the installation and would have to redo it. It seems like the responsible thing to do here is to just go ahead and redo the installation. It can't be worse than the cost to expand to a new city; all they have to do is abandon their current infrastructure and replace it with the new (I know that "all" underestimates, but still).

This is a case where to maintain customer trust, they should suck up the cost themselves rather than trying to recover it through the one-time customer fees that they used at rollout time.

>It can't be worse than the cost to expand to a new city; all they have to do is...

hey guys, i found the project manager!

It can't be worse than the cost to expand to a new city

Google Fiber isn't expanding to new cities.

I'm from Louisville. From my perspective, I never expected to see Google Fiber, and I guess this confirms it. After reading about how all of KC moved so slowly after its initial deployment, it was telling when Fiber rolled out here that it only covered a tiny portion of our downtown-ish areas. This is not a high income area, nor is it all that dense population wise(then again, neither is the city). After the combination of the county/city, Louisville became a physically rather large city. I don't think it would have been worth the investment trying to cover it, and covering only a portion is only bound to irritate people.

That said, I appreciate them in that they really lit a fire under TWC/Charter/Spectrum. If I remember right, we were 10down/3 up when fiber was announced (possibly 30 down, depending on when). Our base is now 200 down.

> This decision has no impact on our operations in any of our other Fiber cities,


Just more evidence you cannot rely on Google for long term product outside of search.

> product outside of search


This is sad news for me even though I'm not from Louisville. Backing out of existing market shows that it's even less likely now for them to enter new ones.

The main issue is that current companies have extreme control over market thanks to regional monopoly. They can squeeze as much of money from users, because there is no real competition. When a new company enters specific region they can up their speeds, lower price (and even operate at a loss in that specific region), so a competitor will have to leave.

The solution to this requires regulation, and basically a law should mandate to allow leasing infrastructure (the fiber) to others, of course not for free, but for a fee.

Currently if we would have great scenario when we would have 10 ISPs available to choose from, we would need 10 fiber cables to every home. That means if someone chooses one, 9 of them are not used, and all the effort to lay the cable and getting permits was just waste of money. If this is expensive to even company that has "unlimited" amounts of money, like Google it's unrealistic to do for everyone else.

Leasing the physical infrastructure is the key to bring competition back in that market.

Seriously. Google messed up REALLY bad on this one. It would really difficult for many customers to come up with a contingency plan under 2 months.

Google should NEVER be trusted with other infrastructure projects going further. Infrastructure isn't some forprofit companies trial-and-error playground, it is about stability and promise to the local residents, it is not about money and certainly not an experiment.

It should've been 6 months to a year. They blew a lot of use trust on this.

They've been doing that a lot.

>>It would really difficult for many customers to come up with a contingency plan under 2 months.

Why? It's just an Internet connection. Customers can call the local cable company and be up and running in a matter of a few days. Might cost a bit more and be somewhat slower, but it's hardly an existential crisis.

At least they tried. Infrastructure such as fiber to the home should be the realm of governments, as everyone can benefit from it.

They tried and blew away my trust in any of such innovative, pseudo-altruist initiative from silicon valley companies. The disruption their action might bring is much more than cancelling their internet service products.

Can't really joke about their words that 'the last 2 months are on us', like it is something worthy of cheering up..

This development implies that one should not rely on Google for anything. Even you're paying them for the service. They are so large they don't have to care about their users/customers.

This is why I'll never use the Google Cloud. Sorry my business does not move as quickly as your decisions.

Although as always they have language giving themselves some wiggle room, most services in GCP would get at least a year of advance notice.

From https://cloud.google.com/terms/ :

7.2 Deprecation Policy. Google will announce if it intends to discontinue or make backwards incompatible changes to the Services specified at the URL in the next sentence. Google will use commercially reasonable efforts to continue to operate those Services versions and features identified at https://cloud.google.com/terms/deprecation without these changes for at least one year after that announcement, unless (as Google determines in its reasonable good faith judgment):

(i) required by law or third party relationship (including if there is a change in applicable law or relationship), or

(ii) doing so could create a security risk or substantial economic or material technical burden.

The above policy is the "Deprecation Policy."

That is a wide enough loophole to drive a truck through. And it would apply to Fiber here as well, fixing the roads and fiber runs in Louisville would be a "substantial economic or material technical burden".

Aka, it costs a lot of money, and they don't want to spend it.

Heck, even the continuation scenario has a major caveat: “commercially reasonable efforts”.

So they can easily decide not to stick with it for a year, or they can decide to continue to provide service without, say, any technical support.

I never wanted to move to Louisville (from Lexington, KY) but fiber was a nice selling point. Now I'm just hoping our own Lexington MetroNet keeps chugging along so I can leave Spectrum when I buy a house.

I'm going to go out on a limb here and conjecture that this isn't a typical Google pullback like with todo list apps. There are significant financial/intellectual capital outlays required for nationwide-scale ISPs, in addition to operations and maintenance expenditures over time. Pulling back is a significant sign Google Fiber has already reached its high-water mark.

Sadly, I think Google's pulling back because it more or less got what it wanted out of the telecoms in terms of network service, repealing net neutrality and regulatory capture increases Google's competitive moats, and because corporate lobbying may be cheaper, more flexible, and have greater returns: https://gizmodo.com/googles-parent-company-spent-more-on-lob...

On that note, here's a how-to on starting your own ISP: https://startyourownisp.com/

"We would need to essentially rebuild our entire network in Louisville to provide the great service that Google Fiber is known for, and that's just not the right business decision for us."

In other words, "we fucked up and so we bailed." Google is like the bad first husband you should have never married in the first place.

Will Google repair the streets? Saw one picture with rubber gaskets coming out and it looked very bad :-(

That's a lot of words to not actually tell you why they're pulling out of Louisville.

"not living up to the high standards we set ourselves" is about as helpful as hearing over public transport intercom "there has been an incident".

As they say, a picture is worth a thousand words.


That's a picture of a road post-fiber install. The groove is where the fiber was placed. That black snaking rubber was what they sealed it with.

Now imagine that, but across their entire service area.


If that's accurate, it's comedic anyone thought that would work.

I live here - that's almost exactly what it looks like outside my kid sister's apartment ;)

For a local news source: https://www.wdrb.com/news/belknap-neighborhood-residents-con...

Here's an article proclaiming that this is how Google would "outbuild their rivals": https://www.techrepublic.com/pictures/photos-how-google-fibe...

I cannot possibly oversell just how terrible the work was here. Yes, that is indeed a fiber optic cable two inches under a city street, covered only by expanding foam rubber. Yes, someone really did think "yeah, this'll be just fine."


I guess they eventually said "screw it, we will just pull out"

Google Fiber has suffered from a lack of imagination, or just good ole' plain market analysis, in figuring out how to roll out fiber to larger areas around their 'fiber cities'. Google Fiber in Austin is limited to a very small area which oddly enough is probably not where most of the residential demand is. Somehow I doubt that they ever asked homeowners what they would be willing to pay for a fiber connection.

What was the problem? When I click the "encountered challenges" link I get a 451: Unavailable due to legal reasons (I am from the EU).

This link makes it sound like their installation work was faulty and they were going to have to redo it all: https://www.wdrb.com/news/google-fiber-announces-plan-to-fix...

Since that was only a few months ago, my guess is they decided to just pull out instead.

The problem was some idiot at Google though burying a fiber optic cable 2 inches below ground was adequate protection.

What's going to happen with all of that now dark fiber sitting in Louisville, I wonder?

A two inch trench was cut on city streets, that fiber added, and then covered by a flexible rubber seal.[1]

I can't imagine anyone would want to touch that "dark fiber". It was in terrible shape six weeks after they put it down, before winter hit the city. The picture on this article looks like it was taken relatively soon after installation [2] - I've seen streets where the rubber seal has expanded and curled up over top of the groove that was cut.

[1]: https://9to5google.com/2019/02/07/google-fiber-closing-in-lo...

[2]: https://www.wdrb.com/news/google-fiber-announces-plan-to-fix...

I guess the city will be responsible to remove the fiber then as they repave roads over time? Or will Google pay for it?

Seems horrible if Google tore up roads to put in Fiber and now they decide to pull out leaving the taxpayers holding the bag.

I wonder if a situation like this was in the contract with the city. Maybe a journalist will FOIA them for a copy, but something other cities should think about when they do public-private partnerships - what happens when either partner doesn't want to be involved anymore.

In the best of dark humor, their installation approach was so ridiculously light touch that it appears almost unnoticeable. I'm far from a traffic engineer, but I've driven over potholes in this city that are deeper than the cuts Google made to bury their fiber, so I can't imagine things will be "that bad" in the long run.

The sober light of day makes it pretty clear that Google was attempting a pilot of a very certain type of installation that failed spectacularly.

Does google fiber not work in cold weather? Seems all the cities are warmer than Louisville.

Not very well.

When it gets cold enough, the water in the ground freezes and expands and basically "squeezes" the fiber. The "1"'s, which are long and skinny, can still slide right through the fiber but the fatter and wider "0"'s sometimes get stuck. This causes the fiber (Internet pipe) to get clogged and, eventually, stopped up completely, similar to how car crashes on the Interstate can quickly bring vehicular traffic to a standstill.

(Source: I was the Senior Network Engineer for a FTTH ISP, until just recently.)

OMG that's hilarious!

I'm sure Google Fiber works fine in cold weather, but fiber optic cables buried in city roads in a two inch trench covered with a flexible rubber gasket have demonstrably poor resiliency to snow, ice, salt trucks and freezing temperatures.

I'm in Kansas City and have Google Fiber. I don't really know how cold Louisville can get but I can tell you that we got pretty gosh darn cold (10-20 degrees below 0F) last week during the Polar Vortex freeze. And we're going to be flirting with those temps again this week. But to answer your question, I have not noticed any disruption or degradation of my service as a result of the cold temperatures.

EDIT: As others have said, I think the problems in Louisville come more from shoddy installations thus making them more susceptible to extreme weather rather than 'fiber not working in cold weather'.

It will be connected into the dark net.

I hope it's biodegradable :-)

Another good example of "smart engineering can't solve every problems".

This is even worse in a way. They specifically caused these problems because they believed their engineers were so smart that they could rely on their novel microtrenching technique to deploy the fiber for a fraction of the cost of the competitors.

I just don't understand why they didn't spend more time testing it on their own resources before rolling out to actual customers in a major metro area. We've all seen potholes, roots coming through streets and sidewalks. Just crazy to think they thought this was ready to be deployed at scale...

It actually seems like the problem was exactly that smart engineering wasn't applied; microtrenching was basically testing expensive (thouhh cheaper than alternatives, if it worked) infrastructure ideas in production without prior testing in sufficiently realistic conditions; it appears to have been an abject failure that left GF in a shutdown or start over position.

"Its not up to our quality standards, so instead of fixing it we're going to pull out". What a corporatespeak way to say you're giving up on your customers.

If I'm not getting an A, then I'm not taking your stupid class.

This decision has no impact on our operations in any of our other Fiber cities, where we continue to sign up and install new customers every day.

This is just irresponsible of Google at this point. This entire project has no momentum and isn't going anywhere -- why are they still signing people up when it's obvious that they'll just cut them off eventually as well? In the meantime, they prevent any of these cities from taking control of their own connectivity.

They should help the cities form nonprofits to run the fiber networks, then gently remove themselves from the situation.

I'm curious how much physical infrastructure Google Fiber is leaving under streets. In Louisville were they using someone else's fiber or will there be actual dark strands under Louisville now?

They're not leaving much fiber under the streets in Louisville, which is the whole reason they're pulling out - they attempted a low cost "microtrenching" strategy where they cut a small groove in the street pavement itself and tried to run the fiber through that. It was pretty much an abject failure and I doubt anyone will want the infrastructure as it's unlikely to function in even a few months.

They're still "microtrenching" in other markets but at a much greater depth (6in rather than <3in) and with a different cover strategy (asphalt replacement over sealant rather than sealant only)

This microtrenching would be amazing if it could make its hands into people wanting to make open municipal networks, if it could actually work.

Apparently from other comments, most of the fiber was really shallow, like a couple inches deep, and poorly insulated so shifts from cold/thaw created a lot of damage and is likely unrecoverable short of re-laying a deeper infrastructure for the entire area.

They were experimenting with real customers, as google often does. Only this time it was what many would consider essential infrastructure.

So what happens to the infrastructure? There's fiber optics in the ground right? Do they give it to a local ISP?

Some of it could be re-used, but from what I understand, lots of fiber could become worthless because of the failure of their micro-trenching "experiment".

I wonder how many of the people saying 'See? Can't trust google with anything' use Gmail accounts.

10 years ago I would have jumped at the chance to sign up for Google Fiber. Today they could not pay me to use the service.

Oh c'mon with the exaggeration. I would still use them (even if I had to pay), it doesn't take that much time to switch. The major issue though will affect most people in Louisville, since it's almost guaranteed that AT&T will increase its prices. Thanks to Google Fiber at least for limited time, people there enjoyed fast and relatively cheap internet access.

TLDR: Google isn't as good at paying off politicians as well as AT&T and other ISPs are.

I fully expect my ATT Fiber to go up in price or down in transfer allotment as a result. What a shame.

ATT offers fiber internet in Louisville. After Google found out they wouldn't be the only provider offering the service, they bailed.

One thing missing from this post: why?

After "Advancing our Amazing Bet", it became clear that the level of regulatory obstacles Google needed to overcome made it too cost-prohibitive to continue their rollouts. But what factors are forcing them to pull out of an already established market?

Is the ISP environment so hostile as to squeeze even the biggest players out of town? Is Google slowly ramping down even existing Fiber operations? What's going on behind the scenes here? If the microtrenching issue is driving this, why can't Google of all companies put up the investment to re-do it right?

There's failing an experiment, and then there's realizing something is sub-par and putting in the work and money to make it right for your customers. I feel like Google has misconstrued the two here.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact