I greatly respect the initiative and scrappy-ness of someone doing this. And the legacy providers are clearly sitting on their monopoly position in a way that makes their pathetic alternative so starkly unattractive.
But isn't it also true that once his network grows above a certain customer base (and gets into the maintenance phase), he will start to see all the effects that eat into being able to do this cheaply?
Namely:
-- customers who don't behave as well or kindly as before
-- customers who need 24 hour customer service
-- maintenance that can't be done himself, and he has to employ people
-- customers and vendors who sue you for breach of contract, or other simply nuisance lawsuits
-- upgrading the network to the next technology requirement, or when he's unable to get 2nd-hand parts so cheaply, etc.
-- or a natural disaster that unexpectedly forces replacement of (and charging for) equipment that wasn't anticipated in the original subscriber price
Maybe none of this rises to the level of making it fundamentally different or unsustainable? But it seems to me the honeymoon phase doesn't last long, and it's got to hit some unavoidable realities soon. At least, if you think you can replicate this, it requires finding people and neighbors who are willing to do actual work and investment/concern to make something like this possible, and not simply pay a vendor a premium to phone it in. It must be treated like a neighbor-to-neighbor community project, not a faceless commercial transaction with its attendant obligations.
I'm going to skate past the fact that difficult customers and maintenance aren't why monopolies are expensive, in fact they're the things that are most amenable to economies of scale, so bigger gets cheaper.
The real question is: why does he have to get larger than the 600 homes in his nearby rural area, ever? Why does his goal have to be to defeat and replace Comcast rather than to supply internet service to his neighbors?
He doesn’t of course. Local/muni/coop last mile is a well worn path. It’s your local volunteer fire department, but for internet, and local self reliance is not a bad thing. It doesn’t have to grow, it doesn’t have to constantly evolve, it just has to work and be reliable. That is what infrastructure does, and when it does so, it’s mostly invisible (and I argue, that is its most beautiful form).
for every small business that "gets by", there are 2 (probably more) that go out of business due to not having grown sufficiently by the time they face some competition.
I think this is a weird framing of the issue. Sure, lots of businesses go under, and maybe being larger would have saved them, but maybe not. Plenty of VC-funded businesses go under precisely because they tried to be too large, when they could have perfectly comfortably served a few satisfied initial clients for enough money to pay all their bills.
I think the idea that companies go under because they aren't ambitious enough says more about modern attitudes towards growth than it does about the reality of business.
When I say growth I mean net profits. Those imploding VC companies were never profitable.
A larger profitable company has more chance of survival by shrinking into a smaller profitable company. It's a buffer. But an already small profitable company doesn't have that option, there's more risk.
> for every small business that "gets by", there are 2 (probably more) that go out of business due to not having grown sufficiently by the time they face some competition.
If your goal is to fight a monopoly, then the mere existence of competition means that you've accomplished your goal.
the same reason one would file for patents without any intent of enforcing them. For defense and security.
I would say that to attempt to have zero growth/shrinkage is difficult in business. The market is always changing, people's preferences change, etc. If you try to stagnate you will likely find yourself shrinking, either because demand changes, or there are mixups in supply (competitors).
If shrinking is the only non-goal, then growth is likely the only prevention since stagnation is hard to ensure.
The reason he exists is because the competition is bad. If the competition is good, he has no reason to exist. The goal is to supply 600 rural households with broadband at a reasonable price, not to own 600 households.
Since we are asking why questions. Why does everyone else have to support them with tax money when it costs 30k to run a wire to a house? If there is no prospect of scaling it further to boost local infrastructure, then they should be footing the bill themselves or use Starlink.
Because we as a society are better when everyone has access to the Internet. Just as we ran electricity and running water to neighborhoods that could never pay back the investment. Just like we build roads, bridges, and tunnels that probably never be afforded by the people they serve.
Maybe the person that lives in that house in the next generation is the next great scientist that discovers how fusion can work reliably for us. If that is the case, then it was worth 30k to link that house to the Internet.
This is the most bizarre comment I’ve seen in a while. You don’t think folks do cost-benefit analysis before building out infrastructure? If so, you are in for quite a surprise when you attend a hearing at your local municipality.
I'm not convinced this is the case. The big thing that makes telco's such profit making machines is that wires in the ground are generally a large capital expense that doesn't really provide a great marketplace for competition. But once you've got that infrastructure, it's hard to duplicate. The rest of the equipment and employees relatively aren't that expensive.
So the power is on the provider here, there isn't really another choice for customers if the article is to be believed, no matter how good or bad the company is. Sure there might be disputes with vendors, but that's just part of any business.
The biggest threat IMO is probably some sort of competition. Maybe a big telco decides to wire up the area, although then they would be the second player in the market trying to steal customers who may not be interested in switching. Or if this really is a rural area, things like wireless last mile (basically LTE), Starlink, OneWeb, etc may start to be more compelling options if they get the capacity, latency, and price point to the right spot to be competitive.
His millions were funded by the government.. and the legacy providers also could bid on the contracts. It’s not clear if he’s expected to pay off those funds or not (I assume not). As the saying goes, the best money is someone else’s.
It seems that the ISP motivation comes from lack of other options. Should a viable competitor emerge, that might be considered a "win" w.r.t rural customers having good broadband choices.
Not sure how Canada compares but these concerns haven't stopped the biggest telecoms in Canada from providing subpar service under very restrictive terms and conditions with no accountability. Namely, a 12 hour complete outage by Rogers to which the reply was basically a big shrug. If they can get away with that I am sure a small independant provider can get away with that as well.
> -- customers who don't behave as well or kindly as before
Easy. Refuse service. You aren't legally obligated to offer your service to assholes. Any business has the right to do or not do business with whoever they want, provided they’re not refusing service for a reason that violates local, state, or federal law.
> -- customers who need 24 hour customer service
Also easy. You are under no obligation to meet peoples unrealistic demands or needs.
> -- maintenance that can't be done himself, and he has to employ people
He already is familiar with third party contracting.
> -- customers and vendors who sue you for breach of contract, or other simply nuisance lawsuits
Frivolous lawsuits are a risk in any business in America.
> -- upgrading the network to the next technology requirement, or when he's unable to get 2nd-hand parts so cheaply, etc.
What is this "next technology requirement"? My area cable company still runs most their network on 30 year old lines.
> -- or a natural disaster that unexpectedly forces replacement of (and charging for) equipment that wasn't anticipated in the original subscriber price
Cost of doing business, doesn't matter the size.
I think people don't understand just how profitable municipal broadband can be. It's why big players spend so much lobbying and bribing so they can keep their established position running and keep the gravy train running, but really the economics of it are fantastic once you've done the initial digging and running the lines, which sounds like he has here.
At $55 /mo for 400 households he's bringing in $22,000 a month plus whatever federal and local government subsidies and grants. The odds of a disaster, or one of the other scenarios you mentioned happening anytime soon is low, so he will have runway to build a decent sized war-chest to be able to easily afford handling any of these scenarios with third party contractors. The more houses he brings on line, the better it gets.
Right, but that's OPs point. If he does what you say, he's no better than Comcast, ignoring customers and telling them to screw themselves at the first sign of trouble.
There’s still a country mile between what gp is suggesting and what Comcast gets away with because of their monopoly position.
Anecdotally, I replaced a router they gave me because it would randomly crap out (probably neighbors using the xfinity Wi-Fi feature I couldn’t turn off), and they kept trying to charge me a monthly rental fee for their router. Every time I would call with confirmation it had been returned, the charge would be removed for just that month and back again the next - this is just the most recent example of a long line of infuriating time wasting schemes I have dealt with from them.
This happened to me as well with Cox Cable in AZ back in the early 2000s. I returned the modem and got the returned receipt. Next 6 months I had to call and get them to reverse the charges. At that point, I started recording all the calls each time I had to call and get the charge reversed. Recorded 5 months of calls, had them transcribed, and sent the transcriptions, recordings, and a copy of the return receipt to the AG’s office saying “I believe Cox is committing fraud, and I wonder how many people they’re doing this to”. Never heard from Cox again. I did actually wonder how many people just continued paying it because “it’s just $5 a month”
I'm with an ISP that is fairly well known for having poor support. I have never had an issue with them. They deal with problems on their end efficiently and without complaint. I would never expect them to deal with a problem on my end, so they never have an excuse to provide me with poor customer service. It all works fairly well, particularly since I am paying about the half the price compared to a major telecom company.
Compare that to a major telecom company. Even if I took the same approach, I would have more issues to deal with (typically issues over billing, rather than technical problems).
I seem to see this a lot especially with American business owners. You don't have to service every customer market. If you offer only certain speeds or certain hours of support, despite being able to support otherwise, that's fine. Not every customers fits your target market.
> Easy. Refuse service. You aren't legally obligated to offer your service to assholes. Any business has the right to do or not do business with whoever they want, provided they’re not refusing service for a reason that violates local, state, or federal law.
Then isn't this a point against the scalability / feasibility of this idea working broadly for others or becoming a model for replacing dumb telcos?
If part of the reason telcos are the way they are is because they have to serve everyone, and at some point if you run a service like this you will run into that requirement, then you will too become like a telco because of those obligations. And this is just one example of a factor that starts to matter.
I try to help out in my HOA of 25 people to manage the utilities, infrastructure, landscaping, and even with this small a group people are uncooperative and 1-2 people are constantly questioning and threatening to sue if we don't do what they say. Hundreds/thousands of people is even more a nightmare.
> threatening to sue if we don't do what they say.
I do love the occasional power trip. I'd look them straight in the face: "here's our lawyers number, have your lawyer give my lawyer a call. Since you seem to be so adamant about suing, you should have no further contact with me. I'll see you to the door." and if they don't go? Arrest them for trespassing.
I'm in a condo here, with an HOA / board, and it was a pain in the ass to get fiber brought in from the local telco. They wasted months sending out letters, waiting for people to give input, votes, etc. until they finally agreed it was a good idea. The telco pays for the whole install: trenching, digging, running fiber between the buildings, etc. That doesn't matter, because you still have people complaining about the utilities messing up their lawn.
It's been over a year now and the project still isn't done. The fiber is right on the street, not even 30 feet from my unit. I'd have paid a couple grand to get my own conduit brought in, if that was an option.
The technical solution would be a QOS that deprioritizes/throttles these people first, with clear wording in the contract. The reality is that these people are a negligible fraction of the users.
>I think people don't understand just how profitable municipal broadband can be.
Operating the network might be profitable. Recouping installation costs are not, when Comcast and other coaxial cable internet providers are sitting there ready to undercut you the second you enter the market. Unfortunately, sufficient customers are not willing to pay more for a reliable symmetric fiber connection yet over whatever the cable company is offering with meager upload.
Also, I assume you mean fiber when you wrote “municipal broadband”. I thought municipal broadband refers to taxpayer funded internet networks, where there would be no profit required (and hence is the only alternative to getting a better internet connection than the cable company).
In Minneapolis there is a local fiber provider which charges about the same for the same level of fiber connectivity. I think it's pretty sustainable.
It looks like his revenue is going to be $50k/mo in not so long and that's more than enough to have a couple of people willing to work on an as-needed hourly rate and to cover whatever issues come up.
Not everything need to scale. A good way to handle this kind of project is keep it at a certain community size, and if people want in, beyond a certain threshold, they need to build their own. This is how federated internet providers work usually.
I, too, greatly respect the scrappy-ness of this individual. Kudos to him for sticking it to Comcast. That said, I'm not wild about the notion of dropping $30K of our collective money on running fiber to a single home out in the country.
I don't see anything wrong with "collectively" deciding that every American citizen should have access to high speed internet access and putting our money where our mouth is. Especially remote areas that are expanding and will become increasingly populated to take advantage of the infrastructure.
For most of us, it doesn't cost anywhere near that much to get access so we can handle the rare costs to build out to remote areas where it's more expensive. That's the benefit of collective money. No one person has to shoulder the burden alone and together we each only chip in a small amount to achieve a massive goal.
America should be heavily investing in building out select remote areas now because we're going to be getting much more crowded in the decades ahead. Climate change is going to force people inland, away from the western US, and cause hundreds of millions of climate refugees from around the planet to seek relocation. The US is going to have to do our part to help take many of them in. MI is a pretty good place to expand.
The United States is not crowded whatsoever. We have no dense cities other than New York. Nobody is forced to live out in such remote areas, not even refugees.
If you choose to live out in the middle of nowhere you’re going to have to pay for all kinds of expensive infrastructure or do without. Why do taxpayers have to cover this particular living expense?
> If you choose to live out in the middle of nowhere you’re going to have to pay for all kinds of expensive infrastructure or do without.
Naturally, people realize that the farther out they get the worse their infrastructure will be. That doesn't mean the US can't or shouldn't set some minimum standards for what's acceptable. Most people think it's pretty reasonable for every American to have access to broadband. Its in everyone's best interest to make sure all Americans can get online. One nice benefit is that people will be able to spread out a little more and still work from home. That means fewer cars on the road, and fewer harmful emissions hurting the environment. Eliminating the need for commutes will be especially helpful in remote areas because their commutes are often longer than average.
I suspect that at a certain point the US will need to start policing where people are allowed to live more aggressively, but rather than trying to get people to move into dense cities to save a little money in the short term I hope it's used to try to limit the number of people living in a given area according to its biocapacity. Encouraging people to leave areas we expect will be hit hardest by frequent flooding, or fires, or droughts would be helpful too. We waste a lot money rebuilding over and over after predictable events. I can't blame the US government for treading lightly though. People aren't used to the idea of being told that they have to leave their homes and move, let alone being told exactly what parts of the country they have to move to. Probably best to start with people legally immigrating. I'm sure a lot of people would jump at the chance to move to the US even if they weren't assigned their preferred location within in.
I have to presume his marketing costs will be close to zero. On tge other hand, in my area (central NJ) both Comcast and Verizon spend a ton on marketing.
He'll also have zero churn. So that's got to help the bottomline.
Finally, I'm willing to bet it helps raise local home prices as those who had to have proper broadband were effectively excluded from that market. The point being, some homes will be able and will to pay more.
Certainly the future will be different, the comparison to traditional ISPs might not be reliable either.
That was my first ISP password assigned to me from San Jose based ix.netcom.com (Also the city I was grounded a month for running a $926 long distance bill calling into BBSs to play trade wars and the pit)
But the best ISP I ever had was a 56K dial-up in Seattle. To play Diablo.
Except, potentially, for locates. In my conversation with one of our local ISPs that for a while was doing fiber builds but then stopped, locates were quite a nuisance for them. This was in a less rural location though.
That's when you call the "Call Before You Dig" number and a bunch of guys with spraypaint show up to mark where the buried gas, electric, phone, fiber... lines are.
Are you saying that Comcast provides decent customer service? because I think it is probably the first or second reason everyone hates them... another one could be the doubling cost yearly unless you call them and are serious about cancelling.
Where I'm at Comcast is very reliable but I've had different experiences.
I recognized his name from providing hosting for the outages.org list[0] – if you haven't subscribed, and you do anything operations at all, go hit the button now.
I'm being a bit lazy here but do you happen to know if there is a way to consume this programatically? I'm thinking RSS or perhaps an API?
Edit: For the benefit of others who might be interested, I've just subscribed using Feedbin's [0] email-to-RSS feature so updates will appear in my RSS reader!
This is a mailing list. Subscribe and point it to something that can ingest messages, similar to how you would pipe support@ to a helpdesk and auto-create tickets.
ASN = Autonomous System Number (https://en.wikipedia.org/wiki/Autonomous_System_Number), it's a number which identifies an ISP in the core Internet routing protocol (BGP). A low ASN usually means your ISP has been part of the Internet for a long time; other than the 16-bit vs 32-bit ASN distinction, it has no practical effect, besides implying that your ISP is one of the "old-timers".
Adding to this, Autonomous Systems, which are identified by ASNs, are the networks that the Internet is internetworking between.
That protocol is called BGP, or border gateway protocol. Most people's familiarity with that initialism, if any, comes from reports of major outages which occur when BGP routing --- effectively the list of peers to which a given AS connects --- gets fuxnored. This happens with somewhat distressing regularity (though not exceptionally high frequency), and along with some other notable failure points in modern telecoms (say, SIM spoofing, DDoS, or good old social engineering) is not-so-charmingly naive in its architecture of implied trust and lack of technical safeguards against either accident or malice.
As originally specified, ASNs ranged to 65,536 distinct systems (16 bits). That's since been bumped up to 32 bits, for 4,294,967,296 distinct systems.
Some old hands would track network abuse by ASN or a somewhat finer gradation, CIDR (classless internet domain routing), which tend to aggregate poorly-behaved networks into identifiable aggregates. That was somewhat more tenable with the smaller number of providers, though power laws and Zipf functions mean that bad behaviour does stil tend to self-organise in useful ways. Growth in indirection (VPNs and Tor) challenge this somewhat, with gateways now being identified as abuse sources, which is ... problematic.
ASN is an Autonomous System Number. An ISP is the primary example of an Autonomous System. There are other organizations that have ASNs like data centers.
The internet is decentralized. Basically, each autonomous system is its own network. This means that they need to connect with one another in order to allow traffic between each other. This is called peering. In order to peer with another network you must have an ASN.
Maybe? Sometimes? It probably depends if what they're doing causes operational issues. BGP is primarily a trust based system. Many upstreams have filters to limit route announcements from their downstream / customer ASes, but at its core, it is trust based.
(Not the guy you replied to, but) it's not, unless I am missing it. ASN 2 is University of Delaware. You can search for yourself at whois.arin.net, just type a number in the search bar in the upper right.
RFC 5575 is a widely adopted specification implemented by router vendors that lets ISPs (think Comcast, Verizon, Deutsche Telekom, Akamai) block certain kinds of traffic at their routers using rules called "Flow Specifications". A rule looks _something_ like "Drop traffic if it's on Port 80 and its packet size is 252 bits". That level of logic is good enough to block many simple DDoS attacks, and since it's done on a router, it's hardware that the ISP has to buy anyway. The more expensive / but also more powerful solution usually involves a dedicated piece of hardware that does packet inspection.
Yeah FPGA's are marketed for packet inspection. Like on xilinx.com, and microsemi.com, they talk about radar and military, defense, on top of AI and fintech. It's just really hard to market FPGA's, it's such a shiny toy but then it never ends up actually selling in volume, like GPUs, there's envy of that success. Especially because in many ways F's have merits that go toe-to-toe with GPU, and defeat them in eg latency, which is why Wall Street prefers F's to GPU's. Just not enough killer apps.
And packet inspection is a good fit for F's [FPGA's] by their very nature, DDoS's are squirrely and ASICs get stale, you need to reprogram you F's on the fly to catch that attack in-progress. So to adapt to new attacks on the fly, or update based on new fashions of DDoS's, patch vulnerabilities, and plus they're harder to reverse-engineer than ASICs, they're strong against that, good crypto to protect the bitstreams that define them. Basically built for that. ASICs on the other hand, can just have the lid scraped, take a photo, done. (Though to some extent they do put functionality on memory that gets lost if the chip is turned off during abduction, that can be done, the line between F's and ASICs is not truly that sharp).
A lot of DDoS's are done by state-sponsored or -affiliated or -harbored adversaries, capturing the ASIC that stops the DDoS is a real thing. Reverse engineering usually happens in another country, another jurisdiction. Under smiling eyes, blind eyes, can't get the police to go there, can't get extradition, maybe sue, maybe get them punished within the country that harbors them.[1]
[1] I read in China there was a Chinese man who traveled to New Zealand and murdered somebody, I think a woman. But he would not be extradited. Instead, the New Zealanders presented their evidence in Chinese court, which found it had merit and credibility enough to imprison the murder, within China, so he paid for his crimes fully. All without extraditing one of their own.
Amidst all the discussion of fpga vs asic vs flowspec, it's probably worth distinguishing two types of attacks: big, dumb volumetric ddos (flow specifications are great and cheap here, if you can match), and more sophisticated layer 5/6/7 attacks where FPGA/packet inspectors are likely necessary (unless you get lucky and the supposedly smart attack has an obvious signature such as a particular packet length combined with other components)
The RFC number is less interesting then the ASN; he has a low ASN, which is for backbone nerds a little like getting a very short domain name; the short ones are long since exhausted, so it's like an O.G. indicator.
(An ASN is a BGP4 network number; think of it as an address in the backbone routing network.)
"I have at least two homes where I have to build a half-mile to get to one house," Mauch said, noting that it will cost "over $30,000 for each of those homes to get served."
That's over $11 per feet. That sounds about right. I paid $18 per feet to have a private fiber optic line of 1000 feet installed at one of my houses (in the US), going down a very long driveway, with 3 patch panels, 2 at each end and one in the middle at a gate. That was just for my LAN, not internet access. I needed the link to hook up intercoms and security cameras. I absolutely wanted 100% reliability of the network link, so wireless solutions wouldn't have been adequate. The previous homeowner had buried a cat5e line in the first 500 feet, with a cat5e repeater (underground), but its electronics failed after a couple years and its exact location couldn't be found. And he had not even put the cable in conduit.
Not only cat6 or cat7 wouldn't have worked at a length of 1000 feet (one would need repeaters, which can and did fail, see my initial post), but it wouldn't have saved much money. The bulk of the cost was in the labor to hand-dig a trench through rocky soil while avoiding multiple buried utilities. Also, cat6 and cat7 would have been slower at 1 Gbps (10 Gbps is impossible at 1000 feet with repeaters). My fiber cable has 12 strands of singlemode fiber and each can achieve 100 Gbps with the right SFP transceiver (1200 Gbps total). Of course I don't need that much bandwidth, but it's a plus to have this potential.
There is basically no reason whatsoever to stick with cat6 or cat7 at such lengths. Fiber beats copper in weight, speed, and reliability.
The whole project was worth it. This allowed me to install video intercoms at two gates, and many security cameras, which is what buyers want at such a property (it's a multi-million dollar estate, and I might want to sell it next year).
If the line were to be damaged somehow, I would need to call my contractors to splice and repair it. But I don't expect it to happen. In fact the line shouldn't need any maintenance over the next 20+ years. Heavy-duty conduit. Heavy-duty weather-proof enclosures for the patch panels. Etc. It's all the same stuff built by the same contractors who build fiber infra for ISPs.
I'm going to put my hand up and say I have absolutely no idea how an ISP works. He runs cables to each house in the area... now where does the other end go?
There is a very good Ars Technica article on how an ISP works. It traces the whole network, from submarine cable through to last mile into a house. It was written in 2016, but I imagine it's still relevant:
Thank you, that was a great link for us uninitiated folks.
Also another great plug for ArsTechnica (even though the main article is them as well, and I'm sure most of this audience is well aware of them) and the excellent technical writing they do.
There are wholesalers that provide "dark fiber", then you buy data services from another "wholesaler". When I looked into it, dark fiber was available through some utilities and through a government funded non-profit. Data to light-up the fiber was available through several different data centers that connected to that dark fiber.
You still had to build-out the last mile though, and thats what will get you. You either need private easements, or be a registered telecom utility to use public utility easements. That last mile is $20k +/-, depending on your circumstances. If your semi-rural or less, there's ROI sucks. Hence, many smaller ISPs are wireless.
At least in area, there are already a number of wISPs, 5G is rolling out, Starlink eventually. and lots of gov't funding going to the big players to expand their networks (and drive the start-ups out of business.)
There some other business models out there too that look interesting. Underline in Co Springs, for example. They provide a basic tier of service, in order to qualify as a telecom, install the fiber and then allow multiple competing ISPs to use their network.
IMHO, any utility that has the benefit of government privilege should be required to allow competors to use the infrastructure that the taxpayers funded.
I'm waiting on one of you brilliant folks to defy the laws of physics to create a decentralized, wireless mesh internet.
These guys have dark fiber right in front of my neighborhood. They service cell sites for Dish Network near me as well. It's interesting to look through their services. For example, you can get fiber service with layer 2, where you're responsible for adding your IP stack over top of it. Or you can buy at layer 3, where Segra is already running a stack, and establish mesh connectivity. So if a fiber is cut, you'll get another working path. Build your network over the top.
Pretty interesting to understand what's available.
Last mile subsidies are super weird. I was looking at a property in montana in the middle of nowhere that had no electricity nearby, but had gigabit fiber. I called the ISP and it was cheaper to get phone+Gb than just Gb due to subsidy rules.
Basically everyone out there (including me) is on starlink now. Turns out the subsidies were not only inefficient, but pretty pointless.
They helped a bit, for a while. Gigabit fiber is less maintenance than electrical power, and it's easier to roll-your-own electrical power than it is to get a Gb connection like how would you do that before Starlink, buy an insane amount of radio spectrum? I heard one HN user who did exactly that in Brazil, got a 20-meter tower to connect by radio to the internet some distance away, and it was a very solid high bandwidth connection. But still much harder than a generator and solar panels, or a tiny little hydropower generator on a stream (a great option in places like Southern Chile, not a joke by any means). Or wind.
I can't find any section of that guide that talks about peering or whatever ISPs are supposed to do to connect to the broader internet. Do you see any step that explains this?
As a small ISP you don't peer - you just buy transit from a bigger ISP. So the basic steps are:
1. Buy a 1G/1G or 10G/10G whatever link to a building you own.
2. Resell that link in parts to customers.
Or you can get yourself into a POP (point of presence) somewhere that multiple providers are also in, and get transit that way. Depends on where you are and what you can get access to.
As a small ISP you definitely can peer and many do, you just aren’t going to get settlement-free peering with any of the big eyeball networks like Comcast.
Something like Seattle IX is a good example of where lots of peering sessions could be established (although I haven’t looked at Jared’s ASN in any detail to see where it’s present).
Any traffic you’re able to offload via peering you wouldn’t be paying an IP transit to haul, so it’s worth seeing if networks like Netflix are on the Route Servers (https://www.ams-ix.net/ams/documentation/ams-ix-route-server...) at any IX nearby your network, seeing if you can negotiate a session over the IX even if they don’t participate in the RS, or seeing if you can do PNI (sling a cable between your networks in a facility you’re both located in).
Wait. The poster above said in point 1 to buy a line,1G, 10g depending on your upstream seller. Why do you need peering then?
If I have 1Gbps line for example and 10 users each are using equal amount 100% of time, it shouldn't matter they send the data to Alaska or Russia or Australia ? Or does it?
You don't "need" peering but it offloads your upstream (transit) links, which are generally much more expensive. In the old days, I worked for couple ISPs and we typically had 3 or 4 upstreams (generally UUNet, Sprint, MCI...) This was back when a T1 was still considered fast.
> Any traffic you’re able to offload via peering you wouldn’t be paying an IP transit to haul
When you're small enough, the difference in price between transit and what it takes to get you to an IX is likely to be pretty small. But, you probably want to be at an IX sooner or later anyway (easier to get multiple transit offers at an IX than on the side of the road), so might as well peer while you're there.
Yes, it can be pretty simple.
Back in the day when DSL and comcast were the options and all of the connections were things like UP TO 5 or even 20 Mbps, but speeds were rarely that - I paid for a dedicated 2Mbs up and down ($180/month) with no restrictions on use and started sharing/reselling it to others in my apartment building, not with wireless, but with cat5 out the window, up the gutter, back inside, etc.
Across the parking lot another guy was sharing his comcast with another building - but comcast was starting to be so slow they couldn't use it. We merged our empires by stringing some cat 5 across the parking lot, around a pole and to his place. Later we added more nearby buildings, all wired until we had 5 buildings and about 20 "subscribers".
Even with 2Mbps, everyone on the network was happier with a guaranteed speed than their flaky "up to" speeds they used to have.
Did I run an ISP? I had subscribers, had to maintain a network, had a proxy server to reduce requests out of the network, had to deal with abuse and collect money - so I'd say yes, a small one, but yes.
Taxes? I was a college student, so I didn't make enough to owe any taxes, but it was mostly cash. I wouldn't have done it if I had to hop through the business hoops.
You definitely can (and should) peer as a small ISP, even if you are buying transit from other providers. This is especially true if you're running an MPLS headend as you'll still have choke points at L2 circuits in your own network. Owning your own peering can be a great way to offload traffic to other circuits that share destinations, most commonly traffic destined for VOD/streaming CDNs.
(N.B. — This is what has worked well for the WISP I cofounded, but YMMV depending on headend infra).
So they're leasing ("buying"?) fiber from the same ISPs they're trying to displace and relying on that payment to provide them with continued internet access? This doesn't sound like a real first-class ISP, but something akin to an MVNO where they're at the mercy of the same companies they're competing with. I get the initial sale might seem fine, and the established ISPs might be fine with this as long as the company is small, but why wouldn't these companies shut them off (or raise the prices, etc.) when they grow too big to become dangerous?
You're misunderstanding this market. There's a wholesale market, which he is buying from. There's a retail market which he is selling into. Some providers service both the wholesale and retail markets, but typically with different divisions, people, tech, resources. It's like saying that if you build a gas station and buy your gas from Exxon then that's bad because Exxon also operates gas stations. It's not like an MVNO where all you're doing is sending the customer a bill, and provisioning API requests to Verizon.
> You're misunderstanding this market. There's a wholesale market, which he is buying from. There's a retail market which he is selling into. Some providers service both the wholesale and retail markets, but typically with different divisions, people, tech, resources.
The difference in divisions/people/tech/resources doesn't explain anything for me. They're both the same company with the same CEO, whether it's one business unit or a dozen. It's not like the executives are oblivious to how much money each unit is making and whether another unit could make more in place of it. If you're the CEO and see you could charge twice as much by doing retail instead of wholesale then you'd obviously try to do that.
Rather, the explanations I'm getting from the other comments seems to be that (a) regulators require some kind of reasonable wholesale to exist to third parties, (b) the big ISPs aren't planning to serve those markets anyway, so they're not missing out on any income by taking money from the last-mile ISPs. And as long as those last-mile ISPs don't try to compete for the same customers then they're fine.
> The difference in divisions/people/tech/resources doesn't explain anything for me. They're both the same company with the same CEO, whether it's one business unit or a dozen.
Then you've not worked in large B2B companies before. Eg Apple pays Google money and Google pays money to Apple, any perceived public rivalry goes out the window as far as business between the two is concerned.
If you're the CEO of Comcast, you've never even heard of this small time ISP, you have far bigger things to spend your time on, and the "upstream" business unit of Comcast really doesn't care what you're doing, so long as your money's green. It's all business. See also: Netflix using AWS despite Amazon having a streaming video service of their own.
Because it's all business to them, and if they did it overtly they could get sued.
But also because once you're in a single location, you can pretty easily get multiple providers to that location for a Price, so there's really no point. Even small rural towns usually have multiple internet connections from different companies, and if they don't you can pay to run fiber if you really wanted to.
People find it hard to believe, but Comcast et al are actually businesses, not Satan's marketing department; and they happily take money even from "competitors".
Comcast would much rather sell a dedicated fiber to a business with capital and guarantees.
Selling to the individual consumer doesn't make a lot of sense business-wise, because of the deployment costs and continued support costs.
Comcast is also abusing their status as oligopoly to gouge costumers financially and qos-wise, but if they're selling to a business that buys large quantities and has staff who's job it is to handle network problems, they actually like that (right up until that business threatens to compete with them in areas where said oligopoly is in place, of course)
In this guide's case yes you will be akin to an MVNO, you won't peer but just buying transit traffic. That's why most of these guides are also focusing on making the network wireless only (easier to build infrastructure)
You can lease fibre/lambda/L2 transport to an IXP (and there peer with other local ISPs and get global transit from Tier 1 providers) from many companies that don't even have any residential offering.
Or if (technically/financially/legally) possible, even run your own fibre to a PoP housing an IXP on your own.
Once you're in multiple PoPs and on multiple IXPs and with multiple upstreams/peers you're pretty much independent from the whims of a single ISP.
Because there are different business units in the upstream company handling the dedicated access vs consumer sides. The dedicated business side have their own sales goals and if you compete with the consumer side, that’s not a problem for them. I’m sure there are some regulatory/anti competitive measures at play here too, but economically, the two sides of the business will act more or less independently.
He's not trying to displace the majors. In rural areas, owning and maintaining a bunch of fiber to service less than a thousand customers isn't a business Comcast really wants to be in unless they get paid a ton for it.
Well there three tiers of ISPs, each one buying service from the one above them. It's ISPs all the way down, and the higher up you go the more expensive the hardware to run it gets.
At the T1 level it's more completely a mesh type setup, but even lower tier ISPs might set up peering agreements to bypass their main higher tier ISP where it makes sense for cost or service quality reasons. Or refuse to to extract more money as in the comcast vs level1 disputes over netflix traffic a while back
gns3 perhaps? I haven't had that setup for a while but I loved it. I had my whole small ISP in it at some point to work as a test/lab env for testing things out. It's a trick to get going but was kinda fun. I took a copy of that when I left and every now and then I fire it up and mess around with my old dsl/dialup ISP from back in the day.
As the other commenters have pointed out, a possibility is simply to "resell" transit from other providers. However, on the Internet all peering networks are somewhat equal and it's entirely possible to extend the "other end" over time to establish dedicated peering with other networks, so that for example traffic from your network to Youtube doesn't have to go through (paid-for) 3rd parties.
There's good chances there are Internet eXchange Points around where you live where for a small maintenance fee anyone can come and place their router and cables to interconnect with others.
So the likely steps are:
1) Find a transit provider, that will serve your trafic to any other network, and where to connect with this provider
2) (Optional) If you don't have the necessary infrastructure, find another provider to get from your last-mile network to your transit provider
3) (Optional) Find other networks to peer with so that you can significantly reduce your transit bill and provide better routes (therefore better service)
Some non-profit ISPs take the problem from the other side, and build a core network without necessarily owning any last-mile infrastructure, which is leased from other operators (opérateurs de collecte) with whom they interconnect at some datacenter/IXP. The most famous example of that in France is FDN.fr which has been operating since early 90s. That approach is more cost-effective in high-density area where the local infrastructure is already quite good, and construction jobs to lay new cables is very costly, but will still set you back 10-30€/month/line.
I think you more or less just buy connections from bigger ISPs, so for example you get a 100 Gbps connection to one location and distribute it to your end users from there.
Most of the equipment you can buy, you can even get a lot of the needed things as a service. You just need to organize all those hardware and software things, and get the economic and legal part right too. And in the end it needs to tie together in a way, that your earnings are bigger then your expenses.
I think it’s not so different to opening a car repair shop for example. Just more nerdy.
Depending on how close they are he could run cables (ethernet) or fiber. Single mode fiber can go 10km according to some Ubiquiti spec sheets I found on google. Ubiquiti also sells AirMax products that can do PTP or PTMP over the air, although some will be affected by rain. They could even rent space from a radio/cell tower. There are probably a decent amount of other products out there I am only familiar with Ubiquiti.
At this distance you would want a good repeater at about half point instead. Don't forget that data has to travel back and other side might not have such a strong signal
The costs for internet in the US still surprises me, how on earth can it be so expensive?! I understand some countries, but in the US, it seems high costs are because "because we can", not because it has to be like that.
In comparison, you get 1 Gbps symmetric fiber connection in most countries in Europe for under ~$30/month. In some, you even get it for under $10/month (like Romania, which has surprisingly awesome internet infrastructure).
>The costs for internet in the US still surprises me, how on earth can it be so expensive?!
Monopolies and regulatory capture. I can't get ANY wired ISP where I'm at. Even AT&T ADSL which was like .5Mbps and ~50% packet loss terminated service to our neighborhood, saying the copper is too degraded. Comcast, for some reason, told us that to wire the entire neighborhood would cost them $73000 dollars, but they won't do it. That was 3 years ago. I'd have paid them 4000 dollars since then for business gigabit by now. I have been kicked off of multiple MVNO's (not for my abuse, but because AT&T/Verizon terminated their ability to sell SIMs for modem use).
My only current option is T-Mobile's home internet service (via LTE/5g), which works well most of the time but has some pretty ridiculous outages at least once a week. I gave Elon my 100 bucks years ago when they said we'd have starlink available by EOY 2021. They're now saying Q3 2023.
Comcast has 189,000 employees who make US salaries. It costs a lot less to dig a trench in Romania than in Seattle.
You can look at the profit margins. 11.3% for Comcast as of June 2022. That tells me they aren't simply collecting the difference between US and Romanian internet prices in profit.
Of course, far be it from me to defend Comcast, but this is basically just the concept of purchasing power parity (PPP)
Costs of deployment and profit margins have nothing to do with it. The US public has subsidized the cost of broadband internet deployment since the very birth of the internet, and continues to do so like clockwork every few years. Private ISPs continue to caress the books to make it seem like they're barely operating at a profit and still need more, without ever having delivered on the last promise. Taxpayers have paid for fiber to every home a few times over at this point.
The thing I can't really understand if this is the argument is where the money is actually going? With an on paper 11% profit margin it's certainly isn't shareholders, and even if the executives rake it would still be a blip in their total revenue.
Share buybacks and acquisitions come from post profit money, not pre profit, so they won't impact the margin. They're not a line item in operating expenses.
Bonuses and other remuneration will, but no executives are getting $5 billion annual bonuses there and ~$million bonuses won't move the needle.
Good question. I briefly looked into it and it seems they do break out the numbers for cable communications division (as well as media and entertainment) but I couldn't find a profit margin figure without opening the whole 10K and my calculator. Worth noting that the great majority of their business is cable communications.
However, Charter Communications is a competitor that is more of a pure play and their margin is 10.8%
On the more measurable side I would imagine the cost of lines correlates with population density. Running wires to 100 single family homes is way more expensive than running the wires to a district of apartment buildings
Depends on how old the apartment buildings are. If the apartment buildings themselves are already wired with fiber (or really good, recent coax) it might be a lot cheaper running a single bundle of wires to services the building. (Keep in mind that ideally you still have one fiber wire per apartment to sell the highest speeds to each apartment, so you aren't necessarily saving on number of cables for 100 apartments versus 100 detached single-family homes.)
Of course, the older the buildings are the more expensive it gets. Running a new line into a single family home is usually a single new hole from the local utility trench or utility pole, which often have existing rights of way and known contact points to do utility work. Running new lines in an apartment complex often requires opening walls and ceilings between, among, and inside units, which then consequentially means doing new drywall and repainting (and maybe high costs to color match historic paints). If the apartments are condos there's even more complex rights of way issues in needing to get the consent of individual unit owners for some of the work.
To be honest, I only have second-hand experience with running Internet lines in a bunch of Soviet-era apartment blocks. On a lot of building designs there's typically a drop going through all floors that exposes the electricity meters in a common area of the stairwell. The cable would go in either through the underground utility way - most likely electricity or heat lines (central heating FTW) or by air from a neighboring house. There would be a switch in the attic where the connections from apartments would terminate.
As much as I hate the high price where I live (Canada) I assume that Internet and wireless phone service is expensive because the country is so large that the build out cost is expensive. The USA is running 3/4 in the list of largest countries by land area and Canada is 2nd[1]. Maybe I'm naive in my thinking but I have family in a teeny tiny European country and they all have 1Gb fibre optic service for cheap-cheap.
>because the country is so large that the build out cost is expensive
Nah, that's just their excuse; most of the country's population lives in urban areas and they don't even bother running fiber or setting up cell towers in more rural areas aside from maybe along the main highways.
Remember, SaskTel (and MTS, before the government sold it to Bell) doesn't have a problem with charging reasonable rates or building out fiber (and turning a profit at the same time) and those are the lowest-density parts of the country. So no, the telcos aren't telling the truth.
I pay ~55$ a month for gigabit in the US. It’s that there are so many different states with different regulations that means we have both extremes generally.
Comcast charges $100/mo for 1Gbps where I'm at in a suburb of Salt Lake City. Our city announced a partnership with Google Fiber that will begin rolling out in 6-8 months. After that happened, I've started getting Comcast adverts to sign a 2 year contract...I also expect to see their prices start dropping soon.
> charges me a few bucks more per month, for 150mbps
And, in my experience, they will slowly ratchet up the cost until you call in and complain or change your plan, so a negotiated 80 dollars slowly can become 160+
I only can use Windstream as the other providers are right on the edge of my area and refuse to move in. I only get “50Mbps” (It's never gone above 45) for $90+ a month, and they have been forever increasing it because well, what choice do we have.
I use a smaller ISP in Washington state and my 1G symmetrical line just went from $79 to $59 a month and they increased my upload, it used to not be symmetrical.
This is kind of an interesting illustration of how little people know about how the internet works, and how news is ultimately entertainment.
Full respect to the man in the article for the hard work and initiative he took in starting a small independent ISP, but this story is the story of thousands of small ISPs in the US and many more around the world.
In a basic sense, this story is not "newsworthy" since there is nothing new about it. It's more of a human interest piece, like if the reporter wrote a story about the lady who started a coffee shop after being overcharged for a Frappuccino.
I'm guessing this ISP has gotten more attention here and on Ars Technica than others because the founder is fluent in the software engineering world, as well as having started an ISP. Ironically there is a pretty big gulf between the world of techies who know how to write the code on the internet and the people who actually build the internet who are more blue collar.
One of my coworkers also did this but went the cell tower route. Had no idea you could just install a cell tower without mountains of red tape and huge expense but hey. Then all his "customers" (i.e neighbors) have antennas on their house pointed right at it and boom, internet. He only had to front the cost of getting the lines run to one location.
There are extremely competent programmers (10x) like there are outstanding players in sports and music. They do have an outsized impact on the projects they work on. However, they are also extremely rare. The problem, IMHO, comes from cult-startups where they think they can (a) identify these people in an interview (b) build a team of only 10x programmers.
This results in (c) calling a whole lot of average programmers they hired as 10x programmers because of (a). After all, they are smart and their interview process is infallible.
So, if you meet one of those rare folks, enjoy the intellectual banter :).
Then good luck hiring a well sized team when you’ve set the expectation that everyone needs to be a genius to contribute. A successful startup needs to either attract only the best engineers or build itself so that most of the work can be done by merely good engineers following the company’s engineering culture.
10x engineers are a myth when it comes to productivity working within a team. There are absolutely 10x engineers when they're working on a project more or less completely solo.
Yeah, but that's also different from how people, especially management, tend to conceptualize 10x engineers. You don't spot the 10x engineer by looking for the one who accomplishes 10x what other engineers do. You spot them by looking for the team that's accomplishing 5x what other teams are, and then finding the "glue" person on that team.
When I look at 10x engineers who look like 10x engineers what I typically find instead is a 3x engineer leaving a path of destruction behind them. If you give everyone else impostor syndrome and difficult processes they slow down, and you look better than you are. Than you deserve.
The real heroes are the ones who make everyone else look better. But some managers only figure out who that is when they quit or when the business lays off the wrong guy because Steve produces less than Sarah, but that’s because Steve is helping people all the time, including Sarah.
I have met 10x engineers. They solve a problem in an hour that takes me all day and which someone else might never be able to solve. They identify and solve problems I couldn't even begin to tackle. In that sense, they're not really 10x but qualitatively superior.
A+ comment. I've been hearing this idea that "there is no such thing as a 10x engineer" for almost a decade now and from the very first moment I heard it I considered it one of the most definitively untrue ideas circulating in the tech industry. In fact, there are 100x engineers.
Most the criticisms of the "10x engineer" thing I've seen were more about this expectation that everyone can be 10x, when they're more the exception than the rule. Your average programmer is just that: average.
The reason people say it's a myth is because the study that purported to identify this concept was found to have an extremely small population and confounding factors. In addition if I remember correctly it tried to do this identification by using a contrived programming problem.
There are obviously software devs who are more productive than the average. This is true of every skill. The myth is thinking that (a) companies can somehow identify these people in advance, and (b) it is better to prioritize building a team with these supposed rock stars than it is to build a team of potentially average developers who know how to work together, and then properly manage, support & motivate them. A team of ten properly supported 1.5x programmers will beat out one 10x programmer every time. And in many cases the "I'm a 10x dev" personality type does not play well with others.
I'm a firm believer that any genuinely interested, motivated and at least mildly intelligent dev can be made highly productive by finding the right fit. It's far more important for companies to focus on fit and on ensuring that their own managers actually know how to manage than on trying to tap into a hidden stream of 10x devs.
I guess it boils down to the fact that I think many companies absolve themselves and their mgmt team of blame for poor performance by saying "well we just haven't been able to identify 10x devs yet." They expect to be able to hire a single employee who will save the day for them, rather than hiring and training good mgmt.
First, the "I'm a 10x dev" personality type is not a 10x dev. Arrogance is a sign of insecurity.
Second, I don't think a team of ten 1.5x programmers will beat out a 10x programmer. You either have the depth of understanding and imagination or you don't. Take Linus Torvalds, for instance -- I would say he is a 100x programmer, or perhaps a 10,000x programmer, since he is the author of both Git and Linux -- good luck trying to replicate that contribution with a "well managed team". It is similar in many areas -- 10 guys with Math PhDs do not make one Einstein.
In the context of hiring for a business that is developing a CRUD app, you're usually trying to differentiate between 1x programmers and 0.1x programmers, however -- 10x programmers aren't often looking for work.
Pretty much, it's not about grinding out assembly line CRUD work but vision. Domas is another good example if you ever watch his Black Hat presentations on x86 backdoors he's able to approach topics that are not just technically challenging but in a manner that simply would not be attainable for many.
The irony is most companies doing routine CRUD/simple business apps probably shouldn't hire such people as it's a waste and likely causes bad outcomes and perpetuates the stereotype.
I’ve met people other people called 10x engineers. Once you looked soberly at the development process that illusion has faded every time.
Part of the problem with the myth is that as originally formulated it’s meant to be between your worst and best engineer, and whoever came up with that idea is an idiot, inattentive, sheltered, or all three.
Why? Because the worst engineers help the team by calling in sick. They have negative outcomes all the time, which means everyone else in the team is infinity times as productive.
What the rest of us think is 10x versus an adequate developer, and there are almost none of those. Are there people who can work solo and produce as much as a team of 10? Sure, but that’s because of the communication overhead. Can that person join a team of ten and double their output? Only if they are a unicorn among unicorns. The easiest way to double the output of a team is to double the output of the team members. And that doesn’t make you look more productive than them. If you’re not very careful it makes you look less productive.
I've long felt that there's a relatively simple formula for productivity:
Productivity = (Time * Effort)^Talent
People like Buckminster Fuller come to mind. Especially because of this quote of his:
>“We should do away with the absolutely specious notion that everybody has to earn a living. It is a fact today that one in ten thousand of us can make a technological breakthrough capable of supporting all the rest. The youth of today are absolutely right in recognizing this nonsense of earning a living. We keep inventing jobs because of this false idea that everybody has to be employed at some kind of drudgery because, according to Malthusian Darwinian theory he must justify his right to exist. So we have inspectors of inspectors and people making instruments for inspectors to inspect inspectors. The true business of people should be to go back to school and think about whatever it was they were thinking about before somebody came along and told them they had to earn a living.”
That also depends on the X, from my experience working at FAANGs, startups, etc... I have never seen a 10x engineer in good teams, I have only seen "10x engineers" on teams without great engineers. The comparison with sports and music is pretty silly, as those are environment where the winner(s) take all (there can only be one Billie Eillish (lol) even tho there are many singers who are better), engineering is often a team effort. In the other hand, the best engineers I have seen, just spend more time than anybody else working on a problem, and often are the ones who like to show off more, and very often lack the skills in other areas of life.
I’ve seen too many prolific engineers who destroy the confidence and productivity of people around them. These are not people you want to aspire to be.
Yeah, I also think is way more common in smaller companies. Also extreme toxicity around tools, editors, .... stuff like "10x engineers" only use Vim (the classic "i dont need a mouse, thats for normies"), the command line, Arch Linux, etc... In bigger companies, with talented engineer, no single engineer can claim to be the best/10x-er or any nonsense like that, because whatever he does, can also be done(and often improved) by the team. That's why I said that depends on the X. You can certainly have a 10Y engineer where Y = 1/10*X;
If we get to expand the definition from a software engineer on a team to a business founder, do we also get to call the fiber optics 10X engineers? Is a truck driver delivering laptops a 10X engineer?
It's a coping mechanism like lying on the couch watching the Olympics and getting angry that some people are able to push themselves to incredible feats instead of being happy for them.
Never understood that mindset, when I see 100x engineering feats like TempleOS or αcτµαlly pδrταblε εxεcµταblεs it inspires me to learn more and think outside the box.
>"I have at least two homes where I have to build a half-mile to get to one house," Mauch said, noting that it will cost "over $30,000 for each of those homes to get served."
is this really a valuable use of taxpayer money? sending a wireless link over a half-mile isn't that difficult, surely there's a better way to spend $60k of public money than delivering internet service to two families. especially now that starlink exists.
i'm all in favour of scrappy upstart ISPs, but this just seems wasteful.
Especially since he's burying the lede about the people he's servicing--its true 'in general' that the area is lower income, but most of the homes he's serving will be millionaires.
Washtenaw is certainly an affluent county, but not that rich. I live in another part of the county and used to work in Mauch's part of it; it's by no means mostly millionaires. I think the simplicity of the project's goal -- every household served, no exceptions -- is one of the reasons that it got off the ground.
without knowing specifics, that's kinda what i assumed. i see the same thing near where i live - the "low income rural areas" that get infrastructure subsidies is cottage country or areas with a lot of retirees who've moved to the country.
the actual low income people living in rural areas have to move to a suburb or a trailer park if their region isn't serviced by a utility, they don't get assistance like this.
If utilities are underground, it can be pretty expensive to install anything. I have an estimate for municipal fiber that's about that much to get fiber a mile or two down the street overhead, and then about that much to go down my driveway underground 400 feet.
It's hard to justify when the local phone company is probably going to roll out fiber in the next few years without a direct charge, at least for the portion on the street. Of course, that'll probably be PPPoE, maybe asymetrical, likely limited to 1G, etc. Comcast won't even quote me to come down my driveway, even though they serve my neighbor across the street from the pole at the corner of my driveway.
Wow, have you considered buying some conduit and renting a trenching machine and giving the driveway portion a go yourself? Might be worth talking over that option with the muni fiber people. Though sounds like the overhead portion would still be $$$.
A friend did this at his farm in central VA, but for power line instead of fiber. It was previously above ground, unsightly, and occasionally damaged by trees. He dug the trench from the road and had the power company lay the wire in it and make connections at each end. I don't remember the total numbers, but he saved thousands by doing the dig himself (with rented equipment) vs paying the power company to manage it.
Of course, this assume you're comfortable with heavy machinery and can work around other utilities (most counties have a "Miss Utility" service that will mark existing services).
Yep, depending on the lay of the land doing the grunt work can save thousands or tens of thousands - if you have the company do it they'll almost certainly bring in a crew of 4 or 5 with a underground "hog" machine that's supposed to work perfectly but doesn't actually so the backhoe appears and then they cut into a buried utility line that was marked but backhoes can't read and then you wait for the power company to come out and then they fight over whose fault it was while the freezer slowly drips onto your floor.
Or you can rent a ditch witch and do it yourself and dig by hand near anything remotely marked by the marking crew.
I've considered it, but I suspect it might end up like bombcar's experience with cutting buried lines. I'm not much of a digger for the manual work either. And there's a seasonal creek to cross which seems like a lot of fun.
I'm not too worried about the overhead portion; in theory, I could group with neighbors and we all pay a share, or I could pay it and consider it a goodwill gesture to my neighbors; they wouldn't need to pay that portion if they wanted to get online (and some of them have overhead drops for electric and what not, so they'd be able to get a cheap drop for fiber, too)
Well, you'd probably start by getting Miss Utility to come out and mark your lines. I think bombcar's point was that you care a lot more about not cutting those than any contractor will, and so you're less likely to do it.
It's funny because he said one of the houses needed 0.5 miles of cable. My jaw dropped when he said it would only be $30K for that.
I'm speaking as someone who has had a few hundred foot trenches dug in my yard for running cable. Extrapolating it to 0.5 miles would come out to a lot more than $30K.
What's the expensive part of a new fibre run? With $30k you could hire an excavator and operator for maybe 15 to 20 weeks straight, but I'm guessing the pits are expensive and dealing with obstacles is hard.
I don't know. I didn't do it. I just know how much money came out of my wallet and how long the trench was. :-)
So that means I paid for labor. But presumably some part of that $30K will be going to labor as well.
Another possibility is that when you get to the scale of 0.5 miles, you start using different tactics or machines than the small little backhoe loader that the guy used in our yard. So, more capital required but overall more efficient.
Anyway, I don't mean to try and offer an accurate accounting of all of this. I mostly just meant to provide a counter-expectation.
There are fixed costs to a job. It doesn't cost much more to dig a bit longer trench. Things like needing to do horizontal boring to cross an intersection would jack up the cost though.
e.g. I used to pay ~$2k for a contractor to come to re-gravel my driveway. Now I own my own excavator and loader and dump trailer it costs me about $200 (plus my time plus equipment depreciation).
Surely with utility plans you can just use a mole? Dig a few trenches and just use a mole to go between them. No need to dig the entire length. I'm pretty sure this is what utility companies use in the Uk if they can't drag the utility through the existing duct/pipe. Imaging installing fibre to a neighbour and having to dig up every single pavement/road to do this.
And you dig smaller trenches with them. Microtrenching digs a foot deep and two inch wide hole for direct bury fiber cable, saving time and money over older techniques.
Only a foot deep? My sprinkler system is deeper than that.
ATT recently did fiber by my house with some kind of machine. It did some kind of U shaped trench, where it drilled down (not sure how deep), then over about 200 feet, and back up. So you only see a hole every 200 or so feet, vs a solid trench. Let them go under driveways and all of that.
A team of 4 guys was able to do my entire neighborhood in a day. Still waiting for ATT to actually wire up the fiber, until them I am stuck with comcast cable (which is fine ,except the data cap doesn't scale with speed, so the faster connections cap cap you out in like 15 minutes).
It isn't, but that's the norm for all internet infrastructure, both last-mile and backbone.
Since time immemorial, the gap between the amortized cost of building it, and anyone's willingness to pay for transport or transit, has been a) huge (that is, commercially insurmountable), and b) traditionally covered by one of two means:
1. Government subsidy, or
2. Attempting to offer services at the high prices necessary to recoup the investment, consequently going bust due to low volumes, selling the infrastructure for a pittance in a fire sale, and the next owner gets to offer services for prices the market is willing to tolerate. With this approach, it merely remains to find some VCs to sucker for the build phase.
It was also possible, back in the day, to run tunnels across your peers since they would announce the IXP networks at each end into their IGP, but folks got wise to that scam.
There is a variation on (2) involving anti-trust laws during M&A but it amounts to the same thing.
Depends on the area. Wireless won't work well in the mountains, and I assume weather could affect some wireless technologies as well.
I live in a mountainous area and we have a local ISP that provides fiber to our entire county. Which is weird, because I recently lived in a major city and couldn't get fiber.
It's way easier to push fiber through the ground in rural areas where there's basically nothing than it is in major cities where there are tons of things and already some form of wired internet.
And if you're within a mile of the destination, that last mile isn't actually that terribly expensive, especially if it's literally rural and that mile is on the property owner's land. They can figure out how to get to the box at the road.
Agreed - that much money could put in a computer lab in a local library for everyone to use. I’m very supportive of rural people and the life they choose to live, but you are right - they should understand the drawbacks.
Its more than he personally was willing to pay ;-)
>Comcast once told him it would charge $50,000 to extend its cable network to his house—and that he would have gone with Comcast if they only wanted $10,000.
Im guessing being a nerd working at akamai he wont be the one spending ~1-2 days on a Ditch Witch/trencher to make those. He probably wont even hire anyone to work a rental from United Rentals. He will subcontract to same company that does trenches for Comcast.
This is literally how societies function - you contribute a small amount to the general pool, you use small amounts from the general pool. In some cases bigger chunks go for bigger works like ISPs or bridges. I certainly hope you don't want a world where every road, bridge and traffic light is independently owned.
Sounds more like how crony politics for personal gain works. Alternatively, you could finance the hospital, fire depart, or whatever without an middle man siphoning off "their fair share"
So if you want to see this theory in action go to developing countries with an elite ruling class where they don't disperse funds to social works and see how nice it is, behold their lame GDP, etc.
The country I live in SE Asia is a good example. It's quite libertarian out here and yeah being able to pay for private hospitals is nice, but generally speaking your quality of life is lower, quality of goods is lower, average person is less educated, traffic is a crippling problem due to poor planning, it goes on and on. And despite labor being super cheap, roads are a mess, sidewalks are few and far between and if you do get one it's crowded with junk.. Only 10% of the country pays taxes, the inequality with the rich is massive, and if you're not in the top 1% you're basically a poor.
I recommend everyone in a rich english speaking country spending at least a year or two living in a developing country to get some perspective
Coordination games and public goods games (which arguably model insurance) work best when people don’t adversely self-select, but coordinate around the social optimum (for insurance, when the risk pool is as large as possible). Whatever can orchestrate such coordination adds value. If people do it on their own, great, but some problems have characteristics like time horizons such that the coordination doesn’t happen without an authority. Yes, this brings in other public choice problems, but the trade-off is not necessarily bad.
Alternatively, you could finance the hospital, fire depart, or whatever without an middle man siphoning off "their fair share"
This has already been tried. People used to subscribe to fire service, or ambulance service. It doesn't work, and is also bad for society.
If you want people to only use the things they directly pay for, and not pay for shared things through taxes, then only drive on your own driveway. Don't drive on any roads outside of your cul-de-sac. Don't get your Amazon order delivered on state and federally-funded highways. Don't fly out of any big airport in America. Don't fly on any commercial airline, since they have all received taxpayer bailouts in the past. Don't use a bank. Don't use money. Hire a security guard to protect your property, and another one to follow you around every day. Get your water from a well on your own property.
For an 88-day-old account to be this stunningly obtuse, I'm going with "troll," rather than "genuinely completely oblivious to how the world works."
I've lived in areas with volunteer fire departments that paid for their operations primarily with "fire dues" for most of my life. As far as I know, most volunteer departments operate like that.
I had no idea they didn't work. I wonder if anyone has told them?
Subscription fire departments and volunteer fire departments aren't the same thing. I know all about volunteer fire departments, and have worked with them in the past.
Subscription fire departments were commercial entities, sometimes run by insurance companies, to which you paid a regular subscription fee. If you house was on fire, they'd extinguish the flames. If your neighbor's house was on fire, and they didn't subscribe, then they let it burn.
Citing a completely different thing does not refute what I wrote. It just illustrates that you don't fully understand the issue.
That’s literally how it works in much of the rural South. You pay fire dues, or they will only act to save human life. Otherwise they’ll watch your house burn to the ground with you.
In this case the "middle man" is literally doing the work. Money doesn't build things. It goes to entities so that they can build things. I suspect you know this, but it _seems_ like you don't.
Ah! - so the next neighborhood over has all-private roads, and the homeowners association there (not the government) pays for all snow plowing, crack & pothole patching, repaving, storm sewer work, etc. that might be needed?
Ah yes, the HOA. A group of elected people that can compel you to do things with your property, require you to pay a share every year or they take your home, and have the ability to fine you if you don't comply with the majority opinion. Very much not a government in any way!
In my corner of Michigan, there seems to be a very clear dichotomy on this:
- Private developer builds a sprawling subdivision with plenty of nice wide roads and lots. (So a very large area of pavement per tax-paying property.) And turns the whole thing over to the city/village/township, to be their public road budget black hole forever more.
- Private developer builds a very compact little development, with houses (or condo's) packed in like sardines along a rather narrow and minimalist Private Road.
This is also why USPS is crucial for rural areas. The Government should be subsidizing this work because if they don't people in rural areas are left behind.
Inflation isn't at 10% but it seems like investing in infrastructure is a good idea. If we needed to pinch pennies we could start at the bloated military budget
Sure, they're helping an ally in a De facto war against Russia, but currently the us spends more on "defense" than both Russia and China combined, when it is technically at peace. In case of a war with China, are you expecting the military budget to not increase at all?
Say it this way. Tomorrow if the american security /defence budget was cut to 0, do you think the rest of the world will storm/attack Americans because they have an eternal blood thirst for them? Don't they have their own problems to deal with?
This is the problem with mitary and security infra of any country. They keep the bogeyman alive because their paychecks depend on it.
How does spending a lot of government money make goods and service more expensive?
EDIT: At least here in Western Europe, we mostly have a supply side inflation, because energy got a lot more expensive, not because the government has been "printing" a lot of money. I suspect it's the same in the US.
Yes, inflation is currently a world-wide issue, and explanations at the world level lead somewhat obviously to the pandemic and Russia's invasion of the Ukraine.
But here in the USA, people like to believe it must be political and local, completely unrelated to the totally-coincidental worldwide issue that happens to be very similar.
It does not unless that money is spent competing with businesses and citizens for resources. However, in this case the money had already been earmarked for rural internet service and is not being used to purchase goods and services that citizens would be buying instead.
Actually Europe has been printing a lot of money by having less than 0% interest rate for loan.
Current inflation is due to many factor, some estimate it has been slowly growing since 2008, plus covid where we printed money to just to keep business alive, plus negative interest rate that allowed countries to loan too much, etc...
But I suspect that subsidies for infrastructure is one of the least impactful factor for inflation.
In most locations in the US any entity can hang wire on utility poles (the poles are often owned by the city, with an open access policy -- this is how CATV and PSTN wires are up there on poles, and more recently 5GUWB base stations). There are certain requirements (e.g. insurance, you have to have assets on hand to repair your cable when someone drives into a pole, you need workers who are certified to work near high tension wires, etc). Usually you can outsource that stuff, for a price, possibly to the same contracting company who does the same work for Comcast.
Friend of mine needed to run fiber across the street. They had to dig up the road. Cost was $50k. This was in a city where there aren’t large pools of money from the government to get people decent Internet address.
You are a private person and you choose to live deep in the country-side / on a desert / on an island / remote location / deep in the forest.
Who should pay for your road, your electricity, your water, your internet connection when you are the one mostly benefiting from it ?
Taxes have to be used primarily with the goal to maximize public interest, not the interests of single private persons.
Perhaps a Starlink connection would have been enough for them and perfectly fine if it's a single family.
Could there have been alternatives that maximize coverage ? For example, by supporting deployment of 5G antennas as public infrastructure (thus, benefiting the whole area).
This family doesn't necessarily need a single fiber cable to reach their house.
> Perhaps a Starlink connection would have been enough for them and perfectly fine if it's a single family.
Oh the irony... Starlink is also tapping (federal) government subsidies to provide internet service to rural areas. Tapping government subsidies is a very important part of Starlink's plan to become profitable.
The difference is that those investments will be usable by anyone who wants the service and can setup the antenna. Where-as a half mile fiber run to your house in the boonies can only ever be useful to you.
The subsidy is for the company though and not this specific fiber run, which was a sort of worst-case. The company is quite limited in geographical scope, so they got a fairly small subsidy, while Starlink is much larger in scope and thus got a larger one.
Also that fiber run will remain useful for far longer than the Starlink satellites.
It's pretty much a one-time cost with negligible operating cost, whereas Starlink will have to continuously keep launching satellites to keep it running.
One way or another, tax payers spent $30K on a fiber run to one house. Yes, they spent less on some other ones too. The indirection just increases cost insensitivity.
It's all about averages though. Some people will be cheap to connect, while others will be expensive to connect. And the subsidies are most likely written in a way, such that the ISPs can't only go for the low-hanging fruit.
Same with Starlink on a bigger scale. Some ground station will have more people near them than others (absent satellite to satellite comms). Some orbits will be used by more people than others..
I don't understand this comment. There are a lot of places in the country where a majority pays for the minority when it comes to infrastructure. Case in point NYC or Chicago, whose populations and tax bases make up a majority of the state, yet their taxes still go to maintaining the state infrastructure as a whole. The state, in order to function, needs some kind of continuity and predictability to plan for population dynamics and spread out taxes accordingly.
Even beyond helping the state as a whole, they are also helping themselves. Good luck getting anything into or out of Chicago or New York without rail, roads, locks, dams, and airports. Infrastructure that connects to nothing isn't all that useful. All that downstate Illinois roadway, railway, navigable rivers, and smaller airports have their uses for Chicago, too. That's what networks - like the Internet - do.
5G base stations have a range on the order of 1000 feet, and need to be connected to a high-speed backbone to function.
In rural areas, a 1000 foot radius doesn't get you very many people, and since you ran fiber all the way to that antenna, you might as well run fiber the rest of the way.
Well it's not a stupid idea at all, that when you pay taxes, you could vote for the 3 or 4 topics that you want support in priority, and they get allocated a more budget in proportion or something like that.
This could even increase support of people to pay taxes (reducing fraud) as the taxpayers would know they would be supporting projects in line with their vision and lifestyle.
I get the idea, but this is basically just admitting that our representative form of government doesn't work. Ostensibly we control our taxes already by who we vote for.
Rural sprawl significantly increases overall infrastructure costs. Their taxes are already being subsidized by more urban tax payers. Those rural areas can't afford to maintain what they have.
Further, they should be forcibly blocked from using any services they refused to pay taxes for. No highways, flood protection, low food prices, or access to the global trade network for you!
Except if they purchase a subscription to these benefits through one of the two companies (same parent company) that provide them. The subscriptions are of course competitively priced, since they only have the best interest of their customers at heart.
I find it's helpful to create a monopoly on purpose, and then give that monopoly for a service an additional monopoly on violence. Then, if someone doesn't want to use the monopoly, they can just send men with weapons of war to force them to fund the monopoly at gunpoint.
It's pretty widely accepted that the government will help people gain and maintain access to infrastructure, even (especially?) in rural areas. Ever heard of the Rural Electrification Administration[0]? The Tennessee Valley Authority[1]? Despite the fact that it is not considered a _necessary_ utility de jure, internet access is hugely important in our modern society and economy. These areas have post offices, electricity, trash service, etc., so why shouldn't they also have access to internet? Those other utilities cost money to install as well.
You should look up the area that Jared is building his fiber network. These homes are probably 10 minutes from the University of Michigan. It's not a remote country-side, it's just far enough out of reach of Comcast that they won't build out. I understand your point if someone decides to build their house on 20 acres of forest, but this is not that. That's why we need these programs.
Some people are farmers. Everyone benefits if there is Internet in remote places in order to help people stay and live where the farming is going to happen.
You are thinking of 5G mmwave which trades off better range for better speed and latency. For maximizing coverage you are looking at something that looks like 4G.
As a general principle it's fine. The issue is a combination of:
* Providing infrastructure and services for rural areas like this is inherently monumentally expensive.
* For most people, rural living is some kind of choice: most could likely move to a cheap suburb that could be served much more easily, but don't want to.
* Far from having more money to fund things like this, rural areas are actually much less economically productive per person, on average. Of course you need some people to farm, but in practice you have many more people than that.
Essentially, society is providing a heavy subsidy for a lifestyle choice for most people, with no compelling government interest there.
While I do think we should make a goal of hooking everyone up to decent internet, any sensible plan has to look at how we can do this efficiently. Absolutely bare minimum, we should be superceding local zoning laws and similar that often make it illegal for people to build more densely in these areas (small town city centers), such that people can individually choose to live in a more efficient way in the same general area if they want. Not talking about skyscrapers obviously, but traditional, walkable downtowns with townhomes or duplexes would be a great thing.
Some Americans may scoff at this, but you don't need large numbers of people to get walkable neighborhoods. I've been through Bavarian villages of a few hundred people that were more walkable than US cities 100x their size.
It's especially nonsensical that we'd heavily subsidize super low density living when that's basically always gonna be worse for the environment. It means you need much more land per person, obviously, so you gotta cut into nature more, plus higher energy requirements.
He's actually connecting hundreds of people that otherwise wouldn't have such access to fiber. The big ISPs took WAY more money [0] and delivered less.
The point of taxes is to provide collective goods, such as infrastructure, defense, education.
One of the first thing the US's founders did was create the postal service, which was to provide mail service to everyone, regardless of location; it literally costs the same to mail a letter across the street as to send it to some house in Whoknowswhere, Alaska. This provides a minimum communications infrastructure.
One of the best things that were done in the New Deal was the Rural Electrification Act, which ensured that electrical service was provided to everyone, providing a minimum availability of a critical energy source.
Also essential was the initial telecommunications acts, which required providing telephone service at the same rates to all addresses. Again, providing this service universally ensures that the entire country has a baseline communications infrastructure.
This is why the telecomm companies have been aggressively stripping copper telephone wires from their system and replacing everything with fiber or coax — because the laws requiring universal service are tied to phone service and copper wires. This is why we wind up with companies like Comcast saying "F*$k-You - $50,000 for 500m of wire" to to everyone that isn't instantly profitable.
These universal service mandates are not to benefit each individual living on some remote farm or homestead, or just more remote suburbs/exurbs.
They are to benefit THE ENTIRE NATION. Everyone benefits from infrastructure, and benefits most when the infrastructure is more universal, when everyone can has power, can communicate and can transport goods.
You live in an advanced society with advanced infrastructure. When that infrastructure gets built out, perhaps notice that it is a good thing, instead of thinking of only your own petty concerns.
Or, go find someplace where there are no taxes and you get to do everything yourself (hey, if you want it done right, do it yourself, right?) - see what you can find and how well you can live with no roads, comms, power, security, etc. Report back.
My parents currently have four options for Internet access. One is only on their phones with no tethering. The other is to dial in over a landline at 33.6k if they can find an ISP that still offers that. There's existing satellite, which is 512k down and like 25k up for hundreds a month. Or there's a wireless 256k plan that costs $2000 to install.
There's no ISDN, no DSL, no Starlink yet, no 5G fixed, no 4G fixed, no power-line Internet. They are not watching any Netflix, and things like Social Security and Medicare are increasingly accessed through poorly performing, bloated websites. They paid taxes more than five decades of full-time work. There's fiber within two miles of them, but nobody's used it to extend what's becoming a modern necessity to their house.
If they lived on the other side of the road, they'd have the area's rural electric cooperative. Then they could get at least 10 Mbps over the power lines. However, they're on a corporate power provider that has 4 to 12 hour outages 3 to 4 times a year besides not offering similar additional services.
With the right negotiations and a few hundred thousand dollars, their moderately densely populated unincorporated area could serve hundreds of homes with broadband. The cable and phone companies were given millions upon millions of subsidies every month for decades now for rural phone and Internet access, but have not served this area. It's time something else is done in these areas to give them the same access to the modern marketplace and to government services as everyone else.
On top of that, Starlink was just adjudged to NOT QUALIFY for this type of service. [1]
You cannot simply assume as you stated, that just because something looks like a viable solution, it is.
Again, the people that decided are doing the WORK of figuring out how to make the system work, in contrast to taking random potshots in an internet forum.
From the article:
>> The Federal Communications Commission (FCC) has rejected Starlink's application to receive $885.51 million in broadband funding
>>The FCC said that both Starlink and LTD "failed to meet program requirements," submitted "risky proposals," and that their "applications failed to demonstrate that the providers could deliver the promised service."
If multiple and more reliable than Starlink existed, maybe.
The point of universal access is just that - UNIVERSAL access.
We are already failing this massively with laws granting territorial monopolies to companies like Comcast AT&T, Verizon, etc., enabling them to give the worst possible service at globally awful prices. Granting another effective monopoly to Starlink is not the solution, UNLESS we are going to regulate all of this like a utility - actual regulated standards of service, by companies with a large in-state business nexus, cost-plus rates approved by regulatory body, etc.
Using Starlink seems fine, but Starlink has effectively zero skin in the game, no in-state nexux. If it is convenient for them to shut off or downgrade service to these houses for some reason, there is essentially zero recourse for these customers or the state to exercise any leverage to cause Starlink to resume service.
This is actually an excellent solution, with a local vendor with skin in the game, providing solid fiber infrastructure.
You really seem to entirely miss the point of UNIVERSAL SERVICE. Yes, the local post office makes a wild profit on delivering a $0.60 1-ounce first-class envelope to a PO box in the same post office, and loses an insane amount delivering the same letter to/from Wherethafakawe, Alaska by bush planes. I'm sure they could be more efficient scanning the letter and sending an email to/from Alaska, but that won't get grandma's fabric sample to her grandkid, or my high-performance sample to my customer. The point is that the same service level everywhere has it's own benefits, and those benefits are to the entire nation, not only to some.
With every general solution, you can point out individual point inefficiencies. What you are failing to notice is that if you optimize for every one of those point inefficiencies, you effectively de-scale the system.
You lose ALL the benefits of a consistent system, as well as losing most of the economies of scale. This is why companies repeatedly go on binges to reduce their supply chain vendor count - sure, some of those suppliers are lower cost at that point, but the overhead of managing many redundant suppliers outweighs the cost.
And you are looking at only one point of the costs, getting bent out of shape, and trowing out a generic "taxes bad" comment. Yes, it looks like a clueless anti-government political comment.
It might even be the case that in some circumstances, a Starlink solution could be best. But you have done none of the analysis to establish that claim, and other people, who are actually 'in the arena' have found a different solution is better. If you want to challenge them, do so with something better than "ugh, taxes and spending bad".
Snark aside, I spent years being angry about every government subsidy until I learned that some subsidies are pork barrel spending and some are just the normal allocations required by a functioning government to maintain the expected standard of living.
I knew a couple who did similar in Seattle. They all got gobbled up one by one, sometimes by Speakeasy, who in turn was gobbled up by others. Briefly theirs was owned by an east coast company which sucked because they had east coast tech support. If your internet went down binge watching a show at 9 pm you were done for the evening because their people were in bed.
I would not recommend doing this business with a spouse. They did not make it for many reasons, but running a 24/7 interest sped up all of their problems. Not unlike a vacation that is going poorly, but every month.
Also fuck Covad. They only had to suck less than Centurylink nėe Qwest and they couldn’t manage that.
Isn't this just called starting a business? Don't get me wrong it's very cool but this just seems like the thing people should do when there isn't enough competition in the market
Sure, at face value you're right about that, but I think the main difference is a lot of people don't get annoyed at , for example, Ford's customer service and turn around and start an auto manufacturer, and for most non-technical people I think they'd consider the two nearly equal in terms of feasibility and effort.
Not only that, but he’s providing a much higher level of service for a significantly smaller cost than ISPs that have been given billions over several decades and have yet to reach the customers he’s reaching.
My biggest fear for him is that comcast will lobby to be able to sell subscriptions on his infrastructure (because competition!), put him out of business and then screw his existing subscribers.
Exactly. It’s noteworthy because it underscores just how uncompetitive the ISPs in the US are. That a small shop can completely eviscerate them on quality and price shows that they just aren’t trying. (Look at ISPs in any developed country and our networks are embarrassing in comparison.)
It’s frustrating because the playbook for how to improve this is very clear; local loop unbundling on telephone lines, allow municipalities to offer broadband in underserved areas, and mandate sharing of poles etc. to make it easier for new entrants to compete. Of course when you can’t innovate, legislate; the ISPs lobby hard to prevent all of this consumer-centric stuff from happening.
He did start an LLC but it's not a business in the sense that he's hiring a corporate structure around it or kicking up VC funding, or even trying to make a profit. It's admirable because how many other ISPs can you point to with this model? I can't think of any.
Most small local ISPs are like this, a labor of love, not something who’s singular purpose is to make the owners unimaginably rich. Cruzio, in Santa Cruz, and MonkeyBrainz in San Francisco come to mind.
> When the federal government money became available, the county issued a request for proposals (RFP) seeking contractors to wire up addresses "that were known to be unserved or underserved based on the existing survey," he said.
The world is full of small businesses doing good work in exchange for pay. theVC bubble isn't everything.
There is a good interview on YouTube with Jared Mauch. I think it may have already been posted on Hacker News previously, but I have included the link below for anyone interested.
> "I have at least two homes where I have to build a half-mile to get to one house," Mauch said, noting that it will cost "over $30,000 for each of those homes to get served."
I did a lot of investigation some years back hoping to start an ISP in a much more dense city where options were still limited. I had quotes from electrical companies of $25k-75k to run 2,000ft of aerial fiber on telephone poles (no drilling even!) The electrical company (who owns the poles) said that only certified installers could do it but that list was rather short and the person I spoke with didn't seem to know what that certification actually was. I wonder if this guy simply figured out how to legally do the infra layout himself.
He's getting $2.6 million to set up access to 417 homes. That works out to $6,235 per home. At $55 per month, it would take 113 months, or over 9 years just to get $2.6 million in revenue.
Horrible economics! What a crazy business to be in. No wonder grants like this are necessary.
The actual price they are offering seems to be $55 or $79/month + ~$200 installation fee. Also missing in your calculation, is a $30/month subsidy from FCCs "Affordable Connectivity Program".
I didn't make the calculation myself, but a sub-10 year horizon for a project someone seems to do from the goodness of their heart, doesn't seem so bad.
Including the installation fee and $30/mo subsidy (I am assuming this means the price he receives is $30 higher than the one customers pay), my quick math shows it would take a bit over 71 months (almost 6 years) to hit $2.6M in total revenue. However, that assumes literally every customer chooses the $55/month plan, if everyone chose the $79/mo plan, it would take almost 51 months, or a bit over 4 years (obviously the number will be somewhere in between that).
Also, this math assumes no growth whatsoever in homes served or other revenue lines. I assume adding another home will be far cheaper than building out the core network, and the article itself notes other lines of business. To be honest, this doesn't seem like a terrible investment to me. There are certainly better ones in a pure ROI point of view, but for government investments? More of these please!
He's serving 600 households afaik, not offering service to 600 households. So the assumption is that there's a 0% attrition rate. Pretty safe assumption for monopolies, or for local services with half the price and 5x the bandwidth of the other choices.
We're talking about the ROI on the 417 houses he's installing access for, not his total customer base. 2.6m / (417*(55+30)) = months to $2.6 million in revenue.
However, the assumption was that all 417 houses connected will become customers. That's a pretty big assumption. The actual percentage could be 50% or 90%. I don't know -- but surely the answer will have a big impact on the time it takes to reach that amount of revenue.
Going from 0 to 1 internet providers is likely to have an extremely high conversion rate assuming affordable pricing and quality. While it's anecdotal, I have personal experience with this. My family is from rural eastern WA and spends the summer in the mountains up there. Until 2020, the only internet service option was HughesNet, which was extremely expensive and very low quality, as it's a notoriously rugged region that's difficult to serve (it's still ineligible for Starlink). In 2020, a local was able to get a 5G tower installed (way cheaper and way higher quality). When I had him set us up, he told me he'd already installed it for 60% of the cabins on the chain of lakes our house is on. He expected it to be 100% by 2022.
All that said, my experience does remind me that many of the people up there turn it off for the winter, when they aren't there. My assumption is that there'd be some amount of desire to do that, which would also reduce returns.
So taxpayer dollars are necessary to make this business viable, and the product of that business is something that, realistically, everyone absolutely needs access to - certainly seems like this should not be a private business at all but a public utility. Have we ever asked this kind of question for interstate highways?
The Grant County Public Utility District in eastern Washington (and presumably PUDs elsewhere in the nation) did exactly that. They built a fiber network throughout the county (physically large but pretty sparsely-populated), although they don't provide service directly to customers--instead, a healthy number of local ISPs still exist in the area. If fiber isn't at your house yet, there are also a few WISPs, which were easy to stand up because of the fiber.
given the state of the roads and streets in most places in the usa I have very little confidence that public internet will keep up with maintenance, upgrading the equipment to the lastest speeds and standards every 5 yrs.
Commercial ISPs have issues and they should not be given local monopolies but even shitty Comcast is better today than it was yesterday. The same is not true of most of the roads in my state.
I disagree with you on the basis that I can get in my car right now and be confident that I'll be able to drive with speed and safety to any city on the map, and that when I get there I'll be able to drink the tap water and to plug my electronics into any wall socket without them getting fried. Maybe some local municipalities aren't that great at keeping up with their last-mile pothole maintenance, and maybe that should be an issue the locals prioritize more when choosing their representatives - but that doesn't represent the average experience.
But also, we're already talking about publicly funded infrastructure. We've subsidized broadband to every home multiple times by now, and we still continue to write those checks. Maybe if we want it to be private we should actually enforce that and then see how it goes.
My experience is a bit different. The roads where I live (San Francisco) are better than my AT&T options. Roads here seem to be repaved every 5-10 years and AT&T still doesn’t offer a plan that the FCC would classify as broadband to my house.
I don't know what parts of sf you're referring to but my experience of sf is it's pothole hell. Market, Misson, anything between market and van Ness, and plenty of others
to add, I lived on the east coast in the 80s and I found some fellow Californians where we co commiserated about how shitty the roads were in Baltimore and how nice they were in southern Orange County but now I drive though southern Orange County and the roads are clearly in need of repair.
Well the US economy has boomed for the last 250 years or so and depends pretty heavily on roads and thirsty humans. Those investments seem to have given more than they took, by a very wide margin.
This is how the ISPs work as well, typically 10 years is common ROI for any neighborhood and 5-10 years for multi-family housing (apartment) runs. This is also the reason AT&T/Comcast won't run new installations to small (less than 40 residents in my experience) or rural neighborhoods since the ROI time gets longer the fewer potential customers they have.
Not that bad. A lot of utility-type businesses expect to have much longer return on investment times, the electric business is usually wanting to get 50 years of life out of a new baseload generating unit, and it might be 30 or 40 to get your investment back.
How long did it take to complete? Central London seems like high density, how many users do you have? Can you share website, I can forward to some central London folks!
Why - 30mbps download and 0.5mbps upload, it was really the upload that was crippling (think video conferencing)
What were the returns - time will tell, but I probably have the best internet in London
What do we peer into - 10Gbps of NTT, two more 1 Gbps full peering sessions, plus the LONAP internet exchange to pick up Google, Netflix, etc. Plus my wife (AS210412) peers with me (AS211289) of course.
I find it a little weird and off putting that thierprivate business is having its expansion funded by state funds for coronavirus recovery. I get that this is generally a good thing, and many ISPs, especially the smaller ones, receive government funds for developing and maintaining infrastructure. However, why is the Coronavirus recovery fund paying for this?
In this specific case there is an easy answer (mentioned in the article): Access to reasonably priced broadband internet was seen as one of the biggest, most easily addressable (with targeted government infrastructure funding) dividing lines between people that were able to easily work from home and those that experienced larger hardships during the height of the pandemic.
Remember the Hacker Manifesto: What could be dirt-cheap if it wasn't run by profiteering gluttons.
The physical infrastructure of cable is not expensive. The fiber itself costs nothing in bulk. Currently a pair of 1Gbps 20km rated transceivers costs <USD$20 in bulk.
The only things that make installs expensive are: (1) regulation; in particular ingrained antiquated systems of land ownership and associated regulatory capture bullshit by established monopolies; (2) switching infrastructure and associated power, land and security requirements; and (3) one-time installation process costs such as trench digging, termination box installation and cable termination.
Once installed, the cables are unlikely to fail unless aggressively attacked with digging equipment.
> Jared Mauch is expanding with the help of $2.6 million in government money.
> Mauch told us he provides free 250Mbps service to a church that was previously having trouble with its Comcast service.
That's interesting, he's taking money from the government and giving free internet to a religious organization? Do all "churches" get free internet or just the ones he prefers? Taxpayers are OK subsidizing a specific church based on one person's personal whim???
An atheist might reasonably do the same for churches in his service area, for P.R. and Marketing reasons.
How is this different from Bob - who (say) the township pays to mow the lawn & plow the parking lot at the township hall - deciding that he'll mow the lawn & plow the parking lot for free at some local church?
I know this is a troll, but I'll respond anyway. If someone can prove he's discriminating against institutions on the basis of religion, he can be sued. Whether he takes money from the government or not doesn't matter in the slightest.
ah you're right, I had misinterpreted, thinking that he got a grant from the government for the public good, i missed that he just won a contract with them. my comment was based on my misconception that the government was subsidizing him as a public utility.
my comment is past the delete button so that's that.
This seems like a fun project to work on but what is the financial game here? Does he invest in building the network, operate at a loss and then sell to someone like Comcast? I assume building a remote fiber network that can reach 600 houses has to incur huge CapEx (way more than $2.6M right?) and at $50/mo a very long payback period.
The initial investment is paid by subscribers or financed by the government grant, both mentioned in the linked article.
The monthly income of $55 or $79 times 70 people is $3850-5530/month gross right now, which is likely not a full-time income, but with potentially 600 more customers soon, it's possible he could achieve a full-time income for himself, which many people would consider a worthy goal.
In 1994 or 1995, I used an ISP in Sioux Falls, South Dakota, that was just one guy providing decent service. If there were issues, I'd call David and he'd fix them. His goal was to have good internet service--which was difficult to come by then and there--and to underwrite it by sharing it with others. I know he made a go of it for a number of years, although I'm not sure how it ended.
People in tech tend to forget that proper tech doesn't actually need 100s of engineers to keep it operational. That's the whole point of a computer. It does what you programmed it to do, and it does so automatically.
No you are incorrect. If you read the article and the original article they did in Jan 2021, the original effort was completely funded by him and his neighbors (as in they'd pay $X that will cover future monthly bills later.) This article is about how he has obtained additional government funding to supplement the efforts he was already doing on his own.
looks like he's been able to find some deals on equipment and stuff since he operates on such a small scale. I guess he can just continue as a small business indefinitely if he gets enough cash flow.
If you had a small robot that could tunnel underground and pull conduit behind it, the only thing you would need to lay fibre would be some electricity and time.
I don't understand why he doesn't use a microwave link for some of the single-endpoint long runs. (I don't doubt there was a reason; just want to know it,)
Why is the article harping so hard on the whole "government grants" bit. Where the investment is coming from is just about the least interesting bit of this story.
I live in a town of ~15k people in a Southern state, and I have symmetric gigabit. Granted, I pay ~$110/mo, but I have the option of symmetric 300Mbps for ~$50/mo. Neither plan has a data cap.
Then again, I chose my home based on the local ISPs' physical network topology. I didn't rely on their service maps, either - I physically went to their installation folks and got a copy of their maps.
$110 absolutely falls under "affordable" gigabit. If I want symmmetric gigabit, I can get it, with a Comcast business plan. But last I checked it was in the $four-figure range.
I pay around $35/mo for 100Mbps symmetric using a microwave satellite on my house's roof.
More frustrating is to chart how close you actually are to gigabyte symmetric. AT&T and Sonic has wired up large parts of the city but if they don't serve you, it's often by just a block or two, depending on where you are in the city. Rumor has it that local ISP MonkeyBrains is also getting in on the fiber game.
Yeah. A couple blocks away Sonic has wired up the entire neighborhood. I've been begging them to wire up my street for years with no luck. I used to be a customer but switched to Monkeybrains because I couldn't take dsl speeds any longer.
Monkeybrains is my favorite ISP, I currently use them. Affordable and reliable. A couple hiccups over the past few years (my internet speed was cut by 80-90% for a day or two) but resolved with a phone call. Nothing so bad as Comcast regularly becoming nigh-unusable at peak times.
2 years ago, Sonic pulled fiber in my neighborhood in the East Bay. Gigabit is $65/month (including taxes/fees + 1 unused phone line). Very happy with Sonic!
The area served is close to Ann Arbor, MI - so remember Starlink's "satellites are in random-ish orbits around the Earth, not magically hovering over areas with more potential customers" issue.
It's possible that the county is trying to get tough with Comcast here - "stop gouging our residents so badly, or we'll help a local competitor (to you) grow into a real thorn in your bottom line". Starlink isn't credible for that.
And the money is from a "State and Local Fiscal Recovery" fund that the county has access to - so spending it on Starlink would probably be a legal non-starter regardless.
great another example of abusing public taxpayer dollars to subsidize rural homes that shouldnt exist
infrastructure outside of dense towns is unsustainable with the extremely low amount in taxes rural areas pay
these people do not deserve the same standard of living as those in sustainable areas
subsidize them to move to urban areas, not their lifestyle that uses 20x the infrastructure load an urbanite does
Amerika can't keep building out the same levels of roads utilities and municipal water to rural areas as it does to cities. this standard of living does not scale. it is not sustainable.
if you don't believe me, go look at 100 year infrastructure costs once a suburb needs replacing. this is why every town in America is failing
I think it's acceptable to expect better. If we didn't, we'd probably have surface level sewer, water, fiber, cable, etc; all laying about, probably causing trip hazards. And these industries would probably lobby and set archaic and asinine rules for how the burial happens, and make you pay 10x the cost of what it really takes to use one of their approved contractors, because you're indulging in the luxury of having hidden basic-needs infrastructure.
Traditionally the solution is to have a tiny outbuilding with your electric meter, water valve (if you're on town water) and landline connection and then let the homeowner deal with the bulk of the length of the line run.
Getting electrical and water in those situations is always a town by town crap shoot because the trades are constantly lobbying to disallow it because they want more work. I assume ISPs are the same way.
There are many reasons why this isn't done and isn't a good idea. One of them is: animals will eat the cable. Another is: people will trip over the cable. Another is: eventually someone will dig the cable up with an excavator, even if the operator of the excavator is the same person who carefully laid the cable a few years earlier. I don't explain how I know that...
> Comcast once told him it would charge $50,000 to extend its cable network to his house—and that he would have gone with Comcast if they only wanted $10,000.
Starts his own company and finds out it costs $30,000 to do it.
You need big trucks, drills, excavating equipment, skilled union workers making good wages, safety concerns around water, gas, sewer, electrical and other communication lines, you can't mess up peoples lawns, you have to go out and maintain these systems after storms.
Its so expensive that Comcast only made a profit of 42 Billion in 2021, while providing a lower quality of service than what a small ISP in Michigan can give you for a one time 2M in government grants.
You should check their quarterly earnings I think you would be shocked. In the second quarter Cable and Broadband account for 7.4 Billion in profit, and have a profit margin near 50%. NBC Universal only accounts for 1.9 Billion and their theme parks are 632 Million. By far cable is the largest driver of profit.
As someone who actually was working in excavation for internet... well, some points to unpack here:
- You don't hire your own workers to dig trenches as an ISP, you sub-contract that stuff out to contractors - they can spread out the cost of, say, a backhoe not over the one year or two you need to build out a district's fiber, but over twenty years.
- Other underground stuff isn't much of an issue in rural areas - you have the central map register of the district which shows exactly where active lines are, and there aren't many. Usually it's the 10 kV/220V electricity line, water mains and the huge POTS cable. Sewers in most cases aren't much of a concern as they tend to be built very deep (here in Germany, minimum 100cm below ground level, and usually it's more like 2-3 meters). In rural areas you can usually get away with shooting a mole through the ground or a plough for a trench that a following tractor immediately closes after the pipe is laid in.
- That pipe or whatever you're building out underground can last literally for decades. POTS cable in many cases is over fifty years old, personally I have seen stuff that was covered in clay protection plates with swastikas meaning it was well over 70 years old. At 50 years, the life time earning of a connection is 33.000$.
- Governments usually subsidize the cost because broadband is an extremely net-positive investment. Assume a small village of 100 people gets broadband Internet uplink - now a small company moves into some farmer's shed because the rent is cheap and now pays tens of thousands a year in corporate and employment taxes.
In many rural places in the US, the majority of homes have their own septic tank and leech field. Some homes (although it's much rarer) even haul in their own water by truck. Power and phone are often on poles. They probably use LP gas brought in by truck. So often the main concern is the water mains.
I live in one of those rural homes. We only get electricity (and natural gas too, but that's unusual around here) brought in. Water and sewer are on-site. Phone is VoIP. Internet is wireless (via an ISP I built). Rural piped water is very rare here.
True. Many of the places that are low-hanging fruit for rural Internet access do not, but it's a mix. Many of these places that the comments are dismissing as irredeemably remote are along secondary highways less than five miles from a city limit. Lots of those places have water mains, but certainly there are places with private wells.
The correct price in cities is $10 a month. The correct price in rural areas is $500 a month plus. But we have to average them because we insist on taxing cities to subsidise rural lifestyles...
The funny thing is I'd be totally okay paying $500/mo for good Internet service outside the city. The problem with this is that even in the city where Comcast has it's headquarters they will lie to you and then not show up at the agreed upon time scheduled 3 months in advance /and paid for/, then try to blame you for it and take no accountability. Which is exactly what Comcast did when I tried to get connected in my move last month. So, sure, organizations have Product teams that focus on pricing strategy, and part of that is amortizing capital costs to serve those customers and also averaging out the per-customer cost of service, but a bigger issue is that Comcast is just really bad at doing it's supposed job.
I wish there was a rural fiber or muni fiber project near me that I could subscribe to, and I'd happily pay 3x-4x what I pay Comcast, if I had some assurance that the person on the other end of the phone would actually keep their commitments and know what they are doing.
But isn't it also true that once his network grows above a certain customer base (and gets into the maintenance phase), he will start to see all the effects that eat into being able to do this cheaply?
Namely:
-- customers who don't behave as well or kindly as before
-- customers who need 24 hour customer service
-- maintenance that can't be done himself, and he has to employ people
-- customers and vendors who sue you for breach of contract, or other simply nuisance lawsuits
-- upgrading the network to the next technology requirement, or when he's unable to get 2nd-hand parts so cheaply, etc.
-- or a natural disaster that unexpectedly forces replacement of (and charging for) equipment that wasn't anticipated in the original subscriber price
Maybe none of this rises to the level of making it fundamentally different or unsustainable? But it seems to me the honeymoon phase doesn't last long, and it's got to hit some unavoidable realities soon. At least, if you think you can replicate this, it requires finding people and neighbors who are willing to do actual work and investment/concern to make something like this possible, and not simply pay a vendor a premium to phone it in. It must be treated like a neighbor-to-neighbor community project, not a faceless commercial transaction with its attendant obligations.