I too ran a small ISP from 1995-1997 or so, and the biggest gift I got from one of my co-founders was the strong suggestion we use this new cheap database called mSQL [1] as the basis of our accounting system. And I did. I wrote a lot of terrible, undergrad-in-cs C code around it, but we never messed up on billing. Thanks, Kent!
Now that time I hadn't verified that our backups were working and lost all of our data, that was a little less good.
I was living in the Chicago and got one of the first DSL lines during the dot com bubble. It took 6 months to get it installed.
They never charged us for it. Roughly a year later, we got a letter from their big four accounting firm saying that they were going bankrupt due to billing irregularities and our service would be terminated soon, but it was easy to get service then.
We were in the first 100 customers to get dsl in STL. It took 1.5 years and escalating to a vice president to get it to work as both a phone line and dsl. Their billing system was miffed as dsl was commercial. My friends worked there and had oracle access (swbell) and could easily see the mismatch, but couldn't fix it. They ended up giving us our old dialup second phone line for free and dsl for free for 3 years to solve it.
Wife also worked for att and dsl was cheap for them except the billing system swore my apt was in another county and would send the installers there constantly. Took 8 months and was only fixed when wife went on a tour of a facility and mentioned it to an engineer and their VP.
In 2000 or thereabouts PacBell was promising ADSL at my apartment. I was well within range of the CO/DSLAM and everything seemed set but the technician could not get the modem to sync with a signal. Turned out the building's phone wiring was a rat's nest of twisted and taped RJ-11.
It really sucked not to get anything faster than dial-up. It also explained why my dial-up couldn't connect reliably at 28k and I'd often be stuck with a 19k connection.
The experience did teach me a lot about Squid as a caching proxy and using wget and screen to do overnight downloads (I didn't learn about whey's background flag until a few years later).
Me too! Although I did web hosting and company ISDN lines and not dialup 1996-1998.
Used Windows NT for web hosting and Slackware Linux with qmail for mail hosting. Sold out at what I thought was the top - which was a year or two too early but that's OK.
We did our billing with great plains accounting software which still exists as Microsoft Dynamics GP.
When 56k came around you had to do ISDN, but for many rural ISPs the tariffs for ISDN was cost prohibitive so POTS was the only option.
I worked for a few ISPs and negotiated telcos to bring in redundant OC12s to support the need, allowing them to use our data centers for the local switching. We never needed that capacity ourselves, but as it was before they started using multiple colors per fiber that was the smallest they would run.
I also started out with stacks of external modems and NT RAS.
I remember that the plastic on stacked modems would blacken due to heat in about 6 months.
I dealt with sendmail, bind, etc and didn't do the account provisioning but it was written in Delphi.
As I had written an entire 7 digit double entry accounting system in college in two weeks backed by dbase, this stories excel and paper based system seems like an anomaly for the time.
NT we hosting was a pain, but front page did make it a popular option you had to support.
While I did introduce Linux, we were mostly on DEC Alphastations because you could support a lot more load for less money than the sun boxes at the time.
It was a fun time. I remember hopping on #nanog on irc because I couldn't get uunet support to pull routs from us and had them pulling from us in less than an hour.
> NT we hosting was a pain, but front page did make it a popular option you had to support.
I don't remember the details, but at the micro-ISP I worked at, I got frontpage extensions working on Apache for customers.
We had two T1s, one for upstream, one for modems (which I never got to play with), we also contracted out nationwide dialup through megapath? or someone... We ran a radius auth server and they did the rest. Some java accounting package; I did an integration for ach billing, and made a module so we could sell domains (via Tucows OpenSRS). At some point, the modem T1 stopped working and we couldn't get the telco to fix it, so we had to move all the customers to the outsourced dialup provider, which wasn't great for the business... I left because of scheduling/poor performance at school and they got purchased by a customer.
What would truly keep you up at night is how many small/medium businesses still do stuff like this.
One small business I have seen just has tonnes of word documents sitting in a single folder, with a mix of customer details, invoices, etc. To write a new invoice they copy an old one, then change the details, save it with a new name with a "system" (that changed over the years).
Up until semi-recently I saw a small-medium sized business (hired quite a few people) that did all of their accounting _by hand_, for hundreds to thousands of transactions. They had an old lady that came in to manually run their accounts on a calculator and it took several days.
A small business I once dealt with had a small black book (A6?) that was all written in note form, scattered, not dated, random important scribbles everywhere, mix of cash and card (not documented) - it was a nightmare. Somehow they had a good reputation in the local area, but behind the scenes it was pure chaos.
Large/medium businesses, too. In finance, we receive countless reports from 3rd parties. I can't tell you how many of them still rely on person A to manually log into our FTP server at a specific time every day and drop an Excel file into a folder. Sometimes it goes on like this for years. I always wonder if that person ever gets sick.. goes on vacation..
A scary amount of the finance world runs on batched csv and whatever those || column deliminited files are called, just being FTPd around, processed in batches. Which then trigger more FTP and more file processing.
A long ago company I worked for that subsequently went through a couple of acquisitions still basically had one person in finance somewhere who was basically who you contacted for anything related to our defined benefit pension. I assume the bus factor wasn't literally one as there were legally required annual statements etc. but I assume there would have been some manner of chaos if this person from a no longer existing company wasn't there one day. (About 7 years ago, the benefit administration--now a responsibility of Dell's--transitioned to one of the big benefit administration companies; OI expect the person who had been handling it retired.)
I remember a friend whose company allegedly was putting money from their paycheck into some kind of retirement plan, but there was never any documentation showing where that money went. It was skettttttch
Yeah, I've seen stuff like that in my role as a system integrator. We've built all these automated processes for transforming and moving data from system A to system B, but occasionally we learn about other teams who are using excel and VBA for everything. It's very strange.
I consulted for a Blue Cross Blue Shield affiliate (big health insurance group for our non-US readers) that used Excel spreadsheets for managing all their plans and driving their claims processing. This was a company that did 2+ billion dollars a year in business back in 2007.
It was mind-boggling. They had a team of 20 people updating spreadsheets. Hopefully they've gotten their stuff together by now - but I doubt it. We tried to get them to buy a product data management suite (they were bad at building software in-house)...they finally agreed...then the recession hit, and they reversed course.
I can relate. I was building a new kind of data management system when I approached a small mortgage company to be one of my first customers. Startups will bend over backwards to get those first few customers on board, so I worked closely with them to show how my software could make their business run much smoother.
They had a separate spreadsheet for every state (all 50 of them). Some states just had a small number of customers in them (e.g. Wyoming). Others had large numbers of customers (e.g. California).
Their support team could only update one spreadsheet at a time. So adding a customer in Texas required locking that spreadsheet until the transaction was complete. Doing reports required compiling data from all the relevant spreadsheets. Queries that could be done on a relational database in under a second, took many minutes (or hours).
I tried to show them how my system could easily load all their data and make everything work much more smoothly (and saving them hundreds of man-hours); but they just couldn't see the value of spending a few hundred dollars a year for a license for my software.
I wonder if it wasn't the value (people-hours aren't hard to calculate) but a risk assessment. If their system has been working just fine, but the new system entailed major risks (startup going out of business, hosted off-site, backups not in their control, PII / security issues) the calculation gets a lot worse.
I can confirm a large retail company in the US ran all of their inventory purchasing through an Excel file that was managed by a small (<10 people) group.
They were bought out by an even bigger company a few years back, but they were managing to chug along until the mid 2010's at least.
I suppose this makes me feel better about the clunky “enterprise” stuff that my former employer (another national retailer) used. I mean, at least it wasn’t just spreadsheets emailed around!
What will be alternative and not-so-expensive solution? Is there a lot of easy to install, support, customize solutions, cheap to use oriented on a small businesses?
Wave (https://www.waveapps.com/) works well for me. Admittedly, I have not explored things like accrual accounting (I just do everything on a cash basis), so I don’t know how well this is supported, and I’m perplexed that there is no easy way to generate EBITDA reports from within Wave itself, but basic tracking of expenses and transfers across multiple vendors and accounts works smoothly.
[Edit] Also, most CPAs I have talked with know and can integrate with Wave.
It's not THAT bad. I'd argue that if after watching a few tutorials on Youtube and playing a bit with the software, someone is not able to grasp the concepts, it's a strong indication that this person shouldn't be responsible the accounting of a small business.
Dial-up World was consumed in a “roll-up”, which was popular right up to about 2005 when the only dialup ISPs were rural.
The problem with pivoting to broadband was, most could not. The only option was reselling DSL, becoming a CLEC or WISP, and taking the dirt nap. The company I worked for tried everything and eventually went out of business after the dialup was sold in a rollup.
I was at a 123.net colo a few years ago, and off in a rack was half consumed by Livingston/Lucent and US Robotics Total Control dial-up T1 endpoints, running. The USR actually showed signs of use.
I at one point was taking calls in 2012 for a variety of ISPs including a small dial-up ISP in PA. [Also mixed in were weird other clients -- I could bounce from a dial-up call to someone asking about Bit9.]
It broke my heart to be taking a call from someone willingly asking to pay for 12 months of dialup in advance. But they did, and it was like $130. Ow.
National Rural Telecommunications Cooperative members were slowly but surely lighting up fiber, at least.
The article appears to build up though a chain of bad business and technical decisions to something "delightfully and poetically dysfunctional", but what that thing actually is stays frustratingly implicit and doesn't actually make sense to me.
As far as I can tell it's this: Their user management consisted of an excel sheet, but their billing was based on physical piles of filled out signup forms. A user cancelling their contract was implemented by (only) removing their physical from, but this was not communicated to the acquiring company, so they started billing customers who had cancelled.
But then how did they stop cancelled customers from using the service? How did they implement a user authentication mechanism in the first place? There must have been a software representation of user records somewhere in their system, which was kept up to date with cancellations. And the acquirer never learned about that, which would have just been a dub mistake, not exactly delightful or poetic.
Unless... they somehow managed to have the dialup system query the excel sheet directly and never stopped cancelled customers from using the service because they assumed those had switched to broadband anway and wouldn't want to use free dialup even if they could.
These would have been a much more interesting WTF than the mere existence of the Excel sheets and piles of physical forms, and I would expect them to be mentioned explicitly.
Usually in these days, auth was done via RADIUS. It's possible that the company handoff was done by not-the-sysadmins, and the operators of the company only had the things they knew of how things worked, which was the spreadsheet, whereas the spreadsheet to the sysadmins was just the tool they used to create the RADIUS account information off of.
Is it possible that the subscription came with a modem lended by the company which the customer had to send back to cancel their subscription? I think nowadays in the US people tend to buy their router (people more knowledgeable than me, please correct me) but in France for instance, one usually rents their router from their ISP as part of their subscription. Maybe it Dial-up World used to do the same in 1999?
No, subscription hardware was not done anywhere to my knowledge.
You bought your modem at a local store like CompUSA or through a mail order catalog. Actually finding a service to dial in to was left as an exercise to the user. (Hence why AOL famously mailed out floppies (and later CDs) to every household in America on a regular basis.)
Hm, interesting idea. And then they might not have done user authentication at all. Also an interestingly bad practice, plausible, but non obvious and not in any way hinted at by the article.
> one of the aggrieved chose to show their displeasure by posting the list of all Dialup World user accounts online – along with every password in clear text.
- When a customer submitted their signup form to Dialup World, they would add a
row for that new customer to their 20,000-row Excel spreadsheet. Then, they
would put that signup form in a pile with all the other signup forms of
customers who had signed up on that day of the month.
- Every day, they would find which one of the 31 piles of literal signup forms
corresponded to that day of the month.
- Then, they’d charge the customers in that pile, one by one, for a month of
service.
Well, if you hop into a time machine and need dial-up access from "Dialup World" back in the 1990s, I guess there's a sweet hack -- sign up on the 31st of the month and you only get billed 7 times a year.
I started and ran a small early (UK) ISP in the 90s! I can't remember how we did the billing but it was better than that. I suspect that it involved awk with : separated fields, because I remember my dyslexic colleague saying that all he could see was swimming : characters... And I (still) like awk.
I had more incoming phone lines than the rest of the street sometimes (this business was run from home), at several addresses around London as we moved. And we broke each telco's billing system in a new and interesting way...
I've never had an ISP that I've liked since the dialup days. I remember liking a couple of the ISPs that I had at that time a lot, but EnterAct in Chicago is the one I remember most distinctly. They were very transparent about what was going on, provided great service, and did things like offer different numbers for different 56k protocols so you could get the best speeds and stability depending on which modem you owned.
Unfortunately EnterAct was eventually sold to some larger company and DSL became more popular.
1: Did the billing system (and its problems) show up in due diligence?
2: Did they ever figure out who leaked all the passwords and sue them? (Honestly, given that this generally soured the acquisition, the perpetrator could have been on the hook for a lot of money.)
(Now, before you jump to the leaker's defense, remember that in tech, things can change fast, and we all will end up working for an obsolete or acquired company at some time. Part of the game isn't pissing on the seat on the way out.)
This story hits home pretty hard. My first gig in the industry, so to speak, was working at one of the last few independently owned dialup ISPs in midwest in 2005. I first started as a phone tech, where I did everything from help folks get connected and configure their email clients to basic computer tech support. Sometimes I'd get dispatched to do an on-site install. There was also the dirty old man customer that was constantly getting his computer infected on shady porn sites, and we eventually turned on our web filter service for him gratis, because his wife was constantly embarrassed to keep paying to bring the computer in for service.
Eventually became a Jr. Sysadmin, cut my teeth on managing Apache HTTPD, MySQL, qmail, built my first Linux server compiling Gentoo from Stage 1 on a dual 700 MHz Pentium 3 box. Learned so much from an awesome old school Linux sysadmin who, like me, got his start as a young kid at this very ISP. Oh, and I had access to 8 bonded T1s, which when the fastest consumer bandwidth I could ever dream of was a consumer cable modem with 3 Mbit down, and 256kbit up, being able to surf day in and day out on 12 Mbit of bandwidth was blazing fast.
However, like every other dialup ISP, much like the story goes in the OP, this little ISP was far too entrenched in their Dialup install. We did offer DSL service, but we were effectively a reseller to the larger ISPs in the area, so for customers that did switch over, we either offered poorer service, or they were paying more for the same service they could get direct from the bigger ISP. We were suffering from consumer attrition as the great ISP consolidation was happening, and the money was getting tight. A startup looked to be our savior - they wanted to deploy WiMAX (this was before Sprint bought up all the WiMAX spectrum for their form of 4G) to the region. What was originally pitched as a full acquisition became an acquisition of our customer list and some of our services like our shared hosting. Spoiler alert, those jokers didn't know what they were doing, people started leaving the company left and right, eventually equipment started failing and the people that knew how to fix those things took their knowledge with them, and eventually, my paychecks started bouncing, so I left.
Personally one of the most rewarding jobs I ever had. I learned so much, got paid quite well for very little "work", and it really set a course for the rest of my professional career. But hoo, the death spiral story that OP showed gave me a stark reminder of the very bad end times of that time in my life.
Also, I wonder if OP is talking about GlobalPOPS as the company that bought all the dialup ISPs that they worked for.
20,000 phone lines in 12 cities would be like 70 PRIs per city? It's a big order, but doesn't seem like one that MCI wouldn't happily cover. Unless "city" means "suburb", and not like major metro market. Just nerding out, because I ran (tech for) a mid-sized ISP in the late 90's.
Little towns were fun. Had a buddy at the regional ISP (that became a CLEC but that was later) tell me how they had bought all the remaining capacity on the switches in 6 of the 10 towns they served. There's wasn't any ISDN out to these places yet, so they had to have old school closets full of modems in buildings near each CO in these towns. They had to do a microwave backhaul from one of the closets to their network because AT&T couldn't sell them any data lines into or out of that CO.
Helped them shuck and rack the USR courier modems for a couple closet expansions. they'd got a local carpenter to make them a standard 16 wide modem board carrier that was very slick. Still, punching down lines and getting all that stuff woven together was tedious.
In the early 2000s, it was pretty uncommon to bring in all those PRIs from the cities, most telcos supported foreign exchanges through a combination of DMS-100 (brand name Centrex in Ameritech world) features and circuits. Near the end of the retail dial-up ISP, it was not uncommon to cover entire states through that and have them come into groups of PRIs.
The outfit I worked for started in 1990 and had actual POPs with aggregated 3Com, Livingston and Micom serial aggregators that went to a central location over leased lines. They had 1500 lines between retail SLIP/PPP and a couple bulletin boards. That was quite the monster.
I don't know if this disagrees with you or not, but we were done with individual dial-up modems by early 1996, and spent most of that year on Ascend's platform (the Livingston PM3's the next year; I left in the middle of 1997 to go work for a vulnerability lab, and I definitely set up a bunch of PM3's, so that's the timing worked out for me).
I really miss the term Information Superhighway, and I don't know why besides nostalgia. It felt like there was so much possibility and road ahead. If only we'd have known where that road ultimately ended.
Because Cable Television with "57 Channels and Nothin' On" had already taught us about channel surfing. Subsequently couch surfing became part of the vernacular.
Level Zero is "we have no idea what our process is". You get a cookie and promotion to level 1 just for writing down your batshit process, which some engineers find particularly cathartic and thus are all too eager to contribute. Pithy things like "and then we play phone tag and finger point for a week until someone gets fed up and volunteers to get it done," can end up in the first draft.
Now that time I hadn't verified that our backups were working and lost all of our data, that was a little less good.
[1] not a typo, it still exists, though I doubt anyone would use it over MySQL, postgres, or sqlite. https://en.m.wikipedia.org/wiki/MSQL