Hacker News new | past | comments | ask | show | jobs | submit login
The BeOS file system, an OS geek retrospective (arstechnica.com)
154 points by Tomte 7 months ago | hide | past | web | favorite | 50 comments

A fun BeOS story from the 1999 Be Newsletter:


A Testing Fairy Tale

Two test engineers were in a crunch. The floppy drive they were currently testing would work all day while they ran a variety of stress tests, but the exact same tests would run for only eight hours at night. After a few days of double-checking the hardware, the testing procedure, and the recording devices, they decided to stay the night and watch what happened. For eight hours they stared at the floppy drive and drank espresso. The long dark night slowly turned into day and the sun shone in the window. The angled sunlight triggered the write-protection mechanism, which caused a write failure. A new casing was designed and the problem was solved. Who knew?

> The angled sunlight triggered the write-protection mechanism

How does that even happen?

Probably similar to how optical key-switches work: the floppy drive would shine a laser or LED through the write-protect notch and watch for it on the other side with an optical sensor to know whether the disk was write-protected.

So, in a not-write-protected disk (no through-hole), the laser light couldn’t make it through to the sensor... but sunlight sneaking in through the drive casing and under the disk still could.

Also interesting: Practical File System Design with the Be File System


Seconded, I owned that in printed form, and it is very well-written

BeOS was delicious all around and it's a crying shame it didn't take off. It really felt like the Amiga's spiritual successor to me.

Apparently a few of the developers are working on Fuchsia so maybe some of the ideas will make it there.

Possibly. The problem is, for some designs to really work, you need someone at the helm not only incorporating the good ideas, but also saying NO to anything that does not fit the vision. (iPhone etc comes to mind.)

Many open source projects and commercial projects too tend to be much more inclusive in what they pick. The end result often becomes something like when Homer Simpson designed a car. Oh that's neat! [1]

1: https://laughingsquid.com/wp-content/uploads/2013/07/homer2....

Heard of the story that BeOS nearly became the OS X for Mac?


It's a reprint of an article from 2010, so perhaps (2010) should be added to the HN title, in case people are expecting to read something new about BeOS in 2018.

Which is typical of the product and the era.

BeOS was like those Transmeta Crusoe CPUs, Segways and other products that gave journalists something to write about even if adoption was not going to happen.

The circus aspect of the media wanting to write something (as they have pages to fill and eyeballs to feed) must be quite costly to small companies wanting to get a legit product out there. I can still remember the name of Jean-Louis Gassée from his frequent soundbites in the tech press about the wonders of what BeOS was going to be. Did he actually do anything in between conferences, interviews and scheduling media appointments? If so, how did he get the time?

The 'look what we could have won' tech retrospectives are also part of this media cycle, a version of garbage-in-garbage-out applies where those companies that got hyped get recycled as article-worthy a decade on...

I find it sad this is being downvoted. I think this rings true.

Perhaps too true?

HN has its bents and focuses, but it's at least more conversationally balanced than reddit (perhaps https://news.ycombinator.com/item?id=15965536 partially explains some of the reasons why). The news media is of course pure bias and content/distribution/agenda masquerading as fact-based objectivity (eg, https://youtu.be/hWLjYJ4BzvI).

Where else can I look for... the kind of apolitical, discussion-focused information sharing that eg used to happen on blogs circa 2007, but in discussion community form, and within a community framework that is bias-resistant?

Most important thing wrong with this article is that it keeps talking about BeFS having a "64-bit address space". That's not so at all.

BeOS has the equivalent of the POSIX O_LARGEFILE permanently enabled. Historically 32-bit Unix systems assumed files were no larger than 2^31 bytes for the purposes of random access, and a feature labelled O_LARGEFILE says that your program wants 64-bit offset values instead so you can seek inside a larger file. In a 64-bit OS this doesn't make any difference to anything, and in modern programs O_LARGEFILE will always be set for 32-bit programs too. This isn't about the filesystem at all, on BeOS, or any other Unix-like system.

BeFS does, like a lot of modern filesystems, theoretically support larger storage than is ever likely to be in the hands of ordinary end users. Many petabytes of storage in a single filesystem are in principle possible with BeFS. But that's not strongly related, in either direction, with the use of a 64-bit integer for file seeking.

Beyond that BFS has lots of annoying problems, which are very understandable in the context of it being rushed into use over such a short period of time and with really only one key person doing much of the work, but they don't vanish just because they have an excuse:

The metadata indices are clearly aimed at end user operations like "Where's that file with the client's name in it?" or "What songs do I have by Michael Jackson?" but they're designed in a way that wastes a lot of space and yet also has poor performance for such queries - because they're case sensitive for no good reason. They also incur a LOT of extra I/O so if you don't need that feature you'd really want to switch it off, but you can only really do that at filesystem creation time.

Fragmentation is a really nasty problem. This is an extent-based filesystem, so that's somewhat inevitable, but BeFS almost seems to go out of its way to make it worse, and provides no tools whatsoever to help you fix it. It's actually possible to get a "disk full" type error when trying to append to a file which is badly fragmented, even though there is plenty of disk space.

Unix files often have an existence that transcends the mere name on the disk, but BeFS takes that a step further, allowing application software to identify a file without knowing its name at all. There are a few scenarios where this is quite clever, but if you ever want to retro-fit actual privilege separation to the OS (which has been a long term ambition for Haiku for more than a decade) this produces a Gordian knot - permissions are associated with names, but software can simply obtain (or guess!) the anonymous number for the file and sidestep such permissions altogether.

The journalling has a nasty hole in it. There's no way to safely delete files using the provided metadata journalling and recover the freed space, so, in practice if you crash while deleting files some of the space is just mysteriously gone until you run a "check" tool (remember the scorn for such tools in the article...) to find and recover it.

4.4BSD systems on all architectures had 64 bit off_t since the early 1990s. It’s a crying shame that Linux didn’t copy more good ideas from the BSDs :-/

Not that I'd want a return to the vendor wars per se, but I feel like the 90s were an interesting time for Free *nixen in part because there were so many competing alternatives to draw inspiration from.

A great deal of mindshare has been eaten up by Linux. And Linux has largely "grown up" and has incorporated many ideas (and in some cases code) from the commercial Unixes. But I do wonder how many people are out there who have broad perspective and ability to push things forward at the OS level.

Also, back in the 90s I feel like systems programming was cooler and had more cachet. Even if people weren't actively writing code for their OS of choice, the icons of programming were the folks working on the guts. Nowadays, I think the cool stuff is all happening pretty far up the stack, and that there are relatively few people working on the substrate.

That's just my impression though. I'd be delighted for someone to look at kernel and system contributions to Linux, IllumOS, and the BSDs and convince me that there is sustainable mindshare to go around, and that all these OSes have a bright future.

It's a crying shame we aren't using BSD more, period.

Some of us are.

A lot of us are (MacOS)

Another quirk is that the file system cache is fixed at boot time to 1/8th of the available RAM, though this is more due to OS limitations than the file system.

At the time that it was in the market, a "64-bit address space" wasn't the default. There's way more to it than the size of off_t, as it permeates the entire on disk filesystem structure.

For instance, the BSDs didn't have 64-bit block offsets until UFS2 which wasn't a default until FreeBSD 5 in 2003. In other systems, HFS didn't have 64-bit file lengths until HFS+ in 1998. NTFS started as full 64-bit, but you couldn't really get that on a consumer system until WinXP (maybe Win2000 for the prosumer market).

"There's way more to it than the size of off_t"

Nope, the change to off_t is literally all that's behind this claim. It's not Be's claim, explicitly, although they were of course in no hurry to clear it up.

BFS is specifically _not_ itself suitable for huge files. Last time I looked the default Haiku BFS settings won't allow files to grow beyond about 140GB. Which is fine for an ordinary file of course, but you certainly won't be happy if you expected to store a filesystem image _inside_ a file.

This happens because it's a rush job. For example there's lots of special corner case code needed for the extent handling in the multiple indirection case. I can easily imagine a week or two of work just to develop, let alone test, this part of the system. So, instead BFS although it labels these "extents" still just uses a fixed size block so that the code is simpler. It's not a problem at all, for small to medium size files on a defragmented or largely empty disk. But as files get bigger and disks get fragmented... oops.

NTFS dates back to 1993, it takes some real dodging and diving to conclude that NT 3.1 (in 1993) isn't a "consumer system" and yet somehow BeOS, available only to people who swore they were software developers who wanted to write software for Be's new BeBox architecture was.

It's apparently too late to edit this comment, so I must annotate: O_LARGEFILE isn't part of POSIX, it's an extension, my error there entirely.

Tidbit: Dominic Giampaolo, the creator of BeOS file system, is also behind APFS (he gave a WWDC talk back in 2016: https://developer.apple.com/videos/play/wwdc2016/701/).

Having never played with BeOS but dabbled with Haiku — the richness of the BeOS filesystem would have won me over in a heart beat. Imagine if Microsoft bought them and then actually shipped a proper reFS that had the same richness? Or if Apple had — meh, Apple should have just paid Oracle the license fees and licensed ZFS. Or co-opted openZFS (not sure if that’s possible either, legally).

Apple hired the author of BeFS to work for them. He has worked on Spotlight, Time Machine, Apple's file systems, etc. http://www.nobius.org/dbg/

The one thing that I never saw make the leap from BeOS to other operating systems in a meaningful way was the "an application is a thin wrapper around a directory of metadata-enriched files" model that BeOS did.

I think the mail application and the media player were largely built this way because Be needed to build a handful of demonstration applications quickly and cheaply. On the other hand, I remember thinking how nice it was that email and multimedia behaved like the file browser with all its search and filtering features.

iTunes always felt like the exact antithesis of the BeOS model. It was a heavyweight application that had its own interaction model and idiosyncrasies. It might as well have been its own OS by comparison.

Maybe that is why macOS has a similar way to access files by ID instead of by path?

The .vol directory: http://www.westwind.com/reference/os-x/invisibles.html

I think that's a legacy from MacOS Classic, where there was a way of keeping a handle to a file that was independent of the file's name (so you'd still point at the same file even if it was renamed).

You might find the file handle stuff comes from the fact that a lot of the Be employees were originally Mac developers at the start, and quite possibly because they wanted to support some of the features that were in the OFS (original File System, the Database like file system BeOS originally shipped with right up to the AA/PR1 releases. DR8.x definitely had the old file system.)

I remember reading some BeOS API documentation where it said something like "paths are UNIX-like, entry_refs are MacOS-like and they each have some pros and cons". Can't find it now, though…

I still remember running the r5 release on my Pentium III 450Mhz..never had such a lagfree and fast multimedia experience on any OS.. ZERO lag.. it's performance was amazing.. unfortunately it's lack of hardware support and multiuser made it problematic to use it as a daily driver. Even Beos Zeta (commercial version based on r5) did not really improve the situation.

I did the same thing on whatever Mac I had at the time. (It was the pre-Jobs-comeback era when Apple was slowly running the Mac into the ground...kind of like now.) It really opened my eyes as to how fast the Mac hardware could be and how slow the Mac OS was.

> It's pretty clear that some of the functionality that existed in BFS does not belong in the FS.

Does anyone know what functionality the anonymous developer is referring to here?

Why would BeOS not be open source at this point?

The source code, with all the stuff removed that was licensed or probably violated some patent or GPL restrictions would be pretty useless.

It also suffers greatly from the lack of engineers working on it - as in some parts have a lot of comments, some nothing what so ever. Some parts look very well done, some parts less so.

Back in the day, I saw the code (those leaks were real) and one zip was able to fully compile (and this is where the PowerPC version of post R5.03 came from, I used to have it compiling on my old BeBox - long gone now) and the other was a much later version, but the zip was corrupt and half the files were garbled.

DISCLAIMER: No I don't have the code anymore, not that I would never share it if I did, and no, I have never worked for or contributed to the Haiku Project.

Bought by Palm, then HP, then ??? I guess, try to navigate that corporate IP nightmare.

I think BeOS went Palm -> PalmSource -> Access (the embedded web browser people).

I assume you are right and I was hasty :)

Because the incentives for going through with that are low.

Most likely it contains some parts of third-party code under non-ooen license (i.e. in device drivers, but not only) and maybe some subsystems were licensed in some form of exclusive form to others. Also they might want to double check that code not by "accident" looks like somebody else (i.e. the ext2 code might be inspired by Linux's) This all leads to review of old contracts and code while serving no business (besides marketing to a niche of nerds)

Basically this.

Access owns it. Even though they aren't doing anything with it, if they just released the code they would probably open themselves up to lawsuits over possible 3rd party code. So they would have to pay someone to go over the code to take out 3rd party stuff. That would take money for which there is no tangible benefit to them so they won't do it.

Is there a real difference to Haiku?

My understanding is that there is not much BeOS in Haiku.

"The only BeOS code that has made it into Haiku are Tracker and the Deskbar (the file manager and the equivalent of the start menu/taskbar, respectively)."

Understood...but then what is Haiku. I stil remember when r5 was released...an unheard of OS. As a tech geek I installed it and was blown away. Especially the media capabilities...two streams playing at once. That OS was way ahead of its time, and a shameful demise. I was excited about Haiku until I saw how far behind it was. And I mean absolutely 0 disrespect to its authors... but today it seems like an attempt to keep stable an already decades old OS that lost its way.

>seems like an attempt to keep stable

No. They rewrote the whole thing (minus the deskbar/launcher). If it looks to you that they're just keeping the old system stable, that'd mean haiku has been very successful in recreating it as open source.

They're close to 1.0beta1, which is defined as feature-complete for 1.0. It's after they do get 1.0 that they plan on really taking the system's design forward.

Not that it doesn't already have a few fundamental improvements, such as the integrated kernel debugger, the support for newer architectures or the package manager.

What would blow you away today? I’m always curious to know what to build next.

A truly convergent desktop. And I don't even know the details to explain what I would like, I probably wouldn't even know until I used it. I do know Android for Desktop is not it, though. Being able to throw apps from my phone to my pc to my tv would be nice too, and not so clunky as casting.

I would continue, where the ideals of the 60s/70/80s/90s left off. Since IT gets dominated (sponsored) by service companies, book and grocery stores, things are a bit fucked up, to say at least ;-)

I want a new Desktop. Let's talk ;-)

Same here, but not Linux Desktop Distro #301, which is pretty much all anyone wants to make.

i3 should be good enough for anyone.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact