> If we drop those markers (1110 and 10 in front of bytes) and keep the remaining bits we're left with 1111111011111111, which evaluates to 65279, which is in hexadecimal 0xfeff. Yes, you recognize it, it's a BOM. Because yes a BOM is just a ZERO WIDTH NO-BREAK SPACE, isn't it beautiful?
Byte Order Marks have stolen hours and days of my life. Anyone suffering the pain of developing on a windows box can relate. Windows puts BOMs by default in the front of every file. Thus windows programs silently ignore it, but then linux machines run the program and choke on the BOM. You have to specifically ask the editor if the BOM is even there, it doesn't show up in the editor by default. I have specific lines in my .vimrc[1] that prevent BOMs from ruining my day/week, but they still pop up often. I often joke there will be a byte order mark on my tombstone, along with avahi daemon.
> Byte Order Marks have stolen hours and days of my life.
Me too, to some degree. I have discovered them in a Ruby code base at work, in the middle of a line of code (copy pasted), where the Ruby interpreter thinks they are undeclared identifiers. When the code runs, it throws an exception every time that complains of “Undeclared identifier `‘”.
The dad-joke of it is that “You gotta sweep for BOMs before they blow up your code.”
I have written a cross platform, stand alone CLI program to inspect a file for BOM8 and BOM16. It also detects if a file uses CRLF or LF. Tab and nul characters are also evaluated. Please see the Examples in my repo:
I've dealt with two elusive bugs which were ultimately caused by Windows stupidly using UTF-8 with BOM by default. Python requires you to take extra steps to decode that garbage, and some C++ libraries can't handle it at all.
I'm sure there were good reasons that BOM sounded like the right idea at Microsoft, but everyone else just used straight UTF-8 and it was fine.
Windows supported Unicode in 1993 (NT 3.1) and 1995 (Win95) via UCS-2, a fixed-width 16-bit encoding.
In 1996, it was realized 16-bit wasn't enough, and was expanded in Unicode 2.0, which also included UTF-16, a variable-width encoding, which required the BOM.
Windows 2000 supported UTF-16 on release.
Why didn't Windows 2000 support UTF-8, which was invented in 1992 and implemented in Plan9 in that same year? Who can say...
Back in 2014 I was trying to set up a Linux machine and bind it to the active directory domain at work. The active directory domain was a .local domain, but avahi Daemon thinks any packet that's bound for a DOT local address is addressed to it. So it would swallow up all the packets that were headed to the domain controller, look at them, think they were weird and not understand them and then drop them on the floor. From my perspective it looked like the firewall just hated me.
It was like a week or two later until I finally went to my friend and said I must be stupid but I can't do this it's not working and he just disabled the avahi daemon and everything started working again.
.local is a reserved domain for mDNS (aka ZeroConf or Bonjour, the stuff Avahi handles), standardized in early 2013.
Then again, 2014 is soon enough after for that for knowledge not to have percolated everywhere, and/or for it to stomp on older networks that had used .local beforehand.
Microsoft recommended using .local for active directory domains since the 1990s, I think because back then it was not reasonable to demand that their customers register a domain name at a time when that was a massive hassle. But it was still wrong to squat on a TLD: there were already moves to expand the number of TLDs at the time, but MS were very slow to correct their mistake.
Then Apple made the same mistake with Bonjour / mDNS, and the IETF standardized Apple’s use of .local and it all became an even worse mess.
My high school english classes would upload any papers students wrote to a site that would check for plagiarism. I figured out that if I inserted random zero-width no-break spaces in the middle of words my plagiarism score would drop to zero.
Presumably the plagiarism system was just looking for exact matches of long substrings.
I mean, it's the shortest _possible_ pull request (since I don't think you can make a git diff of zero bytes, barring some weird quirk), but also probably has the highest PR description : PR diff length ratio of any PR I've seen.
That won't show up in diffs though, while this one does show up albeit it's VERY hard to spot the actual difference, as the different is a space with zero width.
Given that a BOM is three bytes, I don’t really agree that it’s the shortest. How about replacing a CRLF by LF? That one is invisible in many contexts as well.
The most common reason I see it is to create the initial commit of a repo. This is useful both so you have something to push to a remote, PR against, and because several git commands (most notably rebase variants) need weird switches (eg --root) if you _don't_ have an empty initial commit to refer to instead.
I use it extensively for entire PR based development; that way all the code that ended up in the repo has had multiple eyes (and an opportunity to comment) upon it
I've found it useful when interactively rebasing a series of commits to make a nice PR. The code moved to a different commit, but I didn't want to lose the message.
I see `--allow-empty` in git tutorials all the time, to demonstrate the concept of a commit. I'm not sure when you'd reach for it in a real repo though
The most obvious reason not involving automated systems is to create a repo’s initial ref. This can be useful on teams who are super strict about review process and git history.
Once you account for automated systems, many other reasons can arise. Not just the “trigger CI” case mentioned in another comment, triggering builds or processes based on remote content the code accesses, or generating something either random or seeded by the commit hash/timestamp/message/etc.
It can even be a Homer Simpson-style drinking bird button press, so finished software doesn’t get mistaken for abandoned.
Probably whole worlds of things I haven’t imagined because I don’t use git hooks or submodules.
Maybe so you can make an empty initial commit and push it to a remote like Github as a placeholder? Or a lazy way to trigger a CI job when you're gonna squash and cleanup later.
I guess conceptually you could use it to represent "I started from nothing."
I love the title. I like to ask for pull requests with this exact description to influence my coleagues to look at it faster when it's something small like a single character. For example, when it's a two character PR I say "hey, the second smallest pr in the world". Guess I was wrong!
You can easily check in your CI that your files are ASCII (code should probably be) with file(1). There is probably an off-the-shelf-tool that can validate that all characters are printable, ASCII or unicode.
Nice. Would it be possible to have an option to only output the names of files that failed the -f check? i.e. hide the names of files that look "normal" and show the "weird" ones.
Also, does it detect files that only contain CR as EOL characters? Or files that have different EOL characters on different lines?
According to the page, several machines and operating systems used CR as EOL. While the systems are all obsolete, files from that era that use CR as EOL could persist and be transferred to modern systems. Clearly those are weird on modern systems, so they should be warned about in a linting situation, which I would like to use your project 'chars' in.
Having different EOL chars within the same file is definitely a thing, usually by mistake. I had to fix a bug about this recently:
Byte Order Marks have stolen hours and days of my life. Anyone suffering the pain of developing on a windows box can relate. Windows puts BOMs by default in the front of every file. Thus windows programs silently ignore it, but then linux machines run the program and choke on the BOM. You have to specifically ask the editor if the BOM is even there, it doesn't show up in the editor by default. I have specific lines in my .vimrc[1] that prevent BOMs from ruining my day/week, but they still pop up often. I often joke there will be a byte order mark on my tombstone, along with avahi daemon.
1: https://git.sr.ht/~djha-skin/dotfiles/tree/main/item/dot-con...