After years of working exclusively on Windows, I took a job that required me to build file management, except now, on macOS and Linux (along with Windows).
All I can say is, this article is the tip of the ice berg on Windows I/O weirdness. You really don't realize how strange it is until you are actively comparing it to an equivalent implementation on the two other competing operating systems day-to-day.
Each of them has their quirks, but I think Windows takes the cake for "out there" hacks you can do to get things running. I sometimes like to ponder what the business case behind all of them was, then I Google it and find the real reasons are wilder than most ideas I can imagine.
>Windows I/O weirdness. You really don't realize how strange it is until you
"can I get a wut... WUT?" - Developers, Developers, Developers
====
it's just hard for people today to understand what a revolution stdin and stdout was, full 8 bit but sticking with ASCII as much as possible. There was nothing about it that limited Unices from having whatever performant I/O underneath, but it gave programmers at the terminal the ability to get a lot done right from the command line.
The web itself is an extension of stdin and stdout, and the subsequent encrusting of simple HTTP/HTML/et al standards with layer upon layer of goop that calls for binary blobs can be seen as the invasion of Cutlerian-like mentalities. It's sad that linux so successfully took over that all the people who we were happy to let use IIS and ASP and ActiveX had to come over to this side with their ideas. No idea of which is bad, but which together are incoherent.
"No idea of which is bad, but which together are incoherent."
Right, my complaint about filename/path length being exceeded in my earlier post often occurs when a web page is saved by a browser (some web pages have outrageously long filenames).
Incidentally, many years ago I did a tour of Microsoft's operation in Seattle around the time Microsoft introduced subdirectories into MSDOS and the tour guide (can't recall his name but he was responsible for the development of MS's Flight Simulator) gave a considerable spiel about why Microsoft decided to run with the backslash instead of the forward one as per Unix. Even then, I thought 'oh no, here comes confusion', and others with me thought the same. When we challenged him about it, he said we (Microsoft), want to clearly differentiate ourselves from Unix (there was an arrogance about his answer that I well remember).
Flight Simulator was developed outside Microsoft, but within Microsoft you could only be referring to Alan Boyd. I received a similar tour.
I'm sure you heard what you thought you heard, arrogance and all, but iirc it was backslash because IBM insisted the slash be the switch character. Anybody remember $SWITCHAR
I know Flight Simulator was originally developed outside MS and I think my tour wasn't that long after MS acquired it.
It's too long ago for me to associate the name 'Alan Boyd' with the person in question but I do remember that he had a loud, penetrating self-assured voice. (Incidentally, he spent considerable time demonstrating Flight Simulator's new features).
You're right, IBM was a large part of the discussion as back then it was the principal client for MSDOS. However, I came away from the visit with the understanding that MS was in full agreement with IBM's decision despite MS's dabblings with Unix.
I had a particular interest at the time as I had a S-100 Godbout CompuPro computer, and in addition to CP/M, I ended up putting Seattle Computer Products' DOS (SB86 from Lifeboat Associates) on it which meant that I had compatibility with MSDOS.
I can understand why MS would have wanted to differentiate MSDOS and the backslash being one way, what I'm still not clear about is why IBM would have wanted to make such a distinction.
Re $SWITCHAR, very vaguely, but from my point it added confusion. Like some other commands its implementation and architecture appeared to be the result of afterthought rather than good design. I've forgotten much of that stuff.
I rather liked VMS conventions as it was immediately obvious what was the directory part of the path and what was the filename and the device. You could create virtual devices as well, so for example, with VMS TeX, I had TEX_ROOT defined to point to the root of the TeX distribution and you would have TEX_ROOT:[INPUTS] for the input directory, TEX_ROOT:[MF] for Metafont files, TEX_ROOT:[EXE] for executables, etc. and everything was logically arranged. CLD files were another wonderful thing where you could define your CLI interface in an external file and let the OS handle argument and option parsing for you.
On 2 & 3, what if you have a directory? There’s no easy way to tell whether /foo/bar is a directory or a file. FOO:[BAR] must be a directory.
I have a vague notion that there might have been the ability to have a virtual device span multiple directories, but as I think about it, this seems unlikely since it would make it ambiguous where something would be created if a directory exists in both root1 and root2, so perhaps not. It’s been 23 years since I last used VMS and 30 since it was my daily driver though, so it’s hard for me to say too much.
The downside of building in stuff is that you can't easily replace it. I don't have any VMS experience, but the versioned file systems I've studied are very far from being a replacement for what I would consider version control today. They're more like the automatic writing of backup files in Emacs.
Distributed FS sounds like something with a huge design space. It's better to do that in userspace. Was Plan 9 really the first time virtual file systems could be implemented in userspace? It seems like such an obviously useful idea in retrospect.
I’m not familiar with inner workings, but simply moving files feels odd in Windows compared to macOS. If it’s a big lift in terms of data size or file/folder counts it’s most obvious but it feels like Windows literally copies the files into memory, then rewrites them on disk or something similar that has results in negative performance and a long running cut/copy/paste dialog box. I’ve had some of these run for hours on decent hardware (SSD, etc) for what I consider small datasets (couple GB). It’s been a major Windows gripe of mine for years now.
Meanwhile macOS appears to just change an internal link to the data that’s already written on disk. As such, it’s usually so very fast compared to Windows.
Windows File Explorer does a lot of extra work to get a sense of file sizes and other metadata to try to keep the UI looking fresh/interesting/useful to someone watching the job in real time.
If you need to seriously move/copy lots of files or lots of data in Windows it is generally a good idea to use shell commands. Robocopy [1], especially, is one of the strongest tools you can learn on Windows. (It gets very close to being Windows' native rsync.)
Windows does literally copy (parts of) files into memory. More precisely it's Windows Defender Real-Time Protection. It's a real menace when you're dealing with a lot of small files, e.g. node_modules.
Windows Explorer is also slow for an unknown reason.
Doing file operations through the API with Real-Time Protection turned off is several orders of magnitude faster in the case of small files. It's crazy stuff.
A lot of this depends on whether you're crossing devices. If you think of drive letters as mount points it may make more sense - if you're moving between mountable filesystems obviously a move has to be a copy-then-delete; if you're remaining on the same filesystem a move can typically be a rewriting of indexing information only with very limited data rewriting.
One other thing that can be an issue particularly on NTFS with ACLs is that moving files typically retains their ownership and permissions, while copying files typically inherits the ownership and permissions of the destination. This can bite you if as an administrator you're moving data from one user's account to another because a move will leave the original owner still owning the files.
Eh, with moving files on Windows in the same security context it is 'generally' pretty fast on the same drive.... Are you sure you didn't paste in a directory that is setting new security permissions on all files?
> All I can say is, this article is the tip of the ice berg on Windows I/O weirdness. You really don't realize how strange it is until
In one way it is beautiful. "Laid cards lies", you know. Don't mess with the user for your conception of Agile Clean Extreme Code (tm). Each stupid design decision is forever.
Windows .bat files win the awfulness and quirkyness contest with sh with a razor thin margin. And both are awesome.
PowerShell is an absolutely amazing scripting language; it gets things done way quicker than Bash, because it's object oriented, and you don't have to call external tools to get anything done (sed, grep, find, touch, curl, etc.). It can even run raw C# code for you.
This definitely falls into the category for me of "things that I wasn't there for."
Because I learned computers when DOS was a thing, I will always be able to write a .bat or use CMD when necessary, but having been on the UNIX/Linux side since 2003, I didn't learn C# or PowerShell but rather bash, php, ruby. So while I'm friendly with modern Windows now that they closed up the "is it a stable OS" gap with Apple, I don't really know what to do in PowerShell and am more likely to use WSL!
PowerShell is the closest of Xerox PARC REPL experience that ships in the box on modern platforms.
Because not only it is a proper programming language, it is integrated into .NET, COM/DLLs as well, so not only you can script the OS, any application automation library is exposed as well.
Nowadays, it is possible to automate anything on Windows via PowerShell, the same OS APIs exposed by GUIs are also accessible to PowerShell.
On UNIX side there are things like Fish shell that also offer these capabilities, but they aren't widely adopted as PowerShell on Windows.
Sorry, but Powershell is not a programming language. It has way too many quirks and gotchas to qualify as a programming language. It is an interactive scripting language first, and a scripted language second. But a programming language it is not.
To give just one example: its automatic boxing and unboxing of arrays disqualifies it as a programming language. Try to return a one-element array from a Powershell function and you'll see what I mean.
It's worth learning at any age, especially now that it is an open-source, cross-platform shell. The PS Koans [1] that recently showed up on HN seemed an interesting way to try to learn it.
I just want a shell that runs my commands, I don't want yet another language.
The _beauty_ of bash is you can learn the basics of the language very easily, call out to external tools, and _take that knowledge with you_. Those tools exist independently.
I’ve tried to switch to Powershell a few different times and I always find it to occupy this no man’s land between a quality shell and a quality scripting language. As a shell I find it inferior to BASH and as a scripting language I find it inferior to Python.
Not that it’s a particularly compelling feature on Linux with the standard offering, but it’s a good option for cross platform scripts at times, particularly running in docker.
I mean, if I want something that can run on as many platforms as possible without a prior installation, I stick as closely as I can to posix sh. If I want something more flexible that can run consistently, but may require an installation beforehand, I use python. I don’t really see what niche PowerShell would fill for me.
It doesn't have to fill a niche for you. Before cross-platform PowerShell I certainly used Python for some of those kinds of scripts.
I think a lot of it gets down to ergonomics/aesthetics to decide if you find a useful niche for PowerShell for yourself. Python's os module is powerful and lets you run/chain almost any native commands and shell operations you want to spawn, but it is still a very different API and abstraction with different ergonomics and aesthetics than shell-style pipes and redirects.
PowerShell gives you that focus on shell-like pipes/redirects, but then gives you some Python-like power on top of that to also work with the outputs of some commands as objects in a scripting environment. There's a lot of interesting value to comparing/contrasting PowerShell and Python and if you are happy with Python maybe there isn't a big reason to learn PowerShell. PowerShell is there for when you are doing a lot of shell-like processing pipelines and want to write them as such, but have some of that power of a language like Python behind it. It's a lot more powerful than posix sh and it is similarly but differently powerful to Python but it starts from a REPL that looks/acts more like posix sh. I don't know if you have a need for that niche yourself, but I find it useful for that.
`$ErrorActionPreference = "Stop"` is very similar and does that for all of PS1 cmdlets. You still have to check $LastExitCode manually for BAT/EXEs, though.
`$PSNativeCommandUseErrorActionPreference = $true` is an experimental flag [1] as of PowerShell 7.3 that applies the same $ErrorActionPreference to BAT/EXEs (native commands), stopping (if $ErrorActionPreference is "Stop") on any write to stderr or any non-zero return value.
Having to handle exe's and bats separately is _exactly_ the problem with $ErrorActionPreference, and why it's not suitable.
I wasn't aware of $PSNativeCommandUseErrorActionPreference though, seems like it's very new. How does that work with the helpful windows tools that decide to not return 0 on success (hello, robocopy)
The answer to that, including robocopy as the direct example used, is at the bottom of that documentation I linked on $PSNativeCommandUseErrorActionPreference: you set it to $false before calling something like robocopy and then reset it when done.
5.1 was the last "Windows-specific"/"Windows-only" PowerShell (and is still branded "Windows PowerShell" more than "PowerShell") before it went full cross-platform (and open source). It's an easy install for PowerShell 7+ and absolutely worth installing. If you are using tools like the modern Windows Terminal and VS Code they automatically pick up PowerShell 7+ installations (and switch to them as default over the bundled "Windows PowerShell"), so the above command line really is the one and only step.
You can also install the latest PowerShell Core (the open-source, cross-platform releases we're talking about) via Scoop, which is a package manager for Windows that works even if you don't have admin rights: https://scoop.sh/#/apps?q=pwsh&s=0&d=1&o=true
Unless I can rely it being somewhat available, it's not really feasible to use. It's a bit like writing scripts in fish because it's easily installable - nobody is going to use it.
Winget isn't bundled with windows 10 either, (but I think it is with 11), and it's not on windows server.
If I need to install a package manager _and_ a shell, I might as well just install WSL and be done with it.
Winget is auto-installed in Windows 10 by Windows Update and/or Store Update for every copy of Windows 10 with a recent enough build for more than a year or two, so long as that machine doesn't have the Store disabled or blocked. It is bundled inside the "Application Installer Platform" which is a low-level Store package that powers a lot of little things like the "double-click to install an MSIX file" experience and Windows generally keeps up to date quickly if Store updates aren't blocked.
I can't speak to your usage of Windows Server, but provisioning winget and PowerShell 7+ are standard bootstrapping steps in VM images at places I work, because those are generally assumed to be basic equipment at this point.
It also adds it's own special brand of crap .. as in after trying 10x different ways (not kidding: https://social.technet.microsoft.com/wiki/contents/articles/...) of executing an external ffmpeg command over several hrs I eventually wrote a one line .bat file* and was done with it. Never again.
*
for %%a in ("*.mp4") do ffmpeg -i "%%a" -vcodec libx265 -crf 26 -tune animation "%%~na.mkv"
The maximum path length of the NT kernel is 3276-something UCS-2 characters. 260 is a limit imposed by the legacy Win32 interfaces IIRC. I believe the W-interfaces get you the full-fat version, it's just that they're so inconsistently used as to all but guarantee that something you need will have problems.
The user-mode stuff is kind of a mess. The kernel-mode stuff is comparatively orthogonal.
> I think the margin of bat vs sh is a larger margin [...] the entire file is reread for each line
Ye agreed. Separation of data and code always was a mistake.
I wonder if this feature of bat files was like a thing once as a "best practice"? Practically, you should only append lines I guess. When I close my eyes, I can see a DOS batch file doing actual batch job processing, appending to itself and becoming a intertwined log and execution script.
"All I can say is, this article is the tip of the ice berg on Windows I/O weirdness."
Well, then, is there a more detailed summary than this one that's accessible?
This one looks very useful and I'll use it, but to make the point about more info it'd be nice to know how they differ across Windows versions.
For example, I've never been sure about path length of 260 and file length of 255. I seem to recall these were a little different in earlier versions, f/l being 254 for instance. Can anyone clear that up for me?
Incidentally, I hit the 255/260 limit regularly, it's damn nuisance when copying stops because the path is say 296 or 320, or more.
"no amount of prefixing will help you, if random app enforces the limit"
I've noticed that, it's partially the reason for my confusion (I didn't wake up for quite a while as I put it down to the different versions of Windows I was running on various machines). Other pains are caused by apps that still don't run Unicode and crash or stop copying when they encounter a non-ASCII character.
Thanks for the reference, I wasn't aware of those changes in Win 10 (I run mainly Linux and have been weaning myself off Win for some years).
"...(this value is commonly 255 characters)."
I think the word 'commonly' in those notes confirms my point in that it's changed slightly over the years. Also, I recall the internal processing was once 16k and not 32k, come to think of it this may have been with the previous version of NTFS (can't remember which version of Win that was). My interest is now piqued so I'll search it out.
That we're even discussing such matters confirms the thrust of the article.
Explorer can resolve some http urls to network paths, e.g. for SharePoint libraries.
Programs, e.g. python scripts, can then use those paths, but only after explorer has resolved them. Before that, the path will be treated as not existing.
From my experience as an EE, working with serial ports is much nicer on Windows (COM1, COM2, etc.) than on Linux where serial ports are abstracted to files like /dev/ttyACM0 and has a lot more gotchas.
PowerShell is also quite a powerful alternative to Bash/Mingw, although it came out much later.
Windows might do some things differently than UNIX-like OSs, but it does them really well.
Technically, COM1, COM2 etc. are filenames as well. They are just special in that they are available everywhere. That's why you are not allowed to create any file named COM1 or such.
Wow! There's an operating system I ain't heard tell of in a good long while. That's the very first OS I used in a professional context. Got me my first computer store job (in my late "teens") on a CP/M system.
I deal with software that processes files on a Windows system... loves to break when people on other OS's subnet AUX, PRN, COM, File:Name, and tons of other unacceptable names (like 'file ').
I'm glad our new releases work on Linux and we don't have to deal with that crap in 99.99% of cases now.
I've done quite a bit of work with serial ports on Windows, Linux and other unixes. I've also written a serial device driver.
Your comment is very confusing to me. The serial ports are abstracted to a file on Windows just like on unixes - the file is actually discussed in the above article: \COM1
Maybe you're talking about the old days where you would just outb 0x3f8? The modern interfaces are actually fairly similar.
Remember typing in entire programs from magazines and computer manuals and saving them to cassette tape or floppy disc? That was "the good ol' days" for sure… :)
There is also the persistent problem of USB serial adapters being assigned incremental numbers until they're in double digits that many tools don't let you select from their GUI. So you have to go in and manually purge those devices to get back to sane numbering.
I just started with using serial ports on windows while doing some Raspberry Pico hobby projects. Something that I find strange is that every new device gets assigned a new comport, I mean let's say I do this for a while one day I will have a comport 100, 200 and so on. Is that right, or does it somehow reset the comports?
That's how it works and generally it's to the user's advantage. We often set specific parameters based on the device's serial number so getting the same COM port is nice, sometimes the devices are so simple that you cannot query its serial number.
Sometimes I'll do a "blank slate" and delete all my accumulated COM ports in Device Manager (need to enable "Show Hidden Devices").
COM ports on Windows are crap nowadays due to how crappy USB to serial adapters have become. I've seen Windows reassigning different COM names to the same device every single time it was unplugged due to it "remembering" what COM port was used previously. Needless to say, that was an anti-feature if there ever was one.
Windows tries to keep a long term identity of all of the device instances that it knows about (and in the idela world assign the same COM port numer to the same physical serial adapter). For USB this is supposed to be done by combination of VID, PID and serial number in device descriptor. But even early on there was a lot devices that had the serial number empty and thus Windows came up with some heuristics about when this method is unreliable and switches to identifying the device by its physical position on the USB bus. The whole mechanism is well intentioned, but in the end probably somewhat misguided because it is not exactly reliable and has surprising consequences even if it were reliable.
As a side note: on modern Windows NT implementations the so called "registry bloat" is non-issue (anyone who tells your otherwise is trying to sell you something), but keeping list of every device that was ever plugged into the computer in there is somewhat ridiculous.
> As a side note: on modern Windows NT implementations the so called "registry bloat" is non-issue
How modern? I manage Windows 7 (transitioning to 10) machines that are used for QC in a hardware manufacturing environment that enumerate hundreds of devices (with mostly identical VID/PID) every week. We find that if we don't clear old devices out of the registry every so often, the enumeration time slows to a crawl.
In the times when it was a real issue (I would hazard a guess that that means “before XP”) the reason was that the original registry on disk format made every registry access more or less O(n) in bunch of things like the overall on disk hive size, total number of keys, number of subkeys in each of the keys along the path…
It also do this for monitor or usb/bluetooth earphones. So you end up get earphone(2), monitor(2) even you never have a second one. The only way to fix it is delete the hidden device in device monitor and rename it back in monitor/audio manager.
It's really a confusing thing to me that the script I use to change sound output and leveling suddenly didn't work after a bios/mobo software/whatever windows update and noticing the device have an appended (2).
And this is why I hate Windows in an industry automation environment. Dislike having to troubleshoot why that USB NIC or Serial device being destroyed by plugging it into another port. Had to write a PowerShell script for the USB NIC issue to reapply NIC settings with a reboot.
Also, always locking an open file is repulsive. Other OSs allow for renaming an open file. Not Windows! Thumbs.db being locked because File Explorer keeps the file open / locked preventing deleting an empty folder and wastes so much time waiting for Windows to unlock the file.
We do all the time. In industrial automation COM ports are still shockingly popular, although it's usually the USB emulated variety. On a lot of our development and on some of our production tools we end up with COM20 or COM30, not because we have that many running at one time but because over time we've plugged in that many distinct devices. Nowadays most drivers will assign COM(n+1) when they see a device with a new serial number.
UART is available on nearly every microcontroller under the sun, and USB<->UART serial chips are super cheap, so it makes complete sense to me that'd become the defacto system for interfacing the automation controller with a computer
Even beyond that, USB is available on many microcontrollers, a USB CDC device is dead simple to implement, the drivers are baked into every modern OS, and all the software developers operating at that layer already know how to interact with text streams. Add in the ease of debugging when you can just manually send/receive ASCII to operate a device, and you've got the makings of a standard practice.
If you use USB dongles for serial adapters, then each path through USB is assigned a different COM number when you plug it in. For example if you plug into USB controller 2, port 3 which goes to a hub, and then you plug into port 2 that gets a number. Now plug the same thing into a different port and it will get another COM number.
Under the hood this is because the USB devices do not have the (optional) unique serial number (or in some cases they all get the same serial number).
So do I, I find the addressing more consistent, too.
It used to be completely predictable when I was working with drivers on 1994 (patching the code), then less predictable when hardware for more diverse, and predictable again (or at least "always the same") with UUIDs.
It was always amateur/hobby dev or sysadmin so I may have had the wrong impression.
It’s the flip side of the ‘we bent over backwards so SimCity runs’ coin. Even though Windows hasn’t supported programs out of this era since 64bit became the standard, it’s still held back by clinging on to the legacy. Because it doesn’t dare say ‘this is too old, run it in a VM’.
All I can say is, this article is the tip of the ice berg on Windows I/O weirdness. You really don't realize how strange it is until you are actively comparing it to an equivalent implementation on the two other competing operating systems day-to-day.
Each of them has their quirks, but I think Windows takes the cake for "out there" hacks you can do to get things running. I sometimes like to ponder what the business case behind all of them was, then I Google it and find the real reasons are wilder than most ideas I can imagine.
Fun stuff!