To me, the real problem is not the dotfiles, but the regular (i.e. non-hidden) files that applications surreptitiously create in my home folder. (Which the author does mention as being particularly bad.) dotfiles in the home folder are not a new thing on Linux (e.g. .bashrc, .profile, .emacs, etc.), even if it would be better if more applications used the XDG folders. And if I don't see them when I "ls", they don't bother me.
But nowadays, lots of applications seem to think it's OK to put normal files in my home folder without asking first. This is just outright barbarism. In particular, Ubuntu now seems like it's all-in on snaps, and if you use snaps, they create a non-hidden "snap" folder in the user's home folder. Madness! And when people complain, the devs have the gall to suggest that this is all fine:
I like the idea of snap/flatpak in the abstract, but until they get their act together, and stop creating non-hidden files in my home folder without asking first, I am never going to use snaps.
And of course, all this is on top of the new fashion for the OS to "helpfully" create Documents, Music, Video, etc folders in my home folder, and set the XDG environment variables to point to them. Noooooope.
But at least in that case the user can change the environment variables and delete the stupid folders. No such luck with ~/snap.
This is particularly weird, since click (snap's "predecessor" at Canonical/Ubuntu) was fully XDG compliant, even encouraging developers to separate data from cache etc. (unlike flatpak which just dumps everything in ~/.local/share/[...], though that is still infinitely better than using ~/snap).
I personally don't like xdg. I have bin files from pip and other stuff in ~/.local/bin.
I have huge amounts of cache files in both .cache and .config. It's supposed to make my life easier by being able to backup .config and dump everything. but in reality this doesnt happen.
Backing up .config for just the user defined configs is a massive PITA. you have to put so many gitignore exceptions and I'm not even talking about the gigs of data all the chromium derivatives put into that folder.
I think xdg has an issue of being obtuse when I googled it a while ago, it was very confusing, and I didn't find any real definitions. However I didn't look that hard.
You can try. I mount .cache as tmpfs. You need at least 500mb for your browser alone.
Browsers tend to work fine. Some apps just fail badly like rofi (command history stored there oof)
For what it's worth I also tmpfs+asd .nvm, .npm, and */node_modules, without any major headaches. I did have to fix a couple minor pathing dramas but nothing too drastic.
> unlike flatpak which just dumps everything in ~/.local/share/[...]
Here you are referring to separating the config and data?
Is this actually a problem? I can see how for things like games it would be valuable to have a special location for thing like saves games, like how you can move the roaming save for minecraft from a computer to another; but I never understood the net gain from splitting the application files in config/data
Curated configuration generally outlives data. If storage space becomes an issue, there is value in recovering storage space but config is generally small and much more important in the long term.
Yeah it is crazy when I noticed Ubuntu created a ~/snap folder that is required when anything is installed via snap. People defend placing application data/config folders in ~/ because they're old and it's always been like that, but Snap is brand new and they still neglected to follow the modern standard.
Other "new" softwares that fail to follow best-practices by putting top level files into ~/ just on my laptop are: npm, Ansible, pylint, docker, steam, rustup+cargo, vscode, IDEA IDE, Zoom, AWS CLI. These applications have no excuse.
Also I agree on the Documents, Music, Video, etc. I hate upper case letters in my folders, so when I do a reinstall I delete those and restore my backups from the lower-case versions.
Windows actually has the same dotfile infestation problem these days. I mean, you still get stuff in Documents that shouldn't be there, but on top of that you get apps - often Unix ports - that believe that %USERPROFILE% (i.e. one folder above Documents) is the appropriate place for their configs and such (it's not - that should also go into AppData\Local or \Roaming as needed). And they usually use a dot-prefixed name, just because.
I just looked at my Win10 profile dir, and it has 36 dot-prefixed folders, and 7 more files on top level. Offenders include .gitconfig, .rustup, and .vscode, just to name a few.
On Windows, you could move the configs elsewhere, make a symlink to them in the old place, and mark the symlink as hidden so you don't see it in explorer. Too bad this can't work on Linux because the hidden-state is encoded in the filename.
Well, the hidden-bit on Windows is really semantically equivalent to dot-prefix on Unix, and people set up their file managers to show such files for similar reasons (I don't even use Explorer, mostly). In Win10, it's even exposed as a setting in Settings | For Developers
For actual file management, and generally working in the terminal, I'm firmly in the "two blue panes" club, so it's Far Manager on Windows (and Midnight Commander on Linux). Far is probably the most advanced text-mode file manager in general right now.
Compared to 15 years ago I think Explorer is much better now, it could be even called usable.
Nevertheless there are alternatives. One liked by many are Norton Commander alike file managers like Total Commander or even Far. I am not too much in favor of them though.
I like to use Directory Opus. It is powerful and flexible. Costs money though.
It's fair to store documents and saved games in My Documents.
Much better than storing them in C:\Windows or C:\Program Files, as was the case a decade ago, before windows introduced UAC and blocked write access to these.
It's senseless to store files that don't have user-facing meaning in My Documents. I know games that use it varyingly for web caches, shader caches, binary config files, debug logs, Lua scripts, downloaded mods, and even executables. And while arguably most gamers understand what save files are, they can't be double-clicked to open them, so I don't consider them documents (Maybe this is pedantic).
All this kind of stuff is what Appdata\Roaming and Appdata\Local are supposed to be for.
I understand not backing up AppData/Local, but everything in AppData/Roaming is supposedly important enough to sync it over the network (if you use domains or another mechanism to give you the same Windows account on multiple computers). Any default policy which doesn't back up AppData/Roaming seems ill advised.
Nope, Witcher 1 saves are just really large (for some reason in addition to all the flags they also contain some locale data and such). 10GiB is actually pretty reasonable for a Witcher playthrough.
It's similar in the Witcher 2, 3 and Skyrim. The saves start small and they grow to 10MB as you progress in the game. I guess there is a lot of world state and items to hold in an RPG. Count a few hundred saves, manual or autosave and that's a lot of space.
In the witcher 1 there is also an uncompressed BMP thumbnail with each save that is about a megabyte. It's PNG compressed in the witcher 2. They learned from their mistake.
My AppData/Local is 14GB, mostly caches from Chrome, Firefox and Spotify. My AppData/Roaming is 3.3GB, mostly my emails (with my Browser's localstorage a distant second).
Chrome's cache is in AppData/Local/Google/Chrome/User Data and Firefox's cache is in AppData/Local/Mozilla/Firefox/Profiles/. The only big offender on my system is npm with a cache in AppData/Roaming/npm-cache
Thunderbird infamously places the local copy of IMAP mails (i.e. Effectively a cache) into Appdata\Roaming. It's the single largest directory I have there. Everything else is quite reasonable in size.
You can't do it as a runtime check. You have to compile the software with the flags to import and link to the latest windows library. The program will hard crash if run on an older version.
I just tried to use the API. I literally couldn't find a working example on the internet in 2 hours so I guess noone figured it out in a decade.
You won't have the chance to call anything. If the executable is statistically linked to the latest API, the executable won't be able to load on older systems.
I'm surprised that these new IsWindows...() functions claim compatibility down to Windows 2000. Doubt microsoft knew how they would name their next release 20 years ago. A Google search indicates that they are in fact broken.
You won't be able to compile with the function imported using the modern lib (because it doesn't exist on XP) but you can conditionally link it at runtime with LoadLibrary/GetProcAddress.
If IsWindows... doesn't work then you can use the VerifyVersionInfo function directly.
Why did they have to narrow it down to games? Could just go with "Saved config (and data)", so other apps could just dump small config files which can be effectively backed up.
A game's exe dir is a long-lived but potentially replaceable thing, like a car. Saved games may be persisted across game versions, so you don't necessarily want them to be in the same directory. Admittedly not the best analogy, but was just observing that it's strange and incongruent to store the 2 together.
Which is sort of an improvement from the past, where user-level config data was thrown into the actual app directory in "program files." So if you ever wanted to back-up those settings, you needed to track down every single unique location for each app.
If it's the entire game, it might be to allow kids to install the game without administrator privilege (read: parental knowledge/consent), though I agree that letting people choose would be best.
What? I'm not sure how that connects to my comment. I'm saying that some games install to "My Documents" to avoid requiring administrator privilege for users who might not have that privilege.
I have a slightly different pet peeve. I don't like applications that default to saving files to my home folder. I wish they would look for e.g. /home/user/Documents or whatever else seems appropriate and even semi-standardized given the type of file.
But just dropping stuff in my home directory--look, if I'm in a hurry and my defenses are down, you basically just encouraged me to clutter things up.
The last software I found that does this is a "download helper"...even though I have a Downloads folder that my distro already set up, that Chrome and Firefox know to use, this download helper thinks I want those downloads dumped in my home folder. I hope some generous person changes this behavior, meanwhile I forgot to charge it manually for the last batch and I have a bunch of cleanup to do.
That’s what the article talks about. How applications should use $XDG_DATA_HOME, $XDG_CONFIG_HOME and $XDG_CACHE_HOME and some defaults if they’re not set.
I can't find "official" XDG documentation of this, but the Arch Wiki describes, for instance, the XDG_DOCUMENTS_DIR and XDG_VIDEOS_DIR environment variables:
You should be able to change the default settings for these directories by changing /etc/xdg/user-dirs.defaults (system-wide) and ~/.config/user-dirs.dirs (users override)
I strongly prefer apps default to the previously used directory, and the home directory otherwise. IME apps are wrong 99% of the time when they guess their own save location.
I have great difficulty understanding why the snap developers think the excuse “it’s hard to change!” flies.
If it’s hard to change, you should’ve either fixed your architecture from the start, or thought a bit harder about the implications of magically creating a new directory in $HOME.
It shouldn't be hard to change unless the string "/snap" is hardcoded all over the codebase. There should be a function like "get_app_dir" or "get_install_dir" or some other config string lookup that doesn't use a static string.
On Windows Golang defaults to install to C:\. Had no problems running it from Program Files instead. It's crazy that a new programming language cannot follow decades-old conventions!
Python did this until very recently too. Windows has a problem, because people expect to use those programs in the command line and the %PATH% has a size limit and is a bit difficult to set-up properly.
The sum of the length of all environment variables has to be below 32767 characters. That doesn't seem like a good excuse to save 14 characters by putting your program's folder in C:/ instead of C:/Program Files/
If you do deeply nested paths (as npm used to do) then you can run against the old MAX_PATH limit that limits paths to 260 characters, but the correct solution is to either choose a flatter directory structure or use the newer APIs that allow paths with 32,767 characters.
Steam is doing just fine writing into C:\Program Files (x86)\Steam\steamapps from an unpriviledged process. It just gives all users full rights over that folder. (whether that's a great idea is a different question, but UAC isn't a problem if you just set your access rights correctly)
Steam has definitely been handling Program Files wrong for as long as Program Files has existed. It's a marvel that there isn't more historic Malware intentionally targeting Steam install paths, though it looks like that storm is brewing on the horizon as I've seen references to such malware in the process of becoming a problem.
Steam should have worked to figure out a better approach to UAC years ago. At this point it is one of several big tech debt issues with Steam that has my confidence in Valve at an all time low as a consumer.
If the user never had admin priviledges then you can't install in C:\ either and the program should just install in AppData\Local (as my sibling poster correctly stated)
The problem there is "how lazy is VendorX"? With \Program Files\ Microsoft has a ton of top level ACLs to lock down access to the directories that everything just inherits by default. How much work does the OS need to do to allow something like "folders named `data` under \Programs\ have ACLs XData and `code` have ACLs YCode"? Does it need to watch for silly mistakes like Devs that localize `code` (`Código` for instance) or misspell it (`ocde` or `coode`)?
Also would make it easier to backup the program and its data or copy them to a new system.
But it seems because of the idea that some people may want to back up data and config separately, we end up with stuff scattered everywhere, with some even hidden away in the Windows Registry. Makes backups ridiculously complex.
I always thought it was because that space in “Program Files” was troublesome for a lot of programs, particularly command-line ones. IIRC, older versions of MinGW for instance would simply not work at all if installed in a path containing a space, because some of their startup scripts didn’t properly quote all parts of the path.
Default gopath was just a way to let newbies get up and running quickly; it originally required you to specify where to put it. I imagine they put it at ~/go so it would be glaringly obvious. Hopefully modules will render it all moot anyway.
They put it at ~/go because that's what every official document used as the example $GOPATH, before it had a default value, so it's largely just to minimize change from what they previously recommended.
That said, for something like source code, it makes sense to go into $HOME, so I don't understand this particular complaint the other poster made.
Other people's code shouldn't be in my $HOME if I didn't put it there explicitly. If it's part of the dependencies of a project, it should be either in a cache directory of that particular project, or in the cache directory of the build tool (that preferably conforms to XDG), like all sane build tools do. GOPATH is an incomplete (thus incorrect) solution to a problem that was being solved correctly and completely in other languages (like Ruby).
It's weird that the Go developers were working in such isolation that they didn't pick up on this. All programming languages that I know of that work on modern tooling (Haskell, Rust, Node.JS, even C#!) adopted the Ruby style dependency management when it turned out to be correct years ago.
In my comment I say that it's better than the way Go did it, and Go is switching to the Ruby style, so maybe you could add some kind of argument why it's actually not better than that in any other language?
I can't remember the last time I had a version management problem in Ruby, though the ecosystem is so stable right now not many backwards incompatible things happen any more.
Lots of programming languages and IDEs seem to do this. Rosegarden (the sequencer) and Steam (last time I tried anyway) both create folders right under my home folder as well. Barbarism is right.
> And when people complain, the devs have the gall to suggest that this is all fine
From my impression the answer was more "we set up things in a hurry because of deadlines and now we are stuck with this until we implement the epoch system"
> And of course, all this is on top of the new fashion for the OS to "helpfully" create Documents, Music, Video, etc folders in my home folder, and set the XDG environment variables to point to them. Noooooope.
Yuuuup. It baffles me how Linux users can simultaneously complain that 'its never the year of the Linux desktop' and then at the same time rile against basic usability tenets. The average users wants/needs Documents, Pictures, Videos, etc folders, and it wants the OS to be aware and use these as such.
Now I'm okay with being able to set a specific flag that doesn't make these folders during install, but by default they absolutely should be there, unless you're doing a server install.
Well, I know that I hate them. And I know a lot of other people that hate them. And I'm pretty suspicious of 'features' that allegedly make life easier for naive users by definitely making life harder for experienced users.
And I guess I would call it a "basic usability tenet" that the computer does what I tell it to. Documents, Music, and Videos folders violate this, because if I do the seemingly-obvious thing of just deleting them, the applications that want to use them will just recreate them unless I edit ~/.config/user-dirs.dirs. So, if developers really think they need to 'help' their 'average users' by creating these folders by default, they should finish the job and honor my desire to not use them, which I've expressed by deleting the damn things.
Also, if I want to rename them, the system should honor that. E.g. if I rename "Music" to "music", applications should not then go and re-create a "Music" folder.
Or, you know, they could just simplify my life and theirs by not creating these 'special' folders in the first place.
Damn straight. Where does the system get off trying to tell me how I should organize my stuff? Keep system stuff in a folder for system stuff and the rest of the file hierarchy is mine, because it is my goddamned computer. The original Mac got this right and no one else has since.
Snaps sound good in principle and aren't a horrible solution when they work, but despite the tagline ('Snaps are universal Linux packages') it's not clear that they work well outside of Ubuntu: they aren't sandboxed unless Apparmor is present, and despite their touted 'universality' have a hard (and unnecessary) dependency on systemd.
At this point, Guix and Nix seem the most promising for cross-distro packages.
Do you just have a /snap directory and no $HOME/snap? Normally, snap creates /snap for the binaries (shared among all users) and $HOME/snap for the per-user configuration/data files.
I'm not surprised, I left ubuntu a decade ago because their way of doing things was causing me headaches and they did not seem to care, all that seemed to matter was bug 1 : becoming microsoft in place of microsoft.
I don't understand the whole Snap thing. Not a single snap has ever worked in any of my computers in the past 5 years (I've tried maybe 5 or 6), and they always say they'll add themselves to the system. Add where? Where are these beasts hiding?
I just «snap install pycharm» and it shows up ready for work. Never had an issue with a snap. Can’t recall where it’s stored, but that’s easy to determine with «which». This is on ubuntu 1{6,8}.04 fwiw.
Oh, I was talking about AppImage, not Snap. Sorry.
I actually tried the Snap thing too, I don't remember how it went but I would say it was bad. I think I couldn't find a way to search for packages from the CLI.
If you're speaking of AppImage, what do you mean by "where are they hiding"? AppImages are quite literally exactly where you put them and nowhere else.
Probably that was appimaged, the daemon that handles some convenience things like menu integration and attempting (but usually failing) to show appimage files with icons. appimaged is entirely optional.
>And when people complain, the devs have the gall to suggest that this is all fine.
You're complaining on a really high level and nobody suggested "this is fine". They suggested "wishlist" is fine according to their definition of "if it's not immediately broken and a fix is not trivial"
If you have an issue with default folders that are quite normal in a modern desktop environment I suggest you move back to Arch or wherever you came from and install i3 again?
Also, the bug was first reported in 2016, and Mark Shuttleworth himself has commented on it. And yet it's still not fixed. That gives me the impression that the devs don't consider it super-important. Which is odd to me, since I consider not doing this kind of thing to be 'table stakes' for any application that wants me to take it seriously.
> If you have an issue with default folders that are quite normal in a modern desktop environment I suggest you move back to Arch or wherever you came from and install i3 again?
Ummm, I came from Linux before they started having default Documents, Videos, etc folders. And, like I said, the nice thing is that, in Linux, I can change the XDG environment variables and delete the (IMHO) foolish Documents, Music, Videos, etc folders.
I've only been using Linux for around 10 years as my only desk and laptop OS and this is the first I heard of XDG user dirs! This laptop runs Arch. Thanks for the heads up.
I'll drop "/etc/papersize" in return. If you are not familiar with this and do not print to letter paper then you will find it very, very useful.
It really does make CUPS, Libre Office and all the rest believe you when you say "I want to print on A4, yes A4, no not fucking Letter, A4, end of!" Obviously, substitute your paper size of choice. It probably even lends itself to some form of scripting and perhaps could be changed in reaction to events from say NetworkManager. Whatever.
My life at home is now worth living every time the wife hits print. Such a simple idea - brilliant.
The real problem is that any app I run accesss all files I own with equal permissions. App1 and App2 shouldn’t write their data to my home, or read each other’s data, or, please, be able to write each other’s data.
A video game can read my tax forms. WTF.
Mobile & tablet OSs solve this. Everything has to follow.
Until then, basically every app should run in a docker container. Then it can do what it wants.
Mobile & tablet OSs solve this. Everything has to follow.
No no noNO!!!!
Think of all the things you can't do on a mobile device that you can do on a PC, or which require ridiculous workarounds like uploading from one app to some server on the Internet and then downloading again into another. Effortless data interchange is what makes computing great, and stops proprietary walled-garden data-silos from becoming the norm.
There is a middle ground. Universal data interchange formats but sandboxed on a per directory/app level. Imagine how easy it is to do docker run --volume=~/tax_forms. Put some UX on top of that, plus something resembling Android Intents for escape hatches without giving out full access, and you will have a fantastic solution.
Implementation really really is key when it comes to the UX here though. Windows actually already has this feature (Controlled folder access) but it's done so badly you'd almost think it's a joke. Take this as a warning of how bad things can be if you just sandbox everything without thinking. Enabling the feature accidentally is deceptively simple without any warnings on how difficult it is to use: "Protect files from unauthorized changes by unfriendly apps" is all it says, who wouldn't want that? After that havoc breaks loose. You can only have ONE group of folders protected, you must have Documents protected. Whenever a new program tries to write in a protected folder the program just fails, then 15 seconds later you get a confusing prompt saying excel_32_broker_confusing_name.exe tried to write to protected area, clicking the prompt takes you to a page where you can't simply press yes to allow, you have to manually browse to find the exact program location, good luck finding c:\program files(x86)\office\bin or whatever they decide to call the exe.
Agreed, and we’re now in serious danger of “Okay button fatigue” or whatever you’d like to call it.
If every normal app has to ask for permission to use your disk, you’re just going to click “Okay” when malware asks. Then the whole system was for naught.
Maybe just having some groups, like Firefox's containers.
It's less secure, but a simple setup would drastically reduce the number of programs with access to important information.
So just don't do that. The application needs an "Open" dialog already so you can tell it what files to open, just have that dialog be a process run by the system instead of the application, treat the permission as implicit since the user is the one selecting the file, and pass a handle back to the application.
What can you do wrt file-management on your laptop that you can't on your phone? I have a file manager on my phone that can send files to whatever app needs it; why should a random game have access to arbitrary files?
Exactly! I remember the days of proprietary data formats, when people would literally have to print out data from one program to type it back into another. Let's not regress to that.
You can create another user and sudo / gksudo to it when working on sensitive data. It never occurred to me before reading your comment. I could start doing it myself.
Indeed the capability exists (unix/linux pioneered it?), but it relies on manual containment by the user. It should be automated, with prompts at install or execution time, like on mobile OSs.
An idea is each file should have its own user, basically, with both relative and absolute file creation/modification permissions. If you run an arbitrary executable as a user there's great security risk since it can do anything you can (without sudo of course).
It seems like the natural progression from
Run everything as root ->
Separate root and day-to-day user ->
Don't use root and achieve things with sudo ->
Each file has its own permissions (i.e. essentially its own user) set by the parent user
Permissions should be both for individual actions and permanent classes of actions, on the user's discretion (i.e. allow it to create this file or always allow it to create files)
This approach is even naturally hierarchical, executables could create other executables as long as they have permission to do so, and set the child permissions to at most its own. In this context an executable asking for permissions can be akin to sudo: it really is a file-user asking for its parent file-user for expanded permissions.
To give a real life allegory, consider a Technician in a company wants to make a tool purchase, so he asks the Engineer. The Engineer doesn't have permission, so he denies or asks the Manager. Finally the Manager either denies or asks the CEO which has permission over everything.
There are differences between the apps I use on my phone and the programs I use on my PC.
I don't trust many of the apps on my phone (all of Google, FB, messaging, etc). I trust the open source apps (example K9 email) and I'm replacing proprietary apps with open source ones.
I trust almost all programs I use on my PC, which I install with apt-get from Ubuntu or other open source PPAs (no snap, no flatpack). There are a few proprietary programs I might doubt of, for example NVidia driver, Skype, TeamViewer, Telegram. I understand that one is enough to compromise all the PC.
I'm ok with separating untrusted apps/programs but I don't want to get permissions in the way of programs I trust. Software development would be a nightmare if emacs and vim and all GNU userland and interpreters, compilers, etc would run in different environments.
That's because you have to. I trust all my programs too, because I wouldn't install them if I didn't trust them. We've spent decades telling people only to install applications they trust, and not to click on suspicious executables. The result is that they are very resistant to install new programs, unless the programs are from organizations they already know and trust.
That's partly why web applications are so popular: it requires less trust to use a website than to run a native application. If native applications were less dangerous, people might be willing to use more of them.
Valid point. I thought about this, for system apps you would be able to assign lasting permission requests to read/write/modify anything., by default -- so it would work as right now. Just like in browsers, permissions (permission type prompt) could be remembered in a cache. If you're not comfortable with giving any command full reign on your system, you could simply give granular, temporary permissions as required.
A concept I find useful is that there are two kinds of security:
-- Accidental security
-- Malicious security
Permissions help with both. Not only you don't want 3rd parties to invade or disrupt your system, you don't want users to invade or disrupt your system (or their own systems) accidentally.
A classic example I believe is an user wanting to delete all files in current directory, typing
$ sudo rm ./*
and missing the dot, getting
$ sudo rm /* #(delete *all files*)
Ideally in this case you would see two different requests:
> The command rf requests root access for file deletion at /
> Accept? (y/N)
> User's administrative password:
or
> The command rf requests access for file deletion at /home/user/cache/
> Accept? (Y/n)
Note you're not prompted for password in the second case because you already have permissions for this operations (only the process 'rf' doesn't).
You may not be particularly worried about malicious security for system programs, but even professional administrators could be worried about accidental security and best practices (if they're not they could easily select persistent full permissions). I get the impression the last thing e.g. sysadmins want is mess up production systems, and wouldn't mind a few more prompts when doing things manually (which should be rare?), or assgining scripts the exact permissions needed for their job.
They're painful to set up yourself, but that's pretty much true of all current sandboxing approaches. If you just let your distro do it and follow their conventions it you barely notice.
It seems like MacOS has been headed in this direction, at least with the App Store apps. But it's a hard problem to elegantly implement without breaking everything.
I still see a lot of Mac apps that basically say "disable system protection first."
Yes, this entire discussion is part of a whole other problem: there's too many properties of programs one has to trust esp. on desktop (or alternatively be stuck configuring selinux for days).
One should/must/can not build an operating system on trust alone, as especially the mobile software space consistently continues to show with its numerous malware apps (imagine that on the desktop! "oh sure, have root/admin '50 smileys FOR FREE'", "oh sure, be able to read and write all my user's files <insert dubious clicker app>").
Mobile OSs have become vastly better (not perfect still) than this than desktop OSs.
I would like to see desktop Linux's application management adapt a minimum necessary permissions policy: Allow access to some ressource/API/permission only if the app needs it (and the user allows it of course) and offer mechanisms to allow permission management and ways to enable inter-application communication. This also modularises applications more by bundling up their connections to the rest of the platform and their effects in a central location.
For instance sandboxing applications into their own empty folder would make it simpler to operate on files these applications created (backup, migrate, autocreate <- NixOS would benefit from these especially, allowing finer grained declarative control). It would still be okay then for application developers to follow their own conventions within that sandbox.
Nowadays files are all bunched up in arbitrary folders and nobody knows where they're coming from. It's become so much harder these days to coordinate so many projects that need to create files in the home folder on Linux by trying to introduce standards versus taking that capability away from programs transparently; this should also be much more efficient man-hours-wise.
That said, this is going to be a nontrivial project with how many implicit connections between applications exist. Executing untrusted (or buggy!, I'm looking at you, Steam) software should no longer have to make me afraid my system gets messed up.
For now I think it's best to simply not run untrusted programs and creating regular backups... I wish I had the time/skills to work on something like this.
Just changing the HOME environment variable, which would be what such a hypothetical utility would boil down to, does not address what sakoht was talking about, which is Hurd-style subsetting of a user's account and files.
This is already solved by Linux, as you probably know. You can just use groups and users if you want. Also you can run Docker containers if you want, just don't force me to do it, as they're stupid.
There's a very simple solution to this: Don't run apps that you don't trust.
If you simply must, use a VM or AppArmor or SELinux. Don't inconvenience everybody because you're too lazy to be responsible for your own data.
Besides, untrustworthy apps will bypass the protections anyway. Judging by the popularity of "curl <url> | sudo bash" lately, they'll probably just ask for root directly.
I have some degree of distrust of most of my applications. The "open trust" model usually ends up getting abused to the point where we have to lock it down...like early Windows, wide open for easy of administration, leading to things like the notorious "Blaster" worm, or the open architecture of the Internet, where we now have massive spam and DNS amplification attacks and all sorts of problems based on the notion of a trustworthy network.
In general, everything in computing will keep going in the direction of "trust everything as little as possible for it to do its job" forever, I think, and probably has to.
Running untrusted programs is the default way we do computing. No one has time to audit every single program they run. The current desktop security model comes from when people mostly ran the programs that came with the OS and programs they wrote themself. Its vastly inappropriate for modern computer usage.
One thing I’ve been thinking about is that it would be nice to tag files with the program that created them. Aside from the clutter in my home directory, I have a lot of files that I have to guess as to where they originated from based on metadata like modification timestamps or permissions, but it would be nice to see something like “~/.config was created by some-tool”.
Apple's Macintosh did this in 1984. Files were tagged with 32 bit codes: one for the "creator" (the associated app) and one for the type. These were often made up out of readable characters: a so called "fourcc" (four character code). The "fourcc" persists in cross-platform media formats.
The first software I ever sold online was an app that would change the creator code of a file to whatever application you wanted. For some reason, there was no obvious built-in way to do that in Mac OS 8 (besides opening the document in the other application and saving it, or changing it in ResEdit).
I only ever sold one license, but I was 12, so I was pretty excited.
Why wouldn't it to be easy to implement? You could probably make a global LD_PRELOAD library that hooks into all open() CREAT calls and tags the inode's xattr with the program that created the file. You could use an attribute like, user.created_by.
Why such a hack? This is a solved problem already. Try Linux audit API. You can set it up for tracking various actions, and you'll get a log with all the information including a process name.
I believe the question was asking for a method of storing the metadata, I was just suggesting xattrs. This will work at the expense of running a daemon.
Can audit be configured to only log file creation, not modification or access? Last time I didn't find a way to do that and I don't want to write a log file entry for every file access.
Programs can use programs like 'touch' to create the files so you'd have to get meta into a chain of process ownership or something while exempting the user's shell.
I think touch is a generic example, but not a practical one. The idea is that program A can launch program B, and you'd have to recursively search for the parent. But it's not easy to find which parent is the true owner, maybe an exception can be made for init but shells can't have that exception as they are sometimes the owner.
MacOS does something sorta like this. It uses extended attributes to mark where a file came from in some cases, such as when an executable is downloaded from a browser, it will mark it as "unsafe" and when you attempt to run it, this causes it to fail to run and you have to whitelist it via security preferences.
Adding an interface to access the tag store won't be very hard.
It would be harder to make it so that all the tag-oblivious programs, like mv or vi, to say nothing of rsync, would preserve the attributes when moving or modifying a file.
f = open("the_file.new")
write(f, new_contents);
close(f);
rename("the_file", "the_file~");
rename("the_file.new", "the_file");
So the old file is never modified, it's renamed to the backup copy, and an entirely new file is created to take its place.
This has a number of advantages, but does not play well with any extended info the old file used to have, unless they are copied explicitly. And in a tagging-oblivious program, they won't be.
f = open( "the_file.new.$$" );
write( f, new_contents );
close( f );
rename( "the_file.new.$$", "the_file.new" )
If not implemented like this, then something that attempts to read "the_file.new" may get partial contents, normally truncated along the file system block size.
Backups often take place, too, like you mentioned, depending on editor.
If the value matches, you renamed the right file. If it does not match, you renamed a wrong file. Without holding the fd, it is impossible to know if it is the right file.
Thanks for teaching me this! I wasn't aware. So, the original xattr (and the program that created it) would be deleted and replaced by vi for example. Still technically right, but I'm unsure how you'd keep a history.
Please no. This article is literally "Dotfile madness", don't make it worse!
I have to deal already with these on file shares, specifically for Apple: .DS_Store, .Trashes and .AppleDouble, or for Windows: Thumbs.db, $RECYCLE.BIN (for some reason Windows sometimes ignores the fact I've disabled the recycle bin on a share and creates this instead) and desktop.ini. Please don't drop crap around directories where there exist a multitude of tidier alternatives.
Because when you have multiple OSs accessing the same shared folder, they all create their own crap, which is then visible to the other OSs, and fills up directories with stuff that confuses normal users.
I don't run multiple OSes, but I do use USB sticks and SD cards on other people's computers and vice versa. I'm pretty sure that is "normal user" behaviour.
It gets annoying real quickly because you see the "crap" files of all the OSes you're not using. And you delete them, you only have to insert the stick and they're back. This happens in different ways for all of Mac, Windows, Android and certain Linuxes.
Sidecar files for storing file tags make querying the tag system a huge chore; it totally kills performance. I understand the conceptual appeal but it's just not the way to go.
There's an entire research field dedicated to the information flow control which could solve just that (if it were actually used).
In my research team, we used a tainting tracing mechanism to understand the behavior of malware. Basically, we installed a malware on a clean phone and we then traced all information flow originating from the APK to processes, to files, to sockets, etc. It helped reverse-engineering the malware.
I'd love something like this on program level. Given contents of a variable, I'd like to know where they came from - which pieces of code contributed to the result. I'd also love to be able to mark a piece of data, and see what code touched it or derived new data from it. A programming equivalent of injecting radioisotopes into the body.
I’d suggest looking into taint analysis tools, though those are usually aimed more at finding things like unsanitized input ending up in a call to system.
BeFileSystem (Haiku)
HFS+ and APFS
Even NTFS has attributes.
The problem is file manager. Tracker/Finder are several steps ahead of Explorer. Under Linux, its a greater mess even though there are no technical limitations.
What's not quite so straight forward is how you query the system. The worst implementations of file tagging only implement retrieving a list of all files that have a given tag. Slightly better than this are tagging systems that will return the intersection between two or more tags. Most tag systems never go beyond this.
Going slightly further, tag exclusions are powerful and are sometimes implemented (given a set of files from some subquery, exclude all files that have a given tag.) What you rarely see are systems that allow you to exclude one subquery from another.
However what you almost never see implemented is a system for preferential tags; given a subquery, reorder the results such that files with preferred tags are raised to the top, ordered by how many of the preferred tags they have. Once you implement this, the system's UX changes dramatically because the user no longer has to make strong assumptions about how well their files have been tagged. Many files might be missing relevant tags and the user may not be sure if the file they're after is one of these incompletely tagged files. When using a system with preferential tag querying, the user will receive files in their result list that don't match all the tags listed, but most of them. This is a bigger deal than it may sound, since the main drawback of using tags for file management is incomplete tagging. By addressing this drawback, you stand to bring file tagging to the next conceptual level, which is rendering hierarchical file management obsolete.
Consider that hierarchical file management is a strict subset of file tagging. You can model file hierarchies inside a file tagging system. To demonstrate this by example, consider /home/joesixpack/Documents/seventh-novel.pdf For each level of the hierarchy, we can create a new tag, such that this document has the tags: '/home/', '/home/joesixpack/', '/home/joesixpack/Documents/'. But because we're using file tagging, we can also automatically tag that file with things like 'pdf' or maybe even 'Documents'.
Now before I go on, there is something to be said about the number of tags in the system exploding when you model a tree as a tagset. In practice this probably isn't the approach any real system should take, if only because most of those tags will be useless and because there is a great deal of redundancy in trees that wouldn't exist in a native filetagging system. Consider /home/joesixpack/Documents/ and /home/johnsmith/Documents/. We have two different Documents directories because they exist in different parts of the hierarchy. However in a file tagging system we'd ideally only have a single Documents tag and one tag per user, such that querying the intersection between 'joesixpack' and 'Documents' returns the files that would otherwise be in /home/joesixpack/Documents. Some tags, such as the user tag, could be implicit; such that if joesixpack simply queries 'Documents' the tag 'joesixpack' is automatically intersected with the results presented to him.
With that out of the way, consider how we can go further if we apply straight forward statistics to the tag system. Suppose 99% of joesixpack's 'Documents' that are 'pdf' are also 'novel'. When joesixpack creates a new file that's in the tags 'Documents' and 'pdf', what are the chances that file should also be tagged with 'novel'? We could analyze the contents of the file if the system has a semantic understanding of what 'novel' means, but let's not go there. Tag systems created by the likes of Facebook and Google do this sort of analysis, but it's heavy, tricky, goes wrong in ways that create PR disasters, etc. We can get pretty good results by ignoring the contents of the file and looking instead at merely what that file is already tagged with. If a file is tagged with 'joesixpack', 'Documents', and 'pdf', it _probably_ should be tagged with 'novel' as well. Probably, but not necessarily. So the system can expose to joesixpack the suggestion that he tag that particular file with 'novel'. By presenting suggestions like that to the user, you greatly reduce the UX friction needed to tag files well and therefore increase the usability of the tagging system, while at the same time improving the quality of the tag suggestions in the future. What we create here is a 'virtuous cycle' of sorts that gives back to the user more as it's used more. Such a tag suggester can be implemented as multi-label classification using naive bayes; it's fairly straightforward.
Going further; if we have such a tag suggester, and an appropriate caching scheme, tag suggestions can be used by queries. If joesixpack queries for 'pdf','novel', the system could return to him the intersection of 'pdf','novel', but also return to him files that are tagged with 'pdf' but not tagged with 'novel', in cases where the tag suggester indicates there is a high likelihood that the file should be tagged with 'novel'.
That last paragraph may or may not play out well, I haven't experimented with it extensively yet. But getting back to my original point: if you're going to implement file tagging in a filesystem, you should think carefully about how the users will query that system, and whether the query system will be extensible in userspace to facilitate new powerful methods of querying it. It would be a total tragedy if the system only supported basic tag intersections and the only way to extend it was to implement a new kernel module.
This would require a ton of difference from how modern desktop and mobile OSes handle files / documents. Maybe some time later, some research OS would implement it.
I've implemented the above in userspace for my own experimentation (not yet ready for public release, but maybe soon.) Nearly everything I've described above can be implemented with relatively straight forward SQL queries and SQLites performance has been everything I could have asked from it (in the neighborhood of ten thousand tags and hundreds of thousands of files.)
Getting file tagging into the kernel level to replace directory hierarchies would be a huge paradigm shift, a very dramatic departure from Unix. To be honest I'm not sure whether or not getting such a system into a kernel would be appropriate or not. Traditional hierarchical file management seems more than sufficient for "system files". But I'm really interested in replacing hierarchical file management from the users' perspective. More or less, put ~/Documents, ~/Downloads, ~/Desktop, etc under control of the tagging system but leave the rest of the system as-is. At least for the proof of concept.
I think a demonstration system could be implemented as a custom desktop environment running on regular old linux, where the open and save dialogs of GUI applications have been replaced with tagging/querying windows. Instead of the user clicking through a few directories to find a file to open or find a directory to save something, they would instead click or type in tags to add or query. The GUI file manager would likewise be replaced with the GUI frontend to the tagging system.
The story I heard was that performance sucked. Granted it could have sucked for other reasons, as other failed efforts like writing more OS components in C# were happening at the same time.
If we start sandboxing applications and give each one its own UID, like Android does, we get this feature for free. It's kind of silly to associate exactly one security context with exactly one user.
I think saagarjha's point was that an even simpler standard would be for the creating application to use file metadata to annotate the files. And a simpler standard is generally more likely to be followed, especially if it's unilateral.
This doesn't help with installation, obviously, but it helps with cleanup.
not sure if this is the exact solution to your problem, but rcm (https://github.com/thoughtbot/rcm) allows you to create dedicated dotfiles in a .dotfiles directory which are being symlinked back to files, and it has a -t option to create and restore dotfiles with tags. mkrc -t emacs .emacs.d would create a .dotfiles/tag-emacs/ folder and rcup -t emacs restores it.
>One thing I’ve been thinking about is that it would be nice to tag files with the program that created them.
Isn't what you describe basically the file extension? Not so much what program created the file, so much as what program is meant to handle/open the file.
File extensions are more nomenclature than file type indicator. More than one file type might share the same extension, more than one application might be used to open the file according to its extension. They mostly work, but file extensions != file types != file creators.
File extensions are trivial to fake, don’t really have a strong association with the application that is supposed to handle them, and in this case often doesn’t even apply because most of the files in my home directory seem to lack an extension.
The lack of useful names on the files is IMHO the most egregious issue. (OK using names that don’t start with ‘.’ is worse). I don’t ls -A my homedir as much, but what the hell are .517827.padl? Who created .android (I don’t use android and don’t develop for it) etc.
These people are arrogant shitheads and I’d like to know who they are so I can avoid their software. I assume if they do crap like this their software does other nefarious things.
> These people are arrogant shitheads and I’d like to know who they are so I can avoid their software.
Wow, this is really assuming the worst in people. It’s much more likely to be ignorance because the other developer doesn’t know where the file came from either (some compiler flag on a library accidentally enabled), or also doesn’t ls -a in their home.
> Wow, this is really assuming the worst in people.
Yes, I am. I'm fed up with programs that spam my system without asking: install plug ins, menu items, directories on the desktop, dock items etc for "convenience" -- it's convenient for them, not me. It's especially painful when I pay for an app and it sprays shrapnel through my filesystem.
> It’s much more likely to be ignorance
Indeed, but if they can't be bothered to learn basic system hygiene how can I be confident the rest of their package is safe to use?
A big driver for sandbox/containers in user systems is not protection against malign actors per se but against the lazy and ignorant.
Have I written bugs into code? Of course I have, I'm human. But at least I try to get things right up front; to be a good citizen.
(there are additional reasons for containers in server systems, hence my qualification of "user systems" above).
A "compiler flag on a library" that leads to garbage in your home directory => bad library => developer's failure to vet and research their dependencies.
Perhaps having all parts of this chain called "shithead" is extreme but they would all have responsibility and disrespect for the user.
But nowadays, lots of applications seem to think it's OK to put normal files in my home folder without asking first. This is just outright barbarism. In particular, Ubuntu now seems like it's all-in on snaps, and if you use snaps, they create a non-hidden "snap" folder in the user's home folder. Madness! And when people complain, the devs have the gall to suggest that this is all fine:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1575053
I like the idea of snap/flatpak in the abstract, but until they get their act together, and stop creating non-hidden files in my home folder without asking first, I am never going to use snaps.
And of course, all this is on top of the new fashion for the OS to "helpfully" create Documents, Music, Video, etc folders in my home folder, and set the XDG environment variables to point to them. Noooooope.
But at least in that case the user can change the environment variables and delete the stupid folders. No such luck with ~/snap.