
Don't tug on that, you never know what it might be attached to - JoshTriplett
http://blog.plover.com/tech/tmpdir.html
======
krylon
As a sysadmin on a Windows network of ~100 computers, this story makes me want
to cry, although maybe for the wrong reason:

I see weird problems of the sort "It _did_ work before I went on my lunch
break" on a fairly regular basis. How often would I like to go down the rabbit
hole and explore these problems in such depth, but if I did that, I would
hardly get any work done. The frequency at which our users run into these
problems is just too high.

It is so frustrating so restart a computer or maybe re-install it and see the
problem disappear, because now all hope to understand what caused the problem
in the first place is gone. And problems that disappear for no good reason
have a tendency to return for no good reason, most often on a Friday
afternoon, just when you're about to call it a day. ;-/

~~~
MertsA
This is what I hate about closed source software in general. Every sysadmin
out there has complaints about how terrible support is from XYZ vendor, but
it's not just that the vendor can't provide support, it's that by going closed
source, the vendor is the only one who can really track down these sort of
bugs. With an open source stack, no matter what breaks in any software
component, I'm not left high and dry hoping that some vendor who charges
buckets of cash per incident is competent. As a sysadmin, I'm putting my neck
on the line anytime something breaks. I need to be able to fix things when the
vendor fails.

~~~
tonyedgecombe
A good vendor will work with you to resolve the problem, not all closed source
software is bad.

~~~
wpietri
I'm familiar with that theory, but from my personal experience I'd put the
number of good-vendor software packages around 1%. And in retrospect, I'm not
sure those cases were so much "good vendor" as "vendor over whom we had
significant financial leverage".

~~~
Joeri
From my personal experience I've only gotten consistently good support from
Cloudera and Jetbrains, out of some 20-odd companies that I've contacted for
paid support. So, for me the figure is more like 10% instead of 1%.

------
CoconutPilot
The problem with "newish" features like capabilities, file attributes, SElinux
is they haven't been integrated into the traditional _nix utilities and almost
nobody knows what is going on with them. A few examples of the poor
integration:

File attributes override the _nix permissions such that you can set the
immutable flag on a file and even root can't modify it. `chattr +i FILENAME &&
rm FILENAME`

On distros that use capabilities copying a file doesn't copy capabilities by
default. ie copy the ping program and it wont work unless you're root.

When SElinux blocks an action the error message is almost always wrong. ie A
program tries to make a TCP connection which it doesn't have permission for.
Instead of an error message like "SElinux violation" you get an error like "No
route to host". To debug you need to look at the SElinux audit.log and try to
match up timestamps of violations to when your program died.

~~~
GavinMcG
_nix formatting suggests that the asterisk in_ nix needs to be escaped.
Apparently on HN that requires putting a space after the * though, so you end
up with * nix.

~~~
digi_owl
Apparently HN's markdown implementation is supposed to leave the * alone as
long as here is not another the other end. But there seems to be no upper
limit to where that end may be.

Also, it seems to only check if the * is near something else, not if it is
before or after. Nor if the after is after a before (if that made any sense at
all).

~~~
dozzie
> Apparently HN's markdown implementation is supposed to

What made you think it's Markdown implementation? It's pure text, with
paragraphs delimited with empty lines, code blocks being prefixed by space (or
two, I never remember) and emphasis being marked by asterisks. There's
_nothing_ more.

------
userbinator
Great debugging, and an example of the sort of behaviour that long dependency
chains can expose.

 _Or rather, I did until this week, when it suddenly stopped working._

When this happens to me, the first thing that I ask myself is "what changed?",
and I'm usually able to track down the cause to some configuration change.
Incidentally, this is also why I never like modifying a working system unless
it's absolutely necessary.

The fact that it ultimately was caused by some security features that would be
very important for a multiuser shared server but nearly irrelevant for the
(presumably) single-user local machine that he is using suggests that perhaps
we shouldn't be thinking of "one size fits all" paradigm for OSs, since a lot
of problems like this one stem from the unnecessary extra complexity
introduced by such thinking.

~~~
cantrevealname
> Incidentally, this is also why I never like modifying a working system
> unless it's absolutely necessary.

It's why I find auto-updating apps so infuriating. The trend is that every
app, OS, and driver insists on being self-updating. It's going to be very
difficult to maintain a reliable system if you're doing anything complex.

Privacy issues aside, that's another reason I never plan to use the
continuously self-updating Windows 10.

~~~
groby_b
On the other hand, chasing after CVEs is _also_ infuriating, but in the
opposite direction. auto-update makes it possible to live in an environment
where security issues are found by the bucket load every day.

There really isn't a good answer either way, but between "breaks occasionally"
and "needs a full-time admin, but updates are vetted", I prefer option 1 for
my private systems, and option 2 for things that run in production.

~~~
dozzie
This is where supported distributions come in. In Debian or Red Hat you can
practically assume that the OS update will not break anything. (There are
occasions where it does break something, but I don't remember anything major
in a good few years.)

------
seany
No reference to the full quote? :)

"You can check your anatomy all you want, and even though there may be normal
variation, when it comes right down to it, this far inside the head it all
looks the same. No, no, no, don't tug on that. You never know what it might be
attached to. " \- Buckaroo Banzai

~~~
Animats
The movie is quoting that from a neurosurgeon. I first saw that line in a book
about the Massachusetts General Hospital.

~~~
seany
In the movie the lead character is also the neurosurgeon :)

------
lfowles
> This computer stuff is amazingly complicated. I don't know how anyone gets
> anything done.

Indeed.

~~~
mturmon
In particular, the set of exported shell environment variables is a sinkhole
of state that can potentially affect every program we run.

~~~
gosub
If only we had a way to deal with state without it being mutable...

------
wtbob
What's funny is that in this case the dynamic loader was sanitising something
irrelevant to the actual capability granted, which seems to me should be a
bug.

Also, I'll be an advocate for just starting emacs with systemd, and never
worrying about it again.

~~~
JoshTriplett
> What's funny is that in this case the dynamic loader was sanitising
> something irrelevant to the actual capability granted, which seems to me
> should be a bug.

EDIT: fixed incorrect description of the yak-shaving conclusion.

The dynamic loader did so because it ran with an extra capability that the
user invoking it didn't already have. Most of that sanitizing exists to
prevent the user from gaining those privileges themselves by invoking a more
privileged program, such as by setting LD_LIBRARY_PATH. Sanitizing TMPDIR
prevents a somewhat different class of vulnerabilities, such as using those
extra privileges to write to files you normally couldn't. However, I don't
think it makes sense to have a complex special case like "if you only have one
of this subset of extra privileges, allow TMPDIR but don't allow all the other
potentially dangerous environment variables"; that adds a significant amount
of complexity and subtlety to already security-sensitive code.

Giving /usr/bin/perl itself extra capabilities effectively grants them to
every user on the system, since you can use Perl to run arbitrary code. At
that point, it would make more sense to just allow all non-root users to bind
to arbitrary ports. I'm somewhat surprised that there isn't a sysctl to
disable the reservation of ports 0-1023.

~~~
eridius
No, it was the dynamic loader that did the sanitizing. Mark said that he
thought of Perl's sanitizing, and had ruled it out, and then explicitly said
it was the dynamic loader that did the sanitizing in this case.

~~~
JoshTriplett
Thanks for the correction; fixed. I misremembered that bit of the yak-shaving
adventure when I went to write my comment.

The conclusion still holds, though: I don't think special-casing particular
capabilities makes sense. And in the case of the dynamic linker, it doesn't
actually have that information available; it relies on the AT_SECURE bit set
in the process's "auxiliary vector" (see "man getauxval"), which the kernel
sets when the process has _any_ privilege its caller didn't have.

------
GavinMcG
This drives home the point that (almost) no one actually knows the details of
their full stack.

~~~
digi_owl
And that present day computing is damn hard to reason about, because oh so
much happens silently in the background.

------
makecheck
You should definitely tug on things when you have a controlled test
environment and the time to explore what-ifs.

Much like companies should try to replace their own products (before a
competitor does), infrastructure teams need to force “predictable” upgrades in
a controlled environment on a regular basis. For example: look at your
dependencies, _imagine_ what upgrades are _likely_ to be required in the near
future, and try making those upgrades on test systems to see what could go
wrong.

That approach achieves three things. One, since you’re not in emergency mode
and you’ve used a test environment, any problems that you do uncover are not
going to cause a crisis. Two, if you do this semi-regularly then you’re likely
to see only minor issues. Third, exploratory upgrades give you a lot of time
to fix problems (whether it’s time for your own developers to make changes, or
time to wait for an external open-source-project/vendor to make changes for
you).

------
ivank
Environmental variables have caused some of my more troublesome debugging
experiences:

1) On Windows, VisualVM not being able to find the IntelliJ IDEA process I had
running. This happened because IDEA was started with Launchy, which had a
different TMP directory set, because I used both RDP and the Console, or
something.

2) On Linux, ibus IMEs not working at all in my browsers; happened because I
was starting them in a tmux, and the tmux server was started in my previous
login session, so the DBUS_SESSION_BUS_ADDRESS in the tmux was stale.

------
digi_owl
In the end it seems that all complexity stems from trying to enforce access
privileges on a system that do not gives a rats ass about anything beyond 0s
and 1s.

IF we want to avoid said complexity we basically have to go back to running
only one process at a time, loaded directly from dedicated, removable, storage
when needed.

------
davidu
I read this, but seemed to skip over the part where he explains why this
changed suddenly, when the behavior was documented?

What changed to make the perl become capable whereas previously it lacked the
low port capability?

~~~
ars
He usually started emacs directly. In this case he started emacs via a perl
command, very very indirectly.

Also, a recent change was an admin setting TMPDIR, previously it had been left
unset.

So it was a combination of two things that triggered it, and not one alone.

~~~
raarts
> So it was a combination of two things that triggered it, not one alone

And this basically sums up all the troubleshooting time sinks I've experienced
in my career.

------
dataharry
The real question is why in the world you'd move tmp.

------
chris_wot
I almost hesitate to say this, but it seems to me that emacs needs a new
command line parameter to allow the user to specify the location of the socket
file.

~~~
catern
In case you're talking about emacsclient, that has one, --socket-name, as the
article says.

If you actually are talking about "emacs" and not "emacsclient", then the
argument you want is --eval '(setq server-socket-dir "/what/ever")'

------
carterehsmith
I did not quite get why is he starting emacs with "git re-edit" with re-edit
being some Perl script. What is the role of git in this setup?

~~~
borkabrak
Just that, when you run '$ git nonstandard-subcommand', then the git
executable looks for anything executable in the path that is named 'git-
nonstandard-subcommand'. Then it runs it. That's all. But at that point, git
_could_ have been doing something to the environment before invoking the
custom script.

------
legulere
A pretty good showcase that the more features you have the harder it is for
them to keep working correctly together.

------
fchopin
> This computer stuff is amazingly complicated. I don't know how anyone gets
> anything done.

------
kr0
tl;dr Don't use unnecessary amount of scripts and tools to do what vim + git
do just fine.

~~~
mort96
Why even bother to learn programming if you can't use it to automate common
tasks?

