Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open.
I really don't give a shit how it's not "technically" broken - that's broken from an experience point of view. And I haven't re-installed the OS (this was an App Store upgrade from Snow leopard) because that's a major pain in the ass as this is an actual workstation used to do actual work.
I can't believe this is actually advice, either - that's what Windows users used to say in the 90s. Anyway, I guess I'm just ranting. OS X is wonderful except for the fact that it sucks at managing memory to keep a system snappy.
That's not swapping. That delay is /usr/bin/login searching the system logs so that it can display the date and time of your last login.
Create a .hushlogin file in your home directory to prevent that.
1. The default use of /usr/libexec/path_helper to manage your $PATH.
2. An accumulation of log files in /var/log/asl.
For (1), I just edit /etc/profile and disable path_helper altogether. I set the PATH manually. (This also allows me to put /usr/local/bin before /usr/bin, which is my preference. I've never understood Apple's default settings for $PATH. They put /usr/local/bin later - which defeats the whole point of installing, say, a newer vim in there.) For (2), a cron or launchd job can take care of it.
Really? Are you sure path_helper slows things down?
I'm sure that it did, but not sure that it does. The code you ran isn't quite what /etc/profile does. Here's a run of that on an older machine where I work (see below on versions):
$ time eval `/usr/libexec/path_helper -s`
$ time eval `/usr/libexec/path_helper -s`
0) `login -pf`
1) quietlog = 0
2) if ("-q" in argv) quietlog = 1
3) if (!quietlog) getlastlogxbyname(&lastlog)
4) if (!quietlog) quietlog = access(".hushlogin") == 0
5) dolastlog(quietlog) ->
6) if (!quietlog) printf(lastlog)
You can see from this that the "searching the system logs" (which, to be clear, is going to be really really fast: /var/run/utmpx is a small file with fixed length fields) happens in step #3, before .hushlogin is checked in step #4.
If you wish to verify, you can read the code at the following URL. Note that __APPLE__ and USE_PAM are defined for the OS X distribution of this code, while LOGIN_CAP is not.
Look at the code for getlastlogxbyname(). It does an ASL query for last login, and that's the source of the delay.
(edit: I have gone ahead and verified your statements regarding getlastlogxbyname now being based on ASL. Using that knowledge, and based on st3fan's comments about the output of dtrace, I then used dtruss to verify my own assertion regarding the order of events. The result: .hushlogin in fact only affects the output of "last login"; it does not keep login from getting that information in the first place with ASL. To keep it from doing so you must pass -q, something Terminal does not do.)
The correct way to bypass the ASL query is to set Terminal to open shells with /bin/bash (or your shell of choice) instead of the default login shell. Terminal will still use /usr/bin/login to launch the shell, but it passes the -q switch to prevent the ASL query.
When I dug into the source code a couple of months ago, I inadvertently made both changes (Terminal settings and .hushlogin). Clearly it's the Terminal settings that solved the problem and not .hushlogin. Thanks for clearing it up.
(edit: I have gone ahead and checked: thought_alarm is correct, in that getlastlogxbyname is now using ASL instead of utmpx; however, I have also verified my sequencing assertion with dtrace: .hushlogin has no effect on the usage of ASL, but manually passing -q to login does: it thereby cannot be the source of a .hushlogin-mediated delay.)
However, assuming that is the case for some people, we have to look elsewhere than the last login lookup. There are only a few other usages of quietlog: motd (open file, read it), mail (check environment, stat file), and pam_silent.
The first two are not going to cause any kind of performance issue, so we have to look at pam_silent. This variable is particularly interesting, as it is only set to 0 if -q is not passed (and it is not) and there is no .hushlogin (it is not directly controlled by quietlog).
If it is not 0, then it is left at a default value, which is PAM_SILENT, and is passed to almost every single PAM function. It could very well be that there is some crazy-slow logic in PAM that is activated if you do not set PAM_SILENT.
Given this, someone experiencing this issue might look through the code for PAM to see if anything looks juicy (and this is something that will best be done by someone with this problem, as it could be that they have some shared trait, such as "is using LDAP authentication").
(edit: FWIW, I looked through OpenPAM, and I am not certain I see any actual checks against PAM_SILENT at all; the only mentions of it are for parameter verification: the library makes certain you don't pass unknown flag bits to anything.)
kore:~$ ls -l .hushlogin
-rw-r--r-- 1 jay staff 0 Aug 15 2002 .hushlogin
Am I missing something? `w` on my linux system takes well below one second:
[burgerbrain@eeepc] ~ % time w
14:11:18 up 19:41, 6 users, load average: 0.27, 0.10, 0.14
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
w 0.01s user 0.05s system 61% cpu 0.096 total
Here the majority of files being opened are in /var/log or Homebrew related.
Also interesting ... creating a .hushlogin did not change much. It still opens about 50 files in /var/log/asl/
Anyway, instead of running /usr/bin/login, I just use /bin/zsh as my Terminal.app startup command which is much faster.
However, every time I access the file system, even some tab for autocompletion, it takes a few seconds. Even cd'ing into some directory can sometimes take a few seconds. Sometimes ~= it's more than 1h ago that I accessed that dir or so.
Edit: Maybe it's Time Machine or Spotlight or so which destroys the effectiveness of the VFS cache?
Normally Unixes (Linux included) use a pretty efficient binary file called wtmp for that, I'm surprised if OS X doesn't. Reading the last disk block of that file would contain the last login with overwhelming probability.
There has to be a lot of seeks even on a slow laptop rotating hd to get a 5 second delay, with 15 ms seek time
you get 333 seeks in 5 secs.
eklitzke@gnut:~ $ time sudo head -c 1073741824 /dev/sda > /dev/null
Even if the login command does need to sequentially read through logs to find the last login time (and I'm skeptical of that, because that would be a stupid way to implement login), I don't see how that would explain multiple seconds of waiting.
This has never been hard to do on OS X since 10.0 because they followed the Unix convention of separating user data from system files. It is, of course, rarely necessary unless you've used superuser access to seriously muck with things under /System.
I've done this repeatedly over the years when dealing with beta releases & system migrations and it's never taken much longer than the time needed to copy the files.
You should run some dtrace magic to find out what 'it' is. MIght be the OS, might be a badly behaving application. Who knows.
I find it too easy to blame the OS for all of this. One poorly written app can cause a lot of performance damage.
Lion has probably been optimized for SSD since Apple is quickly getting rid of spinning disks in their entire line up.
Is it actively paging to disk at times when there is plenty of free RAM? A common complaint on Linux is "I've closed a lot of stuff but it is still using swap" because even if the pages are read back into RAM when next used they are kept in the swap area in case they need to go out again (that way they don't need to be written unless changed, saving some I/O).
Under Linux you can see how much is found in both RAM and on disk with the "SwapCached" entry in /proc/meminfo - it won't stop counting those pages as present in the swap areas until either it runs out of never used swap space so needs to overwrite then to page out other pages or the page is changes in memory (at which point the copy on disk is stale so can not be reused without being updated anyway).
> and a new Terminal.app window can take up to 5 seconds to open.
Have you monitored system activity at such times to see where the delay is? While it could be due to unnecessary disk I/O it could also be elsewhere such as delayed DNS lookups if anything in your profile scripts does such a thing and there is an issue with your connectivity or DNS configuration.
(I'm not an OS X user and never have been so sorry if these thoughts are irrelevant - but I'm guessing memory management in OS X is similar enough for knowledge of how Linux plays the game not to be completely useless)
At least Windows, having gone through this particular growing pain, is nowadays fairly painless to reinstall. Did OSX ever improve that aspect of their product?
2) The OS reinstall path is identical to the OS upgrade path, making it very well tested. This has been the case since (IIRC) Snow Leopard.
3) The latest few generations of hardware can even (re)install the OS over the internet, meaning you don't even need to carry around media to reinstall. (Assuming you're on a fast connection or are willing to wait.)
It's entirely painless to reinstall OS X.
If you want to replace various bits of the system, reinstall the OS.
These are different scenarios with different use-cases, and I'd argue it's a much worse idea to conflate the behaviours, as in certain other OSes, than OS X's fault for properly treating them as separate operations.
It's a good thing.