Based on perf report, the latest stable version of exa is spending >99% of its time doing grid stuff. I pulled the repo from git, and using the latest master commit, it appears to have a huristic to not bother with the grid when you have lots of files. Now the times are a lot closer. (redirected stdout to /dev/null since otherwise the vast majority of the time would simply be outputting to the shell)
[user@anarchy:~/exa_test]$ time ../exa/target/release/exa >/dev/null
../exa/target/release/exa > /dev/null 0.05s user 0.07s system 99% cpu 0.122 total
[user@anarchy:~/exa_test]$ time /bin/ls >/dev/null
/bin/ls > /dev/null 0.03s user 0.01s system 99% cpu 0.048 total
Edit:
Hmm, interesting. Certainly a lot slower, but not enough that I'd really notice unless I was doing this constantly:
time ls
real 0m0.228s
user 0m0.145s
sys 0m0.082s
time exa
real 0m1.209s
user 0m1.124s
sys 0m0.084s