Based on perf report, the latest stable version of exa is spending >99% of its time doing grid stuff. I pulled the repo from git, and using the latest master commit, it appears to have a huristic to not bother with the grid when you have lots of files. Now the times are a lot closer. (redirected stdout to /dev/null since otherwise the vast majority of the time would simply be outputting to the shell)
[user@anarchy:~/exa_test]$ time ../exa/target/release/exa >/dev/null
../exa/target/release/exa > /dev/null 0.05s user 0.07s system 99% cpu 0.122 total
[user@anarchy:~/exa_test]$ time /bin/ls >/dev/null
/bin/ls > /dev/null 0.03s user 0.01s system 99% cpu 0.048 total
