
A Quick Study on Tokenizing - ingve
http://www.joshbarczak.com/blog/?p=970
======
d0m
The thing is you don't know what's going to be slow and you can't
realistically test various implementations for every functions. IMHO a better
strategy is to code using standard libraries and use a good profiler. Maybe
the tokenizer could be faster, but something else might be 100x slower so it'd
be better to optimize the other thing first anyway.

Also, sometimes the best performance comes when reevaluating the whole way to
do something. Consider your example about wasting time on tokens. Maybe a
better way to optimize the build time is to create the compilers so it
receives "diffs" rather than all files every time.

~~~
yoklov
Pretty much no software high performance software has gotten there by not
thinking about performance/optimization until the end. IME what you do is you
structure your program so that the optimizations you suspect you will need to
make are easy, and can be done without large changes or rewrites to unrelated
parts. This does result in more work than being totally naive, but less work
than optimizing it ahead of time (you also usually won't loose the flexibility
you loose when you optimize if you work this way).

And while using a profiler is required for optimization, it's not the only
part. Optimization is a feedback loop between your assumptions and a profiler.
Make assumptions, change something, validate the result with the profiler.
Assumptions without the profiler can be wrong, and so you need the profiler,
however the profiler tends to have an extremely poor signal to noise ratio
(especially for code that is already fast, which tends to have a relatively
flat profile).

Bit of a rant, but while theres truth to premature optimization being bad, and
needing to use a profiler, most of the people who say these things mean 'never
optimize' and assume if they have to the profiler will magically solve their
problem and tell them exactly what they need to fix, which is rarely true.

