
Writing a Lexer - kingkilr
http://lazypython.blogspot.com/2009/11/writing-lexer.html
======
fadmmatt
I think the coolest way to write a lexer is to use the derivative of regular
expressions:

[http://matt.might.net/articles/implementation-of-regular-
exp...](http://matt.might.net/articles/implementation-of-regular-expression-
matching-in-scheme-with-derivatives/)

It's surprisingly easy to implement, as the prior example demonstrates in
Scheme.

This is the technique I recommend to my advanced compilers class.

~~~
mahmud
The first lexer and recursive descent parser from Levine's "Lex & Yacc" book
has been hammered into my head. I rewrote it from memory so often it's now
part of my internal standard library; as late as this morning I was writing it
in lisp to process the beanstalkd message queue protocol. I wrote it a few
days ago as well, for Redis, and before that by a week for Memcached.

Manual lexer writing is a reusable but non-transferable technique, specially
when regexes are slow or over kill; every little use-case has its small set of
states, own termination condition, constituent characters and tokens :-)

Basic compiler techniques gave me the freedom to write software that _I_
wanted, not software prescribed to me by a book or work specs (let's face it,
writing tools is the most fun kind of programming.) Sometimes I am grateful I
know these things and can make whatever I fancy, other times I am spiteful for
not being good at it; my work has thus far been marked by exuberant
incompetence :-P

[Edit:

Or you can skip lexing and go straight to language design:

[http://www.cs.brown.edu/~sk/Publications/Books/ProgLangs/200...](http://www.cs.brown.edu/~sk/Publications/Books/ProgLangs/2007-04-26/)

]

