On: Bling Fire: Finite state machine and regular expre...

[zone411]

In NLP, tokenization is one of the first steps in the pipeline to break down text into individual words. I don't know about others but for the project I'm doing, the tokenization speed doesn't matter much in the big picture because the following steps take much longer.

[atoav → zone411]

It matters for anything that should be more or less realtime, so where fast feedback is needed.