Projects like llama.cpp/ollama should make it automatic/dynamic, just rely on triple quote sections where you enter those constraint modes automatically.
Ie. every time you enter into section starting with "```json" you automatically switch to JSON BNF.
Every time you enter into "```json:Foo" you enter JSON BNF + JSON-SCHEMA for Foo object definition.
"```python" for python grammar etc.
"```quote:documentRef" you enter trie based constraints.
"```llm:otherllm" you enter other llm.
"```whatever:whatever" you enter whatever you want.
If you want just json output you start output with "```json" and that's it.
As its all inference time it could be plugin based, ie:
1. character based - given input (from the start of opening "```foo") it returns allowed next characters, or
2. token based - same as above but returns allowed native tokens (not sure how performance would behave here, would it be acceptable?)
IMHO also very interesting area would be exploring stable AST representations for programming languages (a'la darklang I believe?) – where variable names are detached from AST itself, ie. differently named functions that otherwise have the same structure have precisely the same AST representation. This would dramatically reduce space to navigate around.
Use trie constraint for quotes.
Use BNF for grammar languages (json, python etc).
Projects like llama.cpp/ollama should make it automatic/dynamic, just rely on triple quote sections where you enter those constraint modes automatically.
Ie. every time you enter into section starting with "```json" you automatically switch to JSON BNF.
Every time you enter into "```json:Foo" you enter JSON BNF + JSON-SCHEMA for Foo object definition.
"```python" for python grammar etc.
"```quote:documentRef" you enter trie based constraints.
"```llm:otherllm" you enter other llm.
"```whatever:whatever" you enter whatever you want.
If you want just json output you start output with "```json" and that's it.
As its all inference time it could be plugin based, ie:
1. character based - given input (from the start of opening "```foo") it returns allowed next characters, or
2. token based - same as above but returns allowed native tokens (not sure how performance would behave here, would it be acceptable?)
IMHO also very interesting area would be exploring stable AST representations for programming languages (a'la darklang I believe?) – where variable names are detached from AST itself, ie. differently named functions that otherwise have the same structure have precisely the same AST representation. This would dramatically reduce space to navigate around.
ps. [0] in case somebody else is interested
[0] https://github.com/ollama/ollama/issues/3019