On: Every Way to Get Structured Output from LLMs

[zora_goron]

The article mentions,

>> “you've tried response_format: "json" and function calling and been disappointed by the results”

Can anyone share any examples of disappointments or issues with these techniques? Overall I’ve been pretty happy with JSON mode via OpenAI API so I’m curious to hear about any drawbacks with it.

[hellovai → zora_goron]

The main drawback is really when you attempt to do more advanced prompting techniques like chain-of-thought or reasoning.

forcing those parts to be json, can be hard and unnecessarily constrain the model. e.g. https://www.promptfiddle.com/Chain-of-Thought-KcSBh

try pressing run tests and you'll see what i mean! this method or doing chain of thought works a bit better

[amake → zora_goron]

We specify an output schema (TypeScript syntax) in our system prompt, and OpenAI gets it right most of the time. With some regularity it will give invalid (per the schema) output like

- Return a single object instead of an array of objects

- Return an array of a single object instead of just the object

On the other hand I personally haven't seen it give malformed JSON; the JSON is well-formed but not compliant with the schema we specified.

[hellovai → amake]

oh thats really interesting, how often do you get errors like that?

fyi, we actually fix those specific errors in our parser :)