Another question is if such pipelines should act on objects, or more structured text streams. Many programs can output json, is this a pragmatic way forward to extend and improve further?
> Many programs can output json, is this a pragmatic way forward to extend and improve further?
Yes, json is a pragmatic way to put a band-aid on this issue. It really is Good Enough™ for many things, and shell tools that optionally emit json are strictly an improvement, imo.
> Another question is if such pipelines should act on objects, or more structured text streams.
"Structured text" is inherently brittle, compared to a self-describing message format. For one, even when you structure your text, you will always end up dealing with escaping problems.
Even worse, imo, is that backwards compatibility becomes a terrible challenge. Let me illustrate with a best-case scenario for structured text: ASCII-only tabular data (nested, binary, unicode data is left as an exercise to the reader, in true unix fashion). The most natural thing to do in unix is to select the relevant fields with awk. More specifically, you select fields by their number. Well, now your pipeline is tightly coupled to column ordering.
In principle you could use awk to select based on field names, those field names are just text and so it becomes incumbent on you, the pipeline author, to make sure they don't get lost in transit. No easy tail or grep for you!
All of this becomes avoidable when you deal with self-describing messages instead of raw bytes.
