
Using JSON Schema to Document, Test, and Debug APIs - joeyespo
https://blog.heroku.com/json-schema-document-debug-apis
======
mendelk
This is precisely the purpose of the OpenAPI Specification[0], an extended
subset of the JSON Schema specification. Plus there a whole family of
supporting tools, such as Swagger UI[1], and lots more[2].

[0]
[https://swagger.io/docs/specification/about/](https://swagger.io/docs/specification/about/)

[1] [https://swagger.io/tools/swagger-ui/](https://swagger.io/tools/swagger-
ui/)

[2] [https://openapi.tools/](https://openapi.tools/)

~~~
why-el
We went with this exact approach in my previous job, using OpenAPI to do
everything the blog post mentions, including code generation, diffs for API
consumers (published to Slack, for instance), and more. I believe this has a
very good shot at success, but it's not complete yet. For instance we had to
do a lot of in house work to make code generation work.

~~~
handrews
Hi- I'm one of the main editors of the JSON Schema specification, and one
thing we are doing with the next draft is making it easier to build extensions
for things like code generation. We expect to work with the OpenAPI folks on
this as it is very relevant to their interests :-)

Also, we and the OpenAPI Technical Steering Committee are actively working
together to re-converge the specifications. OpenAPI 3 was developed while JSON
Schema progress was stalled due to the prior editors leaving and a new group
of us (eventually) picking it up.

OpenAPI 3.1 will most likely include a keyword to allow using standard JSON
Schema as an alternative to their customized syntax, and hopefully we can
achieve full integration on OpenAPI 4. There are also some other ideas being
explored for improved compatibility in 3.x.

Standards work is hard, but the relationship between the OpenAPI TSC and the
JSON Schema editors is quite healthy and we are making good progress.

------
atombender
These days all my needs for "structured data" start with JSON Schema. I refuse
to work with schemaless apps and databases anymore.

The workflow which I use for one app is:

* Define a JSON Schema for all the app's models

* Generate static Go types from these definitions

* Generate corresponding GraphQL types and inputs in the Go app for serving the API

* Generate TypeScript types in JavaScript front ends against the same schema

* Same model structs in Go are used to shuffle data in and out of data stores

Since GraphQL is more limited than JSON Schema (for example, very limited
union ("oneof") support), there are some features I'm not able to fully make
use of, but the common denominator covers most use cases.

I _love_ having end-to-end static typing, validation and consistency.

~~~
smt88
I tried to do exactly this (but using C# instead of Go on the back end). The
tools basically fell apart when trying to do things that were well within JSON
Schema's spec.

Can you link to the libraries you used??

------
Existenceblinks
I was building an app using JSON Schema last year (until Feb this year) and
gave up. I had read JSON Schema specification like 10 times and there are some
undefined behaviors. I had read a lot of JSON Schema issues on Github, there
is basically one man show and the others come to review mostly.

After the point they added all logics to the spec, everything becomes a mess.

Example 1:

{ "oneOf": [ {"type": "number"}, {"type": "number"} ] }

1 or 2 or 3 is an invalid input based on above schema. It doesn't make sense
to me at first glance. (hint: it's XOR)

\---------

Example 2:

{ "oneOf": [ {"minimum": 0, "maximum": 10}, {"minimum": 5, "maximum": 20} ] }

These ranges will pass: [0,4] and [11,20] But you would be surprise it doesn't
reject string, bool, null. Basically it doesn't reject anything except [5,10]
range.

\---------

Example 3:

{ "type": ["object", "array", "null"], "not": {} }

Easy but confusing. This rejects every thing because comma means AND, and
`"not": {}` means false.

\---------

Example 4:

{ "allOf": [ { "type": "object", "properties": { "a": {"type": "string"}, "b":
{"type": "integer"} } }], "additionalProperties": false }

A well-known problem. This rejects (all - {})

This is because the "allOf" is object AND outside it is an empty object.
"properties" at level 0 is empty if not specified. So this schema only accept
{}; an object AND is empty.

They add a new keyword called "unevaluatedProperties" to solve this and I
won't explain it to you!

\-----

If you just use basic stuff like draft-04, you will be fine.

But I will NEVER touch this spec again!

\-----

EDIT: My app was sort of static analysis on schema, and this spec doesn't
suppose to help doing thing like that.

~~~
awwright
It seems like there's two slight misunderstandings:

> "oneOf": [ {"type": "number"}, {"type": "number"} ]

> { "oneOf": [ {"minimum": 0, "maximum": 10}, {"minimum": 5, "maximum": 20} ]
> }

"oneOf" requires exactly one match (hence the name), typically you want to use
"anyOf" or "allOf". And there's no benefit in putting two identical values
inside them.

> { "type": ["object", "array", "null"], "not": {} }

> { "allOf": [ { "type": "object", "properties": { "a": {"type": "string"},
> "b": {"type": "integer"} } }], "additionalProperties": false }

JSON Schema is merely a list of assertions. Some test the type of the value
(that's the "type" keyword), others test the value ranges within a single
type. This way, you can allow values to be one of multiple types, e.g.:
{type:["string","object"], minLength:1} means "Value must be a string or
object; and if it's a string, it must have at least one character."

Some of the assertions are spread across multiple keywords,
"additionalProperties" depends on "properties", for example. So,
{additionalProperties:false} means: if value is an object, then only an empty
object is permitted.

Some implementations can tell you if you're trying to do nonsensical things
(like test the maximum length of a value that's only allowed to be a boolean),
but that's up to the implementation to test for.

~~~
Existenceblinks
Yes, it's my mistake using JSON Schema for something it does not suppose to.
Analyzing schema from user input is very difficult if "type" keyword is not
required (I actually had read the GH's issue about it). IMHO, JSON is already
verbose, no need to make it concise, requiring "type" make it a lot simpler.
I'd also prefer not to allow multiple type on "type" keyword.

The fact that the spec is too flexible that allow users to write all
nonsensical things, and so sensible things that might have hole on assertions.

I understand all your explanation and thanks for creating a new account to do
this, appreciated!

------
pjmlp
And now the circle is complete, with the difference that XML Schema allows for
proper comments.

~~~
Pxtl
I never saw XML schema as a bad idea, just overdesigned and verbose, same as
all things XML.

~~~
pjmlp
I happen to think otherwise regarding XML, and rather enjoy using all high
level tooling for XML processing.

------
z3t4
Im a fan of running these tests live in production. A type check is cheap so
it doesnt affect performence that much. You should not force correctness on
others, meaning their code will fail because of your strictness. But you can
force correctness on yourself so that your code never causes someone else's to
fail. So make the schema validation or typecheck on outgoing responses, not
only in testing, but also in production.

~~~
brandur
Checking responses in production is also a good idea because it reveals all
the edges that may not be exercised in an artificial test suite.

There is still _some_ performance cost for checking a response against a
schema though, so a nice compromise is just to check a small % of outgoing
responses — the vast majority of requests stay as fast as possible, but given
a reasonable traffic load, it's still enough to eventually reveal any places
the schema doesn't match reality. (This is an approach we use at Stripe for
checking our OpenAPI specification.)

~~~
sbr464
Side note @brandur, I always look forward to your blog posts, time for a new
one!

~~~
brandur
Hah, thanks! :) I'll got a few queued up that I hope to get finished up pretty
soon.

------
smt88
Open API Spec (formerly known as Swagger) is purpose-built on top of (or
slightly forked from) JSON Schema and has much wider support among API-related
tools and libraries.

I tried using JSON Schema for some internal (non-HTTP) APIs and found
bugs/inconsistencies in tooling that made me abandon it.

------
tnolet
I'm really torn whether to adopt JSON schema validation for a Saas that I run.

pro: It validates API JSON responses based on an open spec.

cons: It's absolutely terrible to work with. XML all over again. Documentation
is terrible. Finding examples is, well you get it...

~~~
handrews
Note that "Understanding JSON Schema" has been updated for draft-07 and is now
under the aegis of the main JSON Schema project: [https://json-
schema.org/understanding-json-schema/](https://json-schema.org/understanding-
json-schema/)

------
echelon
Coming from protobuf land, this tool seems gross. It's XML schema all over
again.

Why are we trying to shoehorn a schema design language into a format that
isn't good for human authoring? It's annoying to write JSON, yet the language
masquerades as being human-readable.

Frontend engineers should check out protobuf. It's an amazing data definition
language that generates bindings in every language under the sun and has a
compact binary serialization that is much more efficient than JSON both to
encode/decode and transmit over the wire.

JSON should die the same death XML did. It's so bad.

~~~
cageface
Being able to see JSON responses in browser dev tools is extremely helpful for
development and debugging. Is there something similar for protobuf?

~~~
carlmr
I concur, JSON is easy enough to read/write for humans, easier to parse than
XML, too, it kind of hits the sweet spot in between protobuf and XML.

------
dilyevsky
Why not just use something with a natively supported schema at this point like
protobuf/grpc or graphql?

------
theK
The article describes some very good engineering practices. Nevertheless I’d
suggest the json Schema itself is not the root of the documentation but a
higher level integrated system like API Blueprints MSON. You still can produce
Schema from it but it also can tie into a much better API design lifecycle.

~~~
handrews
Yes, JSON Schema describes documents/resources, not entire APIs on its own.

Even Hyper-Schema (to the extent that it's implemented at all yet) is a
resource-by-resource system, not an API-scope system.

------
kiliancs
This is a very powerful approach because there are json schema libraries
available in all major languages. TypeScript is a better source for truth
because it can represent things json schema can't. It is possible to generate
a json schema from a TypeScript definition (see
[https://github.com/YousefED/typescript-json-
schema](https://github.com/YousefED/typescript-json-schema)), which means
TypeScript can be the single source of truth for API definitions, be used
directly in the frontend and, in json schema form, used to test directly
against API responses.

------
sergiotapia
This seems really verbose? Very XML-y.

What's wrong with simple [https://github.com/omniti-
labs/jsend](https://github.com/omniti-labs/jsend)

``` { status : "success", data : { "post" : { "id" : 1, "title" : "A blog
post", "body" : "Some useful content" } } } ```

~~~
awwright
How is that simple? What does "status" mean? I'm personally familiar with the
word "status"—but whatever that property is, I'd have to re-implement it in my
user agent, which seems like a waste because my user agent already understands
HTTP status codes. How is it simple to have to look at and interpret two
different "status" fields? (And that's just the first property!)

What part of the article is verbose? What's wrong with verbose?

------
kemitchell
For JavaScript, I highly recommend the ajv and tv4 npm packages.

------
pingec
Not directly related but I like writing a json schema for my json configs. In
a pinch I can get an interface for editing and validating it very cheaply with
[https://github.com/json-editor/json-editor](https://github.com/json-
editor/json-editor)

It's not perfect but a great time saver for me and users.

------
mcheshier
Well written and informative. Thanks for posting/writing this!

