Pagination is one of those things I feel like so many of these things get wrong. LIMIT/OFFSET (or as MS likes to call it, TOP/SEEK) style results in O(n²) operations; an automated tool trying to pull the entirety of a large collection in such a scenario is not good. I have to again recommend the excellent "Pagination done the Right Way" presentation[1] from Use the Index, Luke (an equally excellent site).
Just return a link to the "next page" (make the next page opaque); while this removes the ability to of the client to go to an arbitrary page on its own, in practice, I've never seen that matter, and the generalized URL format allows you to seek instead of by LIMIT/OFFSET by something indexable by the DB; HTTP even includes a standardized header for just this purpose[2].
I also think the generalized URL is more in line with Fielding's definition of REST, in that the response links to another resource. (I don't know if being in the header invalidates this; to me, it does not.)
If you get the metadata out of the data section of your response, and move it to the headers, where it belongs, this usually then lets you keep the "collection" of items you are return as an array (because if you need to put metadata along side it, you need:
["the", "data", "you", "wanted", "in a sensible format"]
)
(I've seen metadata-pushed-into-the-body turn API endpoints that literally need to return no more than "true" or "false" into object that then force a client to know and look up the correct key in an object… sigh.)
From the parent Microsoft guideline [1], they make the explicit differentiation between Server-driven and Client-driven paging, the Server-driven seems to be exactly what you describe in your first paragraph:
> Paginated responses MUST indicate a partial result by including a continuation token in the response. The absence of a continuation token means that no additional pages are available.
> Clients MUST treat the continuation URL as opaque, which means that query options may not be changed while iterating over a set of partial results.
They then explain Client-driven paging, if the client really wants to do it:
> Clients MAY use $top and $skip query parameters to specify a number of results to return and an offset. The server SHOULD honor the values specified by the client; however, clients MUST be prepared to handle responses that contain a different page size or contain a continuation token.
But if you want to obtain the entire collection, you would not use Client-driven paging, but only rely on Server-driven paging
Big +1 to killing offset in favor of order-by and an "after" or "before" criteria.
However, -1 to the idea of sticking any important "metadata" in headers. For one, it's dramatically easier to interact with response bodies than headers at the terminal or with simple programming libraries and tools. Having only one way of representing data requires fewer special cases. JSON is far superior to key/value pairs in terms of ability to represent structure. Lastly, if you ever decide to use a different "transport" protocol other than HTTP, you'll be glad you are already using a common representation without a side channel.
The actual HTTP status should be relevant. I don't want to get HTTP/200 with {error: not found}. Instead, useful errors should be like:
HTTP/400
{error: invalid foobar in request. Foobar should be a string, but instead got '123'}
Or something like that. Tell developers what the problem actually was, and potentially how to fix it. http://jsonapi.org/format/#error-objects has some good ideas and suggests a standard.
That said, even discarding the body, you should be able to determine if a request was successful by the HTTP status code alone, IMO. If you made a new thing, 201, thing ain't there, 404, your request sucks, 400, etc. Don't make people using your API parse your [especially true if you've invented your own format] JSON errors if they don't want to.
"Success With Info" is something that just causes problems.
The number of examples we found of "Success with Info" going horribly wrong was abundant. Turns out checking return codes is something people have failed to do pretty much since the invention of the return code...
Like the sibling comment says, often tools designed to consume HTTP will throw a more severe class of error upon 4xx/5xx, which then changes the codepath significantly, making it more difficult for the programmer to treat 2xx and 4xx/5xx responses similarly for parsing out the headers and the response body.
Web browsers are also guilty of some of these behaviors. They would inject a 'user friendly error page' upon 4xx/5xx. Since they are graphical applications, they have also been used as rudimentary debug/sanity-check tools at the expense of curl.
Together, these forces combined to drive some APIs to respond with a 200 for any response. Often, faithfulness to spec or faithfulness to the intent of the design takes a backseat to making a solution work. The combination of these traditions survives in the myriad APIs that essentially use HTTP as an browser-accessible RPC mechanism.
Which is irritating as all hell at times... the .Net http client does this, which makes snagging the actual response text and code difficult... Same wrt .Net for response codes, there's a lot of errors that can happen with the JSON deserializer for services, but it returns a 500 series error not a 400 series. I usually wrap the on error in the app file, and replace those responses for .Net projects.
Worth noting, not sure if this is improved/fixed in .Net Core based services.
As I mentioned in another thread, I'm in favor of the response always being an object at the top level, with a data property for any response data, and an error property for any errors, errors having a "message" property that is the error message string, and a "code" property that matches the numeric http response code.
This also helps if your error classes have these codes in them, that way the http response code can be set based on the error, defaulting to 500.
Together, these forces combined to drive some APIs to respond with a 200 for any response.
I once had to work on such an API, which was designed that way because of similar limitations in the (third-party?) iOS networking library used by the app that connected to the API (as it was in 2013, anyway).
One can see how thinking of an HTTP request as an RPC rather than a REST request/response would justify throwing an error instead of returning an object, but since REST is now more common than RPC, one would at least expect an option not to throw.
I think in many ways it's just tradition. Old browsers did wonky stuff with non 200 codes and that just created inertia that made people never quite take the effort to follow the spec.
I tend to prefer a simple JSON object that wraps any response...
{ data: any, error: object }
So that only the presence of an error needs to be checked... within the error should always be a friendly message property and optionally a numeric code, that mirrors the response code (404), etc.
{ error: { code:404, message: 'Not found' } }
This tends to work well with error traps in server-side workflows... As to the data/result etc, it can be whatever is needed as a response to the client. For paged results, data/nextToken or something similar are perfectly serviceable imho.
To me an error should be top-level in the response object, as well as meta, and the data itself should be a consistent property and nested as such. Other platforms add their own wrappers which include the headers with the response object. ymmv.
//use CamelCasePropertyNamesContractResolver
public class ApiResponse<T> {
public T Data { get; set; }
public Exception Error { get; set; }
public ApiResponse(T data) {
this.Data = data;
}
public ApiResponse(Exception error) {
this.Error = error;
}
}
In this way, all API requests return ApiResponse<TypeOfData>
I agree with tracker1 about a top-level object with data+error keys. Always useful to have an "open" / "extensible" object at the top, so you can add extra stuff as needed. Pagination is a classic example, but another one is debug data, such as profiler timings.
I'm not opposed to having meaningful status codes, but I am opposed to 1) overdoing it and 2) omitting success/failure from the body. The only useful status code distinctions are the ones that browsers respect: success vs failure, redirects, etc. Differentiating 200 OK vs 201 Created, for example, causes nothing but headaches. What would I do differently given that distinct information? What if I do a POST to create an get a 200 when I normally expect a 201? Is that suddenly now a failure?
For API use cases, I only use 200 (success), 400 ("you screwed up"), and 500 ("we screwed up"). For HTML responses, I also use whatever 3xx the SEO people suggest these days.
That's not what the vulnerability is here. If you have a server endpoint that returns an array without wrapping it as a property on an object then an attacker can write a webpage that overrides the Array prototype and requests the json from your page as a script, thereby bypassing the cross origin check it would be subject to as an xhr request. So if you have a page which returns sensitive information as an array you (were, it seems browsers probably don't allow you to override the array prototype anymore) are vulnerable to the attacker stealing a signed in users secrets.
On the contrary, that's precisely the vulnerability:
1. The API server needs authentication, which gets cached in the user-agent.
2. The same user-agent, with the same authentication context, is then directed to a malicious site.
3. The user-agent contains a javascript interpreter, and the malicious website serves a script to override the Array prototype.
4. The same script then executes a CSRF request to the API server, reusing the cached authentication context, and therefore stealing the response.
A browser fits all the necessary conditions that must exist for this vulnerability to be exploited, but if you're the consumer of the API, you don't have to use a browser as your user-agent. You can write your own, which you'd do if you're making these requests programmatically in a non-web application.
Regardless as to which user-agent you use, a viable mitigation is to not visit untrusted endpoints with the same cookie jar or authentication context.
Another viable mitigation is to omit including a script interpreter in your custom API client. This is in fact usually the case.
Making the next page opaque as you said is a good thing also because limit and offset will often become unstable or incorrect if the dataset is changing without a proper timestamp. That is you may get more or less records then you are expecting which maybe important if you need to allocate size.
Almost all of my rest services I require sending a timestamp for batch processing (that is only records created/updated/removed before given timestamp are shown).
Sometimes, you can look at what the index is and find a better thing than page numbers to use; for example, if your collection is ordered/indexed on a timestamp, you can navigate to dates× instead of page numbers. (e.g., in a UI presentation you might expose the ability to choose a date to go to, instead of a page)
Microsoft came up with OData about 10 years ago, but it's now steered by a technical committee with a lot of participants. I wouldn't necessarily be so quick to assume that the OASIS committee - or Microsoft's representatives on it - speak for all of Microsoft. Let alone an internal guidelines committee that they apparently convened fairly recently (given the age of the report) when they use that slogan.
This isn't just some poorly labelled content from a 10 year old legacy website from a time before they knew any better, the OData website got a recent redesign sporting new Stock photos and even larger font for their disingenuous labeling which still continue to push OData crapware under the REST banner to fool devs/CIO's into thinking if they adopt OData they're taking advantage of the best form of REST - which is in-contrast and devalues sincere efforts like this where they're actually looking to promote good HTTP API practices.
I recently had the misfortune to have to tackle oData to talk to Dynamics CRM. The choice was between that and SOAP so it was the lesser of two evils.
But compared to the other nice, modern REST API's I'm used it smelt of mould and enterprisey cobwebs. I'd have preferred an old school RPC API over that. At least API's that abuse REST from that direction tend to have the virtue of simplicity.
Sure, but is the smell because of OData or is that smell mostly coming from the messy old enterprise cruft that an OData veneer has been applied over? I can't really speak to how Dynamics did this in detail, but on my crusty enterprise platform with a huge existing api surface we did a veneering project as a rest v1 using OData (intentionally avoiding esoteric bits, but using query, $metadata, and limited relationships). It does have an enterprisy cobweb feel in places but I think that'd be present if we'd done it without OData as well.
You should never have to apologize for delivering value - that's ultimately the purpose of technology. If it's the cleanest/simplest solution to a given requirement, implement it and move on - don't worry about trying to appease cargo cults. Technology should be used as a tool to deliver the best value you can within a given set of requirements. End users don't care how many Internet points from fashionable tech du jour you collect along the way, all that matters to them is the end user experience, invest your energy towards that.
There's nothing wrong with RPC APIs. Other than being "Not Cool" due to the legacy of SOAP, they can deliver some very nice value.
Various RPC mechanisms like Bond and Protocol Buffers (and more recently GRPC) are trying hard to make RPC cool again. Personally, I hope they succeed, as REST (like any technology) doesn't work for everything.
For that matter, communication abstractions over websockets... Which has been most interesting to me in terms of offering some better security choices.
OData is a big enough tent these days that there are good and bad (relative) bits even inside OData. Fortunately one gets to pick-and-choose what bits you use, so just avoid the bad bits.
The example cited as a bad url makes use of OData functions and parameters, which is definitely a more esoteric part of the spec and has spotty implementation (if at all) amongst OData consumers - so discouraging this kind of API seems perfectly reasonable for a REST-centric guideline.
OTOH the OData query syntax is IMO a lot more reasonable; outside of structured queries built in the form of URI path fragments, if you want to provide a generic query mechanism on top of a store you need some kind of generic query language. $filter is a reasonable such language - it is easy to parse, easy to emit, and relatively easy to read. Yes it has some gaps and a couple bizarre edge cases, but they don't get in the way of mainline consumption scenarios - and it's hard to beat being able to provide a reasonable REST API that clients can construct queries by hand for, and also have these same APIs "just work" when plugged into OData consumers (of which there are quite a few in the enterprise).
No $filter is a horrible construct which accepts a free-form expression that doesn't map cleanly to any variable that's effectively being used as a tunnel to pass an opaque query blob from OData client to an OData server resulting in an anti-pattern violating Microsoft's own Service design guidelines: https://msdn.microsoft.com/en-us/library/ms954638.aspx?f=255...
$filter is an unnatural implementation detail leaking both OData implementation and internal table structures to your user-facing API exposing the entire OData query surface area to consumers where as soon as someone binds to it, your API becomes frozen and your API is forever coupled to the OData server implementation given it's much harder to build a compliant OData query parser than it is to parse a simple HTTP path info and QueryString. So when you need to change your implementation for whatever reason, e.g. you want to populate a part of the query with a search index or alternative NoSql data store, you're headed for a rewrite, whereas if you designed your HTTP API naturally behind an intuitive, impl-agnostic URL structure you could easily change the implementation without breaking your API consumers.
There are a number of examples to cite showing what you say isn't true.
Allowing totally arbitrary OData $filter expressions does lead to problems, but recent versions of OData have a nice mechanism to describe (and discover) what can be filtered on. This is both tooling and developer (for consumption) friendly.
There are many examples of teams providing OData API heads without using SQL as the data store. Teams do this using EXO, various Azure data stores, and custom stores.
Using (for example '$filter=productName eq cupcakes') isn't leaking any weird abstractions, but is giving clients and tooling a nice means to filter a list.
The problem isn't for simple inconsequential queries like: '$filter=productName eq cupcakes' it's as soon as you expose an OData endpoint you're exposing the entire OData query surface area in which you have no idea what complex expressions consumers will decide to execute (unless you are the only consumer), so you're left supporting them all and coupling to the OData server implementation for life. If your API only exposed `?productName=cupcakes` you'd have no such implementation coupling.
Now you're saying you can combat this with recent versions of OData that have filter metadata which is a problem in of itself, additional features, additional metadata services, additional training, additional complexity, etc - making OData an endless rolling spec where eventually every OData client except those developed by Microsoft will fall off the upgrade cliff and no longer see ROI chasing an endless spec. OData's just another big, heavy, tooling-dependent, slow and over-architected, complicated framework - you can avoid it's manufactured problems and create cleaner, faster, more interoperable, more human-friendly and future proofed API's by simply avoiding it in the first place.
The problem with just exposing '?productName=cupcakes' is you're assuming just one filter with simple equality. If you need to expose something even just a little more complex, i.e. "A=1 or B=2", or something other than equality like "A>1" or "B!=2", then you quickly find yourself re-implementing a little expression language, syntax for literals, etc. It is a slippery slope, which one can happily go down and succeed with a custom solution - until you then want someone other than your clients to pull data from. Then they need to build their filter in your language, which is of course different than the next guy's language, and so on and so on.
The fix for this is OData. Not all of OData - just a little bit. It lets one standardize the filter expression syntax (as much of it as you choose to support) without making any requirements on the backend.
I'm personally confident in making the claim that OData $filter doesn't require you to bind to a server implementation, because I work on a service built in Node.js and deployed on Linux in Azure that uses OData as a filter syntax and satisfies it's data requirements from three completely different backend servers, none of which is a SQL Server (not that you mentioned SQL, but it's often cited as "all that OData is good for"). One of them is Elastic Search, BTW :), the other is a proprietary aggregated metrics store, and the third is a cloud-scale columnar data store. All three of these can be queried with the same syntax, from tons of off-the-shelf OData consumers, and for clients the choice of what backend to pull data from is literally the only thing they change. From a business value perspective this is pure win, and this is due in large part to using just a little OData, at just the right spots.
I think of OData like salt in a recipe - a little bit is great; too much ruins the dish. Moderation in all things... :)
You never need to drop down into free form expressions to satisfy a query which should never be exposed over a service boundary, this is a very clear Services anti-pattern introducing unnecessary coupling. Find which requirements your Service needs to implement and expose them under the most intuitive self-descriptive label for end users, if you need Orders after a certain date you can expose `?createdAfter=...` or `createdBefore=...`, `?createdAt=...`, etc.
It's cleaner, faster, more interoperable and it's implementation is fully substitutable, supporting multiple data sources over a user configurable Surface Area that as it's just the same as any other ServiceStack Service has access to all its existing features it enables for all Services including support for an end-to-end typed API over multiple data formats, that's able to take advantage of existing metadata services, and also includes an optional AutoQuery UI. Because it's just another Service there's also infinitely less incremental knowledge to learn whilst at the same time being able to take advantage of features of existing Services Users already know.
This is just one approach, most people never need the inherent complexity and ridgity in big heavy fx's like OData which used to have traction when MS had all its marketing momentum behind it but it's hey day is over and abandoned by its high profile early adopters, now it just lingers in maintenance-mode, mostly forgotten outside of MS and held on by the few that haven't moved on.
That particular example was chosen not because of OData, but because it required someone to type in a horribly long set of alphanumeric codes.
There's no way a human can effectively type that, nor is there any real way to look at it and figure out what it's doing. That the URL happens to use ODAta (because that's what Exchange does) is not really relevant.
The overriding factor, as called out in the doc, is that URLs should be human understandable. That's not to say canonical id's can't be used, but a big base64 encoded string is not recommended.
I'd love to see some guidance on how you're supposed to do file uploads (think: uploading an image avatar for a user) and fit it into the REST structure. These guidelines don't even mention "x-www-form-urlencoded" except in a bit of fluff about CORS.
Made more frustrating by Microsoft's own WebAPI2 basically having zero support for file uploads, meaning we had to go way outside the library to code support for it.
Not sure why that's such a blind spot. Doesn't every web service need to upload files eventually?
It turns out that uploading files is hard. Files come in a variety of sizes, networks sometimes don't work, browser processes come-and-go, some users expect restart semantics, some don't...
POST has the semantics of sending data to a resource. What you're essentially saying here is that you're sending an image to the avatar resource. That's not what you're trying to do though. What you're actually trying to do is place a resource at that URI. For that, you use PUT, not POST.
Then we have the problem of conflicts. What if there's a resource there already? Now you need to introduce If-None-Match and things like that.
Then we have the problem that /users/{id}/avatar might be an inappropriate place to put that resource. Maybe the server needs to store it on a CDN or something.
Finally, we have the problem that this isn't a REST API you are describing, since the client and server implementations are coupled together. In a REST API, the API documentation doesn't instruct developers where to place resources, the server instructs clients where to place resources.
One way of handling avatar upload in a REST API would be for the User media type to define an action that sets the avatar image. The client would then follow that action to upload the image.
From an HTTP perspective, a likely implementation would be to POST to a URL that returns a 201 with the location of the newly-uploaded avatar. But the key thing is that the server defines how the client does that, you don't hard-code behaviour into all your clients.
If that's too abstract for you, think of it as <form action="/upload-avatar" method="POST">. It's simple to implement – much easier than hard-coding a load of URI patterns in all your clients and hoping you'll never have to change them.
From a guidance standpoint – for an API, define the structure of common interactions (e.g. file upload), and have your media types include those structures when they need clients to be able to change state. Your API documentation should describe your media types and relationships, and never URI structures. If you predefine URI structures, it's not a REST API.
Well, for one thing, it's a huge PITA to have 99 controllers that all take JSON and one oddball that has to take forms-urlencoded because you can attach a file.
This is out of the scope of this document, but I'd also like to see some guidance on how to implement the controller for that oddball using Microsoft's own WebAPI2 library. I ended up having to go entirely outside the library and implement it as a generic .ashx handler, but is that literally the best way? It feels like a horrible nasty hack.
You're thinking of 'multipart/form-data' with a content-disposition of 'attachment', which is the usual way of uploading binary blobs to a controller resource.
But conceivably you could have controllers that accept a PUT or a POST of an 'image/jpeg'. Why wouldn't you, if you're aiming for image uploads?
You're right, I had the wrong MIME type. It's been ages since I was working with this stuff.
To answer your second question: because I simplified the example. We run a health insurance website and we need to support life changes events either with or without supporting documentation. It makes no sense to have two separate controllers, as that would mean you could have a successful file attached to an errored-out life change, or the reverse.
But look, never mind. I was just asking because our solution to this problem I imagine most REST APIs would eventually come across felt really hacky and awful and I'm really not proud of the code I wrote to support it, but maybe we're the weirdo freaks and nobody else has to do anything more complicated than assign a file to a user ID.
I agree with you, though I'd say that's mostly a problem of JSON, which doesn't cleanly support binary data, not REST itself. Still, base64 ain't so bad, it's only a 33% increase over raw bytes.
Well presented. It would be great if there was a language / framework that made this guaranteed. As-is everything just returns 500 error on any exception, lets you return errors with 200, allows update on GET, etc. Even the Haskell ones.
Handling errors over REST API is something I've struggled with. What's the best way to handle errors? Data validation errors will be different from system/server errors. Tough to establish a universally applicable error response structure.
I used to be in favor of sending 200 responses with error codes but now gravitating back towards relaying the most relevant HTTP error & letting the clients handle it.
Any app of decent size will probably end up passing HTTP's error codes. 200 + error is OK, except it can mess with caching. A 5xx with details in the response is fine.
You might be tempted to map some errors, like "item not found" to 404, and so on. But you still need to provide the real error code. So you're not gaining much.
Honestly, I don't get the obsession with using HTTP features to represent part of an API. It never saves work; you're writing a custom client each time anyways. From a purely code perspective, you're going to deserialize the body or a context object from a header. Moving that data into multiple headers can only require more code, not less. Same for verbs. I've never gotten any benefit beyond GET-for-read, POST-for-write.
Elasticsearch is a good example. The URL space is overloaded with special things, allows you to create objects you can't reference, and so on. They use verbs, except you still have extra parameters tacked on. There's zero benefit to me, the user, of them making it REST like.
Maybe if REST-someone creates a machine usable spec like WSDL (just "simpler") then all these HTTP headers could be put to use.
The advantage is that there is some level of standardization.
404? That means the entity doesn't exist. 302? I should look somewhere else. 401? The server doesn't know who I am.
Accept? I can specify the format. ETag? I can get a faster response if I include the token in the next request.
This stuff is really, really common, and people can learn your API very quickly. A transparent caching server can improve performance.
Sure, with a custom protocol you can get a tight system. Hell, write your own transport layer for even more control. But it will take longer to learn and harder to interoperate.
The time spent reading that 404 in this case means "the object ID isn't found" versus "this path doesn't exist" pretty much negates any benefit - you still have to include sub codes. Same for "access denied because token expired" vs "invalid token". Not mention all the stuff that'll get crammed into 400/500/503.
If your app is simple enough that all errors map 1:1 to HTTP, great. Or if it doesn't need that level of error management. Otherwise HTTP just confuses the issue.
So, you just want to explain the error further? Wonderful. RFC2616
> the server SHOULD include an entity containing an explanation of the error situation
---
The 3-digit status code tells consumers (1) the status category (success, redirect, client error, server error) and (2) a more specific status within that category. It does that in a way that doesn't require me turning to your API docs every 3 seconds.
Send the most appropriate HTTP status code, along with an error resource the client purports to understand (by accept header content negotiation).
If the client isn't declaring to accept a mediatype you produce, you send your fallback error format, which could be anything you choose: text/plain, or some custom format you design, or some generic hypermedia type that defines an error field.
It's great to see Microsoft release these guidelines. It's good work, a broad document with a lot of interesting topics covered. You can always debate the details (the pagination discussion here is very interesting), but having seen first hand at Zalando how much dedication and effort goes into API guidelines to support a large number of service teams, plus releasing them into the public, it's no small feat and they deserve credit for doing this.
There's naturally been some discussion around REST/HTTP APIs and design styles. One of the things we've tried to do with the Zalando guidelines (https://zalando.github.io/restful-api-guidelines) is go into more detail on using HTTP and JSON, and how REST properties are used. Zalando, like everyone else, has had to think through versioning and compatibility, so it was interesting to read what Microsoft do here. The Zalando guidelines take a different approach, using media types for versioning, document just one set of compatibility rules, plus a deprecation model, and it's working very well in practice so far (http://zalando.github.io/restful-api-guidelines/compatibilit...).
Btw, in case anyone from Microsoft working on the guidelines is reading and ever wanted to swap guideline notes or ideas, that would be awesome. And once again, great job releasing the doc :)
In Kibana, everything is done over GET as get parameters, and I find that extremely annoying and a poor design.
A lot of public APIs also don't honor or have any intentions in supporting or using PATCH. Most APIs I have worked with only use PUT for modification. Anything resembles "creation" is automatically a POST.
I have worked with some systems that do not expose all http methods. Having a fallback way to invoke a method as a GET or POST makes your API more usable to more clients.
Very cool document. I kind of got stuck at delta queries, though. How do you implement that? I can't find any reference to delta/removed queries on Mongo, Postgres, or MySQL. Do you just keep all records in the database and add a "removed" field? How would that solution work with user privacy & users truly wanting to remove data?
Delta queries are pretty easy, assuming you have a rich data store behind your data.
As others have said, your data store needs to be able to say, "Show me changes since XYZ". Most of the Big Apps can do that, and from there the problems is one of API Semantics.
This doc addresses the API Semantics, rather than the application design. To try to solve the design problem would be impossible as every application is different.
I understand what the doc is trying to convey, I just hadn't ever thought of the application-level logic for record deltas before, so I got incredibly distracted from the document. It made me wonder how other people, not using temporal tables, are accomplishing this right now.
I don't think I've ever seen delta's done at-scale with actual temporal tables, but rather with transaction Id's of some sort.
For example (based on the doc) when you register a delta link, there's a row written to Azure Tables saying, "owner:iheart2code, deltaToken:foo, lastRecordIdSeen:123".
When you then make the Delta request, we look up "foo" for you, find that the last id you've seen is 123, and then only give you records from the transaction table with an id larger than that.
Temporal is always a can of worms, as clocks are impossible to keep in sync and there are endless race conditions.
Making the delta tokens causal, rather than temporal, is the way to go. Anything else is brutal in the distributed systems world...
Funny thing: I've been thinking a bit about API versioning quite a bit lately, and the best solution I've come up with is the ONE thing not at all covered in this: put an `api-version` header into the request. I've seen both of the schemes recommended here, and I like neither very much. So what's wrong with my (header) solution?
For instance, if client application A talks to the service using version 1.0 of the API and client application B talks to the service using version 2.0 of the API, then those client applications can't interoperate because they are seeing two different sets of resources.
Your solution isn't far off the approach recommended by everybody who rejects the different URI approach. You don't need an API version header. When your representation format for a resource changes, provide a `version` parameter for the media type. For example: `Content-Type: application/vnd.example.foo+json;version=2`.
This is exactly how HTTP and the `Content-Type` header are supposed to work – if your representation for a resource changes, you don't change the URI, you change the media type.
The "?api-version" approach in the doc is there for exactly the reason you call out. Azure uses this approach.
By omitting "/v1.0" from the path it makes the actual URL's far more durable, as they're not version dependent. There are pros and cons to this, as there is with everything. In Azure's case it's great, as you can then use URLs as resource identifiers and put ACL's on them. If they were versioned via the path, this wouldn't work.
Other services, such as XBox, and (very old) Azure, put the API version in a custom header (x-ms-version, for example). This proved to be extremely problematic, and every team that does this had stories of all the problems it caused and then spends time removing it.
I've never seen a detailed proposal for putting API version in the Content-Type header, and will go look.
The content-type header does seem to have many of the same drawbacks as putting the version in a header (any header). For example, people could not make queries from a browser just by entering a URL as there is no way to specify a header. Nor could someone bookmark an API GET request, which is also quite handy.
Ease of use is huge, and I am in the camp that a version in the URL (path or parameter) is much easier in every way than a header. Every with curl, it's easier (I can never remember how to specify headers w/o the man page).
One slight downside of a custom header to specify a version is that OPTIONS calls don't include the value of the custom header, so your pre-flight gets to say yes or no without knowing what version is being called. Putting API version in the URL or query string fixes this.
As for bookmarking a GET request, this is /almost/ doable even following the MSFT guidelines since it says that service implementors MUST also provide a query-string alternative to required custom headers (section 7.8), and that service implementors MUST support explicit versioning. The only fly in this ointment is that the versioning part of the spec only offers two places of specifying the version - URL and query string, and seems to leaves no room for other options.
Personally, I think the Accept header flavor with custom MIME types is the most flexible for minor (backwards compatible) version - see GitHub's API for an example - but it certainly isn't the most simple to work with, neither in client libraries, Curl/WGet command-line use or API consumer tools (almost none let you fiddle with Accept headers). Since API ease of use is such a big factor for adoption, passing versions in the URL or the query string is most likely an OK lowest common denominator for APIs that seek the widest possible reach.
Interestingly enough, putting it in Content-Type is also invalid unless the mime type allows it. (Which, of course, is trivial if you're defining your own.)
Specifically, RFC 2616 specifies that clients SHOULD only specify parameters which are valid for the mime type [0]. This can get more restrictive; for instance, JSON-API (application/vnd.api+json) states that any parameters MUST result in 415 UNSUPPORTED MEDIA TYPE [1].
1. Older proxies used to remove/strip headers they don't understand.
2. Frameworks and libraries don't always give access to n non-standard headers, meaning they just can't used.
3. It's harder for humans to look at a request and see what's going on.
Proxies stripping headers isn't a problem if you use HTTPS or HTTP 2. Additionally, the proper place for this is the Content-Type header, which is a standard header any proxy would understand.
I can't say I've come across a framework or a library that makes it impossible to access a non-standard header, and if there are any, that would be a pretty glaring bug. Nevertheless, the proper place for this information is in the Content-Type header, which is a standard header.
In what way is it hard for a human to look at a request to see what's going on? The information is right there.
For the TLS case, there are enough MITM proxies, both in the Enterprise and elsewhere, to make this a real concern. There are also API Aggrigators which are effectively MITM and need to be taught to "play-well" with custom headers.
Certainly in the consumer case HTTPS would keep a majority of consumer facing ISPs from header-stripping, but there is still a pretty big hole.
An API version in an URI is fine. But better would be to use a new domain, if the API brakes backward compatibility entirely.
If the API does not break the clients entirely, just have new resources for incompatible formats:
/user
/user2
For the first resource, you can just add a "X-Deprecated" header with a link to the deprecation information. If you already know the date when the resource is abandoned, you can also add a header that has a date of removal in it. If removed, you can either have /user serve the new format or have it serve a HTTP 303 to the new format.
Being that guy again, (and sacrificing my karma) but...
This is not REST, it contains nothing about hypermedia, entities having knowledge of their URIs, or any way of discovering features of an API programmatically.
While I'm sure there's plenty of good stuff in here (it looks otherwise fairly comprehensive), APIs will continue to be a disparate system that requires custom code for every integration until we can embrace the full benefits of REST, and develop the technology and conventions/standards for integrating RESTful services.
Edit: for an example of what's possible with 'real' REST, check out this talk: https://vimeo.com/20781278 – better documentation, easy parsing, API optimisation without client changes, discovery, and the API is fully human readable/usable as well.
Not sure about your karma sacrifice, but if someone has doubts:
> What needs to be done to make the REST architectural style clear on the
> notion that hypertext is a constraint? In other words, if the engine of
> application state (and hence the API) is not being driven by hypertext,
> then it cannot be RESTful and cannot be a REST API. Period. Is there
> some broken manual somewhere that needs to be fixed?
It reminds me that GIF is pronounced GIF by most people even if the inventor pronounces it as JIF.
The language is what people use to communicate. REST may mean whatever most people think it means regardless what the inventor of the term thinks it should mean.
Another example is the word hacker. For most people it is "a person who uses computers to gain unauthorized access to data" (according to Google) that is different from the meaning used on this site: "one who works like a hack at writing and experimenting with software, one who enjoys computer programming for its own sake" http://www.etymonline.com/index.php?term=hacker
REST seems like an elephant in a room with some blind folks. Everybody who touches it thinks it's something different than the next guy, and they're all describing only one element of the thing.
That said, I don't actually know what this elephant looks like, either, because everybody I've read on the subject seems to have only a partial understanding of it...thus, I have a partial understanding of it.
If you know what the whole elephant looks like, and have good resources for what makes something actually REST, I'd be interested.
The best way to understand REST is by example. First check out the blog post by Roy Fielding mentioned in this thread which summarizes the constraints [1]. Then let's examine how several popular restful services work versus those constraints.
Here are some popular well-known services to use as examples: Google.com Amazon.com Facebook.com News.ycombinator.com.
- You enter these services through the site root with no prior knowledge about the services aside from standard media types (HTML/CSS/JS)
- Hypermedia is the engine of application state – meaning that you load a webpage a.k.a. resource that contains hypermedia links to other resources, and you navigate between states in the application by following those hypermedia links. For example, the hacker news homepage contains links to news articles and comment threads about them, which contains links to reply to those comments, and so on.
- The application describes how to compose requests using standard media types such as web forms and JavaScript. Because the pages instruct the client about what requests to compose and to which URL, the services have control over their own name spaces which have no fixed resource names or hierarchy.
I consider REST to be most easily understood as a departure to application protocol design prior to the Web, where each application and service had a unique and custom binary protocol, and a client for these services had to be reprogrammed to interact with it. REST is a set of constraints for designing services that run at Internet scale and that highly decouples clients and servers. Fielding's thesis on REST has more detail on the principles by which the REST architectural style was derived.
The big problem with using the hypertext Web as an example of REST is that there is a human operator literally driving that "engine of application state". Most API clients cannot afford to compute their state transitions on a network of 80 billion neurons.
I can't adequately convey what I think is Fielding's position on how machines should interact with RESTful systems, but I will convey my understanding of the purpose and intent of his thesis:
REST is a characterization of the Web architecture itself. It's not an alternative WWW, it is an abstract description of the principles behind the WWW. You can see this theme in Chapter 4 of the thesis in which Fielding characterizes REST [1].
> This chapter presents the requirements of the World Wide Web architecture and the problems faced in designing and evaluating proposed improvements to its key communication protocols. I use the insights garnered from the survey and classification of architectural styles for network-based hypermedia systems to hypothesize methods for developing an architectural style that would be used to guide the design of improvements for the modern Web architecture. [...]
> The early Web architecture was based on solid principles--separation of concerns, simplicity, and generality--but lacked an architectural description and rationale. The design was based on a set of informal hypertext notes, two early papers oriented towards the user community, and archived discussions on the Web developer community mailing list. In reality, however, the only true description of the early Web architecture was found within the implementations of libwww (the CERN protocol library for clients and servers), Mosaic (the NCSA browser client), and an assortment of other implementations that interoperated with them.
> An architectural style can be used to define the principles behind the Web architecture such that they are visible to future architects. As discussed in Chapter 1, a style is a named set of constraints on architectural elements that induces the set of properties desired of the architecture. The first step in my approach, therefore, is to identify the constraints placed within the early Web architecture that are responsible for its desirable properties. (internal citations elided)
All those examples are hard coded algorithms in the bulk copy category. They pretty much just use the GET method to download all or a specific subset of a site.
To truly drive a REST API requires a human or an AI. The AI doesn't need to be human equivalent, just smart enough to inteligently interpret the hypermedia. If the API changes the AI would be able to adapt.
I'm talking about the complexity of the computer on the Enterprise in ST:TNG. You give it a command and it figures out how to go about solving it.
You don't need an AI. A client for automated interaction will hard-code assumptions about content rather than URIs. If the client cannot find a link inside a document, it should just throw an error. A good service would keep links as long as possible, even when deprecated or provide redirects to new resources.
A client for human interaction though (like a native application or a SPA), can just implement the visitor pattern to display the UI.
Instead of moaning about the architecture principles, we must implement solutions / examples of how to write good clients against REST-based services!
I don't know where you got the idea that REST needs futuristic AI, and you haven't really explained why you think that way.
Why does a REST client have to intelligently interpret the hypermedia? Why are hard-coded algorithms a problem? Why does it need to adapt to changes in the API? None of those things are requirements for REST, you've inserted them for seemingly no reason.
For instance, consider an unattended webcam you want to use to show off, e.g. the growth of a flower bed remotely. It has the URI for a web service that returns a resource describing actions it can take:
The definition of the report-problem relationship is "This is the action to take when an error occurs."
The definition of the update-frame relationship is "This is the action to take when a new frame is available to upload."
This API defines one media type and two relationships. It requires no futuristic artificial intelligence or human intervention, it just needs to speak HTTP, parse simple JSON, and trigger the actions based on simple conditions.
It also doesn't provide much benefit over a plain RPC-like interface, because, for practical reasons, the webcam will be coded to send requests directly to https://example.com/report-problem and https://example.com/update-frame (the "bookmarks"). If you want to change those URLs, your best hope is that the webcam will understand a 307 or 308 redirect.
RPC vs REST is an entirely different argument. We're talking about REST best practices here, not "let's do RPC instead".
There's no reason for the webcam to hard-code URIs, and that's a violation of the architectural constraints. Just arbitrarily deciding to disregard facets of the architectural pattern without giving a reason does not add to this discussion.
None of this relates to whether or not you need futuristic AI or human intervention. I think from the example, it's pretty clear you need neither for a REST API.
> We're talking about REST best practices here, not "let's do RPC instead".
I don't think so. Avernar above seems to argue against REST as an architectural style (RPC being the default as everybody uses it). I thought you were trying to defend REST with your example, and I wanted to point out a weakness in that example.
> There's no reason for the webcam to hard-code URIs, and that's a violation of the architectural constraints.
The webcam has to hardcode some URIs (entry points). Consider that it could be using other services: perhaps a social network to post updates on. Then it must hardcode another entry point URI. But at that point -- and since every resource is supposedly independent and only explicit relations matter -- why can't the webcam treat the "problem report service" and the "update frame service" as two entirely separate services to hardcode? Which constraint of REST does this violate?
In your example, insisting that the webcam go through the entry point doesn't even buy the server a lot of flexibility. You reduce the client's knowledge from 2 URLs to 1 URL -- a URL that you may still want to change etc.
An extra HTTP roundtrip every time can well be an unpleasant overhead. HTTP client caches are sparsely supported and notoriously complex (RFC 7234 is 41 pages long). Rolling your own cache is, well, rolling your own.
And then there is the human problem. I was blown away by a discussion [1] where people reported using cryptography to prevent in-house clients from hardcoding URLs. I was also saddened when I read the JSON API spec [2] -- a great example of REST, I think -- only to discover that most existing implementations [3] disregard the linking aspect and hardcode the URL structure that the spec uses for examples. But this is not a complaint against your webcam example so much as it is a general hurdle with REST.
> None of this relates to whether or not you need futuristic AI or human intervention. I think from the example, it's pretty clear you need neither for a REST API.
True, and some of these use cases are very interesting.
jcrites does not mention them, though. I realize now that his comment does not preclude them, either, but still this example would be more convincing if automated clients were emphasized, because that is what most people are actually building and care about.
> REST seems like an elephant in a room with some blind folks. Everybody who touches it thinks it's something different than the next guy, and they're all describing only one element of the thing.
I don't see it that way at all. Sure, there are minor quibbles about various approaches, but you can easily place everybody into one of two camps: the ones who don't know/care about REST's technical definition, and the ones that do. The two groups largely agree about what REST is amongst members of their own group, but disagree with the other group.
I think the above commentor got it right. You map the CRUD actions to HTTP verbs and statuses, as many of them make sense for your API, with links to further resources along the way.
> or any way of discovering features of an API programmatically.
So, this is one of my issues with HATEOAS. When does this _actually_ come up in practice? I mean really, how do we normally consume APIs? We read the docs to figure out which endpoints we need to call, call them in the right order, and do the right stuff with the result. Nothing about that process is automatic; it takes a human to figure out what's available and the right way to use it.
To me the hypermedia part of REST has always seemed like a pointless academic exercise. Yes it's really cool to be able to point a service browser at your service's root resource and have it map the whole thing out but in practice, when is this ever anything but a neat party trick?
> So, this is one of my issues with HATEOAS. When does this _actually_ come up in practice?
For the type of systems REST is designed to address, all the time. Its not just random happenstance that REST was articulated in the context of an existing, well-known, widely-used system of that type (the WWW.)
Arguably, many API designers are neither designing the kind of systems REST is intended for nor designing things for which their priority is for them to be well-behaved components in a distributed system that, taken as a whole, is the kind of system REST is intended for. Which is fine, but then there is little reason to use REST and less to use pretend-REST.
Efforts like the Hypertext Application Language (HAL) [1] are attempts to implement HATEOAS.
The point that's often missed in casual discussions of REST is that 'link relations' are extremely important, because they represent well-specified ways of relating one resource to another. They are as critical to REST as mediatypes (MIME types) are.
In the graph theory sense, link relations are the label on an edge, URIs are the nodes, and mediatypes are formats that can be used as concrete representations of the nodes.
- Never write a "client" again, use your "REST" library and then just traverse relationships and call actions, rather like an ORM or having a local object graph.
- Letting the API provider optimise your usage, by providing different data at different times, inlining at different times, etc, without any changes to your code.
I don't know why everyone wants to downvote this comment. It's absolutely right.
I would add that REST has nothing to do with pretty URIs, JSON formatting, versioning, query strings, and more misconceptions that people seem to mistake for the architectural style. There is nothing that isn't RESTful with URIs like "/a89f30e9" instead of "/:type/:id". Even TBL thinks URIs should be opaque: https://www.w3.org/DesignIssues/Axioms.html#opaque
To be frank, main issue is people are arguing about semantics, one side thinks Fielding's definition is the only thing that can be called REST, and the other larger(?) side could care less about that academic definition and are describing some simple rules for HTTP APIs. While it would be nice if there were two terms, just saying "This is NOT REST" really doesn't clarify anything.
I'm fine with APIs continuing to be disparate systems if they have a simple, sensible design and have comprehensive documentation. Let's just stop calling them "REST" since they're not as you've pointed out.
Why not push for them to become something more? Documentation for humans is good, but you know what's better? Documentation for computers so that we don't have to build integrations anymore!
I would love to see how this could be realistically done. I recall this being attempted with WSDLs, but there was still a ton of man hours required to integrate with an endpoint.
There is still always an impedance mismatch between the server representation and what the client representation that needs to be fixed.
Like ORMs, such integration automation will always be, at best, a leaky abstraction that makes the easy bits easier, and the hard bits harder.
I'm not sure the point about the ORMs is entirely true. I see what you're saying, but the API boundary should be the canonical definition (unlike an ORM where the database might be), and the server and client need to conform to that, so there shouldn't be leakage.
No worries. There are people out there, like me, who agree with you. You have my vote on this.
Lets say Microsoft at least created their own format, that allows some discovery. That is a good approach, because a big upfront design can be harmful. There are few decisions in HTML you can take as an example.
I am currently working on having a format for my company, that is based on Siren. Its no easy thing to do, because I want it to be easy to be used (which is not simple).
> APIs will continue to be a disparate system that requires custom code for every integration until we can embrace the full benefits of REST
Have an example of a client that DOESN'T require custom code to use an API?
I think this entire debate is summed up best by:
> There is no magical "smart client" that somehow knows that rel=comments means that the link leads to comments about the current resource and can figure out it should POST there to create a new comment. It has no idea what the hell a "comment" is. [1]
This is pretty much the crux of the problem that the Semantic Web / Linked Data folks have been working on for years. They have the technology to do it (RDF). This technology is not widely used for a number of reasons, but "because it can never exist (and I'll write that in boldface for extra credibility)" is not one of them.
People have been able to agree on many common vocabularies of stuff, like "<h1>" and "<svg:ellipse>" and "rel=pingback", and have it work more or less. It is not absurd to suggest that such an agreement can be achieved -- to some extent -- in APIs.
The thing is, those servers and clients already exist, people are just to tied to their notion of "API" to see them. And in fact, you used them to post that very comment.
A website is a RESTful service which a generic client can interact with despite not having any custom code. Does it do so by magically understanding what an <a> or <img> tags are or by magically decoding PNG bytestreams? No, the service uses standard formats that the browser knows and which encode these semantics, allowing it to process them despite its original developers quite possibly not even knowing that service existed.
> No, the service uses standard formats that the browser knows and which encode these semantics, allowing it to process them despite its original developers quite possibly not even knowing that service existed.
You're correct.
But that's not what we're talking about. We're talking about a web browser that can understand its posting a Comment, not a form (or a random collection of key-value pairs). The primitives that make the web work are quite different from the high-level concepts that make an application work.
My browser understands how to display links and forms and text and graphics. It does not understand how to display a comment or a user account or the number of votes a comment has, let alone know when I'm allowed to vote up or down, understand the relationship between a user account and a comment, understand a comment being owned by my user account or another, etc, etc.
What exactly is the end goal? That a HATEOAS-style REST API can be created, you can go to some entry point with your browser, and with no one writing any UI code it just understands how to render the Hacker News site? It will magically understand that the main page contains a list of articles, those articles have comments, links to an original source, etc?
This is my understanding of the goal of HATEOAS. If that's not the goal, I don't understand what it gets besides being a way to build a tiny part of documentation necessary to consume an API.
> Have an example of a client that DOESN'T require custom code to use an API?
Your web browser.
Right here. It's talking with HN's API for me to let me post this comment.
Whenever you have an URL that responds to the browser's requests to let the user do something, that's an API. Maybe a shitty, ad-hoc, poorly designed one, but it's an API. You can even write custom code against if if you want to.
No clients know that "comments" means Comments, because no one has standardised it yet. There's no reason that we can't define "rel=comments" as being the comments for something, in the same way that https://schema.org/ has defined schemas for a range of taxonomies on the web already.
I don't have an example of a public API that can be used by a generic client because no one is making them. I've played around with them in a few projects and there are rough edges, but no reason we can't move forward to them. Google already use 'real' REST for some of their mobile apps as far as I can remember.
So all this push to "do it right" and HTATEOAS, is not actually meant to provide immediate benefit? It is only meant to help enable benefit in the future, when everyone is on board, and the usage of "rel" in json responses is agreed upon?
> No clients know that "comments" means Comments, because no one has standardised it yet.
Given that 'Comments' on HN are different than comments on Facebook, comments on a sales invoice, comments in a threaded forum, and comments on a pull request, how will this get standardized? On their face they may have relatively similar goals, but they have different semantics, rules and ways of being displayed.
On top of that, what happens when I create a new application, with a new object concept that's not standardized? Do I have to wait for it to get standardized and implemented in this magic client before I can launch? Or do I have to build a custom client anyway, entirely negating the point of using standardized stuff in the first place?
It's conceivable that with enough effort, some client could understand the semantics of all object types that are contained in a standard, and the standard is big enough to cover ever situation ever possible.. But then what? Who wants to build an app that basically looks and functions identically to your competitors (who are all using the same semantic objects)?
> There is no magical "smart client" that somehow knows that rel=comments means that the link leads to comments about the current resource and can figure out it should POST there to create a new comment. It has no idea what the hell a "comment" is.
Yes there is: that client is a human being. The point of REST and HATEOAS is to write something that works properly for humans, and to which general software can specifically be adapted.
That's part of why I loathe single page apps and JavaScript so much: HTTP & HTML already offer an incredible expressive range, and JavaScript just tramples over the forms and scaffolding they create.
Search for "rest api standard" you will find a bunch.
While I like standard, and I am always an advocate of standard, I also don't agree with standard body's decision on the exact format. This is the challenge. HATEOS is an absolutely great idea, but also painful to implement, making payload larger. One question I ask is "how many kinds of resource links do I need to supply?"
Otherwise, what I've read so far looks like a really good start. Say what one will about Microsoft's products, but there are a lot of smart folks there.
Just return a link to the "next page" (make the next page opaque); while this removes the ability to of the client to go to an arbitrary page on its own, in practice, I've never seen that matter, and the generalized URL format allows you to seek instead of by LIMIT/OFFSET by something indexable by the DB; HTTP even includes a standardized header for just this purpose[2].
I also think the generalized URL is more in line with Fielding's definition of REST, in that the response links to another resource. (I don't know if being in the header invalidates this; to me, it does not.)
If you get the metadata out of the data section of your response, and move it to the headers, where it belongs, this usually then lets you keep the "collection" of items you are return as an array (because if you need to put metadata along side it, you need:
vs. )(I've seen metadata-pushed-into-the-body turn API endpoints that literally need to return no more than "true" or "false" into object that then force a client to know and look up the correct key in an object… sigh.)
[1]: http://use-the-index-luke.com/no-offset
[2]: https://tools.ietf.org/html/rfc5988