What the Rest?

When I talked recently about being collaborative and open, I mentioned that I was weeks into research around building RESTful API's. I've been like Luke training on Dagobah, except replace Jedi powers with REST and Yoda with the Internet, RFCs and dozens of blog posts. And despite everything, my mind is clouded. When it comes to new tech, sometimes the more you learn about the pieces, the less you can see how they could ever fit together. This post is an experiment to see if we can fix that.

REST is as deep as the rabbit hole, with varied approaches, undefined best-practices, and flamewars along the way. Should I use custom hypermedia types or something like HAL or JSON-LD? Should I implement OPTIONS, and if so, should it tell me only what methods to use or much more? What role should the API documentation play and what information should be described inside the API itself? How do custom verbs like banning a user or linking to a friend look?

Eventually, best-practices will rise to the surface. But I (and maybe you) want to build a high-quality "REST" API right now. Curiously, HATEOAS - the missing RESTful piece for many APIs - has made things more difficult for me so far. The idea is beautiful: return links in your API to help the client know what to do next. But links, which are basically a URL with a "rel", only tell part of the story. What methods does that link support? What fields should I send to it? And where should all this information live?

In the next sections, I'll say some things about REST that are just wrong. And hopefully, I'll be corrected. I'll pour out my RESTful heart, and invite you all to crush it and build it back up. I've heard that the Internet is good at correcting people, so I thought, why not tap into that? :).

By the end, I hope to have one, good, well-defined path that can be written up and shared openly.

HATEOAS: What you can do next?

When I get a response from a RESTful API, it should contain links. Like on a webpage, these tell me what I might do next (i.e. what actions I can take).

Links versus Resources

As I understand it, it's not exactly correct to say that links are links to resources. Of course a link has a URI, and each URI is an address to a resource, but you could have multiple hyperlinks to the same URI. For example, imagine there are multiple actions I can take on that resource. Suppose that when we go to the homepage of our API it returns the following links (I'm using HAL to represent the links, but that's not important):

{
  "_links": {
    "self": {
      "href": "/"
    },
    "http://api.example.com/rels/users": {
      "href": "http://api.example.com/users",
      "title": "Users in the system"
    },
    "http://api.example.com/rels/users_reinvite": {
      "href": "http://api.example.com/users",
      "title": "Re-invite unregistered users to the system"
    },
  },
  "...": "... other stuff ..."
}

In this case, we have 2 different links (actions) but each link points to the same resource. This is because there are 2 actions that can be taken on that resource. This is my understanding of links versus resources, a distinction which is important because it means that a link is much more than just a URI.

We don't know exactly what to do with those links yet, but that's next.

Hypermedia Links: What they tell Us

Exactly how these links look depends on your hypermedia type (e.g. HAL something custom, etc), but a link has the following information:

1. A URI;
2. A rel to explain the significance of this link;
3. (optional) A type, which defines the hypermedia type to be returned by the link;
4. (optional) A nice human title for the link;

In a browser, we click links (i.e. GET) or submit a form (i.e. POST). The form contains the fields we need to send right inside of it and there's an web-wide standard of sending that data in a certain format (i.e. application/x-www-form-urlencoded). So when we're using a browser, each link contains all the information needed to follow it.

But that's not true at all with an API. We need more information than we have from simply looking at the link, which I'll call:

The 4 Missing Pieces of a Link

1. What HTTP methods can I use with this link?
2. What fields should I send if I'm POST'ing/PUT'ing?
3. How should I encode that data in the request (e.g. application/json, application/x-www-form-urlencoded, etc)?
4. What media type(s) should I expect back from the response?

HATEOAS tells us to use these links to determine what our next step, or action is (they induce application flow). In the HTTP world, they should tell us what possible HTTP requests we can take next.

So if this information isn't found in the hyperlinks, where does it live? Where can an API client find these details and how does she know to look there? This is at the heart of the trouble I have in understanding RESTful APIs.

Finding the Missing Pieces to make the next Request

Your API also needs to choose how its hypermedia type(s) look like. Will each link have its own type (e.g. application/vnd.com.users+xml) or will you use one hypermedia type like HAL?

Right now, we're missing 4 pieces of information before we can really make the next request. In REST, you often read that the only thing you should need to document is your hypermedia types. In that model, does every link have its own hypermedia type? And do the docs for that hypermedia type really tell us what HTTP methods can be used and what fields can be POST'ed? That would seem odd, especially because the data is not sent (e.g. POST'ed) using the hypermedia type. Usually data is sent with something simpler like application/json, and the special hypermedia is only used in the response.

So let's look at the 2 approaches (custom hypermedia type versus HAL) and try to see how a client would answer the 4 questions standing between us and the next API request:

+--------------+-------------------------------------------+--------------------------------------+ | | Custom hypermedia type | HAL (or similar) | +--------------+-------------------------------------------+--------------------------------------+ | HTTP Methods | Find docs based on the hypermedia type?? | Find docs based on the link rel | +--------------+-------------------------------------------+--------------------------------------+ | Fields | Find docs based on the hypermedia type?? | Find docs based on the link rel | +--------------+-------------------------------------------+--------------------------------------+ | Request | Find docs based on the hypermedia type?? | Find docs based on the link rel | | encoding | | | +--------------+-------------------------------------------+--------------------------------------+ | Response | Read the link type attribute | Assume HAL | | media type | | | +--------------+-------------------------------------------+--------------------------------------+

It seems that both methods will probably rely on some external documentation. This is known as "out-of-band information", which we shouldn't need in theory. But I'd argue that creating an API that's fully self-describing is still very hard, and quite possibly not worth it.

Some pieces of information could probably be answered globally for your API (e.g. Request encoding). Very literally, you might just say on the homepage of your docs that all endpoints support application/json and application/x-www-form-urlencoded.

But usually, we will need to look up a specific API docs page for each link. For a custom hypermedia approach, the docs seem to be for each hypermedia type (right?, wrong?). For HAL, since we only have one hypermedia type, we rely entirely on the link rel. This could all totally be wrong, but if it is, then how should the client be answering our 4 questions for each link? If we have docs, how do they know which doc to look at for each link?

Value in OPTIONS?

What if we made an OPTIONS request to /users? This actually seems very unhelpful, as the OPTIONS is for a single URI, which could actually have multiple links to it. The OPTIONS response may say we can POST, but how would a client know what to POST? In our example, there will be 2 POST actions (one for creating a new user resource and another for "re-inviting" users), each which has its own documentation on how the fields should look.

OPTIONS may be helpful if it returns the links that I would receive if I made a GET request to the resource. In our example, we'd have:

{
  "_links": {
    "self": {
      "href": "http://api.example.com/users",
      "title": "Users in the system"
    },
    "http://api.example.com/rels/users_reinvite": {
      "href": "http://api.example.com/users",
      "title": "Re-invite unregistered users to the system"
    },
    
    "...": "... other links, like first, prev, next, last ..."
  },
  "...": "... other stuff, like embedded users ..."
}

Of course, I could just make a GET for this information. So is there value in OPTIONS?

Problems: When I have only a URI

One important consequence about a REST API is that we're not really supposed to start with (i.e. bookmark) a URI. In fact, with just a URI, we don't really have enough information to know what to do with it.

For example, if I know nothing about /users and I make a GET request, I get back these 2 links (among other things):

{
  "_links": {
    "self": {
      "href": "http://api.example.com/users",
      "title": "Users in the system"
    },
    "http://api.example.com/rels/users_reinvite": {
      "href": "http://api.example.com/users",
      "title": "Re-invite unregistered users to the system"
    },
    
    "...": "... other links, like first, prev, next, last ..."
  },
  "...": "... other stuff, like embedded users ..."
}

Notice self, which is a standard IANA Link Relation, but which no longer includes the helpful http://api.example.com/rels/users "rel". If the rel is the key to finding the docs, we're stuck.

Of course, according to the rules, we shouldn't be hardcoding URIs', we should always start back on the homepage and follow the link, with its nice rel.

But this scenario does show up during normal client-server interaction in at least two places:

1. After POST'ing to create a new resource, the location header gives us the URI to the resource, but without a rel;

1. When GET'ing a collection resource, the embedded children don't have a specific rel value (they have self):

{
  "_embedded": {
    "users": [
      {
        "username": "weaverryan",
        "_links": {
          "self": {
            "href": "http://api.example.com/user/weaverryan",
            "title": "Users in the system"
          },
          "...": "... other links ..."
        }
      },
      "... other users ..."
    ]
  },
  "...": "... other stuff, links, data, etc ..."
}

In both cases, we can GET the resource, but we're never given the pointer (rel in HAL) to the docs, which answer our questions. I need some "in-band" information that tells me where to find ("out-of-band") the API documentation. Said more simply, if I'm presented with a link to this user, how should our client know what to do with it?

FoxyCart's solution is to include this information in the "originating" docs. In the 2 scenarios above, I'm POST'ing and GET'ing to /users and we already have its rel (http://api.example.com/users). This means we also have its docs. On that page, the documentation from http://api.example.com/user (the "rel" for the new resource) could be embedded (see "Embedded Resource: user" at https://api.foxycart.com/rels/users).

Is there a better way? How should the API client know what to do with this new resource? What can be safely assumed?

Opinions, Experiences, Please!

Again, a huge thanks to Luke Stokes and FoxyCart for pioneering so much of this and answering my questions. If you're someone who's been in the trenches with REST, comment, enlighten and correct me please! Anything we accomplish here will ultimately turn into code, a script (both of which will be available publicly) and a screencast.

But one more thought! The further I get into REST, the more rules I see. Things like hypermedia (linking to what I can do next) seem very pragmatic and wonderful. But immediately after, I find myself (as a client) having a difficult time navigating a RESTful API and knowing where to find the docs (whether those docs are part of the API or externally). I want to be able to leverage all the great ideas behind REST, but not be constrained by them. At the end of the day, we need an API that I can build and that our friendly API client can understand quickly. In some ways, then, the path of DHH (Getting hyper about hypermedia APIs) does make sense: use what's good, leave what's complex, break some rules, and move forward.

But, maybe I'm just missing a few, final, beautiful pieces to get out of the swamp and up to blissful RESTland.

Thanks!