Roy Fielding's Misappropriated REST Dissertation

RESTful APIs are everywhere. This is funny, because how many people really knowwhat “RESTful” is supposed to mean?

I think most of us can empathize withthis Hacker Newsposter:

I’ve read several articles about REST, even a bit of the original paper. But I still have quite a vague idea about what it is. I’m beginning to think that nobody knows, that it’s simply a very poorly defined concept.

I had planned to write a blog post exploring how REST came to be such adominant paradigm for communication across the internet. I started my researchby readingRoy Fielding’s 2000dissertation,which introduced REST to the world. After reading Fielding’s dissertation, Irealized that the much more interesting story here is how Fielding’s ideas cameto be so widely misunderstood.

Many more people know that Fielding’s dissertation is where REST came from thanhave read the dissertation (fair enough), so misconceptions about what thedissertation actually contains are pervasive.

The biggest of these misconceptions is that the dissertation directly addressesthe problem of building APIs. I had always assumed, as I imagine many peopledo, that REST was intended from the get-go as an architectural model for webAPIs built on top of HTTP. I thought perhaps that there had been some chaoticexperimental period where people were building APIs on top of HTTP all wrong,and then Fielding came along and presented REST as the sane way to do things.But the timeline doesn’t make sense here: APIs for web services, in the sensethat we know them today, weren’t a thing until a few years after Fieldingpublished his dissertation.

Fielding’s dissertation (titled “Architectural Styles and the Design ofNetwork-based Software Architectures”) is not about how to build APIs on top ofHTTP but rather about HTTP itself. Fielding contributed to the HTTP/1.0specification and co-authored the HTTP/1.1 specification, which was publishedin 1999. He was interested in the architectural lessons that could be drawnfrom the design of the HTTP protocol; his dissertation presents REST as adistillation of the architectural principles that guided the standardizationprocess for HTTP/1.1. Fielding used these principles to make decisions aboutwhich proposals to incorporate into HTTP/1.1. For example, he rejected aproposal to batch requests using newMGETandMHEADmethods because he feltthe proposal violated the constraints prescribed by REST, especially theconstraint that messages in a REST system should be easy to proxy andcache.So HTTP/1.1 was instead designed around persistent connections overwhich multiple HTTP requests can be sent. (Fielding also felt that cookies arenot RESTful because they add state to what should be a stateless system, buttheir usage was already entrenched.) REST, for Fielding, was not a guide tobuilding HTTP-based systems but a guide to extending HTTP.

This isn’t to say that Fielding doesn’t think REST could be used to build othersystems. It’s just that he assumes these other systems will also be“distributed hypermedia systems.” This is another misconception people haveabout REST: that it is a general architecture you can use for any kind ofnetworked application. But you could sum up the part of the dissertation whereFielding introduces REST as, essentially, “Listen, we just designed HTTP, so ifyou also find yourself designing adistributed hypermedia systemyou shoulduse this cool architecture we worked out called REST to make things easier.”It’s not obvious why Fielding thinks anyone would ever attempt to build such athing given that the web already exists; perhaps in 2000 it seemed like therewas room for more than one distributed hypermedia system in the world. Anyway,Fielding makes clear that REST is intended as a solution for the scalabilityand consistency problems that arise when trying to connect hypermedia acrossthe internet,notas an architectural model for distributed applications ingeneral.

We remember Fielding’s dissertation now as the dissertation that introducedREST, but really the dissertation is about how much one-size-fits-all softwarearchitectures suck, and how you can better pick a software architectureappropriate for your needs. Only a single chapter of the dissertation isdevoted to REST itself; much of the word count is spent on a taxonomy ofalternative architectural stylesthat one could use for networkedapplications. Among these is the Pipe-and-Filter (PF) style, inspired by Unixpipes, along with various refinements of the Client-Server style (CS), such asLayered-Client-Server (LCS), Client-Cache-Stateless-Server (C$SS), andLayered-Client-Cache-Stateless-Server (LC$SS). The acronyms get unwieldy butFielding’s point is that you can mix and match constraints imposed by existingstyles to derive new styles. REST gets derived this way and could instead havebeen called—but for obvious reasons wasnot—Uniform-Layered-Code-on-Demand-Client-Cache-Stateless-Server (ULCODC$SS).Fielding establishes this taxonomy to emphasize that different constraints areappropriate for different applications and that this last group of constraintswere the ones he felt worked best for HTTP.

This is the deep, deep irony of REST’s ubiquity today. REST gets blindly usedfor all sorts of networked applications now, but Fielding originally offeredREST as an illustration of how to derive a software architecture tailored toan individual application’s particular needs.

I struggle to understand how this happened, because Fielding is so explicitabout the pitfalls of not letting form follow function. He warns, almost at thevery beginning of the dissertation, that “design-by-buzzword is a commonoccurrence” brought on by a failure to properly appreciate softwarearchitecture.He picks up this theme again several pages later:

Some architectural styles are often portrayed as “silver bullet” solutionsfor all forms of software. However, a good designer should select a stylethat matches the needs of a particular problem being solved.

REST itself is an especially poor “silver bullet” solution, because, asFielding later points out, it incorporates trade-offs that may not beappropriate unless you are building a distributed hypermedia application:

REST is designed to be efficient for large-grain hypermedia data transfer,optimizing for the common case of the Web, but resulting in an interface thatis not optimal for other forms of architectural interaction.

Fielding came up with REST because the web posed a thorny problem of “anarchicscalability,” by which Fielding means the need to connect documents in aperformant way across organizational and national boundaries. The constraintsthat REST imposes were carefully chosen to solve this anarchic scalabilityproblem. Web service APIs that arepublic-facinghave to deal with a similarproblem, so one can see why REST is relevant there. Yet today it would not beat all surprising to find that an engineering team has built a backend usingREST even though the backend only talks to clients that the engineering teamhas full control over. We have all become the architect inthis Monty Pythonsketch, who designs an apartmentbuilding in the style of a slaughterhouse because slaughterhouses are the onlything he has experience building. (Fielding uses a line from this sketch as anepigraph for his dissertation: “Excuse me… did you say ‘knives’?”)

So, given that Fielding’s dissertation was all about avoiding silver bulletsoftware architectures, how did REST become a de facto standard for webservices of every kind?

My theory is that, in the mid-2000s, the people who were sick of SOAP andwanted to do something else needed their own four-letter acronym.

I’m only half-joking here. SOAP, or the Simple Object Access Protocol, is averbose and complicated protocol that you cannot use without firstunderstanding a bunch of interrelated XML specifications. Early web servicesoffered APIs based on SOAP, but, as more and more APIs started being offered inthe mid-2000s, software developers burned by SOAP’s complexity migrated away enmasse.

Among this crowd, SOAP inspired contempt. Ruby-on-Railsdropped SOAP support in 2007, leading to this emblematic comment from Railscreator David Heinemeier Hansson: “We feel that SOAP is overly complicated.It’s been taken over by the enterprise people, and when that happens, usuallynothing good comes of it.”The “enterprise people” wanted everything to beformally specified, but the get-shit-done crowd saw that as a waste of time.

If the get-shit-done crowd wasn’t going to use SOAP, they still needed somestandard way of doing things. Since everyone was using HTTP, and sinceeveryone would keep using HTTP at least as a transport layer because of all theproxying and caching support, the simplest possible thing to do was just relyon HTTP’s existing semantics. So that’s what they did. They could have calledtheir approach Fuck It, Overload HTTP (FIOH), and that would have been anaccurate name, as anyone who has ever tried to decide what HTTP status code toreturn for a business logic error can attest. But that would have seemedrecklessly blasé next to all the formal specification work that went into SOAP.

Luckily, there was this dissertation out there, written by a co-author of theHTTP/1.1 specification, that had something vaguely to do with extending HTTPand could offer FIOH a veneer of academic respectability. So REST wasappropriated to give cover for what was really just FIOH.

I’m not saying that this is exactly how things happened, or that there was anactual conspiracy among irreverent startup types to misappropriate REST, butthis story helps me understand how REST became a model for web service APIswhen Fielding’s dissertation isn’t about web service APIs at all. AdoptingREST’s constraints makes some sense, especially for public-facing APIs that docross organizational boundaries and thus benefit from REST’s “uniforminterface.” That link must have been the kernel of why REST first got mentionedin connection with building APIs on the web. But imagining a separate approachcalled “FIOH,” that borrowed the “REST” name partly just for marketing reasons,helps me account for the many disparities between what today we know as RESTfulAPIs and the REST architectural style that Fielding originally described.

REST purists often complain, for example, that so-called REST APIs aren’tactually REST APIs because they do not use Hypermedia as The Engine ofApplication State (HATEOAS). Fielding himselfhas made thiscriticism.According to him, a real REST API is supposed to allow you to navigate all itsendpoints from a base endpoint by following links. If you think that people areactually out there trying to build REST APIs, then this is a glaringomission—HATEOAS really is fundamental to Fielding’s original conception ofREST, especially considering that the “state transfer” in “RepresentationalState Transfer” refers to navigating a state machine using hyperlinks betweenresources (and not, as many people seem to believe, to transferring resourcestate over the wire).But if you imagine that everyone is just buildingFIOH APIs and advertising them, with a nudge and a wink, as REST APIs, orslightly more honestly as “RESTful” APIs, then of course HATEOAS isunimportant.

Similarly, you might be surprised to know that there is nothing in Fielding’sdissertation about which HTTP verb should map to which CRUD action, even thoughsoftware developers like to argue endlessly about whether using PUT or PATCH toupdate a resource is more RESTful. Having a standard mapping of HTTP verbs toCRUD actions is a useful thing, but this standard mapping is part of FIOH andnot part of REST.

This is why, rather than saying that nobody understands REST, we should justthink of the term “REST” as having been misappropriated. The modern notion of aREST API has historical links to Fielding’s REST architecture, but really thetwo things are separate. The historical link is good to keep in mind as a guidefor when to build a RESTful API. Does your API cross organizational andnational boundaries the same way that HTTP needs to? Then building a RESTfulAPI with a predictable, uniform interface might be the right approach. If not,it’s good to remember that Fielding favored having form follow function. Maybesomething like GraphQL or even just JSON-RPC would be a better fit for what youare trying to accomplish.

If you enjoyed this post, more like it come out every four weeks! Follow@TwoBitHistoryon Twitter or subscribe to theRSS feed to make sure you know when a new post is out.

Previously on TwoBitHistory…