Moderate Blogging

This post has been in draft for months. It’s been in draft for months while watching me go stark raving mad paragraph by paragraph, day by day, for far too long. This needs to see the light of day right now or the world is going to come to an end.

Alright, yeah, I’m a little dramatic. I’m not really going crazy, but I’m really annoyed. The last year of my professional life has been spent on creating a system that ties many dissimilar systems together conveniently for customers. That might not sound like it’s extremely difficult since you’ve probably mashed Twitter and news feeds together with the most popular fainbruck questions on Stack Overflow in the past.

It wasn’t just difficult, it was downright freaking painful. Why was it painful? Because in 2012 programmers working for big companies still have no understanding of REST. That’s a problem that won’t be fixed in the foreseeable future, and we’re to blame.

If you’re short on time, the tl;dr; version of this is simple. HTTP/1.1 has some awesome verbs in its vocabulary like GET, POST, HEAD, PUT and DELETE. It’s also got neat things like headers. The modern world has this thing called JSON. Objects you offer have life cycles from birth, modification and eventual deletion. Combining this knowledge into an API that doesn’t suck is hard, and you need to think carefully before writing a single line of code.

Wow, still here? You’re a sucker for a long winded rant. Let’s start with a bit of background. I spend my days toiling away in the web hosting industry. My job, loosely translated boils down to this:

Come up with and implement unique, useful features and services that make us better than our competition.

Let’s put a little more emphasis there, unique is a very important word. In order for us to not suck and possibly be better than the rack shack down the street, we can’t succumb to the temptation to just run the typical automation suites that everyone else runs with a slightly different look to it. We can’t just toss a bunch of servers on the rack, throw in a semi branded version of the same control panel everyone else uses and call it a day. That’s not being original and innovative, that’s acting like a well housebroken dog that bets on a continuous supply of scraps falling from the dinner table.

A typical web host has the following systems deployed, in no specific order:

• Billing / Provisioning
• Domain registration
• Support / Knowledgebase
• Backup systems (often several kinds)
• SSL certificate issuance
• DNS Management
• Standard shared hosting controls
• Virtual private server controls
• Network monitoring
• IDS/IPS/DDoS protection
• Power management
• Some kind of community portal to let customers kvetch and socialize

I didn’t list cloud there, because it’s basically covered under virtual server management, just a different (and much smarter) stack.

In order to not suck, all of the above has to be brought under a single consistently tooled management interface. If you welcome a customer with more than one set of access credentials, you fail miserably, especially in 2012. One of the reasons that PAAS is gaining such momentum is that people are tired of dealing with stacks comprised of loosely coupled third party stuff that someone threw together on a budget. Step one to awesomeness? Build something that presents only what the client needs from all of these systems in a single, intuitive place. That should not have taken a year. It took a year because vendors in my industry seldom eat their own dogfood, so how would they know that using their API makes people want to eat kittens?

They don’t, so here’s a series of do’s and don’ts complete with nutritious anecdotal rants for the guilty to ponder.

Do understand how people will want to use your API

Without some kind of coherent flow, your API is just a collection of haphazard methods that people will probably use incorrectly. If your API lets me frob thingamadongdongs with foozywidgets, show me how to do that. How do I create the thingamadongdong, or do I need to create the foozywidget first? Can I couple one foozywidget to multiple thingamadongdongs? How can I list my thingamadongdong to foozywidget dependencies? Don’t make me figure this stuff out through trial and error, because I’ll probably find a way of doing it wrong that still seems to work. Then you have to help me sort that out in production later. A simple getting started guide that shows the most common operations that someone would be interested in doing will save your users from hours of frustration and drastically reduce the amount of time you spend supporting them.

Do learn how REST works and adopt better interfaces, right now

Do you have to support something that was glued together with XML, SOAP and puppy dog tails? That’s cool, I can relate. Nothing is stopping you from freezing that and working on 2.0 of your API to offer a proper RESTful implementation. Leave the old one up so all of that legacy crap out in the wild continues to ‘work’ while you focus on new features in the new branch. If you make people send a base64 encoded XML object via GET in order to create a resource and return a response in CSV, stop the insanity!

Do make your API self documenting

Let me send a GET request to methodName.documentation and return a list of fields, type hints, default values and short descriptions of each. A lot of people argue that you should bundle some kind of ‘explorer’ with your SDK, I’m not entirely sure that’s necessary if you make it extremely convenient for me to make my own. Make sure it lets me flip a switch so that deprecated options aren’t returned. Don’t just throw all of this into a massive PDF file to rot on my hard drive as you change or fix things. If you’re just sadistic and insist on doing that anyway, please don’t outsource your documentation to the point where it’s obviously been written by five incompetent people. One incompetent person per PDF is plenty. HTML documentation is great, but only after your methods are self documenting.

Do make proper use of HTTP status codes

If you send HTTP 200 back to me when one of my requests fails, you failed. The status code you set is the only bullet proof way for me to reliably know what happened. Something may have blown up on your side resulting in the returned response being anything but what I expect, a status code of 500 lets me avoid a maze of possible exceptions. For practically anything your API would want to tell me, there is a HTTP response code, even if your API suddenly decides it’s a teapot.

Do offer a well written and maintained SDK, or none at all

It’s great if you offer libraries in many languages to interact with your API, but do you really have the bandwidth and expertise to handle that? If I’m getting started with your service using Python and the first thing I see is a bunch of unhandled exceptions, my confidence is not going to go up. If your libraries don’t have tests, tickle deprecated language features, leak memory or otherwise suck, drop them and put your effort into better implementation, documentation and examples. Making HTTP requests and interpreting the results is stupid easy if you’ve focused on your endpoints and responses, I don’t need a library if you’ve done your job well. If your service is at least not painful to use, experts in various languages will probably give you well written client code to pass onto others.

Do encapsulate transport and method logic separately

HTTP has this thing called headers, they’re awesome. Clients and servers can set them to indicate meaningful things that both parties understand such as authentication data, reusable request tokens and other meta data like how many requests a client has left for a certain period of time. Keep that meta data separate, the actual request / response body should be exclusively used for the business logic of whatever method is understood by both sides.

Do offer a sandbox that precisely mimics production

Don’t make me change my connection string to just test my understanding of a request. Let me flip a header or field to tell you that I just want the request evaluated. I understand that my newly created thingamadongdong wasn’t really created, and that’s fine. I just wanted to make sure I got the details correct. Be consistent, because I’m now going to write tests against this.

Do be explicit in error reporting

Nothing is more frustrating than having a request fail, fixing whatever the error described and then getting a shiny new error in return. Cryptic errors don’t help anyone. Tell me exactly what things I did incorrectly so I can fix them. I’m not saying you shouldn’t use codes that are easy to fetch from a dictionary, of course you should. I’m just saying be human friendly too.

Do not establish prerequisites that can’t be met with a call

Don’t make me e-mail you just to whitelist an IP address so I can make requests. That just makes me wait until you wake up and actually do it, when I could have been doing more productive things. If you can’t let me manage that sort of things via the API once authenticated, at least give me some sort of self service facility to do it. Other than staying within your terms of service, I should not need your permission for anything.

Do not charge me without a programmatically retrievable paper trail

Many people complain about this. Even though you’ve gone through great lengths to give me a proper sandbox, verbose error reporting, standard HTTP responses, separation of transport and method logic and great documentation, I’m going to have bugs. Sometimes, those bugs are going to cost me money. Allow me to retrieve a journal of everything that I paid for, starting from day zero absolutely free. In fact, offer use of this method a sensible number of times each day in a way that it does not count against any request limit you impose.

Do not send e-mail to my customers unless I specifically tell you to do so.

This is a no brainer. If I’m creating or modifying an object that requires the e-mail address of my customer, let me be the one to decide if they should be annoyed with an e-mail or not. Tell me if the request worked, what went wrong if not and let me take it from there. The exception to this is when you must do some kind of validation on your end, for instance gathering verification data to issue a certificate. Still, give me as much control as possible over the process.

Do pick a character set and stick to it

This should go without saying. Unfortunately, all too often, it doesn’t.

Do eat your own dogfood

Normalization – you need to know what this means. You need to understand that I’m probably mixing your responses with lots of other ingredients and probably storing the results on my end. How easy is it to take your responses and get them into a database in 3NF? I reiterate, don’t make me keep iterating. Use what you build so you can make it better. Are your responses consistent enough to be conveniently stored without much additional effort?

Do your best to support me and listen to my feedback

If I ask you how to list my frobulated thingamadongdongs grouped by fuzzywidget dependencies, don’t tell me that it’s simply not possible. Take a minute and figure out what I must be trying to accomplish and offer suggestions on how I could go about it in a better way. Test your solutions before you hit the send button, never say Try setting A to B and see if that helps.

I’m finally publishing this as I wind down my initial hurdle in preparation of launching the thing that consumed most of the last year of my life. What gets me is, even though I pulled it off – it took a year just to build a platform where we could really begin to do interesting things. If this read like a series of anecdotal rants collected over time – that’s exactly what it is. It used to contain the names of the not so innocent, but I decided to take them out.

If I had access to even halfway correct documentation, I’d be telling you about some spiffy new thing that would make your life easier by now. Really, though – who deserves the blame here? The people that write crap, or the people that consume it quietly? If we continue to tolerate, and sometimes honor mediocrity, we short circuit the quality we deserve. As I said in the beginning, we have ourselves to blame for tolerating crap.

If you got this far, thanks for reading.