Behavior-Driven RESTful APIs

2014-07-01

In the RESTBucks example, the authors present a useful state diagram that describes the actions a client can perform against the service.

Where does such an application state diagram come from? Well, it’s derived from the requirements, of course.

Since I like to specify requirements using examples, let’s see how we can derive an application state diagram from BDD-style requirements.

Example: RESTBucks state diagram

Here are the three scenarios for the Order a Drink story:

Scenario: Order a drink

Given the RESTBucks service
When I create an order for a large, semi milk latte for takeaway
Then the order is created
When I pay the order using credit card xxx1234
Then I receive a receipt
And the order is paid
When I wait until the order is ready
And I take the order
Then the order is completed

Scenario: Change an order

Given the RESTBucks service
When I create an order for a large, semi milk latte for takeaway
Then the order is created
And the size is large
When I change the order to a small size
Then the order is created
And the size is small

Scenario: Cancel an order

Given the RESTBucks service
When I create an order for a large, semi milk latte for takeaway
Then the order is created
When I cancel the order
Then the order is canceled

Let’s look at this in more detail, starting with the happy path scenario.

Given the RESTBucks service
When I create an order for a large, semi milk latte for takeaway

The first line tells me there is a REST service, at some given billboard URL. The second line tells me I can use the POST method on that URI to create an Order resource with the given properties.
bdd-rest-1

Then the order is created

This tells me the POST returns 201 with the location of the created Order resource.

When I pay the order using credit card xxx1234

This tells me there is a pay action (link relation).
bdd-rest-2

Then I receive a receipt

This tells me the response of the pay action contains the representation of a Receipt resource.
bdd-rest-3

And the order is paid

This tells me there is a link from the Receipt resource back to the Order resource. It also tells me the Order is now in paid status.
bdd-rest-4

When I wait until the order is ready

This tells me that I can refresh the Order using GET until some other process changes its state to ready.
bdd-rest-5

And I take the order

This tells me there is a take action (link relation).
bdd-rest-6

Then the order is completed

This tells me that the Order is now in completed state.
bdd-rest-7

Analyzing the other two scenarios in similar fashion gives us a state diagram that is very similar to the original in the RESTBucks example.
bdd-rest-8
The only difference is that this diagram here contains an additional action to navigate from the Receipt to the Order. This navigation is also described in the book, but not shown in the diagram in the book.

Using BDD techniques for developing RESTful APIs

Using BDD scenarios it’s quite easy to discover the application state diagram. This shouldn’t come as a surprise, since the Given/When/Then syntax of BDD scenarios is just another way of describing states and state transitions.

From the application state diagram it’s only a small step to the complete resource model. When the resource model is implemented, you can re-use the BDD scenarios to automatically verify that the implementation matches the requirements.

So all in all, BDD techniques can help us a lot when developing RESTful APIs.


HyperRosetta

2014-06-23

rosetta-stoneThe Rosetta stone is a rock with the same text inscribed in three different languages. This allowed us to decipher Egyptian hieroglyphs.

In this post I’ll introduce a similar “stone” for hypermedia formats, using the RESTBucks example.

I think that seeing concrete hypermedia messages in different formats will make the similarities and differences clearly visible, which will hopefully make it easier to choose the right format for your API.

The text that I’ll use for HyperRosetta, the hypermedia Rosetta stone, is Example 5.6 of the REST in Practice book. For those hypermedia formats that can include templates, I’ll use the payment link with the representation in Example 5.7 of the book.

These are the media types available on HyperRosetta:

If you’d like to see another media type added to this list, please add a comment to this post.

application/vnd.restbucks+xml

This is the representation used in the book:

<order xmlns="http://schemas.restbucks.com/" 
    xmlns:dap="http://schemas.restbucks.com/dap">
  <dap:link mediaType="application/vnd.restbucks.com"
      uri="http://restbucks.com/order/1234"
      rel"http://relations.restbucks.com/cancel"/>
  <dap:link mediaType="application/vnd.restbucks.com"
      uri="http://restbucks.com/payment/1234"
      rel"http://relations.restbucks.com/payment"/>
  <dap:link mediaType="application/vnd.restbucks.com"
      uri="http://restbucks.com/order/1234"
      rel"http://relations.restbucks.com/update"/>
  <dap:link mediaType="application/vnd.restbucks.com"
      uri="http://restbucks.com/order/1234"
      rel"self"/>
  <item>
    <milk>semi</milk>
    <size>large</size>
    <drink>cappuccino</drink>
  </item>
  <location>takeAway</location>
  <cost>2.0</cost>
  <status>unpaid</status
</order>

This is an example of a domain-specific media type, so I will not be able to re-use existing client libraries to parse it. Since this format is based on XML, I can use an existing XML parser, but it won’t be able to recognize links in this representation.

This media type doesn’t tell me in the message what HTTP methods to use to dereference the link URIs. So the client has to be programmed with the knowledge that it has to use PUT on the update link, for instance. If the service should ever change that to PATCH, the client will break.

The message does tell me the media type to expect in the response, making it easier for the client to decide whether it should follow a link.

The RESTBucks media type is a fiat standard that isn’t registered with IANA.

application/atom+xml

<entry xmlns="http://www.w3.org/2005/Atom">
  <author>
    <name>RESTBucks</name>
  </author>
  <content type="application/vnd.restbucks+xml">
    <order xmlns="http://schemas.restbucks.com/">
      <item>
        <milk>semi</milk>
        <size>large</size>
        <drink>cappuccino</drink>
      </item>
      <location>takeAway</location>
      <cost>2.0</cost>
      <status>unpaid</status
    </order>
  </content>
  <link type="application/vnd.restbucks.com"
      href="http://restbucks.com/payment/1234"
      rel"http://relations.restbucks.com/payment"/>
  <link type="application/vnd.restbucks.com"
      href="http://restbucks.com/order/1234"
      rel"edit"/>
  <link type="application/vnd.restbucks.com"
      href="http://restbucks.com/order/1234"
      rel"self"/>
</entry>

Atom is a domain-general media type that implements the collection pattern. Since it is not specific to a single service, I can use an Atom library to parse the message and it will be able to find links. It will also know that the edit link will allow me to update and delete the order (using PUT and DELETE respectively).

The Atom library won’t help me with parsing the content, but I can still use an XML library for that. As with the domain-specific media type, I will need to embed knowledge of the HTTP method used for paying into my client.

A disadvantage of this particular domain-general media type is the overhead of things like author that don’t particularly make sense for RESTBucks.

application/xml

<order xmlns="http://schemas.restbucks.com/">
  <link href="http://restbucks.com/order/1234"
      rel"http://relations.restbucks.com/cancel"/>
  <link href="http://restbucks.com/payment/1234"
      rel"http://relations.restbucks.com/payment"/>
  <link href="http://restbucks.com/order/1234"
      rel"http://relations.restbucks.com/update"/>
  <link href="http://restbucks.com/order/1234"
      rel"self"/>
  <item>
    <milk>semi</milk>
    <size>large</size>
    <drink>cappuccino</drink>
  </item>
  <location>takeAway</location>
  <cost>2.0</cost>
  <status>unpaid</status
</order>

XML is a domain-agnostic format, which means it can be used for anything. The downside is that you won’t know anything about the content until you look at it.

Formally, this means I can’t dispatch on the media type alone anymore, but need to look at the namespace inside the message. In practice people dispatch on the media type anyway. This can lead to parsing problems when the expectation isn’t met, e.g. when an error document is sent instead of an order.

Worse, XML isn’t even a hypermedia type, since it doesn’t describe links. There is the XLInk standard for that, but I haven’t seen that used in APIs.

application/json

{ "order": {
  "_links": {
    "http://relations.restbucks.com/cancel": { 
      "href": "http://restbucks.com/order/1234"
    },
    "http://relations.restbucks.com/payment": {
      "href": "http://restbucks.com/payment/1234"
    }
    "http://relations.restbucks.com/update": { 
      "href": "http://restbucks.com/order/1234"
    },
    "self": { 
      "href": "http://restbucks.com/order/1234"
    },
    "item": {
      "milk": "semi",
      "size": "large",
      "drink": "cappuccino"
    }
    "location": "takeAway",
    "cost": 2.0,
    "status": "unpaid"
  }
}

JSON is a another domain-agnostic format without linking capabilities, so the same caveats apply as for XML. Despite these significant drawbacks, this is what high-profile projects like Spring use.

application/hal+json

{ "_links": {
    "self": { 
      "href": "http://restbucks.com/order/1234"
    },
    "curies": [{
      "name": "relations",
      "href": "http://relations.restbucks.com/"
    }],
    "relations:cancel": { 
      "href": "http://restbucks.com/order/1234"
    },
    "relations:payment": {
      "href": "http://restbucks.com/payment/1234"
    }
    "relations:update": { 
      "href": "http://restbucks.com/order/1234"
    }
  },
  "_embedded": {
    "item": [{
      "milk": "semi",
      "size": "large",
      "drink": "cappuccino"
    }]
  },
  "location": "takeAway",
  "cost": 2.0,
  "status": "unpaid"
}

HAL is a another domain-agnostic format, so the same caveats apply as for JSON.

However, HAL is a real hypermedia type, since it standardizes how to find links (using the _links property). In addition, it also defines how to find embedded objects (using the _embedded property).

HAL uses curies to make the representation a bit more compact. This is nice for humans, but another thing to program into your clients.

HAL is currently an Internet-Draft. There is also an XML version of HAL registered.

application/collection+json

{  "collection": {
    "version" : "1.0",
    "href" : "http://restbucks.com/orders",
    "items" : [{
      "href": "http://restbucks.com/order/1234",
      "data": [
        { "name": "item1.milk", "value": "semi" }, 
        { "name": "item1.size", "value": "large" }, 
        { "name": "item1.drink", "value": "cappuccino" }, 
        { "name": "location", "value": "takeAway" }, 
        { "name": "cost", "value": 2.0 }, 
        { "name": "status", "value": "unpaid" }
      ],
      "links" : [{ 
        "rel" : "http://relations.restbucks.com/cancel", 
        "href" : "http://restbucks.com/order/1234"
      }, { 
        "rel" : "http://relations.restbucks.com/payment", 
        "href" : "http://restbucks.com/payment/1234"
      }, { 
        "rel" : "http://relations.restbucks.com/update", 
        "href" : "http://restbucks.com/order/1234"
      }, { 
        "rel" : "self", 
        "href" : "http://restbucks.com/order/1234"
      }]
    }]
  }
}

Collection+JSON (Cj) is the translation of Atom into JSON. Like Atom, it’s a domain-general format for collections.

In Cj, properties are single values only, so I had to cheat and use the item1. prefix to keep the item data in the message. A better solution would probably be to extract the order items into its own collection, but I didn’t want to change the granularity of the messages.

Cj provides a template property that specifies how to add an item to the collection or update an existing one. Unfortunately, that mechanism is specific to collection actions, so we can’t use it to specify how to add a payment, for example. I don’t show the template in this example because it suffers from the same problem as the data property in that it can’t embed objects.

application/vnd.mason+json

{ "@namespaces": {
    "relations": {
      "name": "http://relations.restbucks.com/"
    }
  },
  "item": [{
    "milk": "semi",
    "size": "large",
    "drink": "cappuccino"
  }],
  "location": "takeAway",
  "cost": 2.0,
  "status": "unpaid",
  "@links": {
    "self": { 
      "href": "http://restbucks.com/order/1234"
    }
  },
  "@actions": {
    "relations:cancel": { 
      "href": "http://restbucks.com/order/1234",
      "type": "void",
      "method": "DELETE"
    },
    "relations:payment": {
      "href": "http://restbucks.com/payment/1234",
      "title": "Pay the order",
      "type": "any",
      "method": "PUT",
      "template": {
        "payment": {
          "amount": 2.0,
          "cardholderName": "",
          "cardNumber": "",
          "expiryMonth": "",
          "expiryYear": ""
        }
      }
    },
    "relations:update": { 
      "href": "http://restbucks.com/order/1234",
      "type": "any",
      "method": "PUT",
      "template": {
        "item": [{
          "milk": "semi",
          "size": "large",
          "drink": "cappuccino"
        }],
        "location": "takeAway",
        "cost": 2.0,
        "status": "unpaid"
      }
    }
  }
}

Mason is a superset of HAL. It adds actions and errors (not shown in this example), which turns it into a full hypermedia type (level 3b). Mason also supports curies.

A peculiarity of the actions is the type property, which can be void, json, json-files, or any. I’d expected a media type there.

application/vnd.siren+json

{ "class": [ "order" ],
  "properties": [{
    "location": "takeAway",
    "cost": 2.0,
    "status": "unpaid"
  },
  "entities": [{
    "class": [ "item" ],
    "rel": [ "http://relations.restbucks.com/item" ],
    "properties": {
      "milk": "semi",
      "size": "large",
      "drink": "cappuccino"
    }
  }],
  "actions": [{
    "name": "http://relations.restbucks.com/cancel",
    "title": "Cancel the order",
    "method": "DELETE",
    "href": "http://restbucks.com/order/1234"
  }, {
    "name": "http://relations.restbucks.com/payment",
    "title": "Pay the order",
    "href": "http://restbucks.com/payment/1234",
    "type": "application/vnd.siren+json",
    "method": "PUT",
    "fields": [{
      "name": "amount",
      "type": "number",
      "value": 2.0
    }, {
      "name": "cardholderName",
      "type": "text"
    }, {
      "name": "cardNumber",
      "type": "text"
    }, {
      "name": "expiryMonth",
      "type": "number"
    }, {
      "name": "expiryYear",
      "type": "number"
    }],
  }, {
    "name": "http://relations.restbucks.com/update",
    "href": "http://restbucks.com/order/1234",
    "method": "PUT",
    "type": "application/vnd.siren.json"
  }],
  "links": [{ 
    "rel": "self",
    "href": "http://restbucks.com/order/1234"
  }]
}

Siren is a full hypermedia type that includes the HTTP methods and media types to use. It also allows specifying the application semantics using the class, rel, and name properties. This makes it suitable for level 4 APIs.

Siren specifies the type for fields, so that clients can render appropriate UIs.

application/vnd.uber+json

{ "uber": {
  "version": "1.0",
  "data": [{
    "rel": [ "self" ],
    "url": "http://restbucks.com/order/1234"
  }, {
    "id": "order",
    "data": [{
      "name": "item",
      "data": [{
        "name": "milk",
        "value": "semi"
      }, {
        "name": "size", 
        "value": "large",
      }, {
        "name": "drink",
        "value": "cappuccino"
      }]
    }, {
      "name": "location",
      "value": "takeAway"
    }, {
      "name": "cost",
      "value": 2.0
    }, {
      "name": "status",
      "value": "unpaid"
    }]
  }, {
    "rel": "http://relations.restbucks.com/cancel",
    "url": "http://restbucks.com/payment/1234",
    "action": "remove"
  }, {
    "rel": "http://relations.restbucks.com/payment",
    "url": "http://restbucks.com/payment/1234",
    "sending": "application/vnd.uber+json",
    "action": "replace",
    "data": [{
      "name": "amount",
      "value": 2.0
    }, {
      "name": "cardholderName",
    }, {
      "name": "cardNumber",
    }, {
      "name": "expiryMonth",
    }, {
      "name": "expiryYear",
    }],
  }, {
    "rel": "http://relations.restbucks.com/update",
    "url": "http://restbucks.com/order/1234",
    "action": "replace",
    "sending": "application/vnd.uber.json"
  }]
}

UBER is a full hypermedia format with a lean message structure. It combines in the data property what other media types separate out in e.g. links, actions, and properties, which makes it a bit hard to read for humans.

UBER allows specifying application semantics using the name and rel properties, making it a level 4 capable media type.

There is also an XML variant of UBER.


RESTBucks Evolved

2014-06-16

restbucksThe book REST in Practice: Hypermedia and Systems Architecture uses an imaginary StarBucks-like company as its running example.

I think this is a great example, since most people are familiar with the domain.

The design is also simple enough to follow, yet complex enough to be interesting.

Problem Domain

RESTbucks is about ordering and paying for coffee (or tea) and food. Here is the state diagram for the client:
restbucks-states

  1. Create the order
  2. Update the order
  3. Cancel the order
  4. Pay for the order
  5. Wait for the order to be prepared
  6. Take the order

Book Design

The hypermedia design in the book for the service is as follows:

  1. The client POSTs an order to the well-known RESTBucks URI. This returns the order URI in the Location header. The client then GETs the order URI
  2. The client POSTs an updated order to the order URI
  3. The client DELETEs the order URI
  4. The client PUTs a payment to the URI found by looking up a link with relation http://relations.restbucks.com/payment
  5. The client GETs the order URI until the state changes
  6. The client DELETEs the URI found by looking up a link with relation http://relations.restbucks.com/receipt

The book uses the specialized media type application/vnd.restbucks.order+xml for all messages exchanged.

Design Problems

Here are some of the problems that I have with the above approach:

  1. I think the well-known URI for the service (what Mike Amundsen calls the billboard URI) should respond to a GET, so that clients can safely explore it.
    This adds an extra message, but it also makes it possible to expand the service with additional functionality. For instance, when menus are added in a later chapter of the book, a second well-known URI is introduced. With a proper home document-like resource in front of the order service, this could have been limited to a new link relation.
  2. I’d rather use PUT for updating an order, since that is an idempotent method. The book states that the representation returned by GET contains links and argues that this implies that (1) PUT messages should also contain those links and (2) that that would be strange since those links are under control of the server.
    I disagree with both statements. A server doesn’t necessarily have to make the formats for GET and PUT exactly the same. Even if it did, some parts, like the links, could be optional. Furthermore, there is no reason the server couldn’t accept and ignore the links.
  3. The DELETE is fine.
    An alternative is to use PUT with a status of canceled, since we already have a status property anyway. That opens up some more possibilities, like re-instating a canceled order, but also introduces issues like garbage collection.
  4. I don’t think PUT is the correct method. Can the service really guarantee under all circumstances that my credit card won’t get charged twice if I repeat the payment?
    More importantly, this design assumes that payments are always for the whole order. That may seem logical at first, but once the book introduces vouchers that logic evaporates. If I have a voucher for a free coffee, then I still have to pay for anything to eat or for a second coffee.
    I’d create a collection of payments that the client should POST to. I’d also use the standard payment link relation defined in RFC 5988.
  5. This is fine.
  6. This makes no sense to me: taking the order is not the same as deleting the receipt. I need the receipt when I’m on a business trip, so I can get reimbursed!
    I’d rather PUT a new order with status taken.

Service Evolution

Suppose you’ve implemented your own service based on the design in the book.

evolutionFurther suppose that after reading the above, you want to change your service.

How can you do that without breaking any clients that may be out there?

After all, isn’t that what proponents tout as one of the advantages of a RESTful approach?

Well, yes and no. The media type defined in the book is at level 3a, and so will allow you to change URIs. However, the use of HTTP methods is defined out-of-band and you can’t easily change that.

Now imagine that the book would have used a full hypermedia type (level 3b) instead. In that case, the HTTP method used would be part of the message. The client would have to discover it, and the server could easily change it without fear of breaking clients.

Of course, this comes at the cost of having to build more logic into the client. That’s why I think it’s a good idea to use an existing full hypermedia type like Mason, Siren, or UBER. Such generic media types are much more likely to come with libraries that will handle this sort of work for the client.


REST Maturity

2014-06-02

rest-maturity2In 2008, Leonard Richardson published his Maturity Heuristic that classified web services into three levels based on their use of URI, HTTP, and hypermedia.

Back then, most web services were stuck at either level 1 or 2. Unfortunately, not a whole lot has improved since then in that respect: so-called pragmatic REST is still the norm.

BTW, I really dislike the term “pragmatic REST”. It’s a cheap rhetoric trick to put opponents (“dogmatists”) on the defensive.

More importantly, it creates semantic diffusion: pragmatic REST is not actually REST according to the definition, so please don’t call it that way or else we’re going to have a hard time understanding each other. The term REST hardly means anything anymore these days.

Anyway, there is some light at the end of the tunnel: more services are now at level 3, where they serve hypermedia. A good example by a big name is Amazon’s AppStream API.

The difference between plain media types, like image/jpeg, and hypermedia types, like text/html, is of course the “hyper” part. Links allow a client to discover functionality without being coupled to the server’s URI structure.

JSONBTW, application/json is not a hypermedia type, since JSON doesn’t define links.

We can, of course, use a convention on top of JSON, for instance that there should be a links property with a certain structure to describes the links, like Spring HATEOAS does.

The problem with conventions is that they are out-of-band communication, and a client has no way of knowing for sure whether that convention is followed when it sees a Content-Type of application/json. It’s therefore much better to use a media type that turns the convention into a rule, like HAL does.

Speaking of out-of-band communication, the amount of it steadily decreases as we move up the levels. This is a very good thing, as it reduces the amount of coupling between clients and servers.

Level 3 isn’t really the end station, however. Even with a hypermedia format like HAL there is still a lot of out-of-band communication.

HALHAL doesn’t tell you which HTTP method to use on a particular link, for instance.

The client can only know because a human has programmed it with that knowledge, based on some human-readable description that was published somewhere.

Imagine that the human Web would work this way. We wouldn’t be able to use the same browser to shop at Amazon and read up at Wikipedia and do all those other things we take for granted. Instead, we would need an Amazon Browser, a Wikipedia Browser, etc. This is what we do with APIs today!

Moving further into the direction of less out-of-band communication requires more than just links. Links only specify the URI part and we also need the HTTP and media type parts inside our representations. We might call this level 3b, Full Hypermedia.

Siren gives you this. Uber even goes a step further and also abstracts the protocol, so that you can use it with, say, CoAP rather than HTTP.

These newer hypermedia types allow for the use of a generic client that can handle any REST API that serves that hypermedia type, just like a web browser can be used against anything that serves HTML. An example of such an effort is the HAL browser (even though HAL is stuck at level 3a).

However, even with the inclusion of protocol, media type, and method in the representation, we still need some out-of-band communication.

The HAL browser can navigate any API that serves HAL, but it doesn’t understand the responses it gets. Therefore it can’t navigate links on its own to reach a certain goal. For true machine-to-machine (M2M) communication, we still need more.

ALPSIf we ever get the whole semantic web sorted out, this might one day be the final answer, but I’m not holding my breath.

In the meantime we’ll have to settle for partial answers.

One piece of the puzzle could be to define application semantics using profiles, for instance in the ALPS format. We might call this level 4, Semantic Profile.

We’d still need a human to read out-of-band communication and build a special-purpose client for M2M scenarios. But this client could handle all services in the application domain it is programmed to understand, not just one.

Also, the human could be helped a lot by a generic API browser that fetches ALPS profiles to explain the API.

All this is currently far from a reality. But we can all work towards this vision by choosing generic, full-featured hypermedia types like Siren or Uber for our APIs and by documenting our application semantics using profiles in ALPS.

If you need more convincing then please read RESTful Web APIs, which Leonard Richardson co-wrote with Uber and ALPS creator Mike Amundsen. This is easily the best book on REST on the market today.


Managers and Individual Contributors in Code

2014-04-21

manager-individualContributorI’m lucky to work for a company that makes it possible to have a good technical career, so that I don’t have to become a manager just to get more money.

I like being an individual contributor because it gives me a sense of accomplishment when I get stuff done myself.

Anyway, I believe the distinction between manager and individual contributor is useful when writing code as well.

Let me show that using the example of Conway’s Game of Life.

Baby Steps

At code retreats, you’ll find people who start working on the problem from the bottom (cell) and those who start at the top (universe, world, grid, game, or whatever they call it).

Interestingly, both groups tend to run into a similar problem.

baby-stepsStarting at the bottom makes it easy to quickly implement the rules in a cell. But then the problem becomes how to connect the cells into a universe.

This often looks like too big of a step to properly test, e.g. with a new green test every two minutes.

Starting at the top makes sure we are not doing anything unnecessary or building any parts that would not connect properly. But here too it’s difficult to keep chipping away in baby steps.

The Single Responsibility Principle

To evolve a whole universe, at least these things need to happen:

  1. Iterate over all cells
  2. Determine each cell’s neighbors
  3. Count the number of alive neighbors
  4. Determine the cell’s next state based on its current state and the number of alive neighbors

Depending on some implementation choices (like keeping the universe immutable), there may be more work.

Yet the Single Responsibility Principle tells us that a class should have only one reason to change. You may count items #2 and #3 as one responsibility (or #3 and #4), but clearly there is more than one involved.

Manager Classes and Individual Contributor Classes and How to Test Them

Now remember the manager/individual contributor (IC) distinction. Managers manage ICs; that’s their whole job. We should count managing other classes as a responsibility too.

That would mean that a class either implements some logic (IC), or coordinates with other classes (manager), but not both.

mockI think this also ties into the state-based versus interaction-based testing debate: we should use state-based testing for IC classes and interaction-based testing (using mocks) for manager classes.

That would explain why both code retreat groups seem to get stuck: they have to switch from state-based to interaction-based testing and it seems that many developers don’t have a lot of experience with the latter.

Test-Driving Manager Classes

So let’s try interaction-based testing of a manager class. I’ll keep with the example of the Game of Life.

I’d like to begin with an architectural choice: I want the universe to be immutable, so that it would be easy to introduce multi-threading later. As the game evolves, I’ll have to create a new universe for every generation:

public class GameTest {
  @Test
  public void clonesImmutableUniverseWhenEvolving() {
    Universe universe = mock(Universe.class, "old");
    Universe nextGeneration = mock(Universe.class, "new");
    when(universe.clone()).thenReturn(nextGeneration);

    Game game = new Game(universe);
    assertSame("Initial", universe, game.universe());

    game.evolve();
    assertSame("Next generation", nextGeneration, 
      game.universe());
  }
}

I’m using Mockito to define how the Game interacts with the Universe. Mocking works best with interfaces, so Universe is an interface.

That means the game doesn’t know the type of universe and therefore can’t create a new instance of it. The universe provides a factory method clone() for this purpose.

If the universe is to be immutable, then its state should be provided in its constructor. This constructor is called by clone(), so that method needs the new state as its input.

What does that state look like? At this point we don’t know anything about the universe other than that it contains cells. Let’s keep it simple and provide the cells as input:

public class GameTest {
  @SuppressWarnings("unchecked")
  @Test
  public void clonesImmutableUniverseWhenEvolving() {
    Universe universe = mock(Universe.class, "old");
    Universe nextGeneration = mock(Universe.class, "new");
    when(universe.clone(any(Iterable.class)))
        .thenReturn(nextGeneration);

    Game game = new Game(universe);
    assertSame("Initial", universe, game.universe());

    game.evolve();
    assertSame("Next generation", nextGeneration, 
        game.universe());
  }
}

OK, so who is going to supply those cells to clone()? The game is a manager, so it should only manage, not provide logic. The universe is an IC that is responsible for the cells and their interconnections, so it already has a responsibility.

So it seems like we need a new type. The new type should be managed by the game, since that’s the only manager class we have. Thus we should add a new test in GameTest for this interaction.

The new type should be responsible for determining the new state of the universe from the old one. This is where the game’s rules come into play, so let’s call the new type Rules.

Let’s start with testing that the game manages the rules:

public class GameTest {
  @Test
  public void consultsRulesWhenEvolving() {
    Rules rules = mock(Rules.class);

    Game game = new Game(null, rules);
    assertSame("Rules", rules, game.rules());
  }
}

Now we want to test that evolve() consults the rules. Since the game is a manager, it can only request output from a collaborator and provide that as input to another collaborator. It shouldn’t contain any logic itself, since that would make it an IC.

The only other collaborator that the game currently has is the universe and that seems the correct candidate to provide the cells. But we need more than the cells to implement the rules: we also need to number of alive neighbors of each cell.

The universe is clearly the correct place to determine a cell’s neighbors. Should it also count the number of alive ones?

I guess it could, but I don’t particularly like that: the rules for Life use the number of alive neighbors, but it’s not hard to imagine a different set of rules that also depend on the number of dead neighbors. So I feel that the rules should do the counting.

This means that the input to the rules is a cell and its neighbors. Since Java doesn’t allow returning two pieces of information, we need to combine them. Let’s call the combination a neighborhood:

public class NeighborhoodTest {
  @Test
  public void holdsACellAndItsNeighbors() {
    Cell cell = mock(Cell.class, "cell");
    List<Cell> neighbors = Arrays.asList(
        mock(Cell.class, "neighbor1"), 
        mock(Cell.class, "neighbor2"));

    Neighborhood neighborhood = new Neighborhood(cell, 
        neighbors);

    assertSame("Cell", cell, neighborhood.cell());
    assertEquals("Neighbors", neighbors, 
        neighborhood.neighbors());
  }
}

Now we can make the universe return a neighborhood for each of the cells it contains and verify that those neighborhoods are used as input for the rules:

public class GameTest {
  @SuppressWarnings("unchecked")
  @Test
  public void clonesImmutableUniverseWhenEvolving() {
    Universe universe = mock(Universe.class, "old");
    Universe nextGeneration = mock(Universe.class, "new");
    when(universe.clone(any(Iterable.class)))
        .thenReturn(nextGeneration);
    when(universe.neighborhoods()).thenReturn(
        new ArrayList<Neighborhood>());

    Game game = new Game(universe, mock(Rules.class));
    assertSame("Initial", universe, game.universe());

    game.evolve();
    assertSame("Next generation", nextGeneration, 
        game.universe());
  }

  @Test
  public void consultsRulesWhenEvolving() {
    Universe universe = mock(Universe.class);
    Neighborhood neighborhood1 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    Neighborhood neighborhood2 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    when(universe.neighborhoods()).thenReturn(
        Arrays.asList(neighborhood1, neighborhood2));
    Rules rules = mock(Rules.class);

    Game game = new Game(universe, rules);
    assertSame("Rules", rules, game.rules());

    game.evolve();
    verify(rules).nextGeneration(neighborhood1);
    verify(rules).nextGeneration(neighborhood2);
  }
}

The next step is to make sure that the output from the rules is used to construct the new universe:

public class GameTest {
  @Test
  public void consultsRulesWhenEvolving() {
    Universe universe = mock(Universe.class);
    Neighborhood neighborhood1 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    Neighborhood neighborhood2 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    when(universe.neighborhoods()).thenReturn(
        Arrays.asList(neighborhood1, neighborhood2));
    Rules rules = mock(Rules.class);
    Cell cell1 = mock(Cell.class, "cell1");
    Cell cell2 = mock(Cell.class, "cell2");
    when(rules.nextGeneration(neighborhood1))
        .thenReturn(cell1);
    when(rules.nextGeneration(neighborhood2))
        .thenReturn(cell2);

    Game game = new Game(universe, rules);
    assertSame("Rules", rules, game.rules());

    game.evolve();
    verify(universe).clone(eq(
        Arrays.asList(cell1, cell2)));
  }
}

At this point we’re done with the Game class. All that is left to do, is create implementations for the three interfaces we introduced: Universe, Cell, and Rules. Each of these is an IC class, and thus fairly straightforward to test-drive using state-based testing.

Conclusion

I find that the distinction between manager and individual contributor classes helps me in deciding what the next test should be.

What do you think? Could this be part of the missing piece for a proper theory of Test-Driven Development?


Conway’s Game of Life and the Flyweight Pattern

2014-04-14

flyweightConway’s Game of Life is fascinating, both from a functional and from a technical perspective.

This may explain why it’s often used for code retreats. Code retreats are a fun way to learn.

It’s amazing how working with new pairs gives you new insights virtually every time.

At the last code retreat that I attended, one of my pairs suggested we use the Flyweight pattern for cells:

A flyweight is a shared object that can be used in multiple contexts simultaneously. The flyweight acts as an independent object in each context — it’s indistinguishable from an instance of the objects that is not shared.

When the Design Patterns book (which contains the above quote) came out, I remember having many aha moments. It was so cool to see all these patterns that I had used before and finally have a name for them so I could discuss them with my peers more efficiently!

I did not have an aha moment when reading about flyweight, however. The example in the book, sharing character objects in a text editor, seemed a bit far fetched at the time. This example is not unlike the cells in a Game of Life grid, however, so I happily went along with my pair’s idea to explore the pattern’s applicability in this context.

After the code retreat was over, I gave this pattern some more thought. (This is usually where the code retreat really starts paying off.)

We actually use a potential flyweight all the time: booleans. A boolean is a class with only two instances, and those instances could easily be shared. In Java they are not: new Boolean(true) != new Boolean(true). However, the Boolean class does provide two constants, TRUE and FALSE, for the instances that you could use for sharing.

That got me thinking about using Enums for flyweights. Most of the time, I use enums for grouping related but mutually exclusive constants, like days of the week. However, Enums in Java can define methods:

  public enum Cell {

    ALIVE(true), DEAD(false);

    private final boolean alive;

    private Cell(boolean alive) {
      this.alive = alive;
    }

    public boolean isAlive() {
      return alive;
    }

    public Cell evolve(int numLiveNeighbors) {
      boolean aliveInNextGeneration = alive
          ? 2 <= numLiveNeighbors && numLiveNeighbors <= 3
          : numLiveNeighbors == 3;
      return aliveInNextGeneration ? ALIVE : DEAD;
    }

  }

One of the fun parts of code retreats is that in some sessions, you will have constraints on the way you work. Such constraints force you to be more creative and think beyond the techniques you would normally use.

One constraint that is interesting in this context is to not use any conditionals, like if or switch statements or ternary operators. The idea behind this constraint is to force you to replace conditionals with polymorphism, making your program more object oriented.

The only way that I see to keep the current Cell enum and not use conditionals, is to introduce a map:

  public enum Cell {

    ALIVE(true), DEAD(false);

    private final boolean alive;
    private static final Map<Boolean, Map<Integer, Cell>> 
        NEXT = new HashMap<>();

    static {
      Map<Integer, Cell> dead = new HashMap<>();
      dead.put(0, DEAD);
      dead.put(1, DEAD);
      dead.put(2, DEAD);
      dead.put(3, ALIVE);
      dead.put(4, DEAD);
      dead.put(5, DEAD);
      dead.put(6, DEAD);
      dead.put(7, DEAD);
      dead.put(8, DEAD);
      dead.put(9, DEAD);
      NEXT.put(false, dead);
      Map<Integer, Cell> alive = new HashMap<>();
      alive.put(0, DEAD);
      alive.put(1, DEAD);
      alive.put(2, ALIVE);
      alive.put(3, ALIVE);
      alive.put(4, DEAD);
      alive.put(5, DEAD);
      alive.put(6, DEAD);
      alive.put(7, DEAD);
      alive.put(8, DEAD);
      alive.put(9, DEAD);
      NEXT.put(true, alive);
    }

    private Cell(boolean alive) {
      this.alive = alive;
    }

    public boolean isAlive() {
      return alive;
    }

    public Cell evolve(int numLiveNeighbors) {
      return NEXT.get(alive).get(numLiveNeighbors);
    }

  }

This approach works, but is not very elegant and it breaks down when the number of possibilities grows. Clearly, we need a better alternative.

The only way we can get rid of the conditional, is by getting rid of the boolean state of the cell. That means we need to have different classes for the two instances, so that the type implicitly embodies the state. That in turn means we need a factory to hide those classes from the client:

  public interface Cell {

    boolean isAlive();
    Cell evolve(int numLiveNeighbors);

  }

  public class CellFactory {

    private static final Map<Boolean, Cell> CELLS 
        = new HashMap<>();

    static {
      CELLS.put(false, new DeadCell());
      CELLS.put(true, new AliveCell());
    }

    public static Cell dead() {
      return cell(false);
    }

    public static Cell alive() {
      return cell(true);
    }

    static Cell cell(boolean alive) {
      return CELLS.get(alive);
    }

  }

  class DeadCell implements Cell {

    @Override
    public boolean isAlive() {
      return false;
    }

    @Override
    public Cell evolve(int numLiveNeighbors) {
      return CellFactory.cell(numLiveNeighbors == 3);
    }

  }

  class AliveCell implements Cell {

    @Override
    public boolean isAlive() {
      return true;
    }

    @Override
    public Cell evolve(int numLiveNeighbors) {
      return CellFactory.cell(numLiveNeighbors == 2 
          || numLiveNeighbors == 3);
    }

  }

Indeed, when you look up the Flyweight pattern, you’ll see that the proposed structure contains a flyweight factory that creates instances of concrete flyweight classes that implement a common flyweight interface.

Thanks to the code retreat and my partner, I now know why.


Too Many Open Files, Or Too Few Bounded Contexts?

2014-02-18

A Business Men Climbing a Pile of PapersServer software needs to run unsupervised for long periods of time to be practical.

Release It! is full of horror stories about programming mistakes that get in the way of that lofty goal.

One example is opening files and forgetting to close them.

On some operating systems this will eventually lead to a Too many open files error when the number of open files passes a certain limit.

Of course we want to make sure that doesn’t happen to us. IDEs like Eclipse can give you warnings in certain cases to help with that, but if you’re test infected like me, you will want to write a test to make sure.

This is one of those situations where people less committed to (unit) testing tend to give up. There is nothing in the Java file system API that tells you how many files are open, so it simply cannot be done, right?

Not so fast, my friend!

First, with some proprietary code you can, in fact, count the number of open files in Java.

Second, we can do even better if we step back for a moment and take a look at the bigger picture. Chances are that our application isn’t really about files; that using files is simply a convenient implementation choice.

In the lingo of Eric Evans’ classic Domain-Driven Design, we have (at least) two bounded contexts: your core domain (what people buy/use your application for) and the file system.

A bounded context delimits the applicability of a particular model so that team members have a clear and shared understanding of what has to be consistent and how it relates to other contexts.

Evans describes a number of strategies for dealing with different bounded contexts. In our example, the file system API is not under our control, which rules out the majority of them. The remaining strategies are:

  • Separate Ways, i.e. use something other than the file system. That probably doesn’t make sense in this example
  • Conformist, i.e. follow the Java model slavishly. This is what most developers do without giving it much thought
  • Anti-Corruption Layer, i.e. create an isolating layer to provide clients with functionality in terms of their own domain model. This layer translates in both directions as necessary between the two models

This last strategy gives us more than “just” the opportunity to keep our models clear and to the point. By introducing interfaces that we control in our anti-corruption layer, we also gain the opportunity to mock those interfaces in our tests, making it very easy to verify that we indeed close all the files we open.

This is yet another example where difficulty in unit testing a piece of code points to an opportunity to improve the design. If we consistently act on such opportunities, we will end up with a clean architecture that is a joy to work with.


Follow

Get every new post delivered to your Inbox.

Join 289 other followers