Mocking an API should be somebody else’s problem

11 03 2008

In an interview about his book with Cédric Beust, Next Generation Java Testing: TestNT and Advanced Concepts, Hani Suleiman says:

“I’m fairly strongly against the use of mocks for Java EE constructs. These APIs are often complicated and come with whole swathes of tests to verify compliance. End users never see what a pain in the ass it actually is to certify a product as compatible to a given EE API. Mock implementations on the other hand are most certainly not certified. The danger of using them is that over time, people start implementing more and more of the API, its more code that can go wrong, and more code that’s written just for the sake of making your tests look good. Increasingly, it becomes more and more divorced from the real implementation.”

You might think that I would disagree with that, in view of my current work on an ObMimic library of test-doubles for the Servlet API (in the broad meaning of “test-double”).

But actually I strongly agree with it, and it’s one of the motivations behind ObMimic.

Yes, if you write your own stubs or mocks or use a general-purpose “mocking” tool, it can be extremely difficult to accurately simulate or predict the real behaviour of an API, and over time you’re likely to encounter more and more of the API. You can also find yourself needing more and more scaffolding and instrumentation to serve the needs of your tests. So it can be problematic and uneconomical to do this as part of an individual application’s testing. Even when it remains simple and doesn’t grow over time, it’s still additional thought and effort for each individual application, and very easy to get wrong. Whilst it’s all theoretically possible, and can seem very simple at the start, the long-term economics of it don’t look good.

But it looks rather different if the necessary facilities are all provided for you by a specialist library developed by someone else. Then it’s up to them to figure out all the quirks of the API and provide complete and accurate coverage, and all you have to do in individual applications is to use it.

This is much like the need for a “container” or other such API implementations in the first place. For example, given the Servlet API, it’s neither feasible nor economic for each application to implement its own servlet container, but it’s perfectly reasonable to have separate, dedicated projects that produce servlet containers that everybody else just uses.

My own opinion is that the same goes for test-doubles for APIs such as the Servlet API: it’s not worth everyone doing this half-heartedly themselves, but it is worth somebody doing it well and producing a comprehensive, high-quality library that everyone else can just use.

Of course, this only works if the resulting library is complete enough and of good enough quality for you to be able to rely on it. This points to the kind of criteria on which to judge such suites of test-doubles:

  • How complete is the API coverage?
  • How accurate is the API simulation?
  • How configurable are the test-doubles?
  • How examinable are the test-doubles?
  • How well documented is it?
  • How easy is it to use?
  • What extra features to support testing does it provide? (e.g. strict or configurable validation of API calls, tracking the API calls made, configurable for different versions of the API etc).
  • What dependencies does it have? (e.g. is it limited to only being used with specific tools or frameworks or only in certain scenarios, or is it more generally applicable).

Unfortunately, we don’t seem to have many API-specific libraries of test-doubles at the moment, and in my own limited experience those that we do have aren’t generally good enough.

That’s understandable, as it’s a huge amount of work to do this well for any substantial API that wasn’t written with testing in mind. Especially for APIs as complex, imperfect and subject to change as some of the older Java EE APIs.

Apart from my own ObMimic library for the Servlet API, I’m aware of some other attempts at doing this for the Servlet API, such as HttpUnit’s ServletUnit and the Spring framework’s org.springframework.mock.web package. However, in general these tend to be somewhat incomplete, inadequately documented, and lacking in configurability and test-instrumentation. Some are also outdated or defunct, limited to a particular web-app framework, or are a very minor and secondary component within a broader product that has a rather different purpose and priorities (and is thus unlikely to get much attention or maintenance).

In terms of other APIs, I’m aware of MockEJB for EJB, Mock Javamail for JavaMail, and a few such libraries for JNDI. There’s also a discussion of this issue in James Strachan’s blog article “Mocking out protocols and services is damn useful” (though some of the solutions mentioned are lightweight “real” implementations rather than test-doubles as such). But that seems to be about it.

Does anyone know any more? Or have any general views on the quality and suitability of any of these libraries?

As an ideal, I’d like to see the provision of a suitable library of any necessary test-doubles as a mandatory part of any Java API, in the same way that the JCP demands not just a specification but also a reference implementation and a compatibility kit.

That sounds like a big extra burden on API development. However, most new APIs ought to be designed so that test-doubles aren’t generally necessary in the first place, or can be relatively simple. For example, EJB 3.0 has far less need for this sort of thing than EJB 2.0, due to being more “POJO”-based. As another example, I believe the new JSR 310 Date and Time API is being designed so that the API itself will allow you to “stop” or control the time for testing purposes (for example, see slides 89-91 of Stephen Colebourne’s Javopolis 2007 presentation on JSR 310).

More generally, if this was always tackled as an intrinsic part of each API’s design then it ought to result in APIs that are more amenable to testing, and developing any test-doubles that are still necessary for such an API ought to be far simpler than trying to provide this retrospectively for an API that has ignored this issue. Having any necessary test-doubles should also be helpful in the development and testing of real implementations. In any case, this ought to be a more efficient division of labour than leaving everybody to hack their own way around the absence of such facilities.

As an absolute ideal, I’d want the resulting libraries of API-specific test-doubles to all take a similar form, with common features and facilities, terminology, naming conventions, usage patterns etc. But that’s probably getting into the realm of fantasy.



2 responses

12 03 2008

Having accompanying mock libraries (or mimics) for all API’s sure would be nice. It would mean that we wouldn’t have to manually write mock objects which is a real pain and is very time consuming. I hate writing mock objects and would very much welcome the idea. It is important to note though that mocked tests alone should also be accompanied by end to end integration tests that have at least the same API coverage. Testing against a mimic and the target implementation can yield different results if one or the other is not 100% API compliant. The challenge is compliance. An API can have many implementations.

12 03 2008


I agree with you about compliance being a challenge. Both mocks/mimics and real implementations are subject to compliance bugs. On the plus side, being able to test against mocks/mimics as well as a real implementation can at least help identify and understand or avoid any incompatibilities in the real implementation (as can testing against multiple real implementations – I’ve seen this first-hand in the past, with tests against Tomcat and Orion providing proof of bugs in early Websphere versions, despite IBM attempts to “deflect” the issue).

A related challenge is where the API explicitly permits differences in behaviour, or is simply ambiguous, incomplete or contradictory. This can be as simple as having optionally-supported features, or as blatant as not specifying whether a method argument can or can’t be null (and what actually happens if null is passed). At the other extreme it can be something really subtle and complex in the interaction of different API objects and calls.

My own view is that a mock/mimic should at least detect API calls that hit such situations, and as a minimum should default to reporting them as errors – the API call might or might not work as expected on one particular version of one particular implementation, but you can’t rely on it and should aim to detect and eliminate such calls. But ideally you should be able to configure how the mocks/mimics handle each such situation (e.g. whether the call is rejected as an error or not, which of the various possible behaviours is implemented, what particular type of exception is thrown etc).

In a sense, the mocks/mimics need to provide not just a compliant implementation of the API, but need to be configurable to simulate any compliant implementation of the API. For my own work on ObMimic for the Servlet API, dealing with this (including documenting and testing the various options) has been a very significant part of the overall effort.

As for the broader issue of what types of tests one needs overall, I’ve no particular axe to grind, and certainly don’t see mocks/mimics as the entire and only solution. I just think good mocks/mimics are a useful tool whatever one’s overall approach.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: