Monday, March 16, 2009

Test drive of Spring 3.0 Milestone 2 REST support

I'm developing a proof-of-concept application based on REST architecture, with pure-JavaScript client written in Ext-JS, and server side based on Spring. Until now, I've used Spring 2.5 to create REST service, but it was a bit problematic: it required me to write too many repetitive code. I've read about JAX-RS and it sound very good to me. There are some implementations already available, but I didn't want to learn and add yet another product to the already challenging technology stack. As it was known at that time that Spring 3.0 is going to include REST support, I decided to wait a bit for it, expecting full JAX-RS support in Spring. Now, after reading Arjen's blog entry, I see JAX-RS will not be implemented: that's a pity, because JAX-RS seems very well designed for me; but ok, if Spring's solution delievers the same value, this doesn't matter much. So today, after reading Arjen's introduction, I switched to 3.0 M2 to see how it fits my needs, and I'd like to share my results.


The most wanted thing, the URI templates are of great help obviously. But this is only the minimum of what I expected. After playing a bit with spring 3.0, I must say I'm a bit disappointed. I simply expected much much more. It seems to me that Spring REST is targeted at server-side web applications, the ones I call "half-REST" applications. This is not what I expected: because Spring REST support is done by Arjen Poutsma, the author of Spring Web Services, I thought REST support was going to be comparable with Spring Web Services. I really like the Spring Web Services approach, focused on real services with data, not some HTML-oriented views. Unfortunately, Spring REST doesn't seem to be like this. It doesn't help much writing real web service which return data, instead of HTML-oriented views. For example, shallow ETag support and HTTP-over-POST method conversion are nice things for "half-REST" application, but they are of little value for fully RESTful applications. In my REST service I plan to rather go for deep ETag, based on database versions, cache states etc, and I use real HTTP methods: PUT, DELETE, instead of PUT-over-POST.


Requirements and assumptions


My application consists of two parts: server side, written in Java, based on Spring, and client-side, written in Ext-JS. Server side exposes the web service based on REST principles, following the guidelines presented in the great RESTful Web Services book. Service returns data in JSON format (I decided JSON better fits my needs than XML, as the client is written in JavaScript, so JSON processing will be faster, and bandwidth usage will be smaller; service can be used also by any other tool, as JSON is widely supported in virtually any language). It accepts POST/PUT request body data mostly as form-encoded pairs, but in some more complicated cases it can accept JSON data too.


Now say we have a resource called "Project". It is addressable by http://mydomain/restservice/v1/projects. GET to this address returns list of projects in JSON format. GET to http://mydomain/restservice/v1/projects/MyProject returns details of MyProject. DELETE to this URL removes this project. PUT to this URL creates or updates MyProject, using form-encoded data from request body (during creation, client knows the URL of project, that is why we use PUT; if we have used database IDs in URL, than creation would be done by POST to http://mydomain/restservice/v1/projects).


This URL may be a base for other related resources, e.g. http://mydomain/restservice/v1/projects/MyProject/tasks will return tasks for MyProject, http://mydomain/restservice/v1/projects/MyProject/tasks/Development will return details about Development task in MyProject, http://mydomain/restservice/v1/projects/MyProject/assignments will return assignment list for MyProject, ... etc.


Note, that those URLs actually represent virtual directories. This means that common rule "trailing slash doesn't matter" (especially for URL representing "list of resources") should be respected: http://mydomain/restservice/v1/projects address is equivalent to http://mydomain/restservice/v1/projects/ address. If you consider that each project is also the container for its tasks or assignments, then also http://mydomain/restservice/v1/projects/MyProject should be equivalent to http://mydomain/restservice/v1/projects/MyProject/, etc.


Each request to the service must return proper response code: 200,201,301 (if data in PUT request changes the resource address), 304 (in conditional GETs), 400, etc. In some cases it must contain proper body (here in JSON format) and proper headers (e.g. Location header in case of 301 response). My services misses only one RESTful requirement: connectivity of representations. In my opinion application of this principle to javascript client, that must be already programmed against known service, doesn't add any value, it only increases bandwidth usage (ok, maybe some very intelligent client could take advantage of it; but for a while, it would be too complicated for my case). JSON representations doesn't fit well to handle links: if I have served linked representations, I would probably have to use XML instead. I also don't use content negotiation: all representations are JSON. Again, this would be overkill for my service, which is primary targeted at predefined JavaScript client. If it was more general, public service, this should be probably refactored.


Implementation with Spring 2.5


Let's implement the Project service with aid of Spring. Annotation-based Spring MVC introduced in version 2.5 seemed to match quite well my requirements, and that is why I decided to use it. So let's start with Spring 2.5, and analyse all the problems I faced; then we'll see which of them are solved by Spring 3.0.M2.


I have resource "Project" with some actions to be performed on it: so logical choice is to put all operations related to Projects into one controller. Base URL of this controller is http://mydomain/restservice/v1/projects or equivalent with trailing slash http://mydomain/restservice/v1/projects/. Some actions (GET project list, POST - if we have used it) are done directly on this URL. Some others (GET project, PUT, DELETE) are done on http://mydomain/restservice/v1/projects/{projectName} URL (or, potentially, equivalent one with trailing slash http://mydomain/restservice/v1/projects/{projectName}/). There can be also other controllers in my service, either completely unrelated to Project resource, like User controller (say http://mydomain/restservice/v1/users) or related to Projects, like tasks controller (http://mydomain/restservice/v1/projects/{projectName}/tasks) or assignments controller (http://mydomain/restservice/v1/projects/{projectName}/assignments). I won't cover those controllers, but we should keep in mind that framework should be able to match the most specific controller properly, that is for request for http://mydomain/restservice/v1/projects/{projectName}/assignments the assignments controller should be picked up, not the projects controller.


So, I would like to write the ProjectController, mapped to http://mydomain/restservice/v1/projects (with or without trailing slash - let's call it "base URL" for now), with 5 methods (method names doesn't matter here): list() mapped to GET with base URL, show() mapped to GET with base+projectName URL, createOrUpdate() mapped to PUT with base+projectName URL, and remove() mapped to DELETE with base+projectName URL. Something like this (assume that URL part http://mydomain/restservice/v1 is already mapped to dispatcher servlet) :



@Controller
@RequestMapping("/projects")
public class ProjectEndpoint {

@RequestMapping(method=GET)
public HttpResponse list() {...}

@RequestMapping(method=GET, value="*")
public HttpResponse show() {...}

@RequestMapping(method=PUT, value="*")
public HttpResponse createOrUpdate() {...}

@RequestMapping(method=DELETE, value="*")
public HttpResponse remove() {...}
}

I would like to take advantage of Spring's @RequestMapping annotation ability to merge class-level path value (/projects) with method-level relative paths. But it doesn't work. To make it work, it seems that class-level path must end with wildcard, and all methods must have some URL suffix. But you can't have methods like list() and show() in this case, one mapped to base URL, and onother to base URL + suffix. This is annoying shortcoming of Spring MVC. There are also many other problems here: The star character (*) matches zero or more characters. That's wrong, because method show() will be matched even for request when no project name was passed. There should be something like "+" character, meaning "one or more characters". Also trailing slashes are real problem here. If I mark controller class with @RequestMapping("/projects") (as in the code above) then only request without trailing slash will be matched. If trailing slash is appended to the actual request, we get 404 response. What's worse, method-level annotation will not be appended to base URL in this case. So we have to change it to @RequestMapping("/projects/*"). Now methods will work, but if you send request to base url to get list of projects, and forget the slash at the end, you will get 404 response again. That's very bad in my opinion. JAX-RS handles it properly: see JAX-RS's @Path Javadoc. Finally, the only solution that works is to omit the "base URL" on class level, and repeat it at each method level (without trailing slashes). This works, but requires mapping repetition, and you can't use trailing slashes in actual requests, unfortunately.


Second problem: in method show() we actually need to extract the project name from URL. Spring 2.5 doesn't offer any help for this, we have to manually parse the URL.


For PUT and POST method I need to get access to request body. For form-encoded data it is not that complicated: I may use @RequestParam annotated params, I may use @InitBinder to initialize command object etc, I may also access directly Request object and call getParameter(). However I faced third problem here: getParameter() works only for POST method, and not for PUT, even if request body is the same. I had not dig into this issue too deeply, but it looks like this has a root in servlet API specification flaw (or Tomcat implementation). However, it would be nice if REST support framework handled it properly, for example by wrapping the request object. In my case, I had to write myself the method to extract request parameters from POST and PUT requests in uniform way:



public static Map<String, Object> decodeFormEncodedParamsFromBody(HttpServletRequest req) {
try {
Map results = new HashMap<String,Object>();
String inputString = IOUtils.toString(req.getReader());
// Parse a x-www-form-urlencoded string
String[] pairs = inputString.split("\\&");
String encoding = req.getCharacterEncoding() != null ? req.getCharacterEncoding() : "UTF-8";
for (String pair : pairs) {
String[] fields = pair.split("=");
String name = URLDecoder.decode(fields[0], encoding);
String value;
if (fields.length > 1) {
value = URLDecoder.decode(fields[1], encoding);
} else {
value = "";
}
if (results.containsKey(name)) {
List<String> values = (List<String>)results.get(name);
values.add(value);
} else {
results.put(name, value);
}
}
return results;
} catch (IOException e) {
throw new RuntimeException(e);
}
}

What if POST or PUT body is in JSON form? I would like to bind it automatically to some "object-oriented" representation: either use some JSON unmarshaller and transform it to Project bean; or bind it to JSONObject from JSON-Lib library, or use other JSON library. For example, in my case I would like to be able to write method like this:



@RequestMapping(method=PUT)
public HttpResponse createOrUpdate(JSONObject projectJson) {...}

But Spring's @InitBinder is not able to bind request body, only request params. After some research I found solution: AnnotationMethodHandlerAdapter has a method setCustomArgumentResolver() taking WebArgumentResolver implementation, which can be used for this purpose. One have to write the resolver and register it in spring context. This is how such resolver could look like:



public class JsonLibJSONObjectArgumentResolver implements WebArgumentResolver {

public Object resolveArgument(MethodParameter methodParameter, NativeWebRequest webRequest) throws IOException {
if (methodParameter.getParameterType().equals(JSONObject.class)) {
ServletRequest req = (ServletRequest)webRequest.getNativeRequest();
String inputString = IOUtils.toString(req.getReader());
JSONObject json = JSONObject.fromObject(inputString);
return json;
}
return UNRESOLVED;
}

}

Last problem relates to returning results from my methods. For GET methods I have to return JSON representations. For PUT, POST, DELETE there is actually no representation: only response code. Each response may (or sometimes must) contain also some headers. I could use JSONView for this, which was around for some time, but in fact I didn't like this approach that much. My general feeling is that Spring MVC is a result of the dominating trend in the past of abstracting away the HTTP nature of web applications: hide HttpRequest and HttpResponse, and deal with models, views, controllers. JSF is the best exmple of this approach. Spring MVC, though closer to HTTP request-response model, still tries to put abstraction layer on top of this, with its views and view resolvers. Perhaps it is good for server-side web applications and frameworks, that use forwards and redirects, templates, includes etc. But for REST service, that accepts and returns representations which are mostly data (like JSON), not "views", this abstraction may be unnecessary. In my case, in my controller methods I simply want to say "return this JSON object" or "return status this, with header this, and with JSON body that". I don't need views, view resolvers etc. The perfect example of this approach is Spring Web Services: you have an endpoint with some methods on it, each method takes some XML (or its part specified with XPath, or its unmarshalled representation by autoconversion) and returns some XML (or some object which is marshalled then). You have adapters for different XML libraries and different XOM libraries. That's great, and I expected something similar from Spring for building REST based web services.


I wasn't able to find any solution how to return simply JSONObject or something similar from controller method. See also this thread on Spring forum for details. Finally I came to the conclusion that I will use Spring views, and created HttpResponse view:



public abstract class HttpResponse implements View {

int statusCode;
Map<String, String> headers;

public HttpResponse(int statusCode) {
this.statusCode = statusCode;
}

public HttpResponse(int statusCode, Map<String, String> headers) {
this.statusCode = statusCode;
this.headers = Collections.unmodifiableMap(headers);
}

public abstract String getContentType();

public void render(Map model, HttpServletRequest request, HttpServletResponse response) throws Exception {
response.setStatus(statusCode);
if (headers != null) {
for (Map.Entry<String, String> header : headers.entrySet()) {
response.setHeader(header.getKey(), header.getValue());
}
}
}
}

I have two implementations: NoBodyResponse and JSONObjectResponse. Names clearly define their meaning. JSONObjectResponse takes JSONObject (from JSONLib) in constructor, and writes it to HTTP response body. Similar implementations could be created for other JSON or XML libraries, or for XOM or JOM (Json-object-marshaller) libraries. This is a kind of basic replacement for ResponseBuilder from JAX-RS.


Implementation with Spring 3.0 M2


Now let's look what we can simplify after migration to Spring 3.0 M2. The @PathVariable introduced in M1 is a great help, as I no longer have to parse the URLs manually. This reduces code duplication and is less error-prone. So now my example of hypothetical ProjectEndpoint could look like this:



@Controller
@RequestMapping("/projects")
public class ProjectEndpoint {

@RequestMapping(method=GET)
public HttpResponse list() {...}

@RequestMapping(method=GET, value="{projectName}")
public HttpResponse show(@PathVariable String projectName) {...}

@RequestMapping(method=PUT, value="{projectName}")
public HttpResponse createOrUpdate(@PathVariable String projectName) {...}

@RequestMapping(method=DELETE, value="{projectName}")
public HttpResponse remove(@PathVariable String projectName) {...}
}

BTW: I've found already a bug in @PathVariable processing: if you have method taking for example id, which is long, marked with @PathVariable, then the id from URL will be automatically converted to long. However if the actual URL contains characters nonconvertible to long, then exception is thrown, and you get 500 response. This is wrong, as this is clearly client error, so the response should be 400 (bad request), not 500. I haven't checked yet if it was already reported.


Unfortunely, the code above still doesn't work. Apart from @PathVariable, nothing has actually changed (at least from what I observed): you cannot map class to /projects nor /projects/, you have to map it to /projects/*. Now you cannot map list() method to do GET for base URL, and another show() method to do GET for sub-URL in the single controller - this will not work, for any reason. So still you have to throw away class-level mapping and copy base URL to each method, to make it work. And trailing slashes are still broken: you can't use it in request URL. So actually we haven't improved the code too much. Looks like @RequstMapping processing was broken in 2.5 and is so in 3.0.


Let's check other points: getParameter() for form-encoded PUT request body doesn't work in Spring 3.0 too, so all Spring autobinding feature and @RequestParam will fail for PUT requests. That's really a pity. Next: automatic binding of request body to some method param, like JSONObject, or JAXB binding object: not much progress here too, but fortunately we can use the setCustomArgumentResolver() that I demonstrated earlier. Next: HttpResponse View or something similar, to easily set response code or response header - nothing at all. Arjen mentions also JacksonJsonView, but it is not in Spring 3.0, but in Spring JavaScript - hello? What JavaScript? I'm building server-side web service here, so why on earth should I use Spring JavaScript module? This makes no sense. JSON is not JavaScript: it's data exchange format, which you can use for example to talk to .NET from Java, if you wish: no JavaScript is involved here.


And one more issue. I said earlier that I don't use content negotation, because I serve only JSON. But suppose that I have need to serve also XML now, and use content negotiation, based on Accept header from client. How to accomplish this? JAX-RS has @Produces annotation for this: simply by marking two methods or classes with this annotation with different value, we are sure that framework will pick up the valid method, according to user request. How to do it in Spring 3.0? From what I understood, I should use the new ContentNegotiatingViewResolver. Is it better solution then JAX-WS? I haven't try it yet, so I can't say much about it. But I can think about different use cases. Let's assume that only one method of my controller is supposed to be content-negotiation aware. For example, GET method may return JSON if Accept: application/json was passed, or the PDF document if application/pdf was passed (as it's less likely that PUT method is going to use PDF document as input here). In such a case JAX-RS solution is a winner: I would simply have two methods for GET, first marked with @Produces("application/json") - it could return my JSONObjectResponse view, and second method marked with @Produces("application/pdf"), returning some PDF view (or model). On the other hand, if my service produces/accepts JSONs and XMLs, it's higly likely that each method will be doubled: one producing/accepting XML, and one for JSON. Then perhaps it would be better to have two controllers instead: e.g. ProductEndpointXML and ProductEndpointJSON. In such a case probably @RequestMapping annotation should be enhanced to take also header values into account (currently it takes path, HTTP method, and request params, but no headers), to be able to route request to proper controller.


As you see, the only real asset of Spring 3.0 for REST in my application is @PathVariable. @PathVariable is really nice, but it is very very small improvement for the major realease of Spring, having said for long time that REST support is one of the key additions in this release. (To be honest I have to mention here small improvements from 3.0 M1: new @RequestHeader annotation and default value in @RequestParam - this is really helpful in RESTful applications too.) Especially if you compare Spring Web Services for building SOAP-oriented web services against Spring 3 REST support for building REST-oriented web services, the latter seems very very poor. If I remember correctly, Arjen said some time ago that he considered building REST support based on Spring Web Services, but decided to go with Spring MVC instead. Looking at Spring 3.0 M2 I'm not really sure if this was the best decision. Also comparison with JAX-RS shows some Spring shortcomings. Look for example on Bill Burke's JAX-RS intro atricle: it describes pretty much interesting stuff, many of which having no counterpart in Spring 3.0. On the other hand, it is still only M2 for Spring 3.0. I hope M3 will significantly improve support for real RESTful web services. If not, I will have to search for different solutions: some JAX-RS implementation with Spring integration, or perhaps Grails?

Summary


So what should be done for REST support in Spring 3.0, to be fully functional and in line with JAX-RS and Spring Web Services? Spring 3.0 should meet in my opinion following requirements:

  • One should be able to write single controller for single logical resource, with base URL mapping declared on class, and sub-URLs mapping declared on some methods (GET,PUT,DELETE); other methods without sub-URL mapping (GET,POST) refer to base URL. Note that controllers can create "nested" structure (e.g. one for project, other for project's tasks). Also trailing slashes in "virtual directory" URLs should be handled properly (trailing slashes should not influence the response).
    Some ideas here: first, all the method-level paths, if they are not absolute, should be treated as relative to class-level path. According to @RequestMapping Javadoc in 2.5, it should behave like this already in 2.5; but from my observations it did only if class-level path ended with wildcard; so in fact fixing it should not break backward compatibility, and should be treated rather as bug fix. @PathVariable fixes already problem of * character matching zero characters, that I mentioned earlier (I assume @PathVariable matches one or more characters only). Problem of trailing slash can be solved this way: first the request URL is matched literally. It there is a match, then continue normally. If there is no match, then look at request URL: if ends with slash, remove it, if it doesn't, add it - and then try the whole matching again.

  • @PathVariable with type conversion should give error 400, not 500, if conversion fails

  • Proper handling of Request.getParameter() for PUT (the same as for POST) - probably by wrapping PUT requests

  • Getting easy access to request body, with automatic conversion/marshalling (e.g. to JSONObject, some XML representations, JAXB binding result, ...), preferably as method param, perhaps with some annotation

  • HttpResponse View implementation out of the box, allowing setting response code and headers, with different implementations for "body-full" responses (JSONObject, XML representations, marshalling, ...)

  • Simplified content-negotiation: for example by extending @RequestMapping annotation to be able to match request headers too (at least Accept header)

  • Move JSON support from Spring JavaScript project to Spring core


This is not exhaustive list obviously. It is based on my observations for specific service I'm building. But I believe in the future there will be more and more such applications and "real" REST services, so it would be really good to have Spring support them fully, not shallowly.