Jason R Briggs » REST

Units of Work

jrbriggs — Fri, 20 Mar 2009 22:59:26 +0000

In regard to cross-resource transactions, I’m a member of the camp that wonders whether distributed transactions are strictly necessary in the REST/HTTP world; indeed I wonder whether they represent a design failure in modeling the granularity of your resources.

That said, we don’t live in a perfect world — and even if I can’t envisage why a properly designed RESTful application might require access to a distributed transaction, I can certainly envisage the environment where such an application might evolve. I’ve worked in a few of them. Places where interdepartmental barriers are as solid as the Great Wall; bastions of archaic technology where “one might provide a pseudo-RESTful interface, but one certainly won’t be re-architecting one’s legacy system in the buzzword language of the day”.

But, I think there’s a certain amount of smoke and mirrors in the JBossTS article “Transactional support for JAX RS based applications“:

Certainly it is worth pointing out that if a system cannot be made reliable then it can be of only limited utility. That said it is a worthwhile exercise to show how a REST based system can be made reliable.

Lack of distributed transactions would hardly seem to make a REST based system “unreliable” and, as a consequence, of “only limited utility”. Imagine a hotel booking facility — perhaps a booking resource, which internally might be constructed from a number of components, all governed (again internally) by transaction demarcation. Does the fact that the booking resource is coarse-grained and does not require an external transaction make it less reliable than a number of fine-grained resources which do? On the contrary. The latter sounds more like a WS-* api than a RESTful architecture, nothing to do with reliability.

So… hopefully it’s obvious I think it’s a bad idea. But if I were to write such an API, I think the 8-year old spec mentioned in the article falls short of the mark. Here’s my first-cut attempt at an alternative (which I still think falls a bit short of the mark, but is possibly an improvement):

Resource: tc
Method	URL	Content	Description	Statuses
GET	/tc		Returns HTML containing a summary of all transactions (status), plus an href to the transaction detail	200 – ok
	/tc/{txid}		Returns HTML containing detail for the transaction with id {txid}, including the status and a list of hrefs to the each of the participants. For example: `Transaction ID 12345a Status ACTIVE Timeout 5000` `Participant 1 Order`	200 – ok 404 – if txid is not found 409 – if the transaction has been deleted
	/tc?status={status-type}		Return HTML containing a summary of transactions with a specific status, with href to the transaction detail For example: /tc?status=recovering or /tc?status=active	200 – ok
	/tc/{txid}/participants		Return a list of participants in the transaction (list of hrefs)	200 – ok 404 – if the transaction does not exist
	/tc/{txid}/partipants/{rec-coord-id}		Return HTML containing the detail of a participant. For example: `Participant #2 ID 2 Name Order URL Order`	200 – ok 404 – if the transaction or participant does not exist
POST	/tc	[timeout={timeout}]	Start a transaction (with default timeout) returning the url /tc/{txid} — which is deleted after the timeout or after completion (any HTTP method relating to {txid} thereafter returns 404). Use timeout={timeout} to override the default timeout period.	201 – created
DELETE	/tc/{txid}		Rollback and stop a transaction	204 – ok 404 – if the transaction does not exist
	/tc/{txid}?commit		Commit and stop a transaction	204 – ok 404 – if the transaction does not exist
POST	/tc/{txid}/participants	url={url}&[name={name}]	Enlist {url} in the transaction, returning a unique resource for that participant of the form /tc/{txid}/participants/{rec-coord-id}. If name exists, record against the participant detail	201 – created 404 – if the transaction does not exist
PUT	/tc/{txid}/participants/{rec-coord-id}	url={url}	Replace the participant url	200 – ok 404 – if the transaction or participant does not exist

The resource identified by a participant URL will have the following semantics:

Method	URL	Content	Description	Statuses
PUT	URL/tx/{rec-coord-id}	action=prepare	The participant prepares any work done in the context of the transaction. The Warning header will contain additional info about the state of the prepare (either readonly, or notok).	200 – ok 200 – ok (+ Warning: readonly) 200 – ok (+ Warning: notok) 404 – participant has rolled back
		action=commit	The participant commits any work done in the context of the transaction.	200 – ok 200 – ok (+ Warning: heuristic) 404 – participant has rolled back
		action=rollback	The participant commits any work done in the context of the transaction.	200 – ok 200 – ok (+ Warning: heuristic) 404 – participant has already rolled back

Basic usage might look something like the following:

1. Create a new transaction resource
`POST /tc Host: somedomain.com timeout=5000`	`HTTP/1.1 201 Created Connection: close Date: Thu, 19 Mar 2009 21:01:56 GMT Location: http://somedomain.com/tc/10a23v991 X-Powered-By: TransactionServer/0.1`
2. Create a new resource of some kind (notify the resource that it will operate with a distributed transaction)
`POST /res1?tx Host: internaldept1.somedomain.com some xml describing the resource`	`HTTP1.1 201 Created Connection: close Date: Thu, 19 Mar 2009 21:01:56 GMT Location: http://internaldept1.somedomain.com/res1/100`
3. Update another resource (again notify that it will be operating within a transaction)
`PUT /res2/somename?tx Host: internaldept2.somedomain.com some xml describing the update, perhaps including a reference to the previously created resource`	`HTTP1.1 200 Ok Connection: close Date: Thu, 19 Mar 2009 21:01:56 GMT Location: http://internaldept2.somedomain.com/res2/somename`
4. Enlist the url for each resource in the transaction
`POST /tc/10a23v991/participants Host: somedomain.com name=resource1&url=http://internaldept1.somedomain.com/res1/100`	`HTTP/1.1 201 Created Connection: close Date: Thu, 19 Mar 2009 21:01:56 GMT Location: http://somedomain.com/tc/10a23v991/participants/a01abgv21 X-Powered-By: TransactionServer/0.1`
`POST /tc/10a23v991/participants Host: somedomain.com name=resource2&url=http://internaldept2.somedomain.com/res2/somename`	`HTTP/1.1 201 Created Connection: close Date: Thu, 19 Mar 2009 21:01:56 GMT Location: http://somedomain.com/tc/10a23v991/participants/a01abgv22 X-Powered-By: TransactionServer/0.1`
5. Commit the transaction
`DELETE /tc/10a23v991?action=commit Host: somedomain.com`	`HTTP/1.1 204 Committed and deleted Connection: close Date: Thu, 19 Mar 2009 21:01:56 GMT X-Powered-By: TransactionServer/0.1`
6. ‘Behind the scenes’, the commit results in the following (2-phase commit at this point)…
`PUT /res1/100/tx/a01abgv21 Host: internaldept1.somedomain.com action=prepare` `PUT /res2/somename/tx/a01abgv22 Host: internaldept2.somedomain.com action=prepare` and `PUT /res1/100/tx/a01abgv21 Host: internaldept1.somedomain.com action=commit` `PUT /res2/somename/tx/a01abgv22 Host: internaldept2.somedomain.com action=commit`

when does REST become complex?

jrbriggs — Sat, 17 Feb 2007 08:32:59 +0000

According to this:

The great marketing coup that REST folks are trying to pull off is that WS-* is complex and REST is simple. That’s just nonsense – if you really try to build the type of systems that people build with WS-* using REST, then you’d end up just as complex.

Who’s talking nonsense? So, you take something that is inherently complex and build a system with it — then you take something that is fundamentally simple and build a similar system with that — and the final results are the same in terms of complexity?

Mais non. I don’t think so.

simple REST testing tools

jrbriggs — Sun, 06 Aug 2006 20:55:11 +0000

After a bit of googling, I have yet to find a command-line tool that simply handles HTTP requests. There’s wget for retrieving files, curl for PUT and POST, but what to use for DELETE (or other HTTP methods)? For that matter, I’d prefer to have a single tool, rather than 2 or 3.

I wrote a simple python script a while back — the last time I couldn’t find such a tool — and so now that I need to use it again, it seems like a good time to polish it up and release it (here). Not before chopping out a bunch of specific hacks that really need tidying up before being included in released code, of course.

So if you need a basic tool for handling HTTP requests (GET, PUT, POST, DELETE) you may find it useful… rather than having to write the few lines of code yourself.

apples and oranges

jrbriggs — Sat, 01 Jul 2006 16:35:58 +0000

My own experience suggests that REST and Soap/WS is an apple and oranges comparison. Actually that’s being unfair to oranges. It’s more like an apple and strange-looking-fruit-bearing-no-resemblance-to-anything-palatable comparison.

When I’ve had the opportunity to design a system around REST principles (resource-oriented, POX, basic HTTP, etc), it has had a simplifying effect on the overall architecture — not dumbing things down, just a reduction in complexity. I’ve had to think differently about fundamental design issues — what are my resources, and how should they be interacted with… rather than RPC API details — in a way it’s a little like the difference between thinking OO terms and thinking procedurally.

The same can not be said for the SOAP/WS systems I’ve seen, and worked with, where complexity seems to be an accepted byproduct. BPEL, choreography, discovery, and the myriad other WS-* specifications are, to me, a good reason to sell expensive product, and justify business power lunches, but a bad way to design software. SOAP at its heart was a relatively simple idea, but there’s enough other crap bolted-on around it now to sink even the most bouyant of ships. There will be plenty of proponents who’ll argue the whole tooling angle — that their (obscenely expensive) tools will hide the implementation details from you. Well I’ve used some of their tools, and I’m definitely not sold. Just what the world needs: the equivalent of a Renault Alliance, with worse fuel economy than a Hummer, slower than a Simca 1000 running on 3 cylinders, and a Ferrari price tag to boot.
Not to forget that hiding away implementation details is a good way to cause a mass of problems when the client asks you to do something that fall just outside the capabilities of the tool.

All of which is to say I believe that positions which appear to argue the differences between REST and SOAP down to mere implementation details, are less than convincing. Okay, to be absolutely honest, I have argued something vaguely similar in the past… but I’m wavering on the cusp of thinking I should’ve explained myself a bit better. I still believe that a resource-oriented application could expose some SOAP-style services, but it certainly wouldn’t make sense for an entire application to expose dual interfacing. And I certainly don’t think that a WS-* design can then expose its methods as HTTP verbs and somehow magically satisfy the goals of a REST architecture (usage of HTTP verbs, alone, doth not a REST app make). A bit like an elephant trying to squeeze into a pair of hot-pants and then announcing to the world that it’s a prima ballerina.

Related footnote: I hadn’t read this before, but it’s a brilliant read.

REST illustrations

jrbriggs — Fri, 30 Jun 2006 21:06:20 +0000

Stefan has just posted a nice, simple illustration of the differences between service-oriented and resource-oriented (i.e. REST) design. Something to bookmark (which I’m doing right here… ) for the next time someone asks me what the hell REST is…

unconvinced by WADL

jrbriggs — Wed, 03 May 2006 13:40:00 +0000

I remain thoroughly unconvinced by WADL. I accept there’s a (current) corporate need to over-describe everything in XML, but quite frankly I’d rather describe my REST resources with HTTP OPTIONS plus extended http headers and/or, preferably, human-readable (and also machine parseable) xhtml, ala microformats.

Interestingly, I find no mention of WADL on rest-discuss. Crappy yahoo search algorithm? Or just “phasers set to ignore”?

Resource versus Activity

jrbriggs — Sat, 07 Jan 2006 06:11:00 +0000

In an article on IBM’s developerworks:

Proponents of the REST-based pattern often point to its architectural simplicity compared to the complexities inherent in the broad collection of SOAP-related Web service specifications as being the key to why REST is the superior approach. This argument has flaws in that, as I have illustrated, the two approaches are looking to solve different types of problems. Web-based application services, such as Bloglines, Amazon, flikr, del.icio.us, and so forth, face a different set of issues than say, a hospital looking to automate their internal processes for order procedures and writing prescriptions and therefore demand a different architectural approach. That’s why design patterns exist — different approaches to solve different problems.

I don’t believe he’s successfully illustrated any flaws at all, particularly not with those two examples. Both order procedures and writing prescriptions can fit just as comfortably within a REST paradigm as they can SOAP.

For example, it’s easy to envisage POSTing to a prescription resource such as:

/health/services/prescription

which creates a new prescription identifier (for example, P1RA43) and adds the items in the prescription request. The identifier is returned to the caller by way of the Location header:

Location: http://somehealthprovider.com/health/services/prescription/P1RA43

The prescription could then be viewed in a web browser by calling that URI and specifying a content type of text/html, or used by a pharmacy computer during the fulfillment stage, by requesting text/xml instead. An HTTP DELETE against the URI would get rid of the prescription, a PUT would replace the contents, and subsequent POSTs add to the contents.

It seems to me this looks hellaciously (love that word) more elegant than the equivalent CreatePrescription, GetPrescription, AddToPrescription, ReplacePrescription, DeletePrescription, etc that I assume would be the SOAP equivalents.

TDD, Jython & REST (Part 3)

jrbriggs — Sun, 02 Oct 2005 15:27:00 +0000

With the addition of xml validation and value checking, the testing scripts actually become somewhat useful:


> PUT http://localhost:8080/ws/user ''
    'username=testuser1&email_address=testuser1@test.com'
> GET http://localhost:8080/ws/user/testuser1
< assert status 200
< assert header Content-Type text/xml
< assert xmlschema ../web/user.xsd
< assert xmlbody /user/username=testuser1
< assert xmlbody /user/email_address=testuser1@test.com
> DELETE http://localhost:8080/ws/user/testuser1

Depak Vohra’s article at ONJava, shows how to perform schema validation using xerces/jaxp. Adapting his code to jython is straightforward, by adding the following imports to the test script from part 2:


from java.io import StringReader
from java.lang import System
from javax.xml.parsers import DocumentBuilderFactory
from javax.xml.parsers import DocumentBuilder
from org.xml.sax.helpers import DefaultHandler
from org.xml.sax import InputSource

System.setProperty('javax.xml.parsers.DocumentBuilderFactory',
        'org.apache.xerces.jaxp.DocumentBuilderFactoryImpl')

Depak’s Validator becomes:


class Validator(DefaultHandler):
    def __init__(self):
        self.validationError = 0
        self.saxParseException = None

    def error(self, exception):
        self.validationError = 1
        self.saxParseException = exception

    def fatalError(self, exception):
        self.validationError = 1
        self.saxParseException = exception      

    def warning(self, exception):
        pass

Finally, a new assertion method is added to the Test class:


    #
    # assert that an xml document validates against an xml schema
    #
    def assert_xmlschema(self, linearr, req, res):
        xml = res.getContent()

        factory = DocumentBuilderFactory.newInstance()
        factory.setNamespaceAware(1)
        factory.setValidating(1)
        factory.setAttribute('http://java.sun.com/xml/jaxp/properties/schemaLanguage',
                'http://www.w3.org/2001/XMLSchema')
        errhandler = Validator()
        factory.setAttribute('http://java.sun.com/xml/jaxp/properties/schemaSource',
            linearr[0])
        builder = factory.newDocumentBuilder()
        builder.setErrorHandler(errhandler)

        dom = builder.parse(InputSource(StringReader(xml)))

        if errhandler.validationError == 1:
            errmsg = '%s, %s' % (str(errhandler.validationError),
                    errhandler.saxParseException.getMessage())
        else:
            errmsg = ''

        assert errhandler.validationError != 1, errmsg

Value checking is accomplished with the following method:


    #
    # assert that a particular xml element contains a particular value
    #
    def assert_xmlbody(self, linearr, req, res):
        xml = res.getContent()
        if not self.xml or self.xml != xml:
            factory = DocumentBuilderFactory.newInstance()
            factory.setNamespaceAware(1)
            builder = factory.newDocumentBuilder()
            self.dom = builder.parse(InputSource(StringReader(xml)))
            self.xml = xml

        # get rid of the starting '/'
        # (could leave this out in the test, but it looks better in to me)
        if linearr[0].startswith('/'):
            linearr[0] = linearr[0][1:]

        # split into xml element list and a value based on the first '=' position
        equalspos = linearr[0].find('=')
        tmp = linearr[0][0:equalspos]
        value = linearr[0][equalspos+1:]
        elements = tmp.split('/')

        # loop through the elements in the list and retrieve each named node
        node = self.dom
        for x in xrange(0, len(elements)):
            nl = node.getElementsByTagName(elements[x])
            if not nl:
                raise AssertionError, 'sub element "%s" is missing' % elements[x]
            else:
                node = nl.item(0)
            if x == len(elements)-1:
                eval = node.getFirstChild().getNodeValue()
                assert eval == value, 'element %s (value "%s") does not match expected %s'
                            % (elements[x], eval, value)
                return
        raise AssertionError, 'missing element %s' % tmp

Which takes an assertion line such as “assert xmlbody /user/username=testuser1″ and looks up elements in the dom, where “user” is the root node, and “username” is a child node of user. At the moment this is very basic value checking which won’t work well for more complicated documents — but I’m trying to avoid complication as much as possible anyway, so it’s difficult to justify (to myself at least) going to greater lengths to make something more flexible. The main enhancement I’m looking to make initially is the ability to specify the nth child of a node. For example, something like the string “/list/user[2]/username” (in case it’s not painfully obvious, the second “user” element in “list”).

But for the moment, this will hopefully get me going.

TDD, Jython & REST (Part 2)

jrbriggs — Wed, 28 Sep 2005 17:28:00 +0000

In part one of this rumination, I discussed test-driven-development of a REST-style application written using Jython. An example of a test method would be something like:


def test_getvaliduser():
    req = MockHttpServletRequest('GET', 'user', 'http://localhost:8080/ws/user/testuser1', '', '')
    req.setContentType('text/html')

    res = MockHttpServletResponse()

    user.doGet(req, res)

    assert res.getStatus() == 200, 'incorrect status %s (%s)' % (res.getStatus(), res.getReason())
    assert res.getContentType().startswith('text/html'), 'expected text/html, got %s' % res.getContentType()

The main problem I have with this idea, is the amount of coding (read: effort) required for each test method, and therefore the amount of duplicated effort testing each resource (servlet). Testing other components in this manner, or testing non-standard functionality, is another matter, of course.

I’ve come up with what I believe is a more workable solution. Reducing the previous example to something like the following:


== Get Valid User
> GET http://localhost:8080/ws/user/testuser1
< assert status 200

Which is a serious reduction in the amount of typing required for a test, particularly when you add the setup functions (missing from the 'old' test example):


== Get Valid User
> PUT http://localhost:8080/ws/user '' \
    'username=testuser1&email_address=testuser1@test.com&level=1&password=password'
> GET http://localhost:8080/ws/user/testuser1
< assert status 200
< assert header Content-Type text/xml
> DELETE http://localhost:8080/ws/user/testuser1

Hopefully the script is relatively straightforward to understand: '==' signifies a new test, '>' is an http call, and 'here.

The next step is to add handlers for checking the content in the response (particularly xml), both in form and content.

(If anyone is interested in a working example, let me know and I'll endeavour to post something useable)

TDD, Jython & REST (Part 1)

jrbriggs — Fri, 16 Sep 2005 18:34:00 +0000

I’ve been working on a personal project for a few months now (begun while living in Thailand for a few months), combining a number of different technologies I currently find interesting. I’ve chosen Jython as the programming language in recognition of how powerful I’ve found developing in Python — the reasons for not going completely down the Python route (much as I would like to be using Py 2.4 modules) are mainly because of library issues. In particular I haven’t found a templating library as good as Velocity in the Pythonic world. Of course there is Cheetah, but while it is as close to Velocity as it’s possible to be without ‘sleeping in the same bed’, there is still something about Cheetah that rubs me the wrong way. I suspect it’s simply because, as a long time Velocity user, I have become so accustomed to the techniques we use with that library, that slight changes/quirks in the other become larger issues, completely out of proportion to their actual magnitude.

REST is less a technology than a way of thinking which I’m endeavouring to apply to both personal and professional projects. How successful that application is remains to be seen, but thinking about web services in terms of a resource with a simple set of verbs, has certainly resulted in a re-think about how one approaches component development. It’s difficult to justify a full J2EE deployment (EJBs and so on) when your operations are reduced to a basic four (GET, POST, PUT, DELETE) and the business logic fits comfortably within the framework provided by a servlet (in my case a Jython-Servlet).

Finally TDD (which technically should’ve been mentioned first, considering where it falls in the development cycle), comes from the understanding that developing a project, which I hope to eventually turn into something commercially viable, requires more than just hacking together a bunch of parts in the hope that my frankenstein will live when electricity is applied to its extremities. I have to also consider the future of this system — have confidence in both my design and in any changes/refactoring I might make to it after it has a community of users.

TDD is at the crux of the code sample below. This is just the beginnings of a testing system for my rest+jython servlets.

It may make more sense, to some, to reuse what’s already available (JUnit for example, or the unit-testing modules already included in Python/Jython), but this is as much a learning exercise, as serious development, since I haven’t yet had to opportunity to apply TDD in any professional projects.


#! /usr/bin/env ../bin/jython

import base64
import sys

def authbasic(username, password):
    return 'Basic %s' % base64.encodestring('%s:%s' % (username, password))

global successtests
successtests = 0
global failedtests
failedtests = 0
global errortests
errortests = 0

def contains(srclist, matchlist):
   for i in matchlist:
       if i not in srclist:
           return false
   return true

#
# given a module name and a list of tests, run each test in the module
#
def runTests(mod, testnames=None):
   global successtests
   global failedtests
   global errortests

   exec 'import %s as testmod' % mod
   if hasattr(testmod, 'setup'):
       getattr(testmod, 'setup')()

   for name in dir(testmod):
       if name.startswith('test_'):

           if testnames:
               chkname = name[5:]
               if chkname not in testnames:
                   continue

           try:
               try:
                   # get the method from the module object
                   testmethod = getattr(testmod, name)

                   # run the test, and if successful increment our counter
                   testmethod()
                   successtests = successtests + 1
               except AssertionError, ae:
                   # handle assertion failures
                   failedtests = failedtests + 1
                   print '[%s] failed because of "%s"' % (name, str(ae))
           except Exception, e:
               # handle errors that fall outside of the test boundaries
               errortests = errortests + 1
               print '[%s] failed because of "%s"' % (name, str(e))
               (type, value, traceback) = sys.exc_info()
               print traceback.dumpStack()

   print '''%s tests succeeded
%s tests failed assertions
%s tests failed because of errors''' % (successtests, failedtests, errortests)

if __name__ == '__main__':
   if len(sys.argv) == 2:
       runTests(sys.argv[1])
   elif len(sys.argv) == 3:
       runTests(sys.argv[1], sys.argv[2].split(','))

NB: I’ve hacked out some of the functionality required to set up that particular test (such as creating and deleting the user record), but you get the idea.

A sample scripting testing a valid user of a user management web service would look like this:


from wstesting import MockServletContext, MockServletConfig, MockHttpServletRequest, MockHttpServletResponse

import miscutils
import test

import props

global user
user = None

def setup():
   # change the database connect string to use the test database
   props.db_connect_string = 'jdbc:postgresql:testdb'

   # dynamically load the module
   usermod = miscutils.loadmod('../web','user.py')

   # create a servlet context for testing with
   ctx = MockServletContext.getInstance().setRootPath('../web')

   # load the servlet and then initialise with the config object
   global user
   user = usermod.user()
   user.init(MockServletConfig('user'))

def test_getvaliduser():
   req = MockHttpServletRequest('GET', 'user', 'http://localhost:8080/ws/user/testuser1', '', '')
   req.setContentType('text/html')

   res = MockHttpServletResponse()

   user.doGet(req, res)

   assert res.getStatus() == 200, 'incorrect status %s (%s)' % (res.getStatus(), res.getReason())
   assert res.getContentType().startswith('text/html'), 'expected text/html, got %s' % res.getContentType()

So that’s the basic, lots of typing approach to TDD for my REST app. Next step is to completely scrap that structure, and do it the right way (i.e. a method that doesn’t involve writing the similar code over and over again for each service). More on that later…