Waka: Paddling a new canoe

As I mentioned in a earlier post Roy Fielding has started a ASF Lab for Web Architecture that is intended to be a place to work on documentation regarding Web Architecture. This includes existing protocol improvements and Waka a new HTTP upgrade. Waka is still in the head of Roy Fielding and the changes have been eluded to over eight years in various ApacheCon presentations; in various Apache 2.0 design notes and emails focused mostly around the IO-layer and request-response processing chains in Apache 2.0; emails to rest-discuss and references to various draft RFC and previous HTTP next generation efforts – rHTTP, W3C’s HTTP-NG and Spero’s HTTP-ng.

From a podcast with Jon Udell, Roy describes Waka thus.

Waka is a binary token based replacement protocol for HTTP. The goal is to create a language independent mechanism for exchanging data oriented messages using the ReST architectural style – the principals and design.

An application protocol that is efficient as any custom protocol yet generally applicable like HTTP. The difference is that we get rid of HTTP syntax, make it possible to interleave meta-data and data in independent streams/frames. Doing this in a way that is efficient as possible without being entirely generic i.e. Waka is not attempting to replace TCP. This means Waka would be suitable to application integration inside the firewall and well as outside.

Maori WakaWhy name it Waka? Waka is the Aotearoa (New Zealand) Māori name for Canoe still used for ceremonial purposes. It is one of the few four-letter words suitable for a protocol name. And Roy is part Māori.

From the last substantial presentation Roy gave about Waka the following can be gleaned.


Deployable within an HTTP connection via the HTTP/1.1 Upgrade header field with a defined mapping to HTTP/1.1 proxies.


A uniform syntax regardless of the message type and direction that also allows for padding for 32/64 bit alignment. There is a discussion in Chapter 6 of Roy’s dissertation covering uniform syntax and numerous discussions about the short-comings of layered encodings and MIME.

Messages are self-descriptive using explicit typing for message structure and fields. An indication of mandate and scope of the fields.

Association of meta-data to control, resource and representation. Allows for Premature termination of request or response.

To be more efficient the protocol will use tokens for all standard elements where a URI reference can be used in place of any token. Macros (client-defined syntax short-hand) can be established per connection. Meta-data delivery can be interleaved with data.

New Request Semantics

RENDER: display/print/speak this representation.

MONITOR: set up a subscription to be notified of resource state changes. Notification are sent to a resource specified as part of the request. Another name commonly used for these semantics is WATCH.

Authoring Method: DAV simplified and by that I would assume removal of the DAV property methods. Elimination of non-resource identifiers and reintroduction of PATCH.

Request control data: request identifier to support request pipelining removing the head-of-line bottleneck in HTTP. A Transaction identifier, a URI, meaning transactions can be treated as resources. And relative priority – relative to requests in a connections pipeline.

New Response Semantics

Self-descriptive binding to the request by echoing the request id, method and target URI. This basically enables an asynchronous transport.

Cache key explicitly described; caches no longer need to save request fields or guess about Vary information.

Unsolicited Responses for cache invalidation messages and multicast event notices.

1 comment Digg this

Scaling Yahoo! Pipes

Yahoo! Pipes are CloggedIf you’ve been trying to access Yahoo! Pipes and come up with a request for Mario then have some sympathy for Yahoo! as Dare Obasanjo comments.

As someone who works on large scale online services for a living, Yahoo! Pipes seems like a scary proposition.

Basically the service is likely to have hit the twin sisters of scaling large systems, CPU and I/O bounds.The nature and flexibility that Yahoo! Pipes provides: User defined queries over changing data streams that are activated every time a pipe is “pulled” will create a heavy load. Dare continues.

It combines providing a service that is known for causing scale issues due to heavy I/O requirements (i.e. serving RSS feeds) with one that is known for scaling issues due to heavy CPU and I/O requirements (i.e. user-defined queries over rapidly changing data). I suspect that this combination of features makes Yahoo! Pipes resistant to popular caching techniques especially if the screenshot below is any indication of the amount of flexibility [and thus processing power required] that is given to users in creating queries.

This has always been an issue if you consider a centralised event routing infrastructure. I suspect it is made worse by the “pull” nature of feeds. Even if if-modified headers or etags are used in the HTTP request. To determine if a user-defined feed has changed it would require processing the whole chain effectively adding a whole lot of processing over head. This is however a naive way of approaching the problem. An internal event architecture and caching are applicable. The caching is just different to the way content caching currently works. However the answer, I suspect, is not to centralise. The general infrastructure represented by Yahoo! Pipes should be deployed by millions. I have my own router for IP traffic so why not have my own router for application data traffic?

No comments Digg this

Yahoo Pipes a Cloud based Feed Router

Yahoo Pipes was released yesterday (last night NZ Time). What Yahoo Pipes represents is the next stage in using the Web as a platform. That is: RSS and Atom are becoming the connective tissue of the web and web applications the components. This was the basis of my Talk at Kiwi Foo – “Replacing my Wife with Atom.*” I commented I was surprised this [general routing infrastructure for Atom, not the replacement of my Wife] had not been done yet, after all the ideas of routing and remixing feeds have been discussed for a long time in the early history of RSS. Before that if you consider RSS/ATOM and the infrastructure around it as a Universal Event Bus. As Jeremy Zawodny states “For far too long now RSS has been used in ways that don’t really tap its true potential.” Well now Yahoo is making a start in the right direction.

Unfortunately I can’t get in, just getting “Our Pipes are Clogged!” – missed opportunity or what? Well it shows, to some degree that at least, the info junkies want this sort of thing. As a result I can’t really comment too much on the service itself, but I can comment on the underlining ideas – this was after all the motivation for re-entering the blogsphere.

[Tim O’Reilly has a good write-up as does ReadWriteWeb and Anil Dash. While Brady Forrest looks a little closer. ]

The basis of the talk I gave at Kiwi Foo was this:

Atom and the Atom publishing protocol will disrupt the way ‘network’ applications are integrated and become the way Internet scale Pub+Sub and routable applications architectures will evolve…Therefore make your Web API’s atom based to make integration easier – Look Ma no code!

While to some degree the arguments apply to RSS, ATOM [IMHO] has the advantage in the longer term. The arguments are not new. Rohit Khare has been thinking about this stuff for a long time. He gave a talk about SOAP routing at ECON 2002. I just applied similar arguments to ATOM but tried to include the human component – conceptually feeds are more human friendly than SOAP ever will be. That observation then naturally leads into a Disruptive Technology argument w.r.t. EAI. Just take a look at the dabbleDB demo that Avi Bryant put together at FOO last year, or RSSBus for examples. (there are several other examples that I’ll leave for now.)

An observation I attempted to justify was: prior technologies have attempted to implement Internet Scale Pub+Sub but failed not because the technologies weren’t up to it (they may not have been) but that as a broader community we still haven’t grokked what the Web as a platform really means. As a result we haven’t understood what integrating applications beyond the Point-2-Point mashup type actually involves. Oddly enough though we are use to solving these integration issues at the network-level. Just look at the Internet and TCP/IP. The same principles of message (packet) based approaches apply in the Enterprise as any EAI vendor will tell you. ATOM is approaching the EAI issue from the grass-roots up. Unlike SOAP that was top down even though it professed not to be. More importantly for this sort of service composition to really take off Atom and the Web needs a “joe average” programming model.

Yahoo Pipes is a giant leap in that direction for the Web even though we have been programming the Web, for human consumption, for sometime now. Every time you add a feed URL to a feed reader or personalised home page you are connecting disparate web services together, you are programming. Labview has been doing this sort of thing for a long time. Lego RoboLab and the Lego Robotics programming environment, Microsoft Robotics Studio Visual programming are other great examples. Ben Goodger recently pointed me to another example on OSX, Automator.

Does this mean the programming model for the Web platform is a visual wiring diagram. Why not? Nor does it have to be provided as a cloud based service like Pipes. Since Pipes produces feeds these can then be glued to an instance of Pipes (RSSBus/Automator) running inside my firewall, on my desktop, or in my Browser making the offline and online divide seamless – portable ‘backend’ widgets for the Web (more on this later). Aside from all this, Pipes adds an interesting twist to information personalisation.
* [in case you are wondering the title of my talk refers to a user scenario managing and filtering many information sources and notifying me of things that are important at a given time or place using the most appropriate delivery. A job my partner, Lynne, currently does to some extent and Gmail does for my email.]

No comments Digg this

Recovering from Kiwi Foo

It’s over. I’m shattered. And it was absolutely fantastic. Lack of sleep is however has taken a toll. Kiwi Foo (a.k.a Baa Camp), an O’Reilly Media spin-off unconference organised by Nat Torkington was held in Warkworth over the weekend. The photo-stream over at flickr captures the activities.

Attended by a mixed bag of individuals from geeks, user experience designers, artists generally – we’re all artists in some way – business bods, government; you get the picture. The one shared interest was the emerging technology space, how it is changing our society and making sure that we are doing the changing. It was the cross section of interesting individuals, the interactions, the sharing of perspectives that made the event very special from my point-of-view. I’ve taken away a lot of tacit knowledge I doubt I would have obtained elsewhere – Thanks Nat and Jenine. Also if it wasn’t for the Great Coffee that Russell procured I doubt I could have operated on so little sleep.

Friday evening started with a powhiri that I caught the tail end of. A few formalities of the administration and group introduction type followed. Then the establishment of the schedule for the next few days. In the FOO tradition the schedule is not predetermined but constructed adhoc on the first evening. Participants made a mad rush to large sheets of paper with day and time slots hanging from the Mahurangi College staff room walls. Filling out their particular topic. This was all done a couple of hours before the first session. I managed to get a reasonable time on Saturday afternoon to put my case for Atom based Web APIs to the test [more on this in later posts].

The first formal session I attended, along with many others, was a discussion (and it was a discussion) on broadband issues and policy with David Cunliffe the Communications and Technology Minister. I have to say I was impressed with his understanding of the issues and his willingness to listen and take on what was being raised by the audience.

Not being a ‘networking’ wonk, nor someone who has been following local issues, I didn’t realise how bad the current situation actually is. Especially with regards to peering and the robustness of our backbone. Basically the market hasn’t provided so now the issue is being addressed at the first level: unbundling the local loop. Though that is just the beginning. The point was clearly made there has been massive under investment in the network and unbundling is a great, if belated, start to getting things on track again. However as the network is upgraded policies have to be adopted to ensure that enough space and power is placed in the boxes at street level. This is required ensure third-parties can put their equipment in and to enable peering at the street level. It was clearly articulated by several in the audicence the economic impact the lack of peering has had and how it extended beyond bandwidth issues right up to day-to-day Internet usage. Rod Dury declared the need as the “Internet Efficiency Policy.” Peering is a really big issue that needs to be sorted and David declared “from tonight peering is now a primary issue.” Of course the huge round of applause, with a few hoots, followed such a definitive statement. The next morning I had a follow-up conversation with Andy Linton that clarified some issues from the night before that I had not understood.

Another round of sessions followed but at 9:30pm I decided to socialise and partake in the fruits of the more “interactive” sessions: quadraphonic music and body illumination – if that is what you can call it. Both very cool. I particularly liked the quadraphonic sound arrangement that blasted from four car stereos parked around the school netball courts. Unfortunately I cannot remember the name of the artist.

Saturday started early. I decided to stay in the Whare Nui and despite the cacophony of sleepers’ vocals I had a good four hours of sleep. Up, bright eyed ready for great breakfast discussion.

The first session I attended was Tim Norton giving a description of his trials and tribulations with starting and funding technology businesses. The discussions contributed further “stories” and perspectives from others on both-sides: investors and business start-ups. I always find the in-person “war stories” really helpful as they lend a personal touch to an individuals experience that I just don’t get from other sources other than your own experience. Following this I attended sessions on Rapid Web application development that discussed various languages, frameworks, patterns and approaches to application composition. Flex, Jiffy, Django, Pylons and Rails where all discussed in some way. I also attended sessions on User Experience Design, Copyright Amendment Bill, Ambient Signifiers, User Stories and a demo of Xero. I really wish there was two of me so I could have attend numerous sessions running in parallel.

Saturday night went for a while. After a lot of socialising and meeting new people I end-up playing Werewolf until 4:30am. The game is a stable at O’Reilly gigs but I had never played it before. It was great fun and a fantastic way to meet others. Having started totally green and being identified for what I was – a Werewolf – I ending up being the evil participant four times in a row. At least I learned the game fast – orchestrating a complete clean up of the Werewolves in the final early morning game.

I attended only a single session on Sunday. This was not planned. Instead I ended up talking to Ian Hay (from Orange) and Don Christie for most of the morning about mobile technologies and applications. A fantastic discussion. The final session I attended was on the One Laptop Per Child project it’s aims, issues and expected time frames. Chris DiBona had brought a prototype with him so every one had a good look at the hardware and the sugar GUI. After that I managed to have some interesting conversations with Robert O’Callahan and Ben Goodger both working on Firefox about offline storage , application composition with routing and directions of Firefox. Basically I had many interesting discussions throughout the camp.
Anyway it was great fun. I was absolutely stuffed but stoked to have attended.

No comments Digg this

Waka Apache Lab Established

A couple of weeks ago Roy Fielding established an Apache lab for the development of Waka and related improvements to other specifications. From the DOAP:

The Web Architecture Lab is a work area for specifying and experimenting with the interrelated shared agreements (protocols) that make the World Wide Web a happy place. The goal is to produce a central storage for Web protocol evolution, including both existing Web protocols and new ones, that Apache developers can refer to, comment on, prototype alternatives, and provide examples and test cases. A secondary goal is to experiment with tools for integrating public commentary with the specification process.

Nothing in the lab at the moment but Roy has been discussing Waka recently breaking a long silence after throwing a tempting morsel at ApacheCon2002. As Mark Nottingham has stated:

Is WAKA still vaporspec? Totally. Does it still have to overcome the inertia of HTTP by proving at least an order of magnitude improvement (which HTTPNG and others failed to do)? Yes. Still, Roy has a pretty good track record, and I’d put my money on him (especially when backed
up by the ASF) over a bunch of vendors trying to sell “platform” any day.

The Apache Lab was set up to allow failure. I for one am very interested in playing with Waka using Yaws.

A little more info in Roy’s Lab proposal.

No comments Digg this

« Previous PageNext Page »