Developing a complete client-server application with qooxdoo on Cloud9IDE. Part 2: Integrating

In the previous post, we have chosen the application architecture and integrated a connect server to dish out the frontend code to the clients. We now need to decide how to client and server should communicate.

Choosing the right communication layer: why REST is not enough

The dominant approach in the world of web apps is the use of HTTP requests and the REST paradigm. This allows for a very clean separation of data presentation on the client and data generation on the backend. The use of descriptive and clear URLs to query data increases the clarity of the code and also makes it possible to debug the backend easily. This differentiates REST from, for example, Remote Procedure Call solutions such as JSON-RPC (which is notoriously difficult to debug). Plus you can very easily provide a public API with which the application backend can be queried by third-party clients. Finally, web application frameworks such as Ruby on Rails, or node’s Express server excel in dealing with such requests.

However, there are drawbacks to this kind of solution. The most important one is that usually, the requests are unidirectional. You can query data, but you don’t know when new data is available on the server (unless you use some form of polling). Also, for all of their commendable readability, REST URLs allow very little complexity in the kind of data that can be communicated to the server. In addition, server frameworks introduce an additional layer — the router — between data consumer and data provider, and often force the MVC paradigm on the data interchange, even when this paradigm doesn’t really fit. In many cases, it would be much better if consumer and provider could communicate directly. Finally, REST and JSON-RPC communicate only between a specific client and server, and does not extend the communication to any other client.

Enter realtime, bidirectional messaging

This is why I have always wanted to have a closer look at as a comprehensive solution for all client-server communication requirements. is a bidirectional messaging system that is compatible with a huge number of different browsers/platforms/transports. You get quasi-realtime connectivity in both directions, and you can even use it isn RPC-like fashion (i.e. send a “command” and wait for its execution). Most importantly, is not (mainly) about communication between a client and a server. It is a PUB-SUB (shorthand for “publish-subscribe”) system where clients can subscribe to channels and be notified when the server or some other client publishes a message on this channel. Importantly, the library can be considered very stable, since it is in use in many high-traffic production sites. The API is very simple, and it can be installed by a simple npm install

Setting up with Architect

As with the connect server, we’ll integrate the code as an architect plugin that wraps (which could be always replaced by a different library that uses the same API). Create a new directory in plugins named “socket” with the following package.json:

  "name": "socket",
  "version": "0.0.1",
  "main": "socketio.js",
  "private": true,
  "plugin": {
    "consumes": ["http"],
    "provides": ["socket"]

As you can see, the plugin requires that the http server is already set up, because it will hook into it. Then, add the line

{ packagePath: "../plugins/socket", namespace : "/testapp", loglevel : 1 }

to your config files (“source.js”/”build.js”), so that the plugin will be loaded by Architect. You can choose the level of logging of the server from 0 (= only errors) to 3 (=verbose debug log). One sensible setting would be to have 0 in the “build” config and 2 or 3 in the “source” config.
Finally, we need the plugin code.

// This plugin gives provides a server for the application

// options:
//   options.namespace is the namespace in which to accept messages.
//   options.loglevel

module.exports = function setup(options, imports, register)
    // dependencies
    var socketio = require("");
    var assert   = require("assert");

    // options/parameters
    var namespace = options.namespace;
    assert(namespace && typeof namespace=="string", "You must provide a namespace for the channels");
    var loglevel  = options.loglevel || 0;

    // attach server to http server
    var io = socketio.listen(imports.http.server);
    io.set('log level', loglevel);

    // this will be removed later, serves simply to see if everything works
        console.log('A new socket connected!');
        socket.emit("buttonlabel", "Press this button");
          console.log("Message from client: " + data);
          fn("Hello from the server!");
          socket.emit("buttonlabel", "Button has been pressed");

        // API
        socket : io.of(namespace)
    console.log(" server attached, namespace '%s', loglevel %s", namespace, loglevel);

Setting up the client

To use with qooxdoo, we need to tell the generator to include the code before the qooxdoo library, to make sure it is loaded when the qooxdoo application starts up. This works by inserting the following jobs in the config.json configuration file in the testapp directory:

  "jobs": {
      "source-script": {
          "add-script": [{
              "uri": "/"
      "build-script": {
          "add-script": [{
              "uri": "/"

This loads the javascript files from a virtual path.

Finally, we need some code inside the qooxdoo application to interact with the server through our new communication channel. Change lines 54- inside source/class/testapp/Application.js to read like this:

        Below is your actual application code...

      // set up
      var loc = document.location;
      var url = loc.protocol + "//" +;
      var socket = io.connect(url + "/testapp");

      // Create a button
      var button1 = new qx.ui.form.Button("...", "testapp/test.png");
      var doc = this.getRoot();
      doc.add(button1, {left: 100, top: 50});

      // Add an event listener for the button
      button1.addListener("execute", function(e) {
        socket.emit("buttonpressed", "Hello from client!", function(data){

      // setup socket events
      socket.on('buttonlabel', function (data) {

When you read this together with the socketio.js file, you can see what it does: A button is created with “…” as label. When the client receiveds the “buttonlabel” message from the server, it changes the label. When the user clicks on the button, the client sends a message to the server, and received data back in return, which then is alerted to the user. Finally, another message is dispatched by the server to change the label again.

This example behavior is completely useless, but it shows you that can fully replace any other means of client-server communication.

We now have setup application structure and communication channels. The next thing we need is authentication and authorization mechanisms, which allow to grant users different degrees of access to the application. This will be covered in the next post.


Developing a complete client-server application with qooxdoo on Cloud9IDE. Part 1: Application architecture

Having found out that programming with the qooxdoo (qx) framework on the Cloud9-IDE (C9) was not only possible, but actually quite pleasant, I want to now continue by putting together a node-js based architecture that can be the basis of actual applications. In this post and the posts to follow, I will document the different choices of libraries and my experience with them. The current state of the entire application is available on GitHub ]

The frontend is clear — qooxdoo — but what needs to be chosen are three things: 1) the communication layer that lets the frontend and backend talk to each other, 2) the nodejs modules that help me create a maintanable and scalable backend, 3) the persistence layer that allows to save the application data in a -yet to be chosen – database system.

Points 1 and 3 will be covered in later posts. For now, it is enough to say that all my communication needs will be taken care of by In this post, I want to deal with application architecture on the backend.

Application architecture: using C9’s own “Architect”

One of the most important decisions, one that is later hard to change, concerns the application architecture, i.e., how to modularize the code and decide what part of the code does what. When I was coding with PHP, the buzz word was MVC (Model-View-Controller) and this design patterns could be very well translated into the way PHP works. A dispatcher script would call a controller class to query data from the backend into models, and populate a view with the data. If you like this paradigm, there is backbone.js, which is a kind of javascrit MVC framework that many people use, or many others.

MVC and its various siblings is certainly a good pattern when thinking about data. I think, however, that is doesn’t provide an architectural basis for developing data-driven, asynchronous javascript applications. The problem MVC solves (separation of data and presentation) is not a real problem in client-server applications. “Views” are not needed, because data and presentation are naturally separated: all the frontend needs is raw data with which to populate its widgets. But most importantly: the MVC pattern provides no answer to javascript applications’ most important problem, which is the asynchronous nature of javascript.

For some time, I was intrigued by Nicolas C. Zakas Scalable JavaScript Application Architecture, which combined modularity with safety concerns by strictly separating 1) a application “core” which provided all the basic functionality such as client communication, database access, etc., 2) modules, which implemented all the functionality of the application but had no direct access to the core and 3) a “sandbox” which exposed the resources of the core to the modules in a controlled manner. One feature of this approach is that it doesn’t need any library or framework, but can be simply coded by hand – in fact, it must be coded by hand, because the isolation of core and modules is achieved by using closures. This doesn’t work very well with qooxdoo and its traditional OO-approach where one file contains one class with methods, and those classes cannot easily be put into closures. But the main reason I didn’t use this approach was that I wanted some kind of framework that provided me a little more magic.

This is why I was happy to discover that C9 itself was written using a architectural framework named – fittingly – Architect. The idea is that each and every functionality of an application is written as a plugin which registers itself with the application and exposes its functionality to the application without having direct access to the other plugins. There is no sandbox. Instead, each plugin exports an API and “consumes” the API of other plugins. The idea of making everything a plugin with a small but concise API is very powerful. Also, if all functionality of a typical web application backend is modularized into small and focused plugins, it becomes very easy to reuse these modules in a different application. The library also deals with the problem that the initialization of plugins is asynchronous, and it deals with application configuration, which is a very important aspect that cannot be solved at the plugin level.

One caveat: Architect isn’t very well documented yet, seems to have changed considerably before it was released, and there seem to be very different ways of using it. I had to experiment quite a bit before I got things to work. But, as C9 itself, the project seems to hold a lot of promise and just “feels” right, so I was happy to do a little trial & error.

To use Architect, you’ll need to install it (npm install architect) and then create a plugin for each functionality that your application will need.  C9’s github page contains several architect-related repositories which provide examples how to use the framework. There is one calculator demo that is worth looking at, even though I didn’t get it to work. The following code assumes that you have set up a sample qooxdoo project in C9 as described in the the intial tutorial.

The main configuration

Replace the server.js file with the following code, which is from here:

var path = require('path');
var architect = require("architect");

var configName = process.argv[2] || "build";
var configPath = path.resolve("./configs/", configName);
var config     = architect.loadConfig(configPath);

architect.createApp(config, function (err, app) {
    if (err) {
        console.error("While starting the '%s' setup:", configName);
        throw err;
    console.log("Started '%s'!", configName);

The calculator demo’s server.js shows that there is a much simpler approach, but this setup allows several configs (for example, for “source” and “build” versions of the qooxdoo app) that can be run from the same file.

The server will try to read a configuration file from the “configs” directory, which we will need to create, and in which to put the following “build.js” file:

// build.js
var path = require("path");

module.exports = [
  { packagePath: "../plugins/http", root : path.resolve("testapp") }

The README explains:

Notice that the config is a list of plugin config options. If the only option in the config is packagePath, then a string can be used in place of the object. If you want to pass other options to the plugin when it’s being created, you can put arbitrary properties here.

Even the connect server, which usually is in a central place, should be wrapped in a plugin. This is what we do now. In the (newly created) “plugins” directory, we create a new folder “http” with two files:

// package.json
    "name": "http",
    "version": "0.0.1",
    "main": "http.js",
    "private": true,

    "plugin": {
        "provides": ["http"]

http.js (adapted from here)

// This plugin gives provides a connect server for the application

// @options is the object in the config.js file for this plugin.
//   @options.port is the port to listen on.
// is the host to bind to
//   @options.root is the directory from which to serve static files
// @imports is the various services that this plugin declared as dependencies
//   This plugin doesn't have any
// @register is a callback function expecting (err, plugin) where plugin is the
// provided services and lifecycle hooks.  This plugin exports "http".

module.exports = function setup(options, imports, register)
    // dependencies
    var connect = require("connect");
    var assert  = require("assert");
    var path    = require("path");

    // options/parameters
    var host = || process.env.IP;
    var port = options.port || process.env.PORT;
    var root = options.root;
    assert(root && typeof root=="string", "You must provide a document root for the http server");

    // create server and register with architect when done
    var app = connect().use(connect.static(root));
    var server = app.listen(port, host, function (err) {
        if (err) return register(err);
        console.log("Connect server listening on http://%s:%s, serving %s", host, port, root);
        register(null, {
            // When a plugin is unloaded, it's onDestruct function will be called if there is one.
            onDestruct: function (callback) {
            // API
            http: {
                server : server

When you start the server (open server.js as active tab & press “run” or “debug”), you should be able to open the default qooxdoo app at Ignore the IP/Port that is reported on the console – this information is only relevant when you deploy the finished project to your own server.

To run a “source” version of the qx app, you’ll have to create a file named source.js in the “configs” dir with the following content:

var path = require("path");
module.exports = [
  { packagePath: "../plugins/http", root : path.resolve(".") }

and then open the application from

You get the idea. As said before, this setup should never be used on a production server, since it exposes your entire source tree.

The “http” plugin is a good example to demonstrate how smart this form of modularization is. Should you decide to later exchange the connect server with a different server (which, of course, needs to have the same API), you can do this without any changes to the rest of the application. Step by step, we can now continue hooking in the other parts of the application. The next post will deal with client-server communication and the installation of


Coding qooxdoo on the Cloud9 IDE

Long time no write! I was finishing a big project that has been on my back the last couple of years. But now I hope to have some spare time again for coding… and here we go.

The Cloud9 IDE (c9) has caught my eyes right some time ago. I think it is a brilliant idea to have a full development environment for web applications in the cloud, so I can log in whereever I am, on whatever machine I happen to work with, and  continue coding  without having to install and maintain an IDE plus all of the dependencies. To be sure, C9 cannot offer the same feature set as, for example, Eclipse, but then, Eclipse has given me so many headaches (speed, memory problems, incompatibilities bla bla bla), that I am very happy to let go. Also, it has to be said that C9 still has a LOT of problems. Be prepared to reload the application when things seem stuck, which happens quite a bit and this can be frustrating. But I happily accept that as long as things work more or less and I can work decently, and I am sure they’ll manage to fix the remaining problems. You can sync with your git repositories very easily, which makes it an ideal editor to work on your projects hosted on git (bitbucket is also supported). You can run and debug your node programs inside the virtual machine C9 gives you,  access the VM from outside during development, and even deploy the finished app to a production server of your choice. Very nice. No more need to run and maintain a development server yourself, which has always been a nuisance for me. And you can work on your code on a weak netbook with a meager internet connection.

I am a big fan of  qooxdoo (qx) and have written my main project with it. Qx is a very well written JavaScript framework with a very active community and a dedicated developer team that provides excellent support. Qooxdoo has received relatively little attention compared to its competitors such as jQuery, dojo or YUI. In part, this is due to the lack of marketing. Qx is sponsored by a big internet firm (1&1) and used for their internal needs, so there is no real need of gaining market share (and widgets 1&1 doesn’t need won’t be developed, unfortunately). Another reason that works as a disincentive to newcomers is that the library cannot be just plugged in into website in order to put nice shiny widgets on your page, like for example dojo allows you to do. You need to have access to a python interpreter, in order to call a “generator”. The purpose of this generator is to “compile” your code into a bundle (the “built” code that can be deployed) or – for purposes of development – creates a code which loads in the required files in the right order. This has obvious advantages. You don’t have to worry about dependencies, and you’ll get optimized, compressed files that contain only the necessary code. The downside is, as noted, that you have to do a little more work that you’re used to compared to the other libraries, some of which  don’t even have to be hosted on your server (if you include them from a CDN).

Because of the Python dependecy, until recently, it has been relatively cumbersome to bring qooxdoo and Cloud9 together. It wasn’t possible to do the “compile” step inside C9, so I had to sync to git, sync to my Mac, do the compilation, sync back to git, and back to C9. Not ideal. That is why I was really pleased to hear that my C9 VM now supports python! Unfortunately, you have to buy a subscription ($12/month) to be able to call python scripts from the command line (necessary for the qooxdoo tools). [Update: it seems like it works in the free plan, too. Hooray!]

Today I finally found some time to test this out and am happy to report that the qooxdoo development workflow seems to work in C9 as well. Since my time is short,  I just document the basic steps.

1. Create a C9 project

Either create from scratch or by cloning a project from git.

2. Include qooxdoo as a submodule

For example, to use the master branch:

git submodule add qooxdoo

(If nothing happens, reload the page, it should have created the submodule nonetheless)

3. Install the necessary npm module(s)

We’ll need a server to send the static files to the client. For our purposes, a connect server will be sufficient. When creating more sophisticated backends for the qooxdoo application, you’ll probably want to use express or even a full backend solution. In the command line on the bottom, execute

npm install connect

4. Create a qooxdoo application

In the command bar on the bottom, type

python ./qooxdoo/tool/bin/create-application -n testapp

5. Compile the qooxdoo app

Go into the newly created directory and run the generator:

cd testapp
python ./ --no-progress-indicator build

The console output of the script isn’t very pretty, because the C9 console doesn’t support the Backspace character, so that the spinning animation doesn’t work. [Update: you can avoid this problem using the –no-progress-indicator, see Daniels comment below].

You can also tell that the VM isn’t very powerful, because compiling even the small sample application takes quite some time (6 minutes). I’ll have to see if that will end up as a fatal bottleneck for development. [Update: I was told that C9 plans to have more powerful VMs as part of the paid plans.]

6. Setup the server

We now write a minimal server. On the top level (although it doesn’t really matter where it is placed), create a file “server.js” with the following content:

var connect = require("connect");
var app = connect().use(connect.static('./testapp'));
app.listen( process.env.PORT, process.env.IP);
console.log("Server running...");

Save the file, and, while having it open, start the server by clicking on “Run” (the button with the green arrow. If you see “Debug” with a red arrow, uncheck “Run in debug mode” in the associated menu).

That was easy, wasn’t it, compared with configuring an Apache server?

7. Access your qooxdoo application

You can of course directly call your newly generated application by using access to the workspace that C9 allows by opening (replace USER with your username and PROJECT with the project’s name):

However, this way you can only debug a static qooxdoo application without being able to integrate a data backend. Instead, you want to use the connect server you have put in place to serve the static content. When you started  the server, you should have seen a message saying “Tip: you can access long running processes, like a server, at ‘’.” To open the qooxdoo application this way, you need to open this path:

8. [Update] Accessing the development version (“source”)

After finishing the first version of this post, I found out that the above code really only works with the fully compiled application (“build” version). This is because the development (“source”) version of a qooxdoo app needs to access the qooxdoo source tree, which is outside of the testapp directory and thus not available to the Connect server. In order to start the source version, line 2 in the server script needs to be

var app = connect().use(connect.static(__dirname));

and the corresponding URL would be

You can also start your “build” version with this setup, by replacing “…/source/…” with “…/build/…”. Please note, however, that you shouldn’t do this for a production setup, since the http server should be locked into a subfolder without access to the code that is accessible through the root folder.

That’s it. Now you can code away on your new qooxdoo project. Of course, the really interesting stuff starts when you integrate a nodejs-based data backend, using realtime connectivity, such as that provided by More on this in a later post.


Online bibliographic data management

I am a social scientist with lots of curiosity and a bad memory. This means I need tools that help me remember who wrote what were and why. When I started academic work, I had a box with index cards. Later, I started using reference management software, such as “Reference Manager” and “Biblioscape” on Windows (I never liked Endnote). When I moved to the Mac, I migrated my data to “Bookends“, and I am happy with it in general. But I was always annoyed about the closed nature of these applications. It was hard to get data from one app to the other, there was no way of programmatically editing the data (No scripting, even on the Mac where scripting is built-in), and no or only limited access to the database that is used to store the data. And I wanted to be able to edit and process the data my way, and not just the way the application authors let me.

At the same time, I always wondered why there is no really good open source alternative. OpenOffice has bibliography support built in, but so far, it is not useful. There are many open-source projects which help you maintain a list of bibliographies, but very few of them are truly platform independent or feature-rich enough to compare with the commercial programs. And I wanted a software that was intuitive, multi-user, web-based, and easily extensible. I could find none that would have all that and have a nice, user-friendly GUI.

So I had to roll up my sleeves and write one myself. Bibliograph 1.0 was released in 2006, but could not find a large audience, since it was based on PHP4 which was already outdated back then. Also, I had hacked it together in a hurry, so the architecture was really bad. So the project lingered for a couple of years, while I was busy doing other things. Now, thanks to a grant, I could again work on the software, invest some more time into the design, and now am close to a new release. . If you’re interested, have a look at the public demo. I hope I can publish the first Beta soon. [Update 07/2012: Unfortunately, that never happened. I am close to a rewrite that will ditch the PHP backend in favor of nodejs. The PHP-based code is available, though, if you’re interested.]


Das Ausleihsystem der Stabi – Herr, wirf usability vom Himmel!

Zur Abwechslung mal ein “Posting” auf Deutsch, und eines, was sich stärker mit Wissenschaft beschäftigt (das ist auf diesem Blog bisher ja viel zu kurz gekommen!) und meinem Lieblingsthema, der computergestützten Verwaltung bibliographischer Daten. Wer das Ausleihsystem der Staatsbibliothek Berlin besucht, fühlt sich nicht nur gestalterisch in die Mitte der 1990er Jahre versetzt – auch die Benutzerfreundlichkeit ist eher bescheiden.

Der Gipfel der Umständlichkeit ist aber, dass alle Signaturen, die im Ausleihsystem angezeigt werden, nicht mit dem Katalog verlinkt sind. Das heißt, man muss per Copy & Paste die Signatur in den Bibliothekskatalog übertragen und dort nach der Signatur suchen. Und das, wo ein einfacher Link im HTML-Text genügen würde.

Zum Glück gibt es Greasemonkey, zumindest wenn man Firefox verwendet. Das folgende Userscript erledigt die Arbeit für Euch, indem es Signaturen auf den Seiten des Ausleihsystems automatisch in Links verwandelt:

// ==UserScript==
// @name           Stabi-Ausleihsystem
// @namespace      panyasan
// @description    Fügt den Signaturen einen Link zu den Einträgen im Katalog hinzu
// @include
// ==/UserScript==

// Slightly modified code from 

(function () {
    const urlRegex = /\b(([0-9]A|[0-9]B|Ser\.|[0-9]Per)([0-9\-]+))/ig;
    var allowedParents = [ "td" ]; // tags we will scan
    var xpath = "//text()[(parent::" + allowedParents.join(" or parent::") + ")]";
    var prefix = "";
    var candidates = document.evaluate(xpath, document, null, XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null);
    for (var cand = null, i = 0; (cand = candidates.snapshotItem(i)); i++) {
        if (urlRegex.test(cand.nodeValue)) {
            var span = document.createElement("span");
            var source = cand.nodeValue;
            cand.parentNode.replaceChild(span, cand);
            urlRegex.lastIndex = 0;
            for (var match = null, lastLastIndex = 0; (match = urlRegex.exec(source)); ) {
                span.appendChild(document.createTextNode(source.substring(lastLastIndex, match.index)));
                var a = document.createElement("a");
                a.setAttribute("href", prefix+ match[0]);
                lastLastIndex = urlRegex.lastIndex;
Coding, Uncategorized

Plone, LDAP and buildout

I just spent a couple of hours trying to configure a Zeoserver /Plone 3 setup on Ubuntu, build with the “buildout” mechanism, to use LDAP as an authentication mechanism, and ran into a couple of problems, such as the error message:

ImportError: cannot open shared object file: No such file or directory

which might have brought you here. I was using the excellent tutorials by Jens W. Klein and Veit Schiele (German, here the translated version, which describes an even easier setup than than the one provided here; thanks to cra for pointing this out). There is one small detail missing in  both tutorials, namely, that openldap needs the LD_LIBRARY_PATH variable to be set, otherwise python-ldap cannot find the openldap libraries.

To pull both tutorials together, here are the entries that I added to my buildout.cfg (no guarantees that this works unmodified for you).

Continue reading


Novell Evolution on Mac OS 10.5 (Leopard)

I never understood why the Open Source Outlook replacement Novell Evolution never took off. It would be the perfect cross-plattform and open source candidate for getting rid of the crappy proprietary Microsoft-only standard-uncompliant market leader. Instead of spending millions and millions on licenses, companies could have supported Evolution development years ago and now have a really good and free product in their hands. It never happened – maybe readers have a clue as to why.

Continue reading