Coding

Deploying a qooxdoo+socket.io application from Cloud9IDE to a cloud server: Heroku, OpenShift, Nodejitsu

This is part 5 of my ongoing series on “Developing a complete client-server application with qooxdoo and NodeJS on Cloud9IDE“.

Before, as previously announced, I tackle the “Mighty Database Problem”, it seemed to make sense to cover another topic first: the possibility to migrate your Cloud9IDE-apps directly to a deployment server in the cloud, where the restrictions of C9 (such as: no database server) do not apply, and where the application can be really used and tested by others. I approached the PAAS (“plattform-as-a-service”) topic rather naive and clueless, using a trial-and-error approach – that’s why it took two failed attempts and quite some time to put together this post.

As part of the completely new development experience that cloud services offer (in comparison to the old model of doing everything on your own servers), C9 is cooperating with a couple of different cloud providers and offers a one-click deployment from within the IDE. At the moment Heroku and Windows Azure, but I assume that more providers are in the pipeline. Unfortunately, the heroku deployment didn’t work (see below), so I tried my luck with a different solution, OpenShift (Cloud9 itself is based on OpenShift), which also didn’t work out, to finally arrive at Nodejitsu, which is probably the best tool for the given job. There are a couple of other offerings, which I haven’t tested. If you have suggestions, please write a comment.

To make a long story short, the main problem is that Socket.io is not supported well by either Heroku nor OpenShift, at least not in a straightforward way. Had I read the documentation of Nodejitsu earlier (which notes that they are the only service to fully support Socket.io), I would have saved myself some time. However, rather than discarding my notes on Heroku and OpenShift, I thought that it made sense to document the steps I took and maybe later they will be useful, or someone knows how to fix the problems. If you want to go directly to what worked, skip the sections 1) and 2) on proceed right to 3).

The lesson from this episode is that, contrary to what I said in an earlier post, socket.io might not be the solution to all problems, it might be the problem in some cases. Currently, it seems to be better to stick with REST when using “normal” cloud providers. If you have normal connect/express apps, they will all work with the services I tested.

1) Trying to deploy to Heroku.com: Killed by the slug (size)

Automatic deployment to Heroku as offered by C9’s interface doesn’t work for our application for reasons that will become clear later. To set up Heroku deployment, you have to go through a few steps first, which I will describe in the following. For more details, have a look at this tutorial which describes the complete process of deploying a C9-hosted node application to Heroku, including registration and setup.

The heroku deployment requires that your app contains a valid package.js file that declares the dependencies.

package.json

{
  "name": "qxnodeapp",
  "version": "0.0.1",
  "dependencies": {
    "architect": "0.1.4",
    "async": "0.1.22",
    "connect": "2.4.2",
    "roles": "0.0.4",
    "socket.io": "0.8.7"
  },
  "engines": {
    "node": "0.6.x"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/cboulanger/qxnodeapp.git"
  }
}

Note that we don’t use the current socket.io version (as of 2012/08/19, 0.9.8), because the former introduces a dependency (hiredis) that doesn’t compile on Heroku.

Also, we need to tell Heroku how to start the node process:

Procfile

web: node.js server.js

Finally, when pushing our app to Heroku, we must make sure that we only push those files that are actually needed to run the application. We cannot include the complete qooxdoo SDK because it blows the limits imposed by Heroku (200MB).

.slugignore

qooxdoo/*

This means that we can only use the standalone version that has been produced by “python ./generate.py –no-progress-indicator build”. Neither “source-hybrid” nor “source” version are supported by the setup described here – but the deployed app should be the built app in any case.

As of now, socket.io and Heroku are not an ideal match (as detailed here). I hope that will change in the near future because it imposes serious limitations on the use of socket.io. To make socket.io work, we have to add this configuration to plugins/socket/socketio.js, after line 21:

io.configure(function () {
  io.set("transports", ["xhr-polling"]);
  io.set("polling duration", 10);
});

Heroku compiles a complete image of the application called “slug” before it actually stores and publishes it. In this process, we get fairly descriptive error messages in case something goes wrong.

Because of the fact that Heroku clones your C9 repository, it will not receive the files that are excluded by a .gitignore file. That is true for all the “build” files, which are, however, necessary to run the application. We therefore have to create a deployment branch which includes those files. Luckily, we can use the generator to automate this task. Add a “deploy” element to the “export” section and the following job to the “jobs” section of testapp/config.json:

    "deploy" : { "run": ["build","push_to_heroku"] },
    "push_to_heroku": {
      "shell": { "command" : [
        "git checkout -b deploy",
        "git add -f ./build",
        "git commit -m \"Adding build files\"",
        "git push -f heroku-target deploy:master",
        "git checkout master",
        "git branch -D deploy"
        ] }
    }

This creates a temporary “deploy” branch, commits the build files to it, pushes everything to Heroku, and finally deletes the branch. Note that if something goes wrong along the way, and the script terminates, you’ll have to checkout master and delete the deploy branch manually (git checkout master; git branch -D deploy).

Why it didn’t work

Unfortunately, our socket.io dependency is very heavy and creates a slug that blows the available slug size limit. That is where the attempt to use Heroku had to stop: The application wouldn’t even start and I could not figure out how to make the slug any smaller. If you want to have a look yourself, here’s the state of the code then.

So I had to look for a different solution.

2) OpenShift: socket.io woes, wrongly blamed

Now to the next unsuccessful attempt: the setup almost worked, as you’ll see below…

There already is a detailed tutorial on the deployment of Cloud9-Apps to OpenShift, so I only need to add relevant details pertaining to the application that is developed in this series.

As explained in the mentioned tutorial, after setting up your OpenShift application, you need to add the OpenShift container’s git repo as a remote target, using the git URI that can be found in the settings page of your application (you need to click on the name of the application in your dashboard).

git remote add openshift -m master ssh://XXXX@YYYY-ZZZZ.rhcloud.com/~/git/YYYY.git/

Since the next steps involves overwriting your local files, you must make a backup of server.js now (or revert the changes using git afterwards – I am not very good at git yet, so please tell me how one would do that).

git pull -s recursive -X theirs openshift master

This pulls in the preinstalled files in the OpenShift NodeJS appliance. As noted, it overwrites our server.js file, so we need to restore them now from our backup & recommit or use git to get the old version. It also seems to commit the files contained in node_modules, which is not what we want, so we have to throw them out of the git index:

git rm --cached -r node_modules
git commit -m "Removing files from index"

I am sure there is a better way of doing this, so if your git knowledge is better than mine (which is not much), use the comment section to correct me!

Previously, it was necessary to to add and commit a file deplist.txt with the npm dependencies that OpenShift should install automatically. The current README.openshift file marks deplist.txt as “deprecated” and tells us to use the normal package.json file for listing the dependencies, so we’ll use that file.

package.json

{
  "name": "qxnodeapp",
  "version": "0.0.1",
  "description": "A demo app combining qooxdoo & nodejs",
  "engines": {
    "node": ">= 0.6.0",
    "npm": ">= 1.0.0"
  },
 "dependencies": {
    "architect": "0.1.4",
    "async": "0.1.22",
    "connect": "2.4.2",
    "roles": "0.0.4",
    "socket.io": "0.8.7"
  },
  "private": true,
  "main": "server.js"
}

Now we need to adapt our node server settings to the OpenShift container. Create a new architect configuration file configs/deploy.js (and add/commit it):

configs/deploy.js

var path = require("path");
module.exports = [
  { packagePath: "../plugins/http", root : path.resolve("testapp/build"),
    host : process.env.OPENSHIFT_INTERNAL_IP,
    port : process.env.OPENSHIFT_INTERNAL_PORT
  },
  { packagePath: "../plugins/socket", namespace : "/testapp", loglevel : 0 },
  { packagePath: "../plugins/store" },
  { packagePath: "../plugins/users" },
  { packagePath: "../plugins/acl" }
];

We have to update the server code to pick up this configuration by default (the change is in line 4):

server.js

var path = require('path');
var architect = require("architect");

var configName = process.argv[2] || "deploy";
var configPath = path.resolve("./configs/", configName);
var config     = architect.loadConfig(configPath);

architect.createApp(config, function (err, app) {
    if (err) {
        console.error("While starting the '%s' setup:", configName);
        throw err;
    }
    console.log("Started '%s'!", configName);
});

Since the default configuration now is “deploy”, this means that in the C9 IDE, you can no longer simply press the “run” button with the server.js file open, in order to run the source version. You need to type “node server.js source” explicitly to start the server in the C9 virtual machine. Creating a custom job in the “run & debug” section doesn’t work yet because there is a bug in C9 that prevents passing command line arguments to the scripts (which will be fixed eventually).

As in the case of Heroku, we want to tranfer only the “build” files to the OpenShift container, and not the full toolkit. If you haven’t done so already, add a “deploy” element to the “export” section and the following job to the “jobs” section of testapp/config.json:

    "deploy" : { "run": ["build","push_to_openshift"] },
    "push_to_openshift": {
      "shell": { "command" : [
        "git checkout -b deploy",
        "git add -f build",
        "git commit -m \"Ready for deployment\"",
        "git push -f openshift deploy:master",
        "git checkout master",
        "git branch -D deploy"
        ] }
    }

Unlike Heroku, OpenShift doesn’t seem to provide an easy way of skipping the submodules (qooxdoo and the dialog contrib), which take away unneccesary space. I tried to remove the submodules in the deploy branch using these directions, but that didn’t work. Any ideas?

Our C9 repo now contains all the files necessary for committing & pushing the application to OpenShift, which we’ll do now,using the generator and our deploy job:

cd testapp
python ./generate.py -I deploy

Note that I used the -I command line switch here which is available only in the latest master branch of qooxdoo. If you use an older version, use the longer –no-progress-indicator switch.

Pushing the changes to OpenShift triggers a rebuild of the app in the OpenShift container. The app should now start.

Why it didn’t work

But that wasn’t the (happy) end of the story. However hard I tried, the socket.io connection wasn’t established. All I got from the /socket.io server was “Welcome to socket.io”. So I gave up and tried another service: Nodejitsu, which is described in the next section. In the process of which it dawned to me that the problem might have to do with something else completely (a bug in socket.io v.0.8.7), so it was probably not OpenShift’s fault. But then it was already to late, and I had moved on.

If you’re interested, the code of the OpenShift setup is saved on GitHub.

3. Nodejitsu: so easy it should probably be declared illegal

After trying Heroku and OpenShift, which both demanded a relatively complicated setup to get my app (almost) running, discovering Nodejitsu involved a couple of nice surprises: It turned out to be almost unbelievably easy to deploy my application.

There is no official guide information yet that deploying from Cloud9IDE to Nodejitsu is even possible. I found this obscure gist that contained the information I needed.

If you work on a free plan, use

npm install jitsu@0.7.x -g
mv ../lib/node_modules/jitsu node_modules

from the C9 command line. If you have a paid subscription,

npm install jitsu@0.7.x -g

will suffice. I don’t know why the particular version (0.7) is recommended – I even got a notice that that version is deprecated, but there must be reason, and it worked. I haven’t tried to update jitsu within C9 yet.

Upon registering an account at nodejitsu.com, you will receive an e-mail with a confirmation code.

jitsu users confirm USERNAME CONFIRMATION-CODE

You’ll be asked to provide and confirm a password – you can respond to the command line prompts using the C9 command line with no problem.

Use package.json and server.js from the Heroku section and the following configs/deploy.js:

var path = require("path");
module.exports = [
  { packagePath: "../plugins/http",
    root : path.resolve("testapp/build"),
    host : "qxnodeapp.nodejitsu.com",
    port : 8080
  },
  { packagePath: "../plugins/socket", namespace : "/testapp", loglevel : 2 },
  { packagePath: "../plugins/store" },
  { packagePath: "../plugins/users" },
  { packagePath: "../plugins/acl" }
];

The port 8080 is important, because otherwise socket.io doesn’t work.

Important: we’ll have to adapt line 61 of config.json to read

"uri": "/socket.io/socket.io.js"

(“socket.io.js” instead of “socket.io.min.js”), because there is a bug in socket.io 0.8.7 (which we’re using here) that prevents the minified version from being served. This was the reason why my OpenShift setup didn’t work.

Then the only thing that is left to do is to simply:

cd testapp
python ./generate.py -I build
cd ..
jitsu deploy

The utility will analyse your code, package, and setup the app on one of the nodejitsu servers. You’ll be asked to provide the namespace for the app, in our case “qxnodeapp”. Each time you “jitsu deploy” your app, a new “snapshot” will be created. After some time, jitsu will tell you that the application has (re-)started at a specific URL. Our application should be running at http://qxnodeapp.nodejitsu.com (try to log in with john/john or mary/mary).

There is also a developer dashboard which can be accessed at https://develop.nodejitsu.com, where you can start, restart and stop your application, manage your snapshots, and look at the log files, which is particularly useful.

So at the end of the day, cloud deployment of the app developed here was successful. As usual, you find the complete source code on GitHub. Next time, I will really be looking at databases!

Advertisements

4 thoughts on “Deploying a qooxdoo+socket.io application from Cloud9IDE to a cloud server: Heroku, OpenShift, Nodejitsu

  1. Hi, sorry about the “obscure”, what I meant was that it was unclear how up-to-date it was, and how one was supposed to find it (even though I did find it through the big G.) 😉 Thanks anyways for providing the info, and I am glad it is now part of the documentation!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s