Why we need a NodeJS application server or: writing boilerplate code considered harmful

[Update] A lot of commentators have taken issue with my use of the term “application server”, which made them think of a big monolithic architecture. Even though I do refer to Java, that’s not what I mean. It was the wrong choice of terminology – I am  thinking of a modular, lightweight solution that saves you from implementing the most generic parts of the typical web application. For a better writeup with the same cause, see this post on the hood.ie blog.[/Update]

I love creating web applications, but I don’t do it for a living. I code because I like the challenge of it, but first of all: I want to solve specific problems. When I was writing my dissertation in Political Science, I collected thousands of bibliographic references, some of which I wanted to share with others. This was a time long before Zotero came into existence and there was no decent software that let me do that the way I wanted it. So I started to dabble in PHP, later in JavaScript libraries (in particular, qooxdoo), and out of this came, in several iterations, my pet project Bibliograph. Later, when I was coordinating the organization of a big conference, I had to manage a staff of 50 people working in shifts. The wasn’t any freely available software that was up to the task, so I wrote a qooxdoo-based, interactive staff scheduling web app. I won’t publish the code because it is a mess, but it worked well for the task at hand. Besides my ongoing urge to rewrite Bibliograph to make use of the latest web technologies, I have lots of other ideas for web applications to deal with that odd challenge that I come across occasionally.

There is, however, one big issue that holds me back from starting to work on any of those ideas: the lack of the right end-to-end stack including a decent application server for NodeJS, which will free me from having to worry about boilerplate code for things like authentication, user management, access control, etc.

Professional developers might argue that one should write this kind of code oneself, so that one knows the code inside out and has full control, and can then adapt it for each specific job. If you are not ready to write the boilerplate, they might argue, you should maybe not be in the business of writing applications in the first place.

I respectfully disagree. On the contrary, I think that it is wasteful, if not outright dangerous, for specialized, part-time application developers to implement their own backends from scratch. For Bibliograph, I developed a complete PHP solution. It was an interesting experience, from which I learned a lot. However, it proved impossible for me to maintain it beyond the time that I was actively working on the main application. Later, busy with the things I had to do for my real job, I couldn’t react fast when bugs were discovered in the PHP version I used. Having published the backend code, with people actually using it, I couldn’t respond to questions they had or bugs they discovered in the code. In the first place, I had invested countless of hours in reinventing the wheel, with no real benefit.

I should be using the resources in my disposal efficiently, i.e. the very specific ideas and knowledge of particular technologies that solve specialized problems (such as bibliographic data management for the humanities). I shouldn’t spend my time worrying about how to store passwords securely, or how to synchronize data between the client and the server, or how exactly model data is persisted in what database. There are quite a few tasks that can be solved in very generic ways, in order to provide a platform on top of which specialized applications can be built. This issue also pertains to testing: I should be writing tests for the main business logic, and not spend valuable time on tests for boilerplate code.

I think such a platform would not only save thousands (or more) of hours of developers’ time. This time could then be invested instead in solving the application-specific problems in thorough and creative ways. Such a collaboratively developed platform would make the applications built on top of it more secure and stable (because of collective wisdom, test coverage, and quick fixing of bugs). It would generate enough interest in its long-term maintenance.

There is a wealth of JavaScript frameworks on the client and on the server side, but, as far as I can see, no such thing as an “application server” known from the Java world yet exists. Plenty of libraries exist that can be wired together, but this wiring itself is not trivial, as I learned while writing a tutorial on exactly this topic. The large number of libraries to choose from is both a blessing and a curse – the freedom to choose comes with a price – we need more and more time to research which choices should be made, and the multiplication of APIs and protocols also adds to the expense. How neat would it be if we could agree on a meta-API for the most commonly needed features of a web application and then shims could be used to plug in the different solutions that currently exist.

I see a number of features and characteristics of such an application server (most, if not all of them provided by already existing modules):

  • It should not assume any particular rendering model on the client (DOM-templating vs. widget objects)
  • It should also not force a different programming model on developers (like Opa or Meteor), but let you code in plain old asynchronous javascript. No to-javascript-compilation (except optionally). No magic. Just the right tools.
  • It should have an integrated API for client and server.
  • It should provide a static HTTP server, REST routing, and bidirectional, realtime messaging and broadcasting (such as Socket.io).
  • It should offer async startup/plugin/configuration system like Cloud9′s Architect.
  • It should provide an out-of-the box system and API for user & group management, registration, access control, Password storage/retrieval/update etc, preferrably with a set of built-in templates that can be used for managing the most generic configuration tasks. With this, a pluggable system to use third-party authentication providers.
  • It should also provide an integrated system of data modeling and persistence. I really do not care about database technology. I simply want to store, edit and retrieve my model data.
  • It could also have a toolset that would allow you to deploy your application instantly to a cloud provider such as Heroku or Nodejitsu.

The existence of such an application server, I would argue, would unleash a lot of currently unused creative energy and lead to the development of a lot of interesting special-purpose applications by people who like to code to solve specific problems, but who cannot afford to invest the time to develop full-fledged application backends. It would be great to see such a project take shape, and I am sure others like me would gladly contribute small bits and pieces. Or does it exist already and I have overlooked something?

About these ads

26 Responses to “Why we need a NodeJS application server or: writing boilerplate code considered harmful”


  1. 1 Alex Steitz December 11, 2012 at 1:53 pm

    Nice article!

    Just want to point out that the “Bibliograph” link is broken (at least for me it delivers a 404).

  2. 3 Johan Coppieters December 14, 2012 at 4:53 pm

    Don’t despair I’ve received funding for 12 months research in this area, we’ll not only have an application server, but a multi language CMS, template independant…

  3. 4 Corey December 14, 2012 at 5:30 pm

    Thanks for writing this.

    It really caught my attention, because I have been building something very close to this over the last few months, called NGN (coreybutler.github.com/NGN). It’s still rough, and I’m going through a major refactor right now, but I’ve covered a fair amount of ground already.

    I used to work with ColdFusion a lot, and one really nice feature was the CF Administrator. CF does a pretty good job of allowing developers to do things like define a global datasource connection once, then reference it by name in queries or ORM. Same goes for things like establishing connections to mail servers, logs, etc. CF just sits on top of Java and brings a lot of the best parts of Java together in a simple to use manner. The premise behind NGN is similar.

    So far, NGN has a static (NGN.web.Static) and dynamic (NGN.web.Server) web server, as well as an API (REST) server. There is also a reverse proxy server baked in, and I have mostly completed a “service bus” type of concept for interprocess communication via TCP Sockets. A Socket.io implementation is still on the “to-do” list.

    Additionally, there is an inheritance model for application code, so pretty much everything is extendable. I have completed a generic data model class with some ES6 features (like proxies for firing events when a data model’s attribute changes value) and started modeling common entities like “Person”, “Email”, and “Group”. The idea is to provide 90% of what most people need. Need more? Extend it. At the moment, persistence is partially complete. The idea is to support common databases so all the developer needs is the connection string for the DB.

    I started out with this project thinking it would be a boilerplate framework for my own applications. That’s still a primary goal, but I very recently had a shift in thinking. Now I am calling it a process manager. I write more than just “applications”. Perhaps I’m nit-picking terminology, but I write systems. A lot of my node code doesn’t touch the web. My “apps” may consist of multiple processes running in tandem at the same time. So, I have started directing my work on NGN towards the idea of managing processes that may or may not be part of an application.

    As I’ve gone down this road, I’m seeing how big this effort could become. I’d love to hear feedback/ideas from folks.

  4. 5 Chad Burt (@underbluewaters) December 14, 2012 at 5:56 pm

    This. While all you’re looking for is available as separate projects, wiring them up is a bit difficult. More importantly though, the diversity of pieces in the Node ecosystem means that whole classes of tools just are not being built or are limited to a niche.

    Say I want to develop an open source library for supporting pubsubhubub in Express.js. My potential audience for this lib is sliced up based on the choices I make around the dependencies used (i.e. mongoose for db layer, connect-auth, etc). This reduces the incentives for me to release it, and the chances of a community of users to assemble that could help maintain and improve it. If I were to make such a lib for Django, I could engage 90% of the community because I’m relying on the standard ORM, auth support, etc.

    Some days I’m amazed at what the Node ecosystem, npm, and associated tools like github allow. Other times I feel like its a big step back compared with something more integrated like Rails or Django. I’m certainly sticking with it, because I think once a more integrated system emerges it will make the platform even more attractive.

  5. 6 panyasan December 14, 2012 at 8:30 pm

    @johan: Sounds great – keep us posted!

    @corey: very interesting – I’ll look into your project, it seems like to go a long way towards what I am aiming at…

    @Chad: I couldn’t agree more. It is fascinating how much creativity and talent there is in the NodeJS/JavaScript ecosystem, reflected by the enormous growth of NPM modules. But as you say, this also results in great fragmentation and a duplication of libraries with overlapping or even identical purpose, which in turn makes it very difficult to agree on “standards”.

  6. 7 techwraith December 14, 2012 at 10:17 pm

    Can I suggest taking a look at Geddy? It’s got almost everything in your list:

    - and ORM for data management
    - Built in authentication
    - models that you can use on the client and server
    - socket.io integration for realtime model-related events
    - Just Plain JavaScript ™

    http://geddyjs.org

  7. 8 Khalid Jebbari (@Dj3bbZ) December 14, 2012 at 11:15 pm

    Well, there are several node.js frameworks competing to fill this very gap : Derby, Tower, RailwayJS, Meteor (even though I agree with you about its ‘problematic’ specificity), Geddy, Yahoo Mojito to name some of them. Some of them actually build on top of very popular node.js modules like Express and Socket.io and still respect the ‘node.js’ and ‘javascript’ of doing things. They just need to mature a little bit.

    Other thing : node.js is an environment and a toolbox, not necessarily made for creating web applications. Remember that ‘http’ is just one the ~30 core modules, and you don’t have to use it to make something useful out of node.js. And sometimes you don’t need a whole end-to-end stack to create a web app, just a few modules, so a big framework is not necessary. Think about API servers for instance, where node.js excels at. You basically need only http, auth, database/ORM/ODM.

  8. 10 guigouz December 15, 2012 at 11:40 am

    Maybe the solution could be to just write boilerplate *documentation*, for the different libs available by npm, not a new framework/app server.

  9. 11 avoidwork December 15, 2012 at 12:01 pm

    hi, a web server I cooked up for my day job might be a “starting point” for you: http://turtle.io, it’s built on abaaso, which makes api consumption, etc. very easy.

  10. 12 panyasan December 15, 2012 at 12:35 pm

    @all: Thanks for your comments and suggestions!

    @techwraith I looked at Geddy quite a while ago, and then it didn’t seem to fit my needs. But I’ll make sure to have a closer look again!

    @Khalid I should have mentioned the frameworks that I had surveyed
    before writing my post, and I did look at most of the ones you mentioned. It seemed to me that most concentrated on templating (which I don’t need, I use qooxdoo widgets). They provide you with a lot of the things I mention in my list, but leave the basic building blocks (such as, for example, user management) to the developer. But maybe my research was not thorough enough. Like with Geddy, I should look at them again.

    @Matloob Thanks,I hadn’t heard of Wakanda before. It looks very interesting and does indeed solve a lot of problems that I mentioned. But as with Opa and Meteor, it seems to want too much for my taste (apart from not being based on NodeJS). What I have in mind is basically a server library that exposes an API with which all the elementary tasks of a web application backend can be performed (see my list above). I don’t want it any simpler than that. I want to be able to run this library anywhere where NodeJS runs (for example, in Cloud9IDE) without having to worry about third party dependencies. But I’ll certainly give Wakanda a try.

    @guigouz Documentation is certainly an issue, but as I said, one problem seems to me that the solution to very generic problems is fragmented into many different and in many cases incompatible modules. You have to pick one, but you can never be sure whether it will be maintained for long.

    @avoidwork Thanks, turtle.io could actually be a “starting point”, but of course, alot would have to be added ;-)

  11. 14 panyasan December 16, 2012 at 11:36 am

    The discussion on google groups turned out really interesting (https://groups.google.com/d/topic/nodejs/jvtwLsQUkv0/discussion). Here is my concluding comment from there:

    @everybody: Thanks for taking time to comment on my post! I really appreciate your input. I still think that there are some misunderstandings, which is my fault, because it seems that I haven’t made my intention clear enough. As a summary, let me try to restate my argument – not because I hope to convince you, but to show you that there is a segment of developers that would profit from what I was trying to outline. This segment — part-time and occasional programmers — is a minority, yes, but it would be able to contribute valuable stuff in return, when being relieved of “The Boilerplate (c)”. ;-)

    I did not mean to say that we need something new to replace the existing stuff (express+middleware, mongoose, passport, etc., etc., etc.) with a monolithic new architecture. On the very contrary, my argument was that it could be worthwhile to make something that you, as full-time developers, do for every new of your projects — the wiring together of all these different parts — into a collaboratively maintained project, which I labeled (or maybe: misnamed) “application server”. The claim is — and here maybe you would strongly disagree — that there is a certain amount of application logic that is the very same for 90% of applications, at least the ones I would like to write (see my post).

    It is not that I am too lazy to do this. I have done it in the past and learned my lesson (see my post). It is that I have too little time to be spending it on a) writing it in the first place, b) writing tests to make sure it is stable, and c) most importantly, maintaining it beyond the time I will be paid (or have free time) to develop the main app. Contracting it out to someone else wouldn’t help, because the problem simply shifts. As soon as you stop paying this person (and nobody would be paying her indefinitely), the backend will become obsolete, the libraries used will go out of maintenance, etc, and nobody is there to fix that.

    So, to sum up: the disagreement is not over whether the node community has already created great tools, which individually will do what I need. It most certainly has. I understand what you’re saying is “if you’re not ready to do everything yourself (i.e. wire the different modules together), don’t be in the business of writing node applications”. That is of course a valid opinion, but as I said, I think that a lot of valuable things could be created if one only had to worry about the “real stuff”, i.e, the business logic specific to the application, and not to the generic parts. I love coding with JavaScript and NodeJS and really think the JavaScript/NodeJS community and ecosystem are awesome. That is why, even if I had the time for it, I wouldn’t want to learn Ruby or switch to Java.

  12. 17 James Wyse December 23, 2012 at 3:05 pm

    This also caught my eye as I’m working on something that you might like:

    https://github.com/lemoncreative/nails/

    There’s no code available yet as I’m doing a major refactor but I should have something usable out in the coming weeks.

    I started this because writing all that boilerplate stuff every time was quite frustrating. I want this to be a system that takes care of all the boilerplate stuff (and more with plugins) leaving you to take care of the important stuff, ie the API and the client-side website. I’m trying to keep it extendable so I don’t make any decisions on templating languages, coffee-script, css preprocessors or front-end frameworks.

    • 18 panyasan December 23, 2012 at 3:22 pm

      This sounds pretty cool and seems to be exactly what I was looking for. Please push that code soon, so we can try it out! :-)

    • 19 panyasan April 9, 2013 at 5:52 am

      James – any news on Nails?

      • 20 James Wyse April 9, 2013 at 6:38 am

        Hello! I’ve been super busy with client work for the past few months so nails has unfortunately been left gathering dust, however I’ve done some work on it over the past couple of weeks and laid a pretty solid foundation for what I want to achieve!

        I’m trying to keep some of my time free to work on it over the next few weeks, I’ll keep you posted!

  13. 21 panyasan April 9, 2013 at 5:52 am

    http://hood.ie seems promising, but is still in developer preview and so far works only on the Mac.

  14. 22 panyasan May 6, 2013 at 7:15 pm

    The NoBackend initiative seems to have similar goals: provide developers with simple APIs to do the basic, trivial stuff like user management:

    http://nobackend.org

  15. 23 panyasan May 29, 2013 at 9:36 am

    Josha Holbrook argues that some sort of NodeJS framework would help manage the module fragmentation: http://jesusabdullah.github.io/2013/05/28/the-case-for-a-nodejs-framework.html

  16. 24 panyasan January 20, 2014 at 8:11 pm

    Haven’t tried it out, but looks like it takes care of authentication duties: http://mircozeiss.com/lockit-an-express-authentication-solution/

    BTW, http://hood.ie is no longer “developer preview”, so go check it out.

    Finally, similar point as my article, much better phrased: http://blog.hood.ie/2014/01/the-hoodie-why-1-we-have-a-dreamcode

  17. 25 panyasan February 8, 2014 at 9:52 am

    DreamFactory looks like an interesting option, especially since–unlike other commercial offerings–the code is open source and can be installed on your own server.

    http://www.dreamfactory.com
    http://blog.dreamfactory.com/rise-of-the-service-platform


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

%d bloggers like this: