COMET: A Hack for the Broken Web

This post provides insight into some ingenious engineering at Facebook. Erlang is the perfect language to implement the server side of COMET's long-polling with all its simultaneous connections.

However, all of that and COMET itself, is only necessary because of the fact that the World Wide Web is unfinished! In its current state the Web is BROKEN!

Most of the clients out there are not network nodes. They are half-nodes. They can initiate connections out, but do not accept incoming connections (on the WWW port, 80). Unlike the good old telephone network where every phone can call every other phone, the Web is like a television network; you can request and receive their programming, but they are not interested in receiving your programming!

What does this mean for web applications? Well, a chat application is a perfect example. Each client wants real-time updates of the status of its buddies. But since a server can't really push data to a typical client, the client has to poll. Of course polling is so inefficient that brilliant engineers have had to come up with things like long-polling, COMET, hugely distributed web servers, and the like--all of which are still pretty inefficient.

Wouldn't it be nice if each client were a real node? It could PUT its status to each of its buddy nodes any time its status changed. It could POST new chat messages directly to the buddy node. The centralized server would have far fewer responsibilities, like maintaining your buddy list and the address of the node you're logged in at.

What would be necessary to get the Web finished? Among other things, IPV6 or some other addressing scheme to get all the nodes uniquely addressed; new firewall and security rules and protocols to keep the bad guys at bay, and renewed interest in using HTTP to its full potential (using REST of course).

Want to Be a Better/Faster/Smarter Programmer? Learn a New Language!

For years most of the projects I worked on were written in C. I thought I was a pretty good C programmer. Then I had to move fully into the Java world. I grumbled a little because I was no Java expert, but once I learned it I found I would much rather write Java code than C. But you know what else I discovered? When I did go back to writing or maintaining C code, I did it a little differently. I took a more object-oriented approach and wrote more re-usable libraries. (Like my vector.c module that emulated Java's Vector class.)

After a while I began to learn Ruby and that's when several ideas that should have already been ingrained really started to click. I realized that handling exceptions like this:

try {
some_method()
} catch (Exception e) {
write_to_logfile(e);
return false;
}

is almost always WRONG. You should be probably be throwing that error up the stack to be handled at a high level.

Ruby's awesome Test::Unit module woke me up to value of automated unit testing. Instead of doing unit testing as an afterthought, I began thinking about unit tests much earlier in the process. I've learned to use JUnit for Java and even written tests first and the class second a couple of times!

Coding Ruby has got me wanting to write small, easy to test classes made of small, easy to test methods, and modularize to a degree I haven't in the past.

As a result of better modularization and unit testing, my Java and Ruby classes are much more reliable from the moment of release and require far less debugging (and apologizing to users) post-release.

What next? Erlang! What a great languange. And what a great book Joe Armstrong has written in Programming Erlang: Software for a Concurrent World.

I'm not yet writing production code in Erlang, but now, whatever the language, I'm seeking more elegant solutions using recursion, thinking more about concurrency and shared data, doing actual top-down coding, and trying to think of solutions functionally.