Erlang Makes My Head Hurt

For those of you who haven’t heard about Erlang yet, it is a functional programming language (like Lisp or Prolog) developed quietly over the last 20 years by telecoms giant Ericsson for use in telco switches.

Ericsson has been using it for roughly the last 14 years; it has several properties that make it particularly relevant to many of the problems facing developers today. It’s one of the few languages that is particularly good at letting programmers take advantage of multi-core/multi-CPU systems, and distribute services across multiple boxes. YAWS, a webserver written in Erlang and a poster child of its efficiencies, kicks Apache’s tail from a scalability standpoint. This is no small accomplishment for a high level language.

Today’s scaling strategies revolve less around faster clock speeds and more around adding cores. Scaling out to many machines is also important, but power and space considerations are also more of an issue than ever before.

So Erlang is gaining ground because it addresses scalability for this new age of multi-core systems. Today you might have a dual Clovertown Xeon box with 8 cores, but very little software to take advantage of it. Once you get past 2 or 4 cores, that extra capacity provides little to no benefit. Enter a language like Erlang, and suddenly all that power becomes available to the programmer.

Some of my coder buddies, (Jay Phillips, Rich Kilmer and Marcel Molina) have been looking at Erlang for various tasks, and Dave Thomas’ blog posts on Erlang have also inspired me to take a look at the language for some of my own work.

I picked up Joe Armstrong’s book, Programming Erlang from Pragmatic Bookshelf and started reading it on a recent airplane flight.

Today I put together my first Erlang program, based on knowledge gleaned from Dave Thomas’ postings and from the book.

This very simple program grabs the top_rated feed of videos from YouTube (an XML RSS feed) and then iterates through the result set to get the profile URL for each user. It is a fairly useless and trivial example, but if I can make this work then there are other things I can do down the line.

-module(youtube).  -export([fetch_each_user/0]).  -include_lib("xmerl/include/xmerl.hrl").

 get_feed() ->  { ok, {_Status, _Headers, Body }} = http:request("http://gdata.youtube.com/feeds/standardfeeds/top_rated"),  { Xml, _Rest } = xmerl_scan:string(Body),    xmerl_xpath:string("//author/name/text()", Xml).

 get_user_profile(User) ->  {_,[A|B],_,[],Name,_} = User,  URL = "http://gdata.youtube.com/feeds/users/" ++ Name,  { ok, {_Status, _Headers, Body }} = http:request(URL),  { Xml, _Rest } = xmerl_scan:string(Body),  [{_,[C|D],_,[],Id,_}] = xmerl_xpath:string("//id/text()", Xml),  { Name, Id }.

 fetch_each_user() ->  lists:map(fun get_user_profile/1, get_feed()).

I am pretty sure I am doing this All Wrong ™.

My biggest area of confusion comes in the pattern matching that’s required to match (and thus read) the results from the xmerl_xpath:string parsing. According to Dave Thomas’ examples, xmerl_xpath should produce a #xmlText record (or a set of them) that can then be matched with the #xmlText{} syntax.

In practice, and with the Youtube API data I used, I see no such #xmlText records. Instead I get a flattened tuple from such parsing, along the lines of:

>xmerl_xpath:string("//location/text()", Xml)[{xmlText,[{'yt:location',14},{entry,1}],1,[],"GB",text}]

The only way I can find to match this is something like this:

[{_,[A|B],_,[],Location,_}] = xmerl_xpath:string("//location/text()", Xml)

I am sure I am missing some key step or concept, but that’s how we learn new languages — stumble along til we figure out how to solve the things we want to solve.

There’s an incredible amount of functionality packed into these 19 lines of code. It’ll be even more amazing when I figure out my initial questions and then add concurrent processing of the user profile URLs. In theory I can simultaneously process dozens of URL feeds from Youtube and spider their API data as though through a fire hose. Stay tuned.

Meantime if anyone has any suggestions on my current puzzlements I’d love to hear them.

Erlang is a cool language. It doesn’t give me the aesthetic fuzzies I receive from programming in Ruby, but I do get pretty jazzed up thinking about what should be possible from a performance and economy standpoint. Erlang doesn’t allow for mutable state; variables have fixed values once assigned, and algorithms are generally handled via recursion. This is how it scales out to so many cores/cpus/machines so readily. It’s kinda weird if you’re used to “normal” mutable state languages.

Whenever I learn a new language (human or computer) I generally have weird and overactive dreams. I attribute this to my brain shuffling things around to accommodate new grammar and semantics.

The last few days have produced particularly vivid dreams.

MoMA NY Selects Twittervision & Flickrvision

Yesterday, I received final confirmation that the Museum of Modern Art in New York has selected my mash-ups twittervision.com and flickrvision.com for its 2008 exhibition Design and the Elastic Mind.

I’m certainly very flattered to be included and have never considered myself to be an artist. I didn’t seek out MoMA on this. I am just very, very happy to have an opportunity to participate in a small way in the ongoing dialog about what technology means for humanity. Crap. Now I sound like an artist.

Incidentally, this means that twittervision.com and flickrvision.com are the first ever Ruby On Rails apps to be included in a major art exhibition. I already told DHH.

Anyway, at RailsConf Europe a few weeks ago, Dave Thomas’ keynote speech emphasized the role of software designers as artists. He said, “treat your projects as though they are artworks, and sign your name to them.” Or pretty close to it. I think this is incredibly valuable advice for software designers today.

We’re past the days of using machines as amplifiers of our physical efforts. It’s not enough to jam more features into code just so we can eliminate one more position on the assembly line. We’re at a point where the machines can help amplify our imaginations.

Today, creativity and imagination (what some folks are calling the right brain) are becoming the key drivers of software and design. With imagination, we can see around the corners of today’s most pressing challenges. While technical skill is certainly valuable, if it’s applied to the wrong problems, it’s wasted effort.

Creativity, imagination, and artistry help us identify the areas where we should put our efforts. They help us see things in new ways.

Everywhere I turn (perhaps partly because I am a Rubyist), I hear discussions of Domain Specific Languages, and of framing our problems in the right grammars.

This is hugely valuable because the creative part of our brain thinks in terms of semantics, grammars, and symbols. If we can’t get the words right, our imaginations can’t engage.

Everything stays stuck in the left side of our brains when we have to jump through hoops to please some particular language or development environment.

I hope you all will come out to see Design and the Elastic Mind when it opens at NYC MoMA, Feb 24 – May 12 2008. I’m not sure how we’re going to present the sites but we’re going to see if we can get some partners and sponsors involved to do something really beautiful.

And again, thanks to MoMA for the selection. And here’s to creativity, imagination, and artistry as the next big thing in software design!