Archive for the ‘Ruby’ Category

Job ad: Music service developer

Thursday, August 14th, 2008

We’re hiring!

Playlouder MSP has been working with ISPs and the music industry to develop both an innovative business model for music consumption, and innovative user experiences around music and communication to complement ISPs’ offerings.

As a key addition to our small but growing development team, you will be critical in helping to refine, scale and roll out our application and service to white-label ISP clients.

Key requirements

  • Background in computer science, mathematics, software engineering or similar (degree or equivalent experience)
  • A solid technical all-rounder with software development experience on sizeable projects

The specifics

Our work involves all of the following, and you’ll be tackling problems involving many of them:

  • Dynamic languages (experience with Ruby, on which our current implementation is based, would be particularly desirable)
  • Rich web application development with a large Javascript-based client-side portion
  • A modular, widget-based user interface framework for the above
  • Databases (MySQL at present), data modeling and ORM tools
  • Unix-based deployment environments
  • Large volumes of media, media metadata and usage statistics
  • Web services APIs, and large-scale integration work
  • Server push technologies, and scaling applications with a live messaging component
  • Common agile software development tools, processes and techniques – source control, bug tracking, testing etc
  • A warehouse full of music geeks :)

Some other things which you might get to play with:

  • Other languages – Java, Python, C, possibly Erlang (see ‘messaging’ above), …
  • Messaging technologies like AMQP, ActiveMQ, XMPP
  • Music technology R&D projects
  • Lots more in a fast-growing company

In return we can offer challenging problems in an interesting domain, competitive pay, and a great work environment for music fans in our E2 warehouse!

Enquiries to matthew@playlouder.com

Posted in Business, Development, Jobs, Ruby | 2 Comments »

Of MySQL/Ruby, EventMachine, and the need for non-blocking APIs

Monday, May 12th, 2008

Part of the service we’re building is a socket server which uses Flash’s XMLSocket API to push updates to clients. Initially we developed this using the excellent Twisted library in Python, but as it grew, having to duplicate some of our data model code in another language started to hurt, and it made sense for us to port it to Ruby.

Luckily by that point, the EventMachine library had sprung up, offering something very similar to Twisted for Ruby, and we’ve been using that since.

While it’s well known that Ruby’s threading is non-native and not particularly speedy, event-based libraries don’t actually require much use of threading – one is encouraged to structure ones code in such a way that you write small methods which are called asynchronous and return quickly, yielding back to the event loop. For those with client-side experience, this is quite comparable with Javascript runtimes, where there is no threading but a core event loop, the ability to register event handlers, call setTimeout, and asynchronous APIs for longer-running IO (AJAX anyone?).

For this to work well, it is essential that your event handlers do their business as quickly as possible, and yield back to the event loop – as everything else in the event queue is sat there waiting for you to finish. This is all very well, until you need to deal with IO – other things (pesky database servers and clients) have a nasty habit of taking a while to get back to you, and if the API you’re calling to communicate with them blocks you, then it’s blocking everything else in the event queue too.

One way to get around this (despite the concurrency paradigm being based around an event-loop rather than pre-emptive threading), is to have some spare threads lying around to take care of blocking API calls, and fire off an event to the core event loop thread when they’re done. A way of turning a blocking API call into a non-blocking one, something asynchronous. While Ruby’s threads aren’t native or very performant, this shouldn’t matter too much in this case, as the threads aren’t really being used to do very much – just to sit around waiting for IO.

While this doesn’t require an asyncronous API at the Ruby level, it does at least require that the API calls only block the current Ruby thread, and don’t require an interpreter-wide lock in order to go about their business.

Unfortunately, it seems that many (most?) C-based Ruby libraries, including MySQL/Ruby (rather crucial to many), block the whole interpreter while waiting for IO, because they aren’t able to yield to Ruby’s “green” threading code while calling a blocking C API. This is hard to work around unless there’s an non-blocking C API available (which there isn’t, currently, for MySQL, but is for Postgres, hence the non-blocking postgres Ruby library). It may be possible for the C extension to use a separate OS-level thread for the blocking API calls, but as I understand it, one has to be very careful when using multiple OS-level threads in a process which embeds the Ruby interpreter, as the interpreter is not natively-threadsafe in the least.

Anyway the unfortunate upshot of all this is that you can have as many Ruby threads as you like, but only one MySQL query will ever happen at a time. If you don’t believe me, try firing off a Thread.new { connection.execute(“sleep 10″) } and then see if you have any joy querying MySQL in the next 10 seconds. Even with a connection pool, you’re shit outta luck.

This kind of thing rather removes the whole point and usefulness of event-loop based libraries like EventMachine when used with MySQL, and makes ActiveRecord’s specially-thread-safe “allow_concurrency” option considerably less use when used with the MySQL adapter – if all the mysql query grunt work ends up serialized anyway, why bother using threads?

So, there’s a real need for non-blocking APIs, and for Ruby library writers, and (perhaps more critically) those working on the new round of Ruby implementations, to get serious about this if they want Ruby libraries to  be able to get anything out of sub-process-level concurrency.

There’s also a real need for an asynchronous C API for MySQL which Ruby library authors could use. This project appears to have been trying – looking forward to progress!

Tags: , , , , , , , , , , , , , ,
Posted in Development, Geekery, Ruby | 6 Comments »

An interesting Ruby hash semantics gotcha

Wednesday, May 7th, 2008

Thought this might amuse or perplex some Rubyists (or be useful to know – it’s been the source of a couple of hard-to-track-down bugs in the past).

>> {{} => true}[{}]
=> nil

>> {{} => true, {} => true}
=> {{}=>true, {}=>true}

but yet,

>> {} == {}
true

What’s going on here?

Ruby’s Hashes behave very strangely when you try to use a Hash itself as a key of a Hash.

This acts as a subtle gotcha when you try to memoize a function which takes hash arguments – and so a tricky-to-address bug in libraries like this: http://raa.ruby-lang.org/project/memoize/

Why?

Ruby calls Object#hash on each key of a Hash, using that numeric hash (small h) to allocate that object to a bucket of the underlying hash table data structure. Equality, when it comes to Hash lookups and unique keys of a Hash, will only happen if the keys generate the same numeric hash as a result of their hash methods.

For most ruby data structures, x.hash == y.hash is implied by x == y, and everything works fine.
But, not for Hashes themselves!

(NB. this also affects data structures like Arrays which themselves contain a Hash, since Array#hash must call hash recursively on its contents).

(Interestingly, for things like 1.0 == 1, x.hash == y.hash also fails. Note, x.hash == y.hash is always implied by x.eql?(y), but this equality isn’t a desperately useful one, and seems to have been constructed artificially as an equality for use with Hash which is consistent with .hash)

Why might it have been implemented this way?

Hashes are insensitive to the order of their keys – so, for example, we have:
{:a => true, :b => true} == {:b => true, :a => true}

When you’re actually being given two concrete objects to compare, you can just check that each key from the one has an equal corresponding value in the other, and vice versa.

But, when you’re asked to generate a numeric hash which is constant for the whole equivalence class, you’d have to do something to ensure the hashing isn’t order-sensitive. Like ordering the key/value pairs by their individual hashes before feeding into the hash function.

Some attempts at a fix in the form of a monkey-patched Hash#hash:

(yes, that’s pronounced ‘Hash hash hash’)

  1. Sort key/value pairs by the numeric hash of the pair first: 
    class Hash
 def hash
   sort_by {|pair| pair.hash}.hash
 end

 def eql?(other)
   self == other
 end
end
  2. Use an XOR of the hashes of the key/value pairs (XOR is order-insensitive, and should preserve entropy in the bits of the hash) 
    class Hash
 def hash
   inject(0) {|hash,pair| hash ^ pair.hash}
 end

 def eql?(other)
   self == other
 end
end

These then fix, eg:

>> {}.hash == {}.hash
=> true

>> {{} => true}[{}]
=> true

>> {{} => true, {} => false}
=> {{}=>false}

(Note, overriding eql? is required to make the last two work – it seems the Hash implementation uses eql? to do the equality comparison that follows the more approximate hash comparison)

Now, I’m sure there’s a reason Matz didn’t do it this way – perhaps a performance reason, perhaps a gotcha that I haven’t noticed with my approach. Perhaps it’ll be fixed in 1.9.
But at any rate, it’s useful to be aware of the issue.

Posted in Development, Geekery, Ruby | 4 Comments »

About Brix

Tuesday, March 11th, 2008

Recently I’ve been working on a framework called ‘Brix’, which may interest those who saw my rather hastily-prepared LRUG talk last year. It’s another Ruby web framework – I know, I know – why yet another? Here’s an idea of the philosophy:

  • Ruby needs a component-based web framework, to compete with the likes Tapestry, Seaside and WebObjects
  • Separation, composability and loose coupling of components are more important for agile application development, than rigid separation of View and Controller
  • Components have lives on the client-side as well as the server side, and the server-side needs to handle javascript and css includes, and the instantiation of client-side javascript objects, with the minimum of hassle. The client-side widget tree, in turn, needs to know how to update itself and manage its state.
  • Components take parameters. Components may be nested inside other components. Components may be requested on their own (an Ajax update?) or as part of a bigger component tree. This has big implications for the routing component of a framework.
  • Trying to adhere too religiously to what is typically a muddled interpretation of MVC, is often counter-productive
  • REST is wonderful for APIs, but it is the wrong paradigm for modular web application user interfaces in general. (It copes OK for a CRUD-style UI structure which is closely coupled to the data model, but this isn’t typical of more dynamic web applications in my experience)
  • The back button, and bookmarking, need to work!
  • It would be nice if Google can spider the application, in addition to DHTML clients

The good news is that I’ve found that it’s been relatively easy to get started on, thanks to some of the great tools the Ruby community already has available. I’m only reinventing the parts of the wheel that were creaking badly – for the rest, I’m relying on:

  • Haml (take the hour’s time to learn this, it’s really clean, and especially well suited to programmatic generation of small chunks of DOM tree)
  • Rack ontop of Mongrel
  • ActiveSupport
  • for the time being, ActiveRecord (this is due for the chop once I find something closer to the relational model, or heck, something which can do Class Table Inheritance in something resembling an elegant fashion)
  • Bits cheekily stolen from Merb. I was close to building this entirely ontop of Merb, but I ran into some nasty segfaults on Leopard, it didn’t seem to play well with ActiveSupport, its Router would have needed replacing, and Merb’s controllers, while more lightweight than rails, still got in the way of entirely component-based dispatch. But I’m still really impressed with Merb – It’s like Rails done right – leaner, faster, without the cruft, and with the benefit of hindsight. Easier to extend, too.)

Posted in Development, Geekery, Ruby, Software architecture, Uncategorized, User interface | No Comments »