Archive for the ‘Geekery’ Category

pmacct – my new best friend.

Saturday, October 24th, 2009

How do you manage your transit data? If you source your traffic through more than one upstream ISP – and you should – then you need a way to view those relationships in detail, and make sure you are receiving your best value for money. However, it’s not always easy convincing your manager (or your mother) why Internet traffic costs money (and how it is priced).

You need a way to see what and where your routers are sending traffic. What I’m talking about here is Netflow. There is a wealth of information about this once-upon-a-time Cisco protocol, and it’s neighbours, jFlow, sFlow and NetStream.

I’ve tried so many software packages to make sense from my Netflow. OSS command-line based to huge Java run pay-per-port commercial packages. Not one has ever been agile enough (without substantial hacking) to document what is really happening at my network borders. The reason being that I don’t always control all our traffic on the edge. How can I account for the traffic that isn’t routed via my border routers?

Recently I saw on NANOG (or was it c-nsp list?) another vague request for network management tools. I found pmacct through the author, Paolo Lucente. He convinced me in one email that his OSS product (developed since 2003) would fit my needs. I’m going to show you via this HOWTO, as there isn’t that much newbie information in the googlesphere.

Our network runs like a bit this: border routers have full BGP routes from multiple transits. This services most of our network. network diagram
We also have some servers connected directly to an ISP, as they push out megabandwidth compared with what our routers are capable of. I give the servers a default route to some IP space hosted by the ISP which has bigger routing iron. They provide a connection with an HSRP default gateway that I can plug these machines directly into. This way we don’t need to invest in huge amounts of money for those big name routers.

The problem with these latter machines, is that they are not aware of the Internet. They just punt a packet towards a gateway, and get a packet back. I want to be able to see where these packets are going (ie. the ASN), but our ISP doesn’t provide Netflow for us. This is the crux of why all other Netflow accounting packages haven’t worked for me.

I have a basic setup on these machines. The short story is that within their ufw/iptables rules I insert a ULOG rule for all traffic on their “external” interfaces. Then I run a program called fprobe-ulog which interprets the basics of these ULOG messages and respews Netflow packets towards my collector.

Before, I would script the netflow records from these machines and match it against a BGP table, while my name-brand routers were lovingly supported by name-brand software packages. pmacct will aggregate all this functionality into one lightweight, stable and scaleable system.

Now onto setting up pmacct.

  1. Enable netflow on your interfaces with which you peer:
    Router(config-subif)#ip flow ingress
    Router(config-subif)#ip flow egress
    Router(config-subif)#ipv6 flow egress
    Router(config-subif)#ipv6 flow ingress
    Router(config-subif)#exit
    Router(config)#ipv6 flow-export destination 192.168.1.111 3001
    Router(config)#ipv6 flow-export version 9 bgp-nexthop
    Router(config)#ipv6 flow-export source Loopback1
    Router(config)#ip flow-export destination 192.168.1.111 3001
    Router(config)#ip flow-export source Loopback1
    Router(config)#ip flow-export version 5 origin-as bgp-nexthop
    


    without the last line your netflow will not have the ASN information within the Netflow packets.

  2. Make sure netflow packets are arriving at your collector:
    # tshark port 3001
    Running as user "root" and group "root". This could be dangerous.
    Capturing on eth0
      0.000000  94.228.64.1 -> 192.168.1.111 UDP Source port: 51901  Destination port: 3001
      0.000384 94.228.64.41 -> 192.168.1.111 UDP Source port: 57689  Destination port: 3001
      0.980030  94.228.64.2 -> 192.168.1.111 UDP Source port: 58299  Destination port: 3001
      0.980253 94.228.64.42 -> 192.168.1.111 UDP Source port: 56640  Destination port: 3001
      0.999718  94.228.64.1 -> 192.168.1.111 UDP Source port: 51901  Destination port: 3001
    5 packets captured
    # 

  3. Download and compile libbgpdump.
    # wget http://www.ris.ripe.net/source/libbgpdump-1.4.99.10.tar.gz
    # tar xvfz libbgpdump-1.4.99.10.tar.gz
    # cd libbgpdump-1.4.99.10
    # apt-get install build-essential libbz2-dev
    # ./configure
    # make
    

  4. download the DFZ routes de jour from RIPE.
    # wget http://data.ris.ripe.net/rrc00/2009.10/bview.20091024.0759.gz
    # gunzip bview.20091024.0759.gz
    

  5. parse the binary bview format, strip out any bogus lines, and reformat it as “$ASN,$PREFIX”:
    # ./libbgpdump-1.4.99.10/bgpdump -m bview.20091024.0759 > \
    ~/bgptable.20091024.0759.dump
    # awk -F\| '$0 !~ /INCOMPLETE/ {print $6,$7}' bgptable.20091024.0759.dump\
    |awk '{ print $NF "," $1 }'|uniq > bgptable.20091024.0759
    

  6. Now install pmacct! (i just leave mysql root blank for testing)
    # apt-get install libmysqlclient15-dev mysql-server libpcap-dev
    # wget http://www.pmacct.net/pmacct-0.12.0rc2.tar.gz
    # tar xvfz pmacct-0.12.0rc2.tar.gz
    # cd pmacct-0.12.0rc2
    # ./configure  --enable-mysql --enable-ipv6
    # make install
    # mysql < sql/pmacct-create-db_v6.mysql
    # mysql < sql/pmacct-grant-db.mysql
    


    This gets you up to the point of having a working mysql table for nfacctd to insert the data into, the next step I'll walk through some the settings and how-it-works.

  7. Now you are ready to create your config. pmacct has 3 daemons; sfacctd, pmacctd, nfacctd and the pmacct command line client.
    sfacctd is for sFlow. sFlow usually comes from hardware samplers on L3 switches. I don't use this, but I will have to in the future - glad to know its there.
    pmacctd is for either running on a client machine in promiscuous mode (a bit like i do with the ULOG and fprobe) or using it as a Netflow aggregator which can normalize your flows in all sorts of weird and wonderful ways, and then forward it back onto a Netflow datastore. Useful - but not for my basic setup.
    nfacctd collects the Netflow data. This is what I'm using.
    pmacct is the cli interface to the memory plugin. We are using the MySQL plugin - but to get us up and running we will start with the memory plugin.

    Let's create our config file:

    # vim /etc/nfacctd.conf
    
    daemonize: true
    aggregate: src_as,dst_as,dst_host,src_host,flows,dst_port,src_port,proto

    The values for aggregate are the sources in the netflow datagrams that you are interested in storing. This is pretty self-evident.

    ! plugin_buffer_size: 1024
    nfacctd_port: 3001
    nfacctd_time_secs: true
    nfacctd_time_new: true
    ! read bgp table from here..
    nfacctd_as_new: file
    networks_file: /home/charlie/bgptable.20091024.0759
    plugins: memory
    sql_db: pmacct
    sql_table: acct_v6
    sql_table_version: 6
    sql_passwd: arealsmartpwd
    sql_user: pmacct
    sql_refresh_time: 30
    sql_history: 10m
    sql_history_roundoff: m

    Keep the buffer at nil until you are in production mode.
    Set your listening UDP port (3001).
    Configure the path to the BGP table we built before.
    Set the plugin to be "memory".
    The rest commented will remain unused until you are ready to store data in MySQL. Now let's try it out!

  8. # nfacctd -f /etc/nfacctd-mysql_v6.conf
    # pmacct -s
    SRC_AS  DST_AS  SRC_IP          DST_IP          SRC_PORT  DST_PORT  PROTOCOL  PACKETS   FLOWS   BYTES
    6849    47998   94.179.57.134   94.228.76.67    3153      445       tcp       2         1       96
    786     47998   131.251.141.20 193.34.28.19   4782      80        tcp       5         1       1152
    ^C

    woo! There is data! I can even see the external routes being populated with the correct ASN's. Go back to your config and change memory to mysql and you're done. If you want to see it not add the ASN's then change nfacctd_as_new: file to nfacctd_as_new: false.

    There is a great resource in the distribution called CONFIG-KEYS which will help you in figuring out your nfacctd.conf and README.mysql to understand how to expand upon this basic setup.

  9. From here on in you can start to create reports via the mysql cli or from some pre-made web front ends. Flox is a basic one. Here is a screenshot showing some IRC servers.

    flox

    You really must try this out. Paolo has kept the programs really easy and maleable and he promises me that I can soon discard my ULOG for a native pmacctd ULOG implementation.

    More information about our network is of course available from RIS, DB and here.

Of MySQL/Ruby, EventMachine, and the need for non-blocking APIs

Monday, May 12th, 2008

Part of the service we’re building is a socket server which uses Flash’s XMLSocket API to push updates to clients. Initially we developed this using the excellent Twisted library in Python, but as it grew, having to duplicate some of our data model code in another language started to hurt, and it made sense for us to port it to Ruby.

Luckily by that point, the EventMachine library had sprung up, offering something very similar to Twisted for Ruby, and we’ve been using that since.

While it’s well known that Ruby’s threading is non-native and not particularly speedy, event-based libraries don’t actually require much use of threading – one is encouraged to structure ones code in such a way that you write small methods which are called asynchronous and return quickly, yielding back to the event loop. For those with client-side experience, this is quite comparable with Javascript runtimes, where there is no threading but a core event loop, the ability to register event handlers, call setTimeout, and asynchronous APIs for longer-running IO (AJAX anyone?).

For this to work well, it is essential that your event handlers do their business as quickly as possible, and yield back to the event loop – as everything else in the event queue is sat there waiting for you to finish. This is all very well, until you need to deal with IO – other things (pesky database servers and clients) have a nasty habit of taking a while to get back to you, and if the API you’re calling to communicate with them blocks you, then it’s blocking everything else in the event queue too.

One way to get around this (despite the concurrency paradigm being based around an event-loop rather than pre-emptive threading), is to have some spare threads lying around to take care of blocking API calls, and fire off an event to the core event loop thread when they’re done. A way of turning a blocking API call into a non-blocking one, something asynchronous. While Ruby’s threads aren’t native or very performant, this shouldn’t matter too much in this case, as the threads aren’t really being used to do very much – just to sit around waiting for IO.

While this doesn’t require an asyncronous API at the Ruby level, it does at least require that the API calls only block the current Ruby thread, and don’t require an interpreter-wide lock in order to go about their business.

Unfortunately, it seems that many (most?) C-based Ruby libraries, including MySQL/Ruby (rather crucial to many), block the whole interpreter while waiting for IO, because they aren’t able to yield to Ruby’s “green” threading code while calling a blocking C API. This is hard to work around unless there’s an non-blocking C API available (which there isn’t, currently, for MySQL, but is for Postgres, hence the non-blocking postgres Ruby library). It may be possible for the C extension to use a separate OS-level thread for the blocking API calls, but as I understand it, one has to be very careful when using multiple OS-level threads in a process which embeds the Ruby interpreter, as the interpreter is not natively-threadsafe in the least.

Anyway the unfortunate upshot of all this is that you can have as many Ruby threads as you like, but only one MySQL query will ever happen at a time. If you don’t believe me, try firing off a Thread.new { connection.execute(“sleep 10″) } and then see if you have any joy querying MySQL in the next 10 seconds. Even with a connection pool, you’re shit outta luck.

This kind of thing rather removes the whole point and usefulness of event-loop based libraries like EventMachine when used with MySQL, and makes ActiveRecord’s specially-thread-safe “allow_concurrency” option considerably less use when used with the MySQL adapter – if all the mysql query grunt work ends up serialized anyway, why bother using threads?

So, there’s a real need for non-blocking APIs, and for Ruby library writers, and (perhaps more critically) those working on the new round of Ruby implementations, to get serious about this if they want Ruby libraries to  be able to get anything out of sub-process-level concurrency.

There’s also a real need for an asynchronous C API for MySQL which Ruby library authors could use. This project appears to have been trying – looking forward to progress!

An interesting Ruby hash semantics gotcha

Wednesday, May 7th, 2008

Thought this might amuse or perplex some Rubyists (or be useful to know – it’s been the source of a couple of hard-to-track-down bugs in the past).

>> {{} => true}[{}]
=> nil

>> {{} => true, {} => true}
=> {{}=>true, {}=>true}

but yet,

>> {} == {}
true

What’s going on here?

Ruby’s Hashes behave very strangely when you try to use a Hash itself as a key of a Hash.

This acts as a subtle gotcha when you try to memoize a function which takes hash arguments – and so a tricky-to-address bug in libraries like this: http://raa.ruby-lang.org/project/memoize/

Why?

Ruby calls Object#hash on each key of a Hash, using that numeric hash (small h) to allocate that object to a bucket of the underlying hash table data structure. Equality, when it comes to Hash lookups and unique keys of a Hash, will only happen if the keys generate the same numeric hash as a result of their hash methods.

For most ruby data structures, x.hash == y.hash is implied by x == y, and everything works fine.
But, not for Hashes themselves!

(NB. this also affects data structures like Arrays which themselves contain a Hash, since Array#hash must call hash recursively on its contents).

(Interestingly, for things like 1.0 == 1, x.hash == y.hash also fails. Note, x.hash == y.hash is always implied by x.eql?(y), but this equality isn’t a desperately useful one, and seems to have been constructed artificially as an equality for use with Hash which is consistent with .hash)

Why might it have been implemented this way?

Hashes are insensitive to the order of their keys – so, for example, we have:
{:a => true, :b => true} == {:b => true, :a => true}

When you’re actually being given two concrete objects to compare, you can just check that each key from the one has an equal corresponding value in the other, and vice versa.

But, when you’re asked to generate a numeric hash which is constant for the whole equivalence class, you’d have to do something to ensure the hashing isn’t order-sensitive. Like ordering the key/value pairs by their individual hashes before feeding into the hash function.

Some attempts at a fix in the form of a monkey-patched Hash#hash:

(yes, that’s pronounced ‘Hash hash hash’)

  1. Sort key/value pairs by the numeric hash of the pair first:
    class Hash
     def hash
       sort_by {|pair| pair.hash}.hash
     end
    
     def eql?(other)
       self == other
     end
    end
  2. Use an XOR of the hashes of the key/value pairs (XOR is order-insensitive, and should preserve entropy in the bits of the hash)
    class Hash
     def hash
       inject(0) {|hash,pair| hash ^ pair.hash}
     end
    
     def eql?(other)
       self == other
     end
    end

These then fix, eg:

>> {}.hash == {}.hash
=> true

>> {{} => true}[{}]
=> true

>> {{} => true, {} => false}
=> {{}=>false}

(Note, overriding eql? is required to make the last two work – it seems the Hash implementation uses eql? to do the equality comparison that follows the more approximate hash comparison)

Now, I’m sure there’s a reason Matz didn’t do it this way – perhaps a performance reason, perhaps a gotcha that I haven’t noticed with my approach. Perhaps it’ll be fixed in 1.9.
But at any rate, it’s useful to be aware of the issue.

About Brix

Tuesday, March 11th, 2008

Recently I’ve been working on a framework called ‘Brix’, which may interest those who saw my rather hastily-prepared LRUG talk last year. It’s another Ruby web framework – I know, I know – why yet another? Here’s an idea of the philosophy:

  • Ruby needs a component-based web framework, to compete with the likes Tapestry, Seaside and WebObjects
  • Separation, composability and loose coupling of components are more important for agile application development, than rigid separation of View and Controller
  • Components have lives on the client-side as well as the server side, and the server-side needs to handle javascript and css includes, and the instantiation of client-side javascript objects, with the minimum of hassle. The client-side widget tree, in turn, needs to know how to update itself and manage its state.
  • Components take parameters. Components may be nested inside other components. Components may be requested on their own (an Ajax update?) or as part of a bigger component tree. This has big implications for the routing component of a framework.
  • Trying to adhere too religiously to what is typically a muddled interpretation of MVC, is often counter-productive
  • REST is wonderful for APIs, but it is the wrong paradigm for modular web application user interfaces in general. (It copes OK for a CRUD-style UI structure which is closely coupled to the data model, but this isn’t typical of more dynamic web applications in my experience)
  • The back button, and bookmarking, need to work!
  • It would be nice if Google can spider the application, in addition to DHTML clients

The good news is that I’ve found that it’s been relatively easy to get started on, thanks to some of the great tools the Ruby community already has available. I’m only reinventing the parts of the wheel that were creaking badly – for the rest, I’m relying on:

  • Haml (take the hour’s time to learn this, it’s really clean, and especially well suited to programmatic generation of small chunks of DOM tree)
  • Rack ontop of Mongrel
  • ActiveSupport
  • for the time being, ActiveRecord (this is due for the chop once I find something closer to the relational model, or heck, something which can do Class Table Inheritance in something resembling an elegant fashion)
  • Bits cheekily stolen from Merb. I was close to building this entirely ontop of Merb, but I ran into some nasty segfaults on Leopard, it didn’t seem to play well with ActiveSupport, its Router would have needed replacing, and Merb’s controllers, while more lightweight than rails, still got in the way of entirely component-based dispatch. But I’m still really impressed with Merb – It’s like Rails done right – leaner, faster, without the cruft, and with the benefit of hindsight. Easier to extend, too.)

Minification

Tuesday, January 8th, 2008

In case anyone noticed, we’ve done a bit of client-side optimization. Namely:

  • Javascript and CSS files are now ‘minified’ (I prefer ‘squished’) as part of our build process, using the handy YUI compressor – this shaves a good 40% of bloat off our Javascript and 20% off our CSS, and makes them ever so slightly quicker to parse at the other end too
  • Common Javascript and CSS includes are now packaged up into combined packages, which saves a lot of HTTP requests
  • Javascript includes have moved to the bottom of the page, meaning they won’t delay page rendering

Some stuff we were already doing:

  • Far-ahead Expires headers on all static resources, meaning they’ll be filled from the browser’s cache without any HTTP request where possible
  • Gzipping static files with lighttpd – this shaves a good 76% off our Javascript for example (and still 74% off the minified javascript). It also shaves 82% off CSS, and interestingly manages to shave even more (83%) off minified CSS – indicating that stripping syntactically-irrelevant information actually makes the remaining data more amenable to compression in this case.

Some more still to do

  • Convert PNG24s to PNG8s with alpha. Yes PNG8s with more-than-just-1-bit alpha do exist! and are considerably less bulky than PNG24s. Sadly neither Photoshop nor imageMagick can export them, but Fireworks, or the PNGNQ utility can. They don’t work too well in IE6, but then what does…)
  • Consider using CSS sprites and background-position hackery for some of our icons, where possible, to cut down on requests
  • Serve up static files from assets1.playlouder.com and assets2.playlouder.com, to increase the number of concurrent requests browsers make (they typically limit to 2 per hostname)
  • Optimize our javascript to improve page initialization times – browser-native getElementsByClassName may help here, as may selectively delaying some DOM lookups until they’re needed, and using more bubbled-up event handling to avoid the need for more specific DOM lookups
  • Optimize the crap out of the server-side (another topic for another day…)

Much as I love Ruby

Tuesday, December 11th, 2007

And much as we bend Rails to our will, I am getting a bit jealous of these guys developing web apps with Scala – an elegant hybrid functional/object-oriented language with a powerful type-inferencing type system, Erlang-style Actors and other goodies. It compiles and runs fast on the JVM too and can access Java libraries in quite a native way. It’s kinda like Ruby plus OCaml plus Java minus the suck of Java.

I think it’s because the inner maths and type theory geek in me (the one who can never quite get over how awesome http://en.wikipedia.org/wiki/Curry-Howard_isomorphism is) really misses having a powerful type system – and Scala’s does seem to hit the sweet spot when it comes to a middle ground between the bafflingly powerful Hindley-Milner extensions of Haskell and OCaml, and more accessible Object-oriented type systems with subtyping.

liftweb (or ‘Scala with Sails’ – see what they did there?) seems like a pretty neat framework too. I’m just plugging it so that someone else will do (continue doing) the work of making it sufficiently ‘enterprise-ready’ for me to use in ‘the real world’. ;-)

Meet us at LRUG

Thursday, November 8th, 2007

This monday I shall be giving a talk at the London Ruby Users Group. I’ll be giving a tour through our experiences building a modular composable Widget UI framework ontop of Ruby on Rails. Some of the steps we took along the way, problems we encountered, and a tour of the results.

There’s also a juicy debate to be had comparing the REST-driven web application architecture pushed by the Rails project, with our more modular widget-based approach, and in deciding what’s appropriate for your application. For those in the know, comparisons abound with the approach taken by Avi Bryant’s Seaside framework, and the Apotomo plugin already in development for Rails.

The framework comes with a client-side component too, and means of serializing Widgets to constructors for corresponding client-side javascript classes – so the talk may also interest those attempting to do Javascript in an unobtrusive, object-oriented way with Rails.

After that I believe (sincerely hope ;-) ) there are drinks.

The talk is open to all but the venue ask that you Register here; see LRUG for more details.

The more I have to hack ActiveRecord’s guts

Wednesday, October 17th, 2007

to make it do something right – the more I’m tempted to rewrite it from scratch.

That whole “I never took (or never understood) a database theory course at university, so I’m just going to pretend it doesn’t matter, and that a relational database may be treated as nothing more than a glorified filestore for my Objects” attitude just doesn’t cut any ice with me, but seems sadly prevalent within the Rails community. Yes, good object-oriented design is really important – but you need a sophisticated relational approach in order to get a handle on the data model behind any kind of non-trivial inheritance & mixin hierarchy, and to persist it in a logically sound and efficiently-indexable fashion.

What’s more, I contend that your ORM tool needs to understand something of the relational algebra in order to represent what is going on in a sufficiently elegant, flexible way – otherwise you’ll always be piling hack ontop of hack whenever you want to map the results of a moderately complex query over to the OO side. Joining SQL strings together is not the way forward – these things are syntax trees with structure!

Ahem. Sorry if I sound exasperated. SQLAlchemy on the Python side gets this kind of thing ABSOLUTELY SPOT ON, and dare I say it, so do some of the Java ORM frameworks (shame about the XML config files and Java’s tendency towards boilerplate code and bloated syntax, but don’t throw the baby out with the bathwater Rails-ers)

The problem with Rails’ ActiveRecord is that it’s neither here nor there – neither the kind of lightweight, simple ‘map objects to database rows and nothing much else’ approach originally implied by Fowler’s Active Record design pattern – nor the kind of powerful ORM tool which is capable of turning the kind of tricks that are increasingly demanded of it in anything like an elegant fashion.

It seems the Rails team’s solution to some of the endemic problems with ActiveRecord’s messy guts is to wrap them up in a huge plastic bag known as caching – an acceptable pragmatic approach, I accept, in many situations, but one which would not be nearly so necessary had a different approach been taken to ActiveRecord’s architecture.

I feel that superior approach needn’t have come at the cost of ActiveRecord’s ‘convention over configuration’ and ‘easy to get started with’ benefits either – it just would have required a little more forethought and a little humility in learning about the Relational Model before attempting a tool which maps complex data models to a Relational Database.

Crap, I’m starting to sound like Fabian Pascal now aren’t I.

We moved to Git

Monday, October 15th, 2007

I’ll admit it took a while to convince me of the merits of decentralised version control. But after a really nasty couple of merges, enough was enough. We’ve dropped SVN in favour of Git – which seems to be the biggest contender.

The main selling point for me was that it keeps track of all the metadata surrounding merges – information which SVN forgets about, requiring you to document your own merge metadata in commit messages, and scour the SVN logs trying to figure out which changesets have already been merged, when, whence, whither, by whom.

Other wins:

  • It’s really fast
  • Easy creation and painless switching between local ‘topic branches’, which you can create for each feature you’re working on and merge into eachother easily
  • Easy to swap work-in-progress patches with other developers without having to commit to a centralised trunk
  • Easy to make lots of quick local / offline commits, which you can later crunch down into one whn merging, if you want
  • Have the whole history available locally, and lots of backups of the repository

Some minuses:

  • The git-svnimport tool appears slightly buggy. Don’t count on it to import your SVN branches properly, especially if you moved them around at any point in the SVN history. In our case the branches it created only contained the files which had changed since the branch was created in svn – rather than fiddle around sorting them out, I just deleted them and re-created them from the git master branch by applying an svn diff. Which is OK if the history of your branches isn’t super important, but less than ideal otherwise. I also found that a small number of files were missing from the trunk, and had to be re-added manually – I suspect git-svnimport gets a bit lost when files have been moved around in non-trivial ways in the SVN history.
  • There is something of a learning curve with Git, especially when it comes to more complex merging, branching, tagging, cherry-picking tasks which were the reasons I first wanted to move to Git. I found this set of lessons learned helped a lot, ontop of the ‘Git from SVN’ tutorial. Once you know what you’re doing though, it’s faster and a lot less fiddly at merging than SVN.

Multiple inheritance in Javascript

Friday, September 28th, 2007

Javascript’s prototype-based OO would be so much better if objects could have multiple prototypes. One could then build arbitrary Directed Acyclic Graphs of object delegation, and build pretty much any kind of desired object-oriented semantics, any kind of inheritance scheme or method resolution order in the world ontop of that. Raw Power.

Currently all the approaches to building class-based inheritance in javascript either rely on copying attributes (which rather defeats the point of prototype-based OO semantics in my mind), or are limited to a single superclass (no mixins or multiple superclasses).

I think this is the main weakness in the argument that prototype-based OO is better than class-based. Yes – but only if you can spare me multiple prototypes. In my opinion I think the ECMAScript standards should evolve more in this direction, and less in the direction of hacking class-based inheritance into the language. Just add multiple prototypes and provide class-based inheritance as a library.