pmacct – my new best friend.

October 24th, 2009

How do you manage your transit data? If you source your traffic through more than one upstream ISP – and you should – then you need a way to view those relationships in detail, and make sure you are receiving your best value for money. However, it’s not always easy convincing your manager (or your mother) why Internet traffic costs money (and how it is priced).

You need a way to see what and where your routers are sending traffic. What I’m talking about here is Netflow. There is a wealth of information about this once-upon-a-time Cisco protocol, and it’s neighbours, jFlow, sFlow and NetStream.

I’ve tried so many software packages to make sense from my Netflow. OSS command-line based to huge Java run pay-per-port commercial packages. Not one has ever been agile enough (without substantial hacking) to document what is really happening at my network borders. The reason being that I don’t always control all our traffic on the edge. How can I account for the traffic that isn’t routed via my border routers?

Recently I saw on NANOG (or was it c-nsp list?) another vague request for network management tools. I found pmacct through the author, Paolo Lucente. He convinced me in one email that his OSS product (developed since 2003) would fit my needs. I’m going to show you via this HOWTO, as there isn’t that much newbie information in the googlesphere.

Our network runs like a bit this: border routers have full BGP routes from multiple transits. This services most of our network. network diagram
We also have some servers connected directly to an ISP, as they push out megabandwidth compared with what our routers are capable of. I give the servers a default route to some IP space hosted by the ISP which has bigger routing iron. They provide a connection with an HSRP default gateway that I can plug these machines directly into. This way we don’t need to invest in huge amounts of money for those big name routers.

The problem with these latter machines, is that they are not aware of the Internet. They just punt a packet towards a gateway, and get a packet back. I want to be able to see where these packets are going (ie. the ASN), but our ISP doesn’t provide Netflow for us. This is the crux of why all other Netflow accounting packages haven’t worked for me.

I have a basic setup on these machines. The short story is that within their ufw/iptables rules I insert a ULOG rule for all traffic on their “external” interfaces. Then I run a program called fprobe-ulog which interprets the basics of these ULOG messages and respews Netflow packets towards my collector.

Before, I would script the netflow records from these machines and match it against a BGP table, while my name-brand routers were lovingly supported by name-brand software packages. pmacct will aggregate all this functionality into one lightweight, stable and scaleable system.

Now onto setting up pmacct.

  1. Enable netflow on your interfaces with which you peer: 
    Router(config-subif)#ip flow ingress
Router(config-subif)#ip flow egress
Router(config-subif)#ipv6 flow egress
Router(config-subif)#ipv6 flow ingress
Router(config-subif)#exit
Router(config)#ipv6 flow-export destination 192.168.1.111 3001
Router(config)#ipv6 flow-export version 9 bgp-nexthop
Router(config)#ipv6 flow-export source Loopback1
Router(config)#ip flow-export destination 192.168.1.111 3001
Router(config)#ip flow-export source Loopback1
Router(config)#ip flow-export version 5 origin-as bgp-nexthop


    without the last line your netflow will not have the ASN information within the Netflow packets.

  2. Make sure netflow packets are arriving at your collector: 
    # tshark port 3001
Running as user "root" and group "root". This could be dangerous.
Capturing on eth0
  0.000000  94.228.64.1 -> 192.168.1.111 UDP Source port: 51901  Destination port: 3001
  0.000384 94.228.64.41 -> 192.168.1.111 UDP Source port: 57689  Destination port: 3001
  0.980030  94.228.64.2 -> 192.168.1.111 UDP Source port: 58299  Destination port: 3001
  0.980253 94.228.64.42 -> 192.168.1.111 UDP Source port: 56640  Destination port: 3001
  0.999718  94.228.64.1 -> 192.168.1.111 UDP Source port: 51901  Destination port: 3001
5 packets captured
#

  3. Download and compile libbgpdump. 
    # wget http://www.ris.ripe.net/source/libbgpdump-1.4.99.10.tar.gz
# tar xvfz libbgpdump-1.4.99.10.tar.gz
# cd libbgpdump-1.4.99.10
# apt-get install build-essential libbz2-dev
# ./configure
# make

  4. download the DFZ routes de jour from RIPE. 
    # wget http://data.ris.ripe.net/rrc00/2009.10/bview.20091024.0759.gz
# gunzip bview.20091024.0759.gz

  5. parse the binary bview format, strip out any bogus lines, and reformat it as “$ASN,$PREFIX”: 
    # ./libbgpdump-1.4.99.10/bgpdump -m bview.20091024.0759 > \
~/bgptable.20091024.0759.dump
# awk -F\| '$0 !~ /INCOMPLETE/ {print $6,$7}' bgptable.20091024.0759.dump\
|awk '{ print $NF "," $1 }'|uniq > bgptable.20091024.0759

  6. Now install pmacct! (i just leave mysql root blank for testing) 
    # apt-get install libmysqlclient15-dev mysql-server libpcap-dev
# wget http://www.pmacct.net/pmacct-0.12.0rc2.tar.gz
# tar xvfz pmacct-0.12.0rc2.tar.gz
# cd pmacct-0.12.0rc2
# ./configure  --enable-mysql --enable-ipv6
# make install
# mysql < sql/pmacct-create-db_v6.mysql
# mysql < sql/pmacct-grant-db.mysql


    This gets you up to the point of having a working mysql table for nfacctd to insert the data into, the next step I'll walk through some the settings and how-it-works.

  7. Now you are ready to create your config. pmacct has 3 daemons; sfacctd, pmacctd, nfacctd and the pmacct command line client.
    sfacctd is for sFlow. sFlow usually comes from hardware samplers on L3 switches. I don't use this, but I will have to in the future - glad to know its there.
    pmacctd is for either running on a client machine in promiscuous mode (a bit like i do with the ULOG and fprobe) or using it as a Netflow aggregator which can normalize your flows in all sorts of weird and wonderful ways, and then forward it back onto a Netflow datastore. Useful - but not for my basic setup.
    nfacctd collects the Netflow data. This is what I'm using.
    pmacct is the cli interface to the memory plugin. We are using the MySQL plugin - but to get us up and running we will start with the memory plugin.

    Let's create our config file:

    # vim /etc/nfacctd.conf

daemonize: true
aggregate: src_as,dst_as,dst_host,src_host,flows,dst_port,src_port,proto

    The values for aggregate are the sources in the netflow datagrams that you are interested in storing. This is pretty self-evident.

    ! plugin_buffer_size: 1024
nfacctd_port: 3001
nfacctd_time_secs: true
nfacctd_time_new: true
! read bgp table from here..
nfacctd_as_new: file
networks_file: /home/charlie/bgptable.20091024.0759
plugins: memory
sql_db: pmacct
sql_table: acct_v6
sql_table_version: 6
sql_passwd: arealsmartpwd
sql_user: pmacct
sql_refresh_time: 30
sql_history: 10m
sql_history_roundoff: m

    Keep the buffer at nil until you are in production mode.
    Set your listening UDP port (3001).
    Configure the path to the BGP table we built before.
    Set the plugin to be "memory".
    The rest commented will remain unused until you are ready to store data in MySQL. Now let's try it out! 

  8. # nfacctd -f /etc/nfacctd-mysql_v6.conf
# pmacct -s
SRC_AS  DST_AS  SRC_IP          DST_IP          SRC_PORT  DST_PORT  PROTOCOL  PACKETS   FLOWS   BYTES
6849    47998   94.179.57.134   94.228.76.67    3153      445       tcp       2         1       96
786     47998   131.251.141.20 193.34.28.19   4782      80        tcp       5         1       1152
^C

    woo! There is data! I can even see the external routes being populated with the correct ASN's. Go back to your config and change memory to mysql and you're done. If you want to see it not add the ASN's then change nfacctd_as_new: file to nfacctd_as_new: false.

    There is a great resource in the distribution called CONFIG-KEYS which will help you in figuring out your nfacctd.conf and README.mysql to understand how to expand upon this basic setup.

    9. From here on in you can start to create reports via the mysql cli or from some pre-made web front ends. Flox is a basic one. Here is a screenshot showing some IRC servers.

    flox

    You really must try this out. Paolo has kept the programs really easy and maleable and he promises me that I can soon discard my ULOG for a native pmacctd ULOG implementation.

    More information about our network is of course available from RIS, DB and here.

Tags: , , , , , ,
Posted in Geekery, Networking, Statistics, Tools, Unix | No Comments »

Hellfire for Filesharers…

October 16th, 2009

The Guardian newspaper today has this very pertinent story on the challenge of getting music reviewers’ interest while at the same time preventing pre-release leaks:

http://www.guardian.co.uk/music/2009/oct/15/illegal-file-sharing-promo-copies

“In the war against albums being illegally uploaded on to the internet before they are released, David Tibet of the underground band Current 93 may have struck a minor, if resounding, victory. “This is a promotional CD,” announces a little girl on the promo copy of Current 93′s new album Aleph at Hallucinatory Mountain. “Anyone illegally selling, copying, uploading or downloading this material is condemned to eternal hellfire. Happy listening, God is love.” Then Tibet – a devout Christian with strong views about the impending apocalypse – intones “murder” over a guitar riff heavy enough to terrify Satan. It makes you wonder whether a casual upload is really worth being cast into Hades for.”

I declare an interest – state51, one of the businesses that came together to create Playlouder MSP, works with David Tibet.

When we were putting together the original plan for Playlouder MSP we built a beta ISP partly because we believed strongly that the ISP music bundle was very compelling and felt a retail strategy had chances, but also because we had to demonstrate to two cynical industries that it was possible for the technology to work to deliver consumer friendly services as well as to protect the value of music.

Ironically, while we were the only ISP in the world that was able and willing to identify music being illegally shared, and then intervene to stop it, one of the accounts on our network was used to upload a pre-release copy of an album. We were in beta phase, and were handing out IP addresses dynamically without keeping records, so when we received a notification from the BPI there was nothing we could do, short of the rack and thumbscrews for all our trial account holders, to identify the culprit – even if it happened on the wire rather than on an open wifi.

Even more ironically that transfer would and could have been stopped automatically if the record label involved had supplied us with their music, as it would have gone into our content recognition systems which in those pre-encryption days were reliably catching high 90s percentages of music transfers.

It is a sign of an industry in distress when the relationships between artists and the honoured few who are offered the chance to hear and review new music are so weak that the trust is routinely flouted; and when there is more kudos for upping on waffles than for delivering a thoughtful review. When, where, and how to introduce a work to the public seems like a reasonable privilege to reserve to the creator, even if we recognise that once released other people have many other rights that need to be respected. It is also one of the foundations on which value in music is built.

So respecting the release date is one of the principles we hold to at Playlouder MSP, and we have invested heavily over the years into making sure that as far as possible we can back up that respect with real action. The final irony is that the label itself was the cause of our failure to stop a pre-release leak on our beta network, but that did not stop them circulating some of the correspondence to ISPs we were trying to interest in solutions to help prevent new leaks in future, in an attempt to damage our reputation. My conclusion – if one is needed – is that trust is a conversation built on mutual respect.

Posted in Blogroll, Business, Industry, Music | No Comments »

Lobbying by Numbers – The Record Industry Credo

July 27th, 2009

A perfect exposition of the difficulty of adapting what’s inside executives’ and industry leaders’ heads in a period of change appeared in the FT this morning, in the form of a letter from BPI head Geoff Taylor responding to a somewhat downbeat piece on the futility of fighting piracy.

The figure 13% is a bit of a bloody disaster if you ask me – the fact that it is so low shows far greater success in holding back the electrification of the music industry than it does ‘substantial progress’. But this picaresque tour of the problems and concerns shows that the lobbyists either do not wish, or are not equipped, to deliver a nuanced and complex message which reflects the complexities of the world they are trying to change.

The fact that they are not willing to spend more than a few million per year to recapture the £200m claimed foregone revenue shows that either they do not believe the number themselves, or that they do not believe that enough of them believe it.

There is a strong case to be made on behalf of all the affected industries that incentives are misaligned to the point that public value in digital networks is being destroyed almost as fast as it is being created, but it won’t come out of something that relies, Chauncey Gardener-like, on hovering along a continuum between cab-driver common sense and a fundamentalist credo.

The letter is below, and online here.

UK record labels have refashioned their businesses

Published: July 27 2009 03:00 | Last updated: July 27 2009 03:00

From Mr Geoff Taylor.

Sir, The suggestion that “the music industry is no closer to solving the problems created by digital piracy” overlooks substantial recent progress made in the UK (“Pirates on parade”, Analysis, July 22).

UK record labels have refashioned their businesses to bring music to consumers online. Digital music services such as iTunes, Amazon mp3, WE7, Spotify and Comes With Music now contribute 13 per cent of overall revenues, and groundbreaking new offerings, led by internet service providers, are coming online.

The memorandum of understanding signed with the government last year saw ISPs publicly accept their responsibility to deal with online filesharing. The UK government explicitly committed to achieving a 70 to 80 per cent reduction in illegal downloading in two to three years, and the forthcoming digital economy bill will propose new duties on ISPs, enforced by Ofcom, to tackle illegal filesharing. It will also give Ofcom the power to impose stronger regulation if that is needed to solve the problem.

For the record, independent research for BPI by Jupiter shows that this year music labels will forgo £200m ($329m) in lost revenue due to illegal downloading. Between 2007 and 2012 the cumulative loss will be £1.25bn. These figures discount heavily for downloads that may not have been purchased, contrary to the assumption frequently levelled at the industry that we treat every illegal download as a lost sale. The jobs that are being lost as a result, and the threat to future investment in creativity, should be of concern to everyone in the media industry, including the FT.

Geoff Taylor,

Chief Executive,

BPI (British Recorded Music Industry), London SE1, UK

Posted in Business, Industry, Music | No Comments »

Commissioner Viviane Reding Makes Important Statement

July 9th, 2009

Viviane Reding, EU Commissioner for Telecoms and Media, has made an important statement about the approach Europe will be taking towards the balance between consumers’ and creators’ interests and rights as the EU forms and delivers its digital policy over the next five years. It is very clearly in favour of the development of new content licensing and distribution models, even where rights owning industries have been themselves reluctant to come forward into the digital marketplace.

Playlouder MSP has been at the forefront of this movement for the last 6 years, working with the music industry and ISPs to develop a licensing and service model that can truly deliver the benefits of a networked digital economy to consumers and business alike. With support from the EU the promise looks closer than ever to being fulfilled.

This is a key excerpt:

It is necessary to penalise those who are breaking the law. But are there really enough attractive and consumer-friendly legal offers on the market? Does our present legal system for Intellectual Property Rights really live up to the expectations of the internet generation? Have we considered all alternative options to repression? Have we really looked at the issue through the eyes of a 16 year old? Or only from the perspective of law professors who grew up in the Gutenberg Age? In my view, growing internet piracy is a vote of no-confidence in existing business models and legal solutions. It should be a wake-up call for policy-makers.

If we do not, very quickly, make it easier and more consumer-friendly to access digital content, we could lose a whole generation as supporters of artistic creation and legal use of digital services. Economically, socially, and culturally, this would be a tragedy. It will therefore be my key priority to work, in cooperation with other Commissioners, on a simple, consumer-friendly legal framework for accessing digital content in Europe’s single market, while ensuring at the same time fair remuneration of creators. Digital Europe can only be built with content creators on board; and with the generation of digital natives as interested users and innovative consumers.

The full speech can be downloaded here: Vivianne Reding Speech, Brussels, July 9th 2009

Posted in Business, Industry, Music, Playlouder | 1 Comment »

Is The Playlouder MSP Model Gaining Credibility?

May 12th, 2009

Market research and academic studies are no substitute for the informed judgement of experienced business people; but in a time of doubt and a shrinking recorded music market the conservatives and zero-sum boys seem to dominate. Anyway, we get a mention in this article by Volker Grassmuck who has an impressive CV in the social impact of information systems.

Of course we know how well our model stands up to scrutiny, in its ability to deliver a substantial and relatively risk free income stream to the music industry on the one hand, and in its impact on ISP economics on the other hand. We did go through a very stringent due diligence process with a very large broadband company in the UK over the second half of 2008. So we obviously feel we have nothing to fear from the endorsement of ‘academics and so-called thinkers’.

Posted in Business, Industry, Playlouder, Press | No Comments »

Javascript application developer for music services

April 6th, 2009

Work on our music app is coming along nicely, and we’re hiring more client-side talent!

http://mediaserviceprovider.com/jobs.html

As the ad says: the role would suit a Javascript guru who enjoys the challenge of developing a real thick-client application in the browser. It may equally suit a GUI application developer with solid experience on other platforms who’s confident in their ability to pick up the necessary Javascript skills.

Tags: , , , ,
Posted in Development, Jobs, Music | No Comments »

Many Opinions, Not Much Information

August 18th, 2008

I was asked recently to write a guest blog at UK national newspaper website, the Telegraph. Here’s the result.

In it I predicted, perhaps rashly but we shall see, some kind of breakthrough in the uneasy relationship between ISPs and the music industry. It’s long overdue: as anyone can discover with a few internet searches we have been working on building this bridge for the last five years. I was adding my voice to a small wave of comment about us, most of it ill informed about not only our own business model, but also about the enabling technology and the respective positions of ISPs and record labels.

One of our favourite comments was that our enterprise is ‘naive, flawed and doomed to fail’, which we liked so much that we thought we’d make a t-shirt with it.

And one of the reasons I liked that comment so much was that in some way you can apply it to all human endeavour, including blogging itself. It seems that we have created a proposition that some people struggle to accept as possible let alone probable or inevitable. Here’s Jupiter’s Mark Mulligan:

My take is that if they are close to announcing something, it will be significantly watered down from the proposition they’ve been trying to get labels to sign up to for years. (here)

and Paul Resnikoff:

the service represents a step down from current, free acquisition options (here)

You can’t please all the people all the time. I shall continue to predict a breakthrough, sooner rather than later, we shall continue to develop our platform (see the recent job advert on this blog), and we shall continue to make ‘no comment’ to public speculation about our business and partnerships.

Posted in Industry, Press | No Comments »

Job ad: Music service developer

August 14th, 2008

We’re hiring!

Playlouder MSP has been working with ISPs and the music industry to develop both an innovative business model for music consumption, and innovative user experiences around music and communication to complement ISPs’ offerings.

As a key addition to our small but growing development team, you will be critical in helping to refine, scale and roll out our application and service to white-label ISP clients.

Key requirements

  • Background in computer science, mathematics, software engineering or similar (degree or equivalent experience)
  • A solid technical all-rounder with software development experience on sizeable projects

The specifics

Our work involves all of the following, and you’ll be tackling problems involving many of them:

  • Dynamic languages (experience with Ruby, on which our current implementation is based, would be particularly desirable)
  • Rich web application development with a large Javascript-based client-side portion
  • A modular, widget-based user interface framework for the above
  • Databases (MySQL at present), data modeling and ORM tools
  • Unix-based deployment environments
  • Large volumes of media, media metadata and usage statistics
  • Web services APIs, and large-scale integration work
  • Server push technologies, and scaling applications with a live messaging component
  • Common agile software development tools, processes and techniques – source control, bug tracking, testing etc
  • A warehouse full of music geeks :)

Some other things which you might get to play with:

  • Other languages – Java, Python, C, possibly Erlang (see ‘messaging’ above), …
  • Messaging technologies like AMQP, ActiveMQ, XMPP
  • Music technology R&D projects
  • Lots more in a fast-growing company

In return we can offer challenging problems in an interesting domain, competitive pay, and a great work environment for music fans in our E2 warehouse!

Enquiries to matthew@playlouder.com

Posted in Business, Development, Jobs, Ruby | 2 Comments »

80k of client-side-only storage for javascript, without browser extensions

May 27th, 2008

Thought I would share this hack.

The problem – you want to maintain some state on the client, but you don’t want to send this state on a pointless round-trip to the server with every request, as typically happens with Cookies.

There is a way around this though!

  1. add an hidden iframe to your page, with src=”/client-side-cookie/blank.html”
  2. from this directory, you serve a static empty HTML file, and, crucially, you serve this with Expires headers way into the future (see this Yahoo tip for some info about this technique)
  3. This file will not (typically) be re-requested before the time given in your Expire header
  4. Set cookies for the document in the iframe from your javascript code, with path=/client-side-cookie/, and with whatever expiry time you like. eg, iframe.contentDocument.cookie = ‘test=long_data_which_we_dont_want_to_send_to_the_server; path=/client-side-cookie/’
  5. When you want the data back in future – again, create the iframe (the HTML file will NOT be loaded from the server because of the Expires header, and so no cookies will be sent to the server). Then inspect iframe.contentDocument.cookie to get the data.
  6. Because you have restricted the path of the cookie, it will never be sent with requests for files outside of your special /client-side-cookie/ directory.
  7. Profit!

Problems:

  • This can’t be relied on for security or privacy purposes not to send the data to server. The user could purge their browser cache, do a hard refresh on the file, etc.
  • Even a far-future Expires header will expire eventually – and browsers may limit the length of Expires headers.
  • So you should be prepared for the event that this data might, albeit very infrequently, get sent to the server.
  • You are still limited to approx 4k per cookie (including key and value – google for detail on precisely what is supported cross-browser but it is at least very close to 4k)
  • You are limited to 20 cookies per domain (in older IE versions at least, others allow more)
  • So that caps it overall at about 80k, with some fiddling around to distribute the data between 20 separate cookies. Still, not to be sneezed at!

Posted in Uncategorized | No Comments »

Mysterious Flash bug on change of background

May 15th, 2008

Just incase anyone else runs into this and Googles.

If you have a flash movie which mysteriously seems to reload itself during some fairly innocuous and un-connected Javascript execution – take note:

Dynamically setting document.body.style.background was the culprit for us. Don’t ask how I managed to identify this as the culprit, suffice to say it involved a lot of logging statements and patience. Doing this immediately caused the flash movie to reload itself, causing havoc in our case as we use it to play music and connect to a socket server.

You may find, like us, that setting separate background properties, eg document.body.style.backgroundImage, worked around the issue.

Anyone hazard a guess as to why the Flash runtime feels the need to implement this behaviour? Perhaps something related to wmode=transparent? (although we’re not using it)

Tags: , , , , ,
Posted in Uncategorized | No Comments »

« Older Entries