Andy Piper

Yesterday, YouTube turned seven.

I’ve recently become aware of just how pervasive YouTube has become. It’s available on a range of “computer” platforms – desktops and laptops, mobile phones and tablets. I’m able to access it via AppleTV, XBox, and a Sony Bluray player. Friends who recently updated their home media setup have it on their internet-enabled TV as well as via their Virgin Tivo box on the same “system”. Alongside BBC iPlayer, it’s actually more pervasive in the UK across the various devices many of us have in our living rooms, than the broadcast DTV/satellite/cable channels themselves. It’s also noticeably more present across a very broad range of devices, than alternatives like Vimeo which are arguably better at presenting more beautiful, longer HD content on the web itself.

This is both exciting, and also potentially problematic.

For those of us who have been seeing a multichannel, multimedia future ahead for some time, it’s a validation of the success of streaming web video in breaking the monopolies of the existing broadcasters and media companies. Over time, Google has added some tremendous value to YouTube – enabling creators to rapidly upload, perform simple edits, add soundtracks, and share content all within a rich HTML browser experience. It is also easy to reach a wide range of devices simply by ticking the “make this video available to mobile” box on the video management page – Google does all the heavy lifting of transcoding, resizing, and deciding on whether Flash or HTML5 is a better delivery mechanism, etc.

However, at the same time, it’s kind of… well, clunky. In order to consume content from YouTube on any of the platforms I mentioned before, you have to visit a dedicated YouTube widget, app or channel and then navigate around content within that box (oh, and each platform has a slightly different way of presenting that content). It’s not integrated with the viewing experience – I can’t just say to my TV or viewing device, “show me videos of kittens” and have it aggregate between different sources which include YouTube. Not only that, but we all know just how variable YouTube content can be, both in terms of production quality, duration, and the antisocial nature of comments and social interactions around videos. For some of the most popular videos I’ve posted on my channel, I can’t tell you how long I spend on moderating the most unbelievably asinine comments! Oh and, when we consider the increasing use of streaming video online – be it iPlayer, YouTube, Netflix or any other source – we constantly have to consider the impact on available bandwidth. Bandwidth and connectivity are not universal, no matter how much we may wish they were.

The other side of this is the group of voices who will point to the dominance of Google and their influence over brands and advertising. All very well, but I like to remind people that for all of the amazing “free” services we enjoy (Facebook, Twitter, Google and others), we do have to pay with an acceptance of advertising, and/or sharing of some personal data of our choice – or go back to paying cable and satellite providers for their services. It’s really a simple transaction.

I guess I don’t really have a message with this blog entry, other than to share my observation of the amazingly rapid rise of the new media titan(s). If I was going to offer any further thoughts or advice, it would be the following:

• explore online video services more – you probably have access to them in more places than you think.
• remember that video you produce may be viewed on any device from the smaller mobile handsets, to a nice HD television – so always try to produce your content at the highest quality setting possible, and let YouTube or the other video hosts do the rest.
• richly tag and describe your content to make it easier to find. “Video1.mov” tells me nothing.
• learn about the parameters which control how your content is displayed. I’ve previously written about this; the content is still useful but I should probably create an update.

This omnipresence across platforms is one of the reasons why I’ve started to primarily use my YouTube channel as the canonical source for all of my video content. Previously I’d used Viddler and Vimeo and occasionally posted a clip to Facebook, but now that I am able to post longer movies, I’ve also posted the full videos of various talks that I’d only previously been able to host at Vimeo. I’m not abandoning all other sources, but a focus on one channel makes a certain amount of sense.

I joined Twitter on February 21, 2007, at exactly 15:14:48, and I created my account via the web interface. As you can see, my first tweet was pretty mundane!

overheating in an office with no comfort cooling or aircon. About to drink water.

—
Andy Piper (@andypiper) February 21, 2007

I remember discussing this exciting cool “new Web 2.0 site” with Kim Plowright @mildlydiverting in Roo’s office in Hursley a couple of days before, and before long he, Ian and I were all trying this new newness out. It was just before the 2007 SXSWi, where Twitter really started to get on the radar of the geekerati.

But wait a moment! It’s impossible to pull back more than just over the last 3,000 tweets using the API, so how was I able to get all the way back to 5 years ago and display that tweet when I’ve got over 33,000 of them to my name?

It’s a relatively little-known fact that you can ask Twitter to disclose everything they hold associated with your account – and they will (at least, in certain jurisdictions – I’m not sure whether they will do this for every single user but in the EU they are legally bound to do so). I learned about this recently after reading Anne Helmond’s blog entry on the subject, and decided to follow the process through. I first contacted Twitter on April 24, and a few days later faxed (!) them my identity documentation, most of which was “redacted” by me Yesterday, May 11, a very large zip file arrived via email.

I say very large, but actually it was smaller than the information dump that Anne received. Her tweets were delivered as 50Mb of files, but mine came in nearer to 9Mb zipped – 17Mb unzipped. I’d expected a gigantic amount of data in relation to my tweets, but it seems as though they have recently revised their process and now only provide the basic metadata about each one rather than a whole JSON dump.

So my entire archive of Tweets back to Feb 2007 is a ~8Mb text file (140 chars per tweet+metadata).

—
Andy Piper (@andypiper) May 11, 2012

So, what do you get for your trouble? Here’s the list of contents, as outlined by Twitter’s legal department in their email to me.

- USERNAME-account.txt: Basic information about your Twitter account.

- USERNAME-email-address-history.txt: Any records of changes of the email address on file for your Twitter account.

- USERNAME-tweets.txt: Tweets of your Twitter account.

- USERNAME-favorites.txt: Favorites of your Twitter account.

- USERNAME-dms.txt: Direct messages of your Twitter account.

- USERNAME-contacts.txt: Any contacts imported by your Twitter account.

- USERNAME-following.txt: Accounts followed by your Twitter account.

- USERNAME-followers.txt: Accounts that follow your Twitter account.

- USERNAME-lists_created.txt: Any lists created by your Twitter account.

- USERNAME-lists_subscribed.txt: Any lists subscribed to by your Twitter account.

- USERNAME-lists-member.txt: Any public lists that include your Twitter account.

- USERNAME-saved-searches.txt: Any searches saved by your Twitter account.

- USERNAME-ip.txt: Logins to your Twitter account and associated IP addresses.

- USERNAME-devices.txt: Any records of a mobile device that you registered to your Twitter account.

- USERNAME-facebook-connected.txt: Any records of a Facebook account connected to your Twitter account.

- USERNAME-screen-name-changes.txt: Any records of changes to your Twitter username.

- USERNAME-media.zip: Images uploaded using Twitter’s photo hosting service (attached only if your account has such images).

- other-sources.txt: Links and authenticated API calls that provide information about your Twitter account in real time.

Of these, let’s dig a bit more deeply into just a few of the items, no need to pick everything to pieces.

The “tracking data” is contained in andypiper-devices.txt and andypiper-ipaudit.txt – interesting. The devices file essentially contains information on my phone, presumably for the SMS feature. They know my number and the carrier. The IP address list tracks back to the start of March, so they have 2 months of data on what IPs have been used to access my account. I’ve yet to subject that to a lot of scrutiny to check where those are located, that’s another script I need to write.

I took a look at andypiper-contacts.txt and was astonished to find out how much of my contact data Twitter’s friend finder and mobile apps had slurped up. I mean, I don’t even have all of this in my address book… given the fact that the information contained the sender email addresses for various online retailer newsletters, I’m guessing that Google’s API (I’m a Gmail user) probably coughed up not just my defined contact list, but also all of the email addresses from anyone I’d ever heard from, ever.

Twitter has on record 981 phone numbers and 8741 email addresses from my contacts / when I've used Find Friends :-o deleting that...

—
Andy Piper (@andypiper) May 11, 2012

Fortunately, there’s a way to remove this information permanently, which Anne has written about. I went ahead and did that, and then Twitter warned me that the Who To Follow suggestions might not be so relevant. That’s OK because I don’t use that feature anyway – and in practice, I’ve noticed no difference in the past 24 hours!

I use DMs a lot for quick communication, particularly with colleagues (it was a pretty reliable way of contacting @andysc when I needed him at IBM!). That’s reflected in the size of andypiper-dms.txt, which is also a scary reminder to myself that I used to delete them, but since Twitter now makes it harder to get to and delete DMs, I’ve stopped removing them and there’s a lot of private data I wish I’d scrubbed.

Taking a peek at the early tweets in andypiper-tweets, I’m trying to remember when the @reply syntax was formalised and when Twitter themselves started creating links to the other person’s profile. Many of my early tweets refer to @roo and @epred and I don’t think they ever went by those handles. 5 years is a long time.

I mentioned that the format used to deliver the data appears to have changed since Anne made her request. She got a file containing a JSON dump of each tweet including metadata like retweet information, in_reply_to, geo, etc etc.. By comparison, I now have simply creation info, status ID (the magic that lets you get back to the tweets via web UI), and the text itself:

********************
user_id: 786491
created_at: Wed Feb 21 15:43:54 +0000 2007
created_via: web
status_id: 5623961
text: overheating in an office with no 
comfort cooling or aircon. About to drink water.

It’s a real shame that they have taken this approach, as it means the data is now far more cumbersome to parse and work with. However, using some shell scripts I did some simple slicing-and-dicing because I was curious how my use of Twitter had grown over time. Here’s a chart showing the numbers of tweets I posted per year (2012 is a “to date” figure of course). It looks like it was slow growth initially but last year I suddenly nearly doubled my output.

Still considering what other analysis I’d like to do. I can chart out the client applications I’ve used, or make a word cloud showing how my conversational topics have changed over time… now that all of the information is mine, that is. It is just a shame I have to do so much manual munging of the output beforehand.

Oh, and the email I received from Twitter Legal also said:

No records were found of any disclosure to law enforcement of information about your Twitter account.

So, that’s alright then…

Why did I do this? firstly, because I believe in the Open Web and ownership of my own data. Secondly, because I hope that I’ll now be able to archive this personal history and make it searchable via a tool like ThinkUp (which I’ve been running for a while now, but not for the whole 5 years). Lastly… no, not “because I could”… well OK at least partly because I could… because I believe that companies like Twitter, Facebook, Google and others should be fully transparent with their users and the data they hold, and that by going through this currently-slightly-painful procedure it will encourage Twitter to put in place formal tools to provide this level of access to everyone in a frictionless manner.

If you’ll excuse me, I’m off to dig around some more…

In which I get misty-eyed and nostalgic, geek out over electronics, and think about mobile and the cloud.

Then

On Saturday I went along to the Horizons 30th anniversary of the ZX Spectrum event, organised by Paul Squires and Leila Johnston and held at the BFI in London. The event ran on both days but I wasn’t able to stay on the Sunday, so I missed at least half of the fun!

Although I’m full of nostalgia for the 8-bit era, I have to confess I never actually owned a Speccy or any Sinclair hardware. My friends did, but I was primarily an Acorn enthusiast and our first home computer was an Electron (although the first computer I used at primary school was a Commodore PET).

Me: c.1985 Acorn Electron, A3000, Risc PC, Acorn A4, and then onto Dell PCs, homebuilt Linux PCs and Thinkpads. Apple from 2007. #sflzx

—
Andy Piper (@andypiper) May 05, 2012

I fondly remember some of the hacks I did on/with/to the Electron, including soldering a pair of headphones into the motherboard to avoid annoying my parents with the music from various Superior Software titles

Regardless of “allegiance”, Horizons was a really great day. Highlights for me included a fantastic history of computing by PJ Evans from The National Museum of Computing at Bletchley Park (if you haven’t been there yet, you should visit!); Spectranet, an Ethernet adapter for the Spectrum which had me wanting one for no good excuse that I can come up with; and the mind-blowing live composition of a chip tune by Matt Westcott which I saw, but I struggled to comprehend. Matt’s ability to reverse engineer a tune in his head was remarkable.

Oh, and if you haven’t downloaded or bought MJ Hibbett‘s Hey Hey 16k yet, or at least streamed it, you really should.

aside: since Horizons was part of SciFi London, I tried to get Micro Men director Saul Metzstein to drop some hints about his upcoming S7 Dr Who episodes. All he would say was that the western episodes were filmed in Spain (knew that), and that the script for the Christmas episode hasn’t been written yet (didn’t know that).

Now

After the event on Saturday evening, I found it a real struggle to avoid crazy, nostalgia-fuelled eBay purchases, but I did manage to resist! Instead, I resolved to finally get around to building the Fignition I’d picked up at the Hack to the Future event a couple of months ago.

For those who are not familiar with it, the Fignition is a credit card sized build-it-yourself 8-bit computer based around the ATMega chip (the same one used in the Arduino and Nanode Open Source hardware boards). It’s really a remarkable little device – I guess it took me about an hour to assemble and solder, although your mileage may vary. The build guide is excellent and very clear. After performing a couple of power on tests with and without the ICs inserted, it was time to connect up to the TV – and it worked first time. It boots into a simplified Forth environment, which was reminiscent of that BBC BASIC> prompt I am so familiar with from my childhood. The only real downside is that the keyboard – built from 8 clicker buttons – is a bit fiddly to get to grips with, but hey – I just assembled a complete 8-bit computer including video out and keyboard! It’s hard not to be excited.

The board I built was a RevD – the new RevE board has onboard audio in/out (get ready for some fun loading stuff from audio cassettes, again!), and is also slightly modified so that in principle, it is possible to add Arduino-footprint shields. That’s kind of cool, as it means that it might be possible to add a PS/2 keyboard or a network interface.

What’s “the point” of something so simple, by today’s standards? Well, actually – the simplicity. I went from a bag of components, to a fully working computer in the palm of my hand – no surface-mount components – to a programmable device. It’s “primitive” by the standards of today’s machines, but it’s not that hard to understand how an 8-bit “brain” works, in comparison to the 32 or 64-bit mulitcore CPUs and GPUs in modern laptops and mobile phones. In my opinion, the Fignition, Arduino and Nanode fulfil an important role in helping youngsters to understand the basic principles of electronics and computing.

Next

Last night I headed along to the fantastic Mozilla offices in London.

The main LJC event was Simon Maple from IBM showing off the new WebSphere 8.5 Liberty Profile running on a Raspberry Pi. I’d hooked Simon up with Sukkin Pang recently so that he could get one of the smart enclosures he provides for the Pi. It was pretty cool to see a full Java app server running on such a small computer – actually almost exactly the same size as the Fignition, only considerably more powerful of course.

The whole talk was live streamed on Mozilla Air – but if you missed it, there’s a video available (complete with semi-professional heckling from yours truly!)

What stole the evening for me, though, was two other glimpses of what lies ahead. First, Tom Banks from IBM Hursley came on stage after Simon and showed off the Liberty profile running on a mobile phone. Let me clarify – he was running Android 2.3 on a Nexus One (an “old” phone), running Ubuntu Linux as a virtual image inside of that, and WebSphere inside of that. Kind of mind-blowing! A proof-of-concept and arguably not very useful… not sure when I would want to put a full JEE app server in a phone… but extremely cool. Finally, @cyberdees let Tom and I have a play with Boot to Gecko – Mozilla’s new mobile play. B2G was something I’d heard about, but not touched. I have to say that even in an early form, it’s looking very slick, boots extremely fast – much more quickly than any Android or iOS device I’ve seen – and the device integration (GPS, camera, access to hardware settings, etc) was impressive.

With the Open Web as the platform, ubiquitous mobile devices, and increasingly sophisticated cloud-based backends to interact with, the future is looking pretty cool.

There’s a cool Cloud Foundry fan site called preshavedyak.com - and last week at SourceDevCon London, we challenged a bunch of developers to earn themselves a nice new preshavedyak hoodie by registering for a Cloud Foundry beta account and seeing how quickly they could get a “hello world” app up-and-running in the cloud. The event saw a bunch of new signups and some great discussions.

The “pre-shaved yak”, of course, is one aspect of what a polyglot open source PaaS is all about – delivering a ready-made, ready-to-host, application runtime environment. We shaved the yak, so you can just go ahead and get productive with your development tool of choice, be that vi or emacs, Notepad or TextMate, or Eclipse / a.n.other IDE. Grab a micro Cloud Foundry VM image and take your pre-shaved yak with you when you’re not connected!

I actually started to write this post in order to comment on something that’s a bit more hairy that, though! I’ve been playing around a little bit with MonoDevelop and ASP.NET (for reasons that will become apparent during this week, I suspect). I’m using the current stable Mono (2.10) and MonoDevelop (2.8) packages on Lion, and they seem to work well. I’ve also recently been learning about Sinatra, the lightweight web framework for Ruby, and one of the node.js equivalents called Express. It turns out that the .NET world has a bunch of Sinatra-wannabes, the most popular of which appears to be Nancy (see what they did there…? dive into the world of Sinatra-themed name-related web frameworks…!).

Nancy’s site recommends installation via NuGet, which is evidently really well integrated into Visual Studio (NuGet is the equivalent of gem in Ruby, or npm in node.js). Unfortunately there’s no MonoDevelop equivalent. Here’s where the yak shaving started! The NuGet FAQ claims that the command line NuGet.exe will run and can be compiled under Mono, but in my experience, that’s not quite true – I could not get the source to compile in MonoDevelop on OS X. I grabbed the pre-compiled version and followed the instruction to get it to update itself (basically you just run it, and it bootstraps and downloads the latest available)… that went fine, but after that, it would no longer work and produced a huge stack trace.

So here, after getting most of a yak’s fleece all over me, is the secret. The prebuilt NuGet.exe will work under Mono on OS X, but it does require a Windows .NET 4.0 DLL (Microsoft.Build.dll) to be in the same directory / locatable in the path – I grabbed mine from my Windows VM install. It also requires that you tell Mono to present a v4.0 runtime. So I whipped up a tiny script to avoid having to type a bunch of paths and switches each time.

Further results of this recent dalliance in .NET land will be coming soon…

As my new role continues, a podcast I recorded with Michael Surkan over at Uhuru Software has just gone online. Uhuru provide hosting based on the Cloud Foundry platform, and add first-class support for .NET applications. They also have some really neat add-ons for MMC and Visual Studio to make deployment easy. We talked a little about the role of a Developer Advocate, the groups I’ve been talking to about adoption of Cloud Foundry, and some of the “gotchas” to consider when taking an application to a Platform-as-a-Service environment.

(if you can hear any background noise on this one, it was because I was at the Scala Days event in London on the day we spoke, and not Michael’s fault at all! I don’t think it sounds too bad)

Coming up this week, there’s the big Cloud Foundry Open Tour London on Tuesday (based on the numbers I’m hearing about, it sounds like that is going to be busy). Many of us from the engineering and developer relations teams will be speaking at that one. The rest of the week, I’ll be at SourceDevCon in London where my head honcho Patrick will be speaking on Thursday afternoon.

To round the week off, there’s Horizons at the BFI on Saturday and Sunday, to celebrate the 30th anniversary of the ZX Spectrum. I was always more of an Acorn guy myself, but there’s no denying these classic machines really kickstarted my interest in this role space – and I can’t wait to hear MJ Hibbett perform “Hey Hey 16k” in person!

Related articles

• First week with VMware and Cloud Foundry (andypiper.co.uk)
• My next steps – joining the Cloud Foundry team (andypiper.co.uk)

Well, it’s that fun Ubuntu release day again, and around the world, I’m sure there are parties aplenty…

I grabbed an ISO this morning (64-bit desktop version, natch), and quickly setup a new virtual machine to run on my Mac. A nice feature of VMware Fusion 4.0 is the “easy install” option which lets you rapidly pop in the basic information needed to setup the system, and the rest is taken care of for you.

In my case, I specified a user ID, selected that I wanted my home folder read/write shared into the VM, and then customised the machine to up the memory and add a CPU core. In a few minutes (my machine has an SSD…!) I had a new virtual machine, running full screen in a separate space on Mission Control. Fusion even took care of installing VMware Tools so it was able to do the file sharing, use the full screen resolution etc etc straight away.

So… first impressions? Much more slick than 11.10 which I was using on a daily basis until recently. In particular, the configuration options have been streamlined really nicely. I’m still struggling with discovery of applications in Unity but in general, it’s not bad at all.

Browsing through the available packages, I was interested to find that the Cloud Foundry tools are available in the default repositories:

That’s awesome! you can just run sudo apt-get install cloudfoundry-client and get the main tool for deploying apps and administering Cloud Foundry right from the repository using the regular apt method (actually this is simply a convenience package – under the covers it installs a package called ruby-vmc, which installs the vmc command-line Ruby gem… it’s nice that the Ubuntu guys have made it easier to discover, though).

So what’s the problem? Well – no big deal, but actually, if you want to keep up with Cloud Foundry as a developing platform, you might want to grab something slightly newer than what is available on tap in the repository. As I write this, the version of vmc available via the cloudfoundry-client package is 0.3.10 and the one we’re currently working with is at least 0.3.16.

My suggestion, therefore, is to do the following:

1. Install rvm (Ruby Version Manager). That way, you can have different versions of Ruby itself installed, manage gems for the different versions independently, and also – importantly – not require root privileges to do your development work and install additional gems. A handy guide to installing rvm on Ubuntu is here, and it still works fine on 12.04. Just follow the commands shown in the Installing RVM and Installing Ruby sections and you should be all set and rocking ruby-1.9.2 on your new 12.04 setup.
   (I’m using rvm and JewelryBox to manage Ruby versions on OS X, incidentally. Great tools)
2. Run either gem install vmc or gem install vmc --pre (the latter option will get you the very latest pre-release of vmc, if you like the bleeding edge). Note that, if you installed rvm and Ruby successfully, you should not need root permissions to install gems.
3. There is no step 3 — vmc target http://api.cloudfoundry.com and then vmc login and you should be good to go. Looking for a sample app to deploy? You could take a look at the Sinatra example I added to the cloudfoundry-samples repository on Github last week…

It’s fantastic that Ubuntu is moving towards strong desktop / development environment support for Cloud Foundry.

Oh, there’s another story with Precise Pangolin,  too – you can rapidly install the server side pieces to build your own cloud using the Juju Charms which provide Cloud Foundry support. But… that’s a story for another post, in another time and place….

Update 15/05: I raised bug #998111 against Ubuntu to ask for the ruby-vmc package to be updated, in case you feel like tracking progress via Launchpad.

Hello, VMware.

Well – that was bracing!

I don’t expect to be posting “week notes” like this on a regular basis, but as a one-off it seems like a nice way to encapsulate just how much happened in the first week of my new role.

Tuesday

Joined VMware. Met new colleagues in London office. Started to look at the User Account and Authentication component in Cloud Foundry. Ran samples against cloudfoundry.com, modified the documentation. Issued first GitHub pull request

Wednesday

More hacking on samples. Updated the Cloud Foundry Google+ page. Agreed to present Cloud Foundry at the London Real-Time hack weekend for the RabbitMQ guys. Watched (and tweeted) the live webcast of the Cloud Foundry first birthday event in San Francisco – very exciting news! “more clouds, more community, more code“, including a broader range of partners, a new governance process around the cloudfoundry.org Open Source project, and the announcement of BOSH being released to the community, too – multi-cloud deployment, here we come!

Thursday

Trip to the VMware Frimley office for some HR stuff. First call with full Developer Relations team for event planning. Briefing with the two directors I work for. Nothing to see here, move along…

Friday

Setup new laptop (custom order from Apple so there was a small delay). Prepped for demo at London Real-Time. Started making a lot of noise on VMware Link (aka Socialcast, the social sharing/discussion platform) internally

Saturday

My first speaking gig – a Lightning Introduction to Cloud Foundry, taking Chris Richardson’s much more comprehensive Boot Camp presentation and cramming the essentials into ~15 min including a live demo, for a bunch of hackers at London Real-Time.

Oh, and it was caught on video.

Four or five of the hacks ended up running on Cloud Foundry, too, which I think was rather nice I was also interviewed on realtime and the importance of cloud at the event, but I’ve not seen that video appear just yet.

Monday

More laptop setup, HR stuff. Prep for Scala Days. Started to improve a sample app (Ruby/Sinatra) I’d used in the past by adding Twitter Bootstrap and restructuring the code.

Tuesday/Wednesday

Scala Days in London – helping to man the sponsor stand talking about Cloud Foundry, answering questions, and meeting many new colleagues from the US who were presenting on Spring, Scala, and Cloud Foundry (including an announcement that Play 2.0 framework support and standalone apps are coming to cloudfoundry.com Real Soon Now). Recorded a podcast interview with Uhuru about what a Developer Advocate does.

Summary

I’m pleased that I was able to be so productive so quickly. I’d had a little previous experience with Cloud Foundry but it’s a testament to how quick it is to learn the basics and get moving that I was able to rapidly start playing with a bunch of code. It was also exciting to be out on Github on my first day – not something I could have done in a former life… it’s nice to be working in an organisation that is innovating with Open Source at this level.

There’s much to learn, and to be honest, a couple of the key aspects of Cloud Foundry actually make it more challenging (and interesting) for me to get to grips with. It’s open, and with BOSH, can potentially target different IaaS offerings (initially vSphere and the beginnings of AWS support; a hackathon yesterday aimed at adding OpenStack to the list) – so suddenly I need to know about those. It’s a polyglot platform, which means I need to broaden my language knowledge – I’m already making a start on Ruby and node.js, to complement existing Java and PHP knowledge.

It’s also exciting to learn more about what VMware does, the layers of technology that they offer, and their vision. My previous experience has primarily been with the desktop virtualisation technology, but there’s a huge and vibrant community around the server-side virtualisation tools, and products like Socialcast, Sliderocket and Zimbra in the collaboration space too.

There’s a lot of exciting stuff happening in this space. It’s thrilling to be here. Thanks to all of my new colleagues for a warm welcome and support – looking forward to working with you!

I’m very excited to announce that, from April 10th, I will be joining the Developer Relations team for Cloud Foundry at VMware.

This is a thrilling opportunity for me for a number of reasons.

• from a technology perspective: Cloud Foundry is very, very, very cool. In my opinion, it really comes from a different set of thought processes than the other Platform-as-a-Service offerings out there, which make it unique and compelling.
  • the operating system stuff gets out of the way (why should it matter?), but multiple language runtimes and backend resources are available for easy scaling. Seriously, the first time I walked through the command-line tutorial and scaled a Ruby app to 6 load balanced instances with a single command, I was instantly impressed.
  • it is Open Source. The code is on Github. You can run your own cloud if you like. You can add support for your own languages and frameworks, much as AppFog have done for PHP, Tier 3 and Uhuru have done with .NET in Iron Foundry, and so on. This provides a huge amount of flexibility. Oh, and of course mobile and cloud go hand-in-hand, so last week’s announcement of FeedHenry providing tools to develop HTML5 apps to deploy on Cloud Foundry was really significant, too.
  • you can take your cloud with you using Micro Cloud Foundry – so the development and deployment model remains the same whether you are online or offline. I love this idea.

• for me, personally: it’s a natural evolution of much of the work I’ve been doing over the past few years – focusing on developer communities and promoting technology adoption, as much as top-down solution selling. As my good friend James Governor is fond of saying and as his colleague Steve O’Grady wrote, developers are the new kingmakers – and with trends like mobile, cloud, and devops, nurturing those communities is more important than ever. You don’t impose technology on a community – you explain it and earn your place and reputation.
• I’m looking forward to more speaking, more writing, more mentoring, and more online community building. These are things I’ve grown to enjoy (and in the case of the latter, appear to do naturally).
• I’ve followed Patrick Chanezon, the Senior Director of the team, since he was setting up the developer advocacy programme back at Google – I have a lot of respect for what he’s achieved and the way he operates, so I’m delighted to have the chance to work closely with him. I’m excited to join everyone in the team, of course – I have spoken with most of the group already and I’m really looking forward to learning from their diverse range of experiences and backgrounds.

Between now and April 10th, I have a few things planned including a vacation (!), heading to EclipseCon to talk about MQTT and M2M topics, and some other speaking engagements. After I start the new role, I expect I’ll join in on the Cloud Foundry Open Tour and start to meet folks. I’ll also be on the team for the GOTO conference in Aarhus in October – exciting times ahead!

Since the announcement of Eclipse Paho (an Open Source project under the Machine-to-Machine umbrella at Eclipse) there has been a fair amount of excitement in the MQTT community about the availability of IBM’s C and Java client code under an Open Source license.

The initial proposal and setup stages have taken a little while, but this week the initial availability of the C client code was announced on the Paho mailing list (Java will follow shortly).

Paho Quickstart

This is not intended to be a comprehensive guide – better documentation etc will emerge over time – but I thought I’d post a quick guide as a kickstart for anyone wanting to give it a look. I did this on 64-bit Ubuntu 11.10 – similar steps will apply on other Linux or UNIX platforms (note, the initial code contribution has a Makefile with rules which should work on UNIX, Windows, or z/OS).

Install the necessary packages to build code. NB git is for grabbing the source from Eclipse; build-essentials is a metapackage providing gcc etc on Ubuntu; and doxygen and optional graphviz are used for generating the documentation.

sudo apt-get install git build-essentials doxygen graphviz

Get the code from the git repository:

git clone git://git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.c.git

Quick build for the client library and documentation:

cd org.eclipse.paho.mqtt.c.git/src

make -f ../build/Makefile all

doxygen ../doc/DoxyfileV3ClientAPI

Once these commands complete, you should be left with subdirectories called <platform> and docs. In my case, <platform> was 64-bit Linux, so I had a binary at linux_ia64/libmqttv3c.so. There’s no “make install” rule at the moment, nor is there a rule to compile the docs so I had to run doxygen directly. In the future it would be nice to automate all of that, and also to build some test applications.

Opening docs/html/index.html in a browser reveals very nice documentation describing the client library, including some examples of how to use it. For example, in docs/html/pubasync.html there’s a complete listing for an asynchronous publisher application. I extracted that code into pubclient.c and decided to check that it worked!

gcc -Wall pubexample.c -L./linux_ia64 -lmqttv3c -lpthread -o pubexample

That command successfully built a binary called pubexample. All I needed to do was test it. The sample application assumes that an MQTT broker is available on localhost port 1883 – if you want to change that, simply modify the value of the static variable ADDRESS in pubexample.c – in my case I simply apt-get installed the mosquitto and mosquitto-clients packages onto my system, but I could equally have unzipped and run Really Small Message Broker – both start on port 1883 by default if not given alternative configuration.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./linux_ia64/

./pubexample

Waiting for publication of Hello World!
on topic MQTT Examples for client with ClientID: ExampleClientPub
Message with token value 1 delivery confirmed

It was trivial to test that a subscriber (mosquitto_sub in my case) also received the publication. Job done!

Getting involved, and other news on Paho

I mentioned that the Java client contribution should appear soon. One other piece of news this week is that the project’s sandbox broker implementation – based on mosquitto – has been spun up. That was posted on the Paho mailing list, and if you want to get involved you should definitely subscribe to that; start to track the Eclipse Bugzilla for Paho; watch the Paho wiki; keep an eye on the source repositories; etc.. I’m already thinking about getting an OS X build rule sorted out. If you want to test your sample code now, you’ve got the option of a local broker, the Eclipse Paho sandbox, the mosquitto sandbox, or various other implementations.

Oh – and please leave a comment on this post if you find this information interesting, or want to discuss where things are with Paho. I’ll be hanging out on the mailing list as well.

What about Bob? (or Andy, even!)

Well, although I’ve left IBM, I’m delighted that MQTT is now going Open Source – in fact that was one of the things that I really wanted to help to achieve before I moved on. I am really pleased that I will be able to continue to contribute to both Paho and the broader Eclipse M2M Industry Working Group. I’ll be helping to update the mqtt.org community site, and heading over to EclipseCon in Virginia in a couple of weeks’ time to talk about M2M and work with our friends from the Koneki project. If you are attending EclipseCon please come say hi to me – and you may be interested in Wes Johnson’s session on MQTT and Eclipse tools.

There are very cool times ahead!

My friend and former colleague, Peter Anghelides, is a rather good writer. In particular, he’s written a number of books and audio plays set in the Dr Who, Sarah Jane Smith, Torchwood and Blake’s 7 universes.

The Christmas 2011 Big Finish special subscriber-only release, The Four Doctors, contained characters named “Lady Cowen” and “Whitmore” – a lovely little reference to Laura and Tony, also huge fans of Dr Who (as well as hosts of the Ubuntu UK Podcast[1], and some of the folks behind OggCamp). Incidentally – I reckon The Four Doctors is probably one of the best and cleverest Big Finish plays, and certainly it’s my favourite. If you’re into Dr Who, then it is worth a listen.

A subsequent Companion Chronicle for Big Finish – Ferril’s Folly – contained a brilliant line which referenced another friend, my mentor Dr Andy Stanford-Clark (yes indeed, he of MQTT, mousetraps and ferries fame).

I’ve just listened to Peter’s new Blake’s 7 audio play, Counterfeit, performed by Gareth Thomas and Paul Darrow. I’d been tipped off via some tweets that there should be a nod to me in this one… initially, I’d jokingly whinged that it was only a concealed reference in the dialogue rather than a namesake character, but then I heard the line itself:

… but then he twittered on, about chronon bridgebuilding, and deep hyper messaging connections…

Genius! Thank you, Peter – very nice, and I’m honoured

[1] this also gives me a handy opportunity to mention that I was a guest presenter on the first episode of season 5 of UUPC, which was live-broadcast on the interwebs on Tuesday, and released as a download yesterday. Check it out!

Warning: long post! the first in a series covering some of the events I’ve attended or been involved with lately.

Background

At January’s London Internet of Things meetup, I had the privilege to hear Haiyan Zhang speak with passion about various topics, including how she had collaborated with hackspaces in Japan in the aftermath of last year’s earthquake and subsequent nuclear disaster.

It was only the first time I’d come across Haiyan, so I was surprised but delighted that she invited me to the OpenIDEO Make-a-thon this past weekend, after tweeting about events like the London Green Hackathon.

I had only a vague idea what to expect of the Make-a-thon. When I saw some of the project briefs being published ahead of the day, I knew that it would be a little different to hackathons and other tech events I’d been to in the past. The briefs spanned issues such as improving local communities, bike safety, and several supporting campaigns by Amnesty International to use technology to support human rights activities.

My initial impression that it would not be a “run of the mill” tech event was reinforced when I arrived at the IDEO offices in Clerkenwell on Friday afternoon – it was a very different crowd to the ones I typically encounter – full of product designers, makers, human factors specialists, as well as web coders and developers. I rocked up with a bunch of Nanodes and other electronics with a vague thought of doing something hardware-related, but in the event I didn’t get that far!

IDEO’s typical approach to design revolves around prototyping and directed brainstorming, and in the event we divided into 8 teams of around 6 each, with diverse skills but with common interests around the briefs on offer. Friday afternoon was spent first understanding and exploring the brief, and then rapidly prototyping a rough idea before presenting it to the rest of the group. Saturday was spent refining the idea and producing an “experience prototype” which was intended to have been tried out “in the real world” if possible.

Evolution of the “Karma Phone”

Several of the briefs interested me, but I joined the team focused on the concept of Postcode Gangs – how could we build something to develop and improve community facilities within a postcode – essentially an arbitrarily-delineated area – in London? We spent some time brainstorming ideas around what “makes” a community before needing to rapidly decide on something to build for our rough prototype.

“What if” – there was a ringing phone in the middle of the street – and on the end of that phone, someone who knew something, had something to offer, or who was needed something? “What if” – we could create a new local hub with current and historical information about an area, enabling people to explore and meet their neighbours?

So the first prototype of what came to be called Karma Phone involved a lamppost (named, erm, Dan!) with a phone on it, which would randomly ring as people passed by – people could call it with a need and that others could then try to address. On the other side of the lamppost (also known as, Dan’s back) we imagined a large touch display with information about current events, realtime information, historical maps, and so on.

Karma Phone – The Outcome…

The team changed overnight, as Hayley and James were not able to stay for Saturday, Lydia joined us, and Victoria could only be involved for a short time on Saturday. We weren’t all convinced that a ringing phone would be answered, that the system wouldn’t be abused, that there wasn’t a social barrier around providing home address and asking for help, etc. How could we get an actual phone into the street and ringing, too? So, Saturday morning involved some rapid rethinking of what we wanted to build!

We settled on what turned out to be a subtle evolution of the original idea – a public phone which could act as a hyperlocal information service and skills exchange.

While we were brainstorming how to hack a physical phone, run a long cable into the street, use a mobile chained to a metal box, etc etc I remembered Twilio, which I’d been following for a long time, but never had the chance to hack on in anger. Within about 30 minutes I’d demonstrated the ability to initiate calls between two users from a web page, to the rest of the team.

Steve and Dan set about implementing the web UI; Tim started working on a physical enclosure; Victoria and Lydia managed to source a real “traditional” phone handset; and I remained hard at work writing PHP to talk to Twilio.

A couple of minor wrinkles along the way:

• network issues meant that I had to use Tim’s phone to tunnel through to my webserver’s console, since it was apparently impossible via the event wifi. Evidently IDEO had just had a network provider change, so it was just an awkward time, but I lost some time fiddling with hosting in the early part of Saturday.
• at a certain point on Saturday afternoon, I realised that attempting to call from the Twilio web client on the iPad was never going to work… since it requires Flash. I thought of a number of workarounds, but the one that finally stuck was that we were able to use Skype on the iPad, and use the skype:// URI scheme to launch the app from the web client. It wasn’t seamless as we needed Skype credit, and also had to tap an extra “call” button in order to start the call, but it was good enough for a prototype.
• I’d wanted to make the web app, a standalone launchable web app on iOS. Weirdly, adding the usual meta tags to the page header to instruct iOS to treat the app as standalone launchable, meant that it was no longer possible to invoke Skype from within the web UI… so I backed off from that idea. The only cosmetic issue that presented was an inability to hide Safari “furniture” like the header, but that wasn’t a big problem for a prototype.

Here’s how the final system hangs together:

Impressive Outcomes…

I spent so long coding and tweaking on Saturday (the commented and documented code is here – ignore how short it might seem – it was an intense few of hours!) that I missed most of the physical assembly. Tim and Dan did an amazing job of creating an enclosure for the iPad and handset. It might have been made from foam board, a box folder, and vinyl, but the final result was beautiful. And most importantly – it was fully functional!

We would have loved to get the prototype out on the street for public testing (I suspect none of us more than Steve and Lydia!), but time worked against us. The final experience prototype was presented as a live demo with willing audience volunteers – one example call going to an answering service, and the other redirected to the local expert on Scotch Eggs (Tim!).

I’m happy to say that Karma Phone won Best Digital Prototype at the event, and I was (apparently!) Best Tweeter. Nice accolades

So – conclusions? I really enjoyed the way we worked together as a team of very unique and different talents; and seeing the Karma Phone prototype realised so brilliantly. However, I also think the experience of the Make-a-thon was humbling… listening to the experiences of people illegally detained abroad, and seeing some truly brilliant ideas from all 7 of the other teams, was wonderful.

A huge thank you to everyone involved in the first IDEO Make-a-thon – a really unique hackday. The IDEO team in particular looked after us brilliantly, with superb facilities, a great welcome, and more-than-adequate quantities of the hacker staples (coffee, sweets, pizza and beer).

Read a full recap including information on all of the project briefs on the OpenIDEO site. There’s a gigantic set of photos from the IDEO team, and a much smaller one from me shot on an iPhone at lower quality.

Tim, Dan, Hayley, Victoria, Steve, James, Lydia – Thank You. It was a pleasure!

All of the other teams – you rocked. You did great things. I salute you!

I’ve fallen seriously behind on blogging, and I’m less busy with work than I have been recently, so here are some of the posts you can expect to read in the next week or three… (I’ll come back and link this post to them once they are written!). Think of this as both a trailer, and an incentive to me to get these things written!

Event reports:

• London Green Hackathon
• Monkigras
• Hack to the Future
• bcs Oxford talk on Connected Planet
• IDEO Make-a-thon

Other “stuff”:

• what I’m doing next (!)
• Project Nanode
• Using MQTT for 2-way device control
• In defence of the Nintendo 3DS
• Defining the Empty Room Problem

This week, I tendered my resignation at IBM, after 10 years and 4 months, to a manager who has been my team leader and friend for the past 3 years. I can honestly say that it was a really hard moment; but also the right moment to make this particular transition.

As I’ve repeatedly written over the past few years – IBM has been a company I always aspired to work for, and once I had the chance, one that I’ve been immensely proud to represent. It’s a company that has endured over a century, and one that I was able to spend time with for a tenth of its existence – it was really the age of both WebSphere and the rise of IBM Software Group, and I’m grateful for the opportunity to have been there.

I have brilliant memories of the past decade. IBM is an amazing company and I will always value the chance to be a part of it, particularly in a wonderful location like the Hursley Lab. The people I’ve worked with, and with whom I’ve formed what I believe will be enduring friendships, have been simply outstanding. There were so many opportunities to do great things, not only in “the day job” but also as a BlueIQ Ambassador and social collaboration advocate, with IBM developer communities, in the universities programme representing IBM at careers fairs and as a guest lecturer in degree programmes, and the schools and community programme as a BlueFusion volunteer and mentor to kids at schools in deprived areas. I’ve also loved the chances to learn from others formally and informally, and to act as a mentor to others.

This will sound like a total paean, but it’s very true that there are amazing talents around IBM. In 7 years in IBM Software Services, and more than 3 years representing the development, strategy and product management teams in the lab back out to the field, I amassed a list of friends and colleagues from across continents, business units, and brands. It’s amazing to think of the broad reach of my network and I can’t help but be grateful for that.

My next steps are still forming; but I’m looking forward to spending more time with Open Source communities, with developers, with new technology, with connected systems and the Internet of Things, and as a speaker and writer. I’m also grateful to a range of friends for their support, particularly in taking over initiatives like eightbar, and in enabling me to remain involved in strands like Eclipse and MQTT.

Thanks for following me, reading my blog, sharing my thoughts, and joining the journey. I hope what comes next will be a continuation of the path I’ve been on; and an exciting next step in developing the direction I’ve been headed in.

For the past couple of weeks I’ve been thinking about the company I’ve spent the last 10 years of my life with – IBM. A few months ago I wrote about the company’s centennial. As an historian this has had me extremely engaged, excited, and interested.

In the last week I’ve had a couple of interesting experiences related to IBM.

First of all, I visited the Oxford University Careers in Computing Event.  I’d been up to Oxford in November for the wider University careers event, but I enjoyed the opportunity to talk with science students about what the company is all about. IBM helped to invent modern computing; to put man on the moon; it invented the PC, the floppy disk, various storage advances; it helped decode the human genome; it built the machines that defeated humans at chess and at Jeopardy; it is helping to build a Smarter Planet. It’s a great place to be.

Secondly, I helped to host some US colleagues in our UK lab at Hursley. I love Hursley and I’ve been enormously privileged to work there for the past few years. I remember my first experience of visiting IBM there as a customer in ~2000 – seeing the wonderful Wedgewood Room, the IBM consultant I was working with dropped the thought that one day I could work there into my head, and I’ve spent a long time wanting to work there, getting to work there, and then learning the history and showing it to others. Wonderful place.

I’m proud to have had the chance to work with an organisation that has helped to reshape and change the world. The quality of the people, the history of the organisation, and the amazing technology, has transformed my life.

Tagged: deep blue, history, hursley, IBM, oxford, University of Oxford, watson

The undead corpse of @dogearnation, rising again as new Games at Work podcast! bit.ly/AtPY2Z Braaains... I mean: Gaaaames!

—
Mikael Haglund (@0xdeadbeef) January 15, 2012

I’ve  been podcasting fairly regularly for the past few years, primarily with my good friends Michael Martine and Michael Rowe over in Durham, NC on a weekly show called Dogear Nation.

As I’ve travelled more, and as we’ve all got more busy, it became harder to keep that momentum up. I know I, for one, was tired and looking for some fresh inspiration. When we reached episode 200 last year, we announced a hiatus.

This year, we’re starting to ramp up something new. Same presenters, different format – going back to basics, if you will. Probably not weekly, more likely every two weeks… but continuing to explore some of the themes we’d been looking at around how gaming technology and concepts can influence business, work and productivity.

We’re still working on branding etc but you can grab the first episode of Games at Work right now via the existing site. We’d be interested to hear your thoughts.

This site will go dark tomorrow (18th January 2012) as part of the worldwide protest against the proposed US Protect IP Act (PIPA) and Stop Online Piracy Act (SOPA – currently shelved).

These are misconceived and threatening to the free and open operation of the Internet – and they have far-reaching implications that go far beyond the shores of the United States (got to love asymmetric extradition…). They need to be stopped.

It began, as these things sometimes do, with a childhood passion.

One of my earliest memories is of kneeling on the floor at the back of my bedroom making LEGO cars – it was in version 1.0 of my bedroom as I grew up, before new furniture and decoration. I must have been about 4, or 5. I had a castle, knights, some space stuff including base boards with little moulded “craters”… lots of fun as a child.

When I became a man, I put childish ways behind me.

I’d long known that many of my friends and colleagues have remained huge LEGO fans (Cerys has just blogged about her interest; Ben made some fun timelapse videos of building his Christmas present). For me, a key moment was Roo‘s 3 minute masterpiece of a paean to the medium at Interesting in 2008, embedded here for your enjoyment. Listen to the audio slidecast – closest you can get to having been there, and Roo did a wonderful (and amusing!) job.

Also, a memorable talk at the CRIM Crystal Ball Conference in Montreal in April 2010 (at which I also spoke) came from then Professor of Innovation at LEGO Group, David Robertson – a tale of Rebuilding LEGO, and how the company had saved itself from bankruptcy by refocusing on its core values and customer needs. It was a fantastic story and I was rapt.

More recently, I went along to the Internet of Things meetup in London last month, and was delighted to see Ken “monsonite” Boak – creator of the Nanode, a fantastic UK-grown prototyping platform akin to Arduino – use LEGO as his metaphor for a talk exploring Open Source electronics. Ken was kind enough to pop his slides up on Slideshare today, so you can take a look. He’d just been out to get some LEGO the previous weekend…

I want some LEGO. I own a single minifig. This is not good enough. Where can I get LEGO in Guildford.

—
Andy Piper (@andypiper) January 14, 2012

That talk was more-or-less the moment when I realised – I needed some LEGO. I wanted some. Both as a way of seeing where things had gone to, and to help me to prototype things, and just… well… just because! I’d already started to use dioramas featuring minifigs in a couple of presentations recently and had good feedback, so I figured that was another excuse

So, on Saturday I decided to dip back into my passion for LEGO. It started with a bucket of bricks from the nearest toy shop… but then I noticed the LEGO Star Wars sets with slight discounts[1]… and I figured well, obviously I’d need some wheels of some kind so picked up some City sets… and some of the foil-bag Minifigures…

The splurge quickly developed into a binge via a @darachennis-inspired trip to the LEGO store in Westfield White City on Sunday… picking-and-mixing bricks from the back wall, and signing up for the VIP program. There may be no hope left for me…

So what have I learned?

• Minifigs are brilliant. The aforementioned David Robertson gave me his business card, his details printed on a minifig resembling him, in Montreal in 2010 and that reawakened my interest. When I was a kid they all had the same pair of staring eyes and identical pleasant non-threatening smile, but the range of looks and expressions now available make them as much fun to customise as the full sets.
• People talk about the beauty of Apple’s designs – both inside and outside of the product (not that I’ve ever cracked open an iPhone to look inside). LEGO is blocky and “harsh”… but the designs and assembly process is beautiful. Assembling little cars and other sets on Saturday evening, following simple pictorial instructions, I realised that every piece had a place and it all fitted together wonderfully, perfectly. That (re)discovery had me as delighted as an adult, with a more architectural and design-oriented brain, as I was as a kid with the sheer enjoyment of being able to build and modify things.
• In my opinion, all kids should be given some LEGO, and allowed to build the models from the boxes themselves (much though I’m sure as an involved adult I’d be itching to take over!). I’ve blogged recently about my excitement for the maker culture, and this is really where it can all begin.
• I need to keep an eye on my bank balance, and a check on my excitement. I love it, but I bought it for “professional” reasons…

"children don't need to spend a year learning PowerPoint, and get a GSCE for it... They need digital Meccano" @ian_livingstone #futureteched

—
Andy Piper (@andypiper) January 12, 2012

Last week, the UK Government announced that ICT courses would be replaced with Computer Science, including a programming element (one of the campaigns I’ve been passionate about). At an event from The Education Foundation in London the next day – The Future of Technology Education – I was privileged to hear one of my personal heroes Ian Livingstone (of Fighting Fantasy books, Games Workshop and Eidos fame) speak and refer to “digital Meccano” – and I owned Meccano as a child too.  He also highlighted the need to combine science and art to push the digital boundaries.

Here’s what I think: we should be giving children a choice of physical LEGO, Meccano, and other toys; encouraging their creativity and building skills; and helping them to bridge between both the digital and physical worlds. No child should be excluded, and none should be pushed down a particular path. We should be supporting and helping every child to discover their passions and explore them; recognising that not every individual will want to program, or draw, paint, build, or write – but never belitting anyone for their talents or interests.

I’ve rarely been as excited about the future than I have been right now!

[1] as a child in in the 1980s I owned significant numbers of the Palitoy Star Wars figures and vehicles[2]. Whoever thought of combining LEGO and Star Wars is a genius – so much MORE FUN than the original, inflexible, non-customisable toys. So much more interactive, and through the video games, adding a humorous new twist on the Star Wars saga. LOVE.

[2] … I never had the Millennium Falcon or the Death Star, though… always wanted those…

Hot on the heels of the latest goodness in WebSphere MQ, it’s the turn of IBM’s Enterprise Service Bus – WebSphere Message Broker – to get a major new update.

WMB v8.0 was announced back in early October and has just arrived ready for  download in versions for distributed platforms, System z mainframes, and as a Hypervisor Edition for Linux and AIX (to be provisioned via the IBM Workload Deployer appliance).

As I did with WMQ last month, I wanted to take a moment to break out and highlight some of the key things in this release that you may have missed from the announcement letter. This won’t be a comprehensive list of everything, but I  want to point out some of the cooler features that you’ll want to be aware of. So, here we go…

(I’ve included a few screenshots to whet your appetite, click for larger versions!)

A simpler development experience

Version 8 brings a number of enhancements to the development experience, but one worth highlighting is what we call “Apps and Libs” – the idea that sets of message flows may be grouped into a unit called an Application which can be deployed, stopped and started as a whole. With Libraries, there are also truly re-usable assets like .esql files, or sub-flows, which can be deployed and updated separately, and invoked dynamically at runtime. This is a key change in the way that the Broker works – previously, sub-flows were compiled into the main flow and changing one required redeployment of all flows using it… they are now dynamically linked when needed, so they can be deployed and replaced more easily.

A new standards-based parser and message modeler

A new Data Format Description Language (DFDL, which you’ll sometimes hear called “daffodil”) enables any text or binary data to be understood within the message model. The Broker has had the “MRM” for many years, so of course could already do this, but DFDL is a new industry standard which can supersede the MRM (of course, you can continue to use your existing flows and message formats – you’re not forced to use DFDL). There’s a new mapper, too.

More importantly, coming along with DFDL and the mapper is a really, really nice set of utilities for testing message models inside the Toolkit – you’ll now be able to confirm that the model matches the test data without having to go through a full model->deploy-> test-at-runtime cycle. I saw this demo’ed at the WebSphere Technical Conference in Berlin during October and was blown away by it – it would have saved me a lot of time back in my consulting days!

Comprehensive .NET support

If you have .NET applications, assemblies, or services on the Windows platform, and you want to access those from your message flows – you can. If you want to write your message flow logic using C# or VB.NET or any .NET 4.0 CLR-supported language, using Visual Studio – you can.

If you don’t know how to get started with this stuff, the Toolkit has a new .NET Pattern to lead you by the hand and get you going quickly, and project wizards for Visual Studio.So, if you want a high-performance ESB platform that connects “anything to anything”, with minimal need to learn new skills, and run it on Windows with deep .NET integration – this release is going to cover your requirements.

Web administration

Delivered in version 8 is a first stage in making the Broker more easy to administer from a lightweight client – a web browser. Whilst power users and existing administrators can continue to use the Message Broker Explorer GUI, there is now an easy way to enable an optional web interface for basic administration tasks. Continuing the theme of simplicity the product has followed for a while, no additional moving parts (app or web servers) are required! Version 8.0 provides read-only views of running Applications and access to the log – more capabilities will be rolled into this interface in the future.

Record and Replay

Sometimes, when you are dealing with a set of end-to-end flows of data between applications, you may want the capability to record what is going on, and to replay specific scenarios and sets of events. This could be the case in audit, test, and many other scenarios. Another of the massive enhancements in version 8 is the Broker’s response to this requirement – again delivered using the same simple, lightweight interface offered by the web administration tool.

This also builds on technology around monitoring that has been progressively built into the Broker over the past couple of releases, so there are some really solid foundations and it is straightforward to set up.

Richer, yet easier to use

Just as I highlighted in my piece about WebSphere MQ 7.1, the Hursley teams have been strongly focused on “consumability” (translation for non-IBM-speakers = UX) for a number of years now. WMB continues to add capabilities that make it a richer, stronger integration platform, but also smooths out rough edges seen in earlier releases and is just… well… more productive to use. There’s even a drive to reduce the jargon and make the Broker logs more easy to understand, with new Activity Logging which aims to explain what a flow is doing in plain language (“GET message queue X”, “Update DB table Z”, and so on).

Taken together, the new wizards, web interfaces, integrated testing tools, message modelling tools, reduced dependencies, lightweight deployment with apps and libs… the combination just makes it a much more enjoyable experience for developer and administrators. And there’s a new installer, too.

The “papercuts” and node additions lists are huge: new JMSReceive node; new options for the File nodes; new Connect:Direct nodes; WS-ReliableMessaging support in the SOAP nodes; ability to install without root privileges; dynamic configuration of services without the need to restart execution groups… the list just goes on! Check out the product Information Center for more details on all of the features I just don’t have space to list.

… and finally…

Huge congratulations to some hard-working development teams in Hursley, Toronto and Bangalore in getting this release out there. As I’ve said before, I’ve been using the Broker for 10 years now and it just keeps getting better, and better. These guys are a very strong set of developers who turn out a fantastic, high quality product every time. Special thanks to MGK, @mqmatt, and @domstorey for some of the screenshots in this post

Footnote: version 8.0 is friendly to developers who use Ubuntu, too! Anton (my go-to guy on all things Debian – listen to him!) has some good advice about running WMB or WMQ on Ubuntu and Debian.

Related articles

• WebSphere MQ 7.1 is out – here’s why it’s cool… (andypiper.co.uk)
• European WebSphere Technical Conference 2011 (andypiper.co.uk)
• WebSphere MQ and Ubuntu (and other developer resources) (andypiper.co.uk)

Interesting situation recently. As long-time readers know, I’ve been a big fan of the Stop Blocking campaign for a number of years, and I tend to find it frustrating when I come across blocked networks. Trust and empowerment make me feel great in my job.

I’ve spent most of October and November travelling to speak with customers and present at a couple of conferences around Europe. In that time, I generally had very few problems with network access.

On one occasion though, I realised just how tricky things are becoming, as “social” elements become increasingly baked in to the fabric of the Web. I was in Switzerland, and the plan was for me to present locally during the morning, and then to host and facilitate a conversation with a number of my colleagues in the Hursley lab during the afternoon. The hosts arranged guest wifi network access for me, so that we could make this work. I’d be able to use Sametime to receive files to present locally (we couldn’t access LotusLive), to clarify questions with the remote team, and to coordinate other team members to join the conversation as we went along.

This plan was initially all looking good, until I found that the VPN connection I was using to tunnel in to the corporate network would suddenly and apparently randomly, drop in the middle of a conversation.

After a while these VPN disconnections became more frequent, I became more frustrated, and the meeting became less productive.

… and that’s when I looked at the piece of paper I’d been given with my guest network credentials. To summarise, it said that guests would be subject to all of the same restrictions as employees regarding network access and specific sites were disallowed including “Personal email: Hotmail, Gmail etc; IM: Skype, Google Talk, etc; Social networking: Twitter, Facebook, etc”.

The penny dropped that my browser was sitting there with tabs open on sites like Gmail and Twitter. I shut them, reconnected, reconnected to the VPN, and things…. were better…. well, better, for a while.

I still wanted to use the Internet, of course, so I continued to do so – searching Google for relevant issues when questions were asked in the workshop. That’s when the VPN started flaking out again…. and that’s when I realised that with the Google redesign, the +1 features in the header bar were accessing Google+ when I loaded the Google page, treating that as a “social network”, and silently dropping my wifi connection.

This was a case where a heavy-handed filter, no doubt designed to “protect” the users from themselves and the organisation from inappropriate behaviour, actually impaired real work getting done. Either this technology needs to get a lot, lot smarter; companies need to reconsider these blocking rules, and trust an increasingly savvy workforce to behave responsibly; or the Web just needs to stop getting so darned social and… troublesome. Which option do you prefer?

This may seem like a total non-sequitur after my past few blog posts – but it is something I feel absolutely driven to post. Via a tweet from Cory Doctorow, I learned that Anne McCaffrey has died.

I’m 35 years old. More than 20 years ago, I was at school, studying for my GCSEs and later my A-levels. One of the subjects I studied was English Literature. I love reading. I love literature. I love imaginative, creative writing.

There was, obviously, a set curriculum of texts I was expected to read, learn, and internalise. Shakespeare, Dickens, Hardy, and others. I’m glad I have that grounding. I was also allowed to read anything I wanted, from an early age – and I gravitated towards novelisations of Star Trek, of the Neverending Story, the Lion the Witch and the Wardrobe, and other science fiction and fantasy stories.

Around the age of 12 or 13 I stumbled upon the Dragonriders of Pern series. At the time, I was interested in Games Workshop and Warhammer… a fantasy world involving dragons and, ultimately, a rediscovery of technology, was an obvious step.

So, I started to read the Pern chronicles. I remember reporting on them in my “reading diary” aged around 12 or 13 – a series of books about a near-mediaeval planet where dragonriders saved the population from the deadly Thread. It wasn’t until I read The White Dragon that I really appreciated that this wasn’t just a trash teen fantasy series – themes of erotic passion, love, independence, adventure, and intelligence were involved (and would connect to science fiction, computing and other directions beyond that tale).

I’m deeply saddened to learn that Anne McCaffrey has passed. Her tales and her books truly did light up my early teenage years. I loved the Dragonriders of Pern stories and I hope that others will connect with them in the same way in the future. Thank you, Anne.

I’ve been fairly quiet about the latest software from the Hursley lab here on my blog – although, over the past few weeks since the announcements back at the start of October during the European WebSphere Technical Conference, I’ve definitely been speaking about WebSphere MQ v7.1 and WebSphere Message Broker v8.0 – two exciting product releases.

I’m going to spend this post talking about WMQ 7.1, which became available in electronic download form for the distributed platforms last Friday (z/OS will follow shortly). I’ll return to talk about all the (über)-coolness in Message Broker a little closer to the release date for that product.

So what is the big deal in this release?

It brings parallel / multi-version install

From version 7.1 onwards, there is now the capability to install more than one copy of WMQ on a system, for Windows and UNIX platforms. This includes installing alongside WMQ v7.0.1.6 (fixpack 6 on v7.0.1, the minimum level for multi-version install to work) – you can have one copy of v7.0.1.6, and multiple copies of 7.1, for example – and future versions will also be able to be installed in parallel, should the need arise. This should make migration and testing simpler. Applications can now point to their “own” install of WMQ if required. The GSKit installation, which provides some of the security functions for the queue manager, now gets installed “inside” the main installation as well, to make the whole thing more self-contained, and potentially easier to embed into other solutions if needed.

Here’s a teaser image from a Windows system that my colleague “mqjeff” sent me earlier today he has 7.0.1.6 and 7.1 on the same machine.

It’s (even more) secure

WebSphere MQ has always had a number of strong security capabilities, including SSL for channel authentication and encryption, and fine-grained access control of queue manager objects via the Object Authority Manager. It has also been possible to add transparent, per-message / per-queue / per-policy on-disk encryption and signing of message data via the Advanced Message Security feature. In v7.1, a renewed focus on end-to-end security adds the ability to authorise on a per-IP/user connection basis, as well as adding more crypto algorithms and additional authorisation options, and making much more of that security function available via the MQSC administration tool. T-Rob has a much more complete post about these changes so I won’t go into any more detail here.

It runs better, on bigger systems

Bigger systems… like the z196 mainframes? Well, that’s one example, yes, but WMQ v7.1 has been more optimised for big and multicore systems in general. On the mainframe, there are a bunch of great enhancements such as increased resilience in dealing with shared queues in a coupling facility, and the introduction of Shared Message Data Sets (SMDS) to significantly improve performance there as well. Let’s just say that the performance numbers for z/OS are looking really, really good… which brings me on to…

It continues to push the performance envelope

A major focus on performance in the v7.1 cycle has produced some fantastic results, and when the performance reports appear (as SupportPacs, within the next few weeks), you’ll see the “fastest WMQ ever”. This theme runs throughout everything: not just the base runtime messaging, but also things like making the WMQ Explorer tooling significantly snappier to operate as well (oh, and that’s now 60% smaller, and more sleek!)

There is also a new option for publish/subscribe applications – the ability to publish on a topic via multicast. This re-uses some of the technology from the WebSphere MQ Low Latency product so that it can run very fast. After the initial application startup, it means that applications can also operate when the queue manager is not available.

It adds Telemetry to the base install

No surprise that I’d highlight this one (it is also an important part of the overall story, per the next heading!) – I’ve been talking about the IBM implementation of MQTT, the open protocol which is being standardised and which it was just-announced will be part of the Eclipse Paho M2M project, for the past couple of years.

In WMQ v7.1, there is no longer a separate installation to run in order to add this support. On the platforms where the Telemetry feature is supported – Windows, Linux IA64, and (new in v7.1) AIX – this is now an optional part of the base installation. That means it is very easy to try out. Oh, and as well as being integrated with WMQ Explorer, the full range of Telemetry objects can now also be administered via the MQSC command line.

It brings the family together

This is a big one, in my opinion. I’ve mentioned that WMQ “base” can now interoperate with WMQLLM via the multicast publish-and-subscribe support; and the WMQ Telemetry functionality is “in the box” as part of the installer on the relevant platforms.

Why do these things that matter? Well, as I mentioned in my recent MQTT FAQ, something that IBM has observed over a number of years of building and delivering production-ready messaging middleware is that one size does not fit all. There’s the fundamental transactional messaging backbone (WMQ base) which needs to be solid, reliable, and easy to administer through comprehensive scripted and graphical tools… but beyond that, there are some additional qualities of service that need to be considered. There’s the very high speed, low latency use case which may be very specialised (WMQLLM), and there’s the need to deal with small and constrained devices and less-reliable networks (WMQ Telemetry / MQTT). Of course, you may also want to perform file transfer over that infrastructure (WMQ File Transfer Edition), secure your messaging (WMQ AMS), or route and transform your data and connect with “foreign” systems via different protocols (WebSphere Message Broker). I’ve been talking about this as part of IBM’s Messaging Vision for a number of years and it is really showing through in this release of WebSphere MQ. It’s a complete story.

It addresses many “papercuts”

On top of all of that… the team has really tried to address many of the common papercut issues, by which I mean the gotchas, annoyances, and the “wouldn’t it be so much better if….”s. Things like, gosh, I wish I knew what version of WMQ that client is using to connect to me? (yep, you can find out now).  How about “bind on group” for messages in a cluster? The ability to backup / dump and restore the configuration of a queue manager without needing to use a SupportPac? There’s a real sense of “fit and finish”, and I believe that shows that the development team have been listening to feedback and making the tweaks that users have been asking for where possible.

So – all-in-all, there’s a lot in this release that makes it worth a look, either from the perspective of users who are looking at an upgrade to gain performance, security and usability benefits; or for those looking for a solid, dependable messaging platform which can support modern applications. There’s a lot of excitement and innovation going on in the “traditional Message Oriented Middleware” space at the moment and WMQ and the related protocols like MQTT are right at the heart of those trends.

To learn more about the features I’ve talked about, and some that I haven’t, check out the online Infocenter. You can also check out the “What’s New in WMQ v7.1″ presentation from the WebSphere Technical Conference, via T-Rob’s blog.

There has been a lot of coverage over the past couple of days of some exciting announcements that I’ve been involved with at work. I’ve spent the past three days at EclipseCon Europe 2011, which doubled as the 10th birthday celebration for the Eclipse initiative. It was a funny feeling, because Eclipse started just a few weeks after I first joined IBM, and although I’ve used it and watch it “grow up”, I’ve never done EclipseCon before. The reason I’ve been out there for three days this time (as a WebSphere Messaging guy rather than a Rational tooling or build person, for example) was to get involved with activities around these announcements.

It’s all about machine-to-machine (or M2M) communications, Smarter Planet, and the Internet of Things.

Before I dive in to this, a few clarifications. First, I’m being described in a couple of news stories as “an IBM distinguished engineer”, and whilst I wish that was true, I’ve yet to ascend to those heights! Also, there are various numbers being quoted – note that the figures in the press release were not invented by IBM, the headline number of an expected 50 billion connected devices by 2020 comes from a recent study conducted by Ericsson AB. Oh, and this isn’t about a “new” protocol – MQTT has been in use since 1999.

The other clarification is that some articles seem to suggest that IBM is out to create some kind of new, alternative, Web – that’s not what has been announced, and I’m certainly not aware of any such plan! It’s about connecting “things” – sensors, mobile devices, embedded systems, even small appliances or medical devices for example – to the Web and the associated platform and ecosystem of technologies, not about reinventing or recreating them. I’m personally a huge fan of the Web as a platform

Oh, and of course, the obligatory “all opinions expressed are my own” – this is my understanding of where things are going, although of course I’m talking about events I’m directly involved in!

So what is this all about?

Two things.

1. On Nov 2, IBM, Eurotech, Sierra Wireless and Eclipse formed a new M2M Industry Working Group at Eclipse. Sierra had already started the “Koneki” project at Eclipse to work on M2M tools, and the Working Group will look at a range of topics together, such as M2M tooling, software components, open communication and messaging protocols, data formats, and APIs.

2. On Nov 3, IBM and Eurotech announced the donation of their C and Java clients for MQTT to a new Eclipse project called “Paho” which is under proposal in the incubator – with code expected to hit the repository within the next couple of months. MQTT is being given to Eclipse to live within the M2M ecosystem that is emerging there, and to provide an avenue for adoption of the protocol as a more pervasive standard for connected devices.

How is that news? Isn’t MQTT already open / free?

Technically… kinda, sorta

The MQTT specification has been published under a royalty-free license for some time, and that has led to a fantastic community contributing a range of different projects. IBM and Eurotech took this approach from early on, because it wouldn’t have been possible to compile and support code on every embedded platform that might come along – far simpler to set the protocol free.

Initially the specification was hidden away in the WebSphere Message Broker documentation, but last year it was republished, moved to a new home on developerWorks, and the license was clarified.

In August, IBM and Eurotech announced their intention to take MQTT to a standards organisation. The specific organisation has not yet been finalised, but this is also an important step in ensuring that MQTT is not “just” an IBM protocol, but something of general use which the community can feel comfortable with. If you’d like to join that discussion then there’s a Get Involved page on the mqtt.org community site.

The missing piece was code – a reference implementation, if you like. That’s one reason why the Eclipse Paho announcement is significant.

Why else is this significant?

Well, here are some of my musings on that one:

• it shows IBM is serious, by committing code and open sourcing it (as with the original Eclipse donation in 2001);
• the M2M Industry Working Group exists to foster the discussion in this space;
• it makes high-quality reference Java and C client implementations freely available in source form, with a good Java implementation something that has been particularly lacking;
• it creates an opportunity for Eclipse projects to use MQTT, and to develop tools on top of it.

The press release and Paho project proposals aren’t clear (to me) – what exactly is being donated?

IBM is seeding Eclipse Paho with C and Java client implementations of MQTT. Eurotech is donating a framework and sample applications which device and client developers can use when integrating and testing messaging components.

Why C and Java clients (aren’t they “dying” languages?) Where’s my Perl and Ruby code?!

IBM had previously made some C and Java code available in some SupportPacs, but those are outdated and the license for reuse was never clear.

It’s important to realise that this stuff came from the embedded world of 10 (and more) years ago, and continues to be applied in that industrial space. That category of device typically runs some kind of realtime Java-based OS, or a Linux-based or other runtime with a GCC toolchain for the CPU in question. C and Java are genuinely the most useful implementations to get out there. Oh, and on that “those old languages” thing – I think you’ll find they are very widely used (Android, iOS etc run variants of sorts, most non-web app development is likely to be in one or the other).

We’re very fortunate that clients libraries for a wide range of languages already exist thanks to the MQTT community – see the list at mqtt.org!

Hold on… don’t we need a broker / server / gateway?

Yes. But, one step at a time!

There are brokers available for free today, either as precompiled binaries or as full Open Source implementations, so this is not a dead end from day one.

The Paho project scope outlines the intention to add a broker to the project in the future, and to host an M2M sandbox for developers as well. That is where we are today, and this position will evolve over time.

Why Eclipse?

The Eclipse Foundation has been a fantastic success story (oh, and, Happy 10th Birthday, Eclipse!). As the scope of their mission has broadened beyond an IDE to the web, build environments, and all kinds of other tools, it was a good place for Sierra Wireless to kick off the Eclipse Koneki M2M tools project, and is now a natural place for this primarily M2M protocol to be hosted under Paho. As James Governor notes in his write-up of the news:

… the Eclipse Public License is designed to support derivative works and embedding, while the Eclipse Foundation can provide the stewardship of same. One of the main reasons Eclipse has been so successful is that rather than separate software from specification it brings them together – in freely available open source code – while still allowing for proprietary extensions which vendors can sell.

How quickly will the code donation happen?

The Paho proposal tentatively includes dates in November and December 2011 – there will need to be various approvals as code is accepted into Eclipse, so that may “flex” a little, but it is all in the pipeline.

OK… Why MQTT? Why not HTTP/XMPP/AMQP/PubSubHubbub/WebSockets/etcetcetc?

To answer this one adequately I’d probably end up addressing each individual difference between protocols in turn, and if you’ve heard me speak about MQTT I’ve covered some of this before – so I’ll keep this answer relatively brief. I will admit that I’ve been asked about all of these by journalists in the past couple of days.

There is space for a range of protocols to coexist, because they address different areas. In the messaging space, we’ve found over time that whilst efforts to create a single protocol have been made, that has often ended up as focused around a particular set of qualities of service, and not optimised to cover the the whole range of them.

For example, if we look at IBM’s own messaging protocols – there are several. There’s WebSphere MQ which is all about reliable, transactional, solid, clusterable, enterprise, JMS and other APIs, etc etc.. WMQ itself isn’t ideal for very high-speed in-memory or multicast scenarios, so there is also WMQ Low Latency (interoperable with the new multicast feature in WMQ 7.1, but a separate protocol). Neither WMQ LLM or WMQ scales down to unreliable device networks and embedded systems, so there is WMQ Telemetry (aka MQTT), which was specifically designed for constrained devices and networks, and that can interoperate with the main queue manager, too. Oh, and sometimes you want to deal with files (WMQ File Transfer Edition), or access message data via HTTP (WMQ HTTP Bridge). You need to address a range of requirements in a messaging story.

So why not those others? In this case, IBM believes that MQTT is ideally-suited to the Smarter Planet Instrumented->Interconnected layer – it’s tiny, not synchronous and brittle, isn’t specific to the web as it is all about data rather than documents, XML etc etc. In these scenarios, REST principles may add an overhead. Oh, and it has been around for over 10 years, and has been proven across a range of industries and in a range of extreme conditions. IBM’s commercial implementation is known to scale to hundreds of thousands of connected devices, and we know that is the direction that this space is heading.

Congratulations! / Thank you!

Thanks, but don’t congratulate or thank me! I’m familiar with this stuff, I’ve coded with this stuff, but I didn’t invent it and I didn’t write it. There are some amazing folks at both IBM and Eurotech (and some who have moved on) who started this all off in 1999, and who have helped to implement solutions using this protocol since then, and who have of course developed it. Several of them are on Twitter if you want to say hi! And huge thanks again to the community of folks that formed around mqtt.org and contributed client and server implementations – that absolutely helped to move things forward to this point.

HERE ENDS TODAY’S Q&A!

That, hopefully helps to clarify a few things and answers some of the questions I’ve seen via Twitter, forums, and mailing lists over the past few days. It has been something of a blur, to be honest, but a lot of fun. I’m looking forward to the next stage – working with the community more, working with our friends at Eurotech, Sierra Wireless and elsewhere, and making the M2M space much more real.

For more, here are a bunch of stories I’ve seen in the past couple of days… no particular order, just my cut-and-paste list!

• Smarter Planet blog
• mqtt.org news
• Bob Sutor
• ReadWriteWeb
• PCWorld
• VDC Research
• WebWereld (NL)
• Technolenta (RU)
• ConnectedWorld
• Simon Phipps at ComputerWorld
• Complex.com
• Developer.com
• James Governor
• H-Online
• mosquitto blog
• Design Thinking Thoughts

Although I realise that it seems as though I do little other than spin around “the conference circuit” at the moment what with the various events I’ve blogged about lately, that isn’t entirely true! However, it is just about time for another European WebSphere Technical Conference – something like a cut-down IMPACT run in Europe, a combination of the popular WebSphere and Transaction & Messing conferences we used to run – with plenty of technical content on the latest technologies.

I’ll be in Berlin next week 10th-14th October, participating in at least one panel, speaking about MQTT, and also covering the latest on IBM MQ messaging technologies as they relate to cloud and web. There’s a Lanyrd event page where I’ll try to collate information relating to the individual talks.b

I have a feeling that by this time next week there could be quite a lot to talk about…

Related articles

• WebSphere MQ and Ubuntu (and other developer resources) (andypiper.co.uk)

Last week I was privileged to be invited to give the closing keynote at an event called ReLIVE 11 (Research and Learning in Virtual Environments) at the Open University. This was certainly a big deal for me as I was in the company of some brilliant academic minds and some tech celebrities – plus, the OU is an important and well-known institution (despite the fact that I heard Leo Laporte say that he’d never heard of it on the MacBreak Weekly podcast I was listening to as I drove to Milton Keynes last Tuesday evening!).

I’d previously explained to the organisers that I hadn’t spent so much time exploring virtual worlds lately as I was doing three or four years ago at the height of IBM’s involvement with platforms such as Second Life and our own internal Metaverse. Having said that, I have spent more time with gaming platforms such as XBox and the Nintendo 3DS since then, and more recently also Minecraft. Naturally I did have that business perspective and story to share… and, as the closing keynote I had the interesting task of pulling together the threads we’d covered during the breakout sessions at the conference, as well as attempting to look ahead to what trends might be important in the future.

The video is online via the Open University website and the talk with Q&A lasted for about an hour. More coverage of ReLIVE 11 is aggregated on Lanyrd.

Summary

As I noted in the opening and closing sections of the talk – predictions of the future are a hit-and-miss affair. We may now have tablet computers arguably even cooler than the Star Trek padds and communicators, but I’m still waiting on my hoverboard. Nevertheless, I tried to frame the story of IBM’s exploration of virtual worlds and 3D environments with some discussion of trends. It also gave me an excuse to talk about Back to the Future, and a cool ad that Nike recently released tying back in to the movie.

I want to reiterate (as it may not have been clear from tweets that emerged during the event) that these were very much my own thoughts and not the views of my employer – in fact, I was attending the event in a personal capacity. So, per the presentation, my thoughts on trends to watch in the next five years:

1. 3D Printing: I’ve seen RepRap and other 3D printers more often in the past couple of months than ever before, and it is clear that prototyping and fabrication are coming within financial and technical reach of more than just the early adopting minority. That’s not to say this is something I see going “mainstream” – but as access opens up, expect to see many more interesting things happening here.
2. Social broadcast: I think “TV” is rapidly giving way to a more generalised broadcast media that is being consumed across multiple devices, remixed, shared, etc. I also think that social streams are adding to the experience of how these media are being consumed, as evidenced by hashtags broadcast on BBC programmes, and the ways in which conversations form online around events and video streams.  A nod to my friend Roo Reynolds too, a man constantly way ahead of his time…
3. Touch and Gesture: we already know that the ways in which we interact with technology is evolving fast. Watch any child approach a large screen and attempt to press the screen, expecting their cartoon hero to become interactive. This is not going to stop – Microsoft have some amazing technology in this space with Kinect and we should get used to and embrace the changes as they happen if we want to evolve.
4. Big Data: a nod to my own organisation’s Smarter Planet story, and an acknowledgement that every one of the major tech firms is investing in ways to store, mine, slice and analyse the increasing amounts of data flowing in from the environment and our personal signals. This is just a continuing story, but we’re at a point where it is a red hot topic. It would have been a good point to mention Watson, if I’d thought on my feet quickly enough!
5. Identity: this is not so much something where we will see technical progress necessarily, as an area I think will be a threat, and difficult to resolve. The nymwars of Google+ are one edge of the issue. I believe that there is a real tension between the freewheeling days of the earlier Internet, the desire of individuals to make their own choices about identity (often for valid social reasons, other times for vanity), and corporations and political entities that want to close this situation down. This is going to be a tricky one.

So what of virtual worlds? Three words: Not Gone Away. They may have morphed, lost their early shine, the bubble burst – but we have a range of immersive experiences (and social, but not necessarily immersive ones) through which we interact. I mentioned Minecraft and how that is being used for teaching. I talked through IBM’s work with serious gaming. I spoke about the IBM Virtual Center briefly, and that’s online and used today – in fact Jack Mason just posted a nice deck on that which carries some statistics, if you want to learn more.

Thoughts on education

I clearly was not the most experienced individual in the room when it came to discussions about teaching and education, and I particularly enjoyed hearing different presenters at ReLIVE11 talk about how they are using OpenSim, OpenWonderland and other platforms. However – after my recent post on Raspberry Pi and my exploration of the Brighton Mini Maker Faire I’ve been thinking increasingly about Maker culture and how we could bring technology teaching back around to practical matters.  I was disappointed to read the Government’s (lack of) response to John Graham-Cumming’s recent letter on the same subject, though.

One of the things that I called out as a barrier to the adoption of immersive worlds and new technologies at work is something I’m calling The Empty Room Problem – the fact that unless you build it and then populate it, they will not necessarily come. I’ll be writing about this some more shortly, prompted by Derek Jones’ great blog post.

During the Q&A session I gave an answer to one of the questions which contained some ideas I’ve had on a possible curriculum – I’ll try to expand on those in the near future as well.

Chris also talks about his upcoming session at the East Coast Gamers Conference… Check it out here!

Links:

Zynga Buys OMGPop

Is Google Being Evil?

Where’s the FUN?!?!? – That’s the question that Michael and Michael ask this week, while Andy is out and about.  We focus on a few game examples including the new Matching game called InstaMatch.  We also look at fun and interesting progress systems, and some not so fun… Spinning beach ball we’re talking to you!

Next episode we are looking forward to talking to one of the Authors of the book “GameStorming”

Bonus Links:

Nest thermostat

ePredator’s thermostat stats

Games at Work goes back to the early roots of what brought your hosts together in the first place — a love of games and how the intersection with business and social technologies make them more than simple pastimes. The podcast will explore where games are being used for business results today and how existing games may be adapted to new ways of creating social, economic and other values.

We’d like to thank Zebes System for their awesome version of the MoonPatrol theme.

Your hosts are eager to interact with you via multiple paths — you can use 140 characters on Twitter, send an email or even a gamecenter friend request! Check us out at twitter.com/gamesatwork_biz .

One thing that we agreed on;however, was that we should start blogging on the site to let you know what is going on. And perhaps even include some of the exciting things that we may have talked about on the show, had we recorded a show.  So I decided to share first.

Back in September I was lucky enough to participate in the very first vPEARL conference.  This event, co-sponsored by e426.org, IEEE, and VeCoLab, was all about the topic that founded this podcast – leveraging gaming technologies and virtual worlds for education, business, and entertainment.  I participated as part of VECoLab, helping to define and organize the event, but everyone there also was an active participant in the activities.  We had a bunch of virtual worlds provide environments so that the participants could experience them, while coming up with innovative solutions for real world problems.  The challenges included nutrition, extremely large virtual events, and many others.  The speakers were a fascinating array of Gaming, Entertainment, and Social innovators – highly recommend that you check them out at VeCoLab. There was also a documentary being filmed of the whole event, which I can’t wait to see.

The event ended with a set of presentations by each team on how they could use this technology to solve their challenges, and the VeCoLab team captured all the practices as the foundation for a database of best practices.  All participants agreed to provide these as part of a Creative Commons license.  Another event is tentatively being planned in 6 months.  Can’t wait to see this unfold.

We hope that you will come along for the ride with us.  We are currently working on a logo, intro music, etc.  We would love to hear from you with your recommendations.  So Follow us on twitter at GamesAtWork_Biz.

1) We think the show has too much breadth of topics, causing us to compete with shows like TWiT.  We’d like to focus on a major theme.  This is where we started.  Do you agree?

2) We wondered how do you (the listener and tagger) tell your friends about our little show?  Do you tweet when one of your stories get talked about?  Do you comment on the blog?  Have you rated us on iTunes?  If not, why not?  We need your help to grow the show.

3) What are some of your favorite podcasts?  Drop us a comment to this posting and let us know.

There’s a lot of exciting discussions going on between the co-hosts and we are all looking forward to doing more episodes after this break.

[reposted from my personal blog]

We’re just about to record episode 200 of Dogear Nation, a regular (weekly, barring a few more recent gaps) podcast summing up opinions of what’s new online and in tech.

I actually only joined Michael Rowe and Michael Martine (the co-hosts) as a regular element of the show around January 2009, and by then they had already clocked up 80+ episodes. Either way, I reckon on around 250 hours of recording time, and a little more than that when I include the few episodes I also edited; 2 and a half years of regular podcasting.

So what have I learned from this exercise?

• Michael Rowe and Michael Martine are extraordinarily generous, friendly, and wonderful guys – I’ve enjoyed working with them and sharing ideas and opinions. I have two amazing friends for life, built through a digital foundation.
• It’s difficult to keep a weekly podcast going, even with the regular input from listeners. It’s also difficult to drive and expand an audience. We’ve had some great contributors and regular listeners and I’m grateful to them.
• Preparation is (nearly) everything. Over the course of the past couple of years we’ve evolved the way that we put the show together, finally arriving at a shared Google document where we co-edit the show notes to build the structure of the show. That’s really helped us to build momentum. It’s still good to have some ad-libbing and free discussion of course.
• Technology keeps evolving. We knew this of course – one of the premises of the show has always been about the bleeding edge of technology and where things have been moving. Well, I think it’s fair to say that over the course of the past few years on the show, we’ve seen various products go from science fiction to the beginnings of science fact.
• Go with what you know. I’ve had a lot of fun talking about the things that fascinate me and that I’m passionate about. I know that Michael, Michael, and before me, Matt Simpson, have also made their best contributions based on the areas that they’ve known about or have been most curious about. For example, amongst other things, Michael Rowe is a space / NASA buff and a hardcore gamer; Michael Martine is very business-focused, and hates anything that threatens to go near his eyeballs

It’s been a fun couple of years.

We’re just going now through the listener tags for the show today.  The first 200 episodes has been a great learning expereince for me — and I’ve strived to make it for each one, sometimes from airports, sometimes from parking lots, from both home and “in the field”.   I’m looking forward to taking a short break, and then getting started on the next 200 episodes.

Thanks to our listeners, our taggers and our families for their support, encouragement and tolerance (especially tolerance!) over the past few years.  And a huge thanks to Michael and Andy — I couldn’t have a better couple of cohosts to work with.

To me it means that we hit a milestone that very few podcasts achieve.  When Michael and I first started this podcast, we didn’t think of it as a long term project.  Between episode 1 and 2 (before we actually started counting them) it was months.  One of the pieces of advice that I got back then was that if you are going to start a podcast was that if you don’t have ideas for 40 good shows, you should start, because you will run out steam.  I have started two other podcasts over the years, neither of which survive to this day.  The first one was ‘Trippin’ the verse” – a podcast about getting started in virtual worlds, and “Random Abstractions” – a Kindle book review podcast.  Trippin’ the verse had 16 episodes, when it ran out of steam.  The goal was to answer the basic questions, and well.. there were about 16 of them.  Random Abstractions got to episode 10.  Reading a book a month, between work, podcasting, and other activities, was fun, but the angle that was unique was about the technology of the Kindle and ebooks in general.  Again, it just ran out of steam.

Dogear-Nation is all about what you – the listeners – tag.  So we never really run out of steam.  However, the subjects have changed over time.  We started around playing games at work, i.e. using gaming technology in a business setting.  Michael’s prior post about gaming alone harkens back to that basic premise.  The gamefication of business is a logical outcome of some of our earliest thoughts and episodes. I am personally looking forward to tomorrow’s recording, and the next 200 episodes, wherever they may lead us.  Thank you for being a part of this process.

A spirited conversation with my family two nights ago centered on the classic argument that games are addictive and because of this, can have very bad outcomes for the people that play them.  As you may have expected, I found myself very quickly on the pro-gamer side of the equation, since I am a believer in the power of games to help people collaborate, develop leadership skills, and be enouraged to try new, innovative ways in solving problems.

That conversation got me thinking again about the song that I’ve been kicking around for a while – a fun parody on several levels, since playing games alone, while retaining some of the benefits of gaming, does not get all of them.  While writing the song, I was reminded of the competitive Mario Bros games that my friends and I used to play on the Apple II (thinking it must have been the Apple IIe from the wikipedia timeline  – object of the game was to get more points than the other player – by any means.  A favorite trick was hitting the “POW” block just as your opponent was about to kick a turtle off the ledge, to flip it back over and have it bite your friend, causing him to lose a life in the game.

There are times when I do prefer to game alone too, and come to think of it, it’s been a very long time since I’ve been on a raid with others…  Must fix that.  Enjoy the song – hyperlinks added for your extra enjoyment.

I Game Alone
sung to George Thorogood’s I Drink Alone

I game alone, yeah
With nobody else
I game alone, yeah
With nobody else
You know when I game alone
I prefer to be by myself

Every morning just before breakfast
I don’t want no mana, coffee or tea
Just me and my Infocom text adventures
That’s all I ever need
‘Cause I game alone, yeah
With nobody else
Yeah, you know when I game alone
I prefer to be by myself

The other night I lay sleeping
And I woke from a terrible dream
So I called up my pal Duke Nukem
And his partner Leisure Suit Larry
And we game alone, yeah
With nobody else
Yeah, you know when I game alone
I prefer to be by myself

The other day I got invited to a party
But I stayed home instead
Not to be grinding or crafting,
Just me and my pal Sid Meier
And his brothers Techmo and Mario
And we game alone
Yeah, with nobody else
Yeah, you know when I game alone
I prefer to be by myself

Yeah, my whole family done give up on me
And it makes me feel oh so bad
Don’t want to be PvPing in a MMORPG
Just give me my plain ol’ Plants –
versus Zom-om-bies
And I game alone, yeah
With nobody else
Yeah, you know when I game alone
I prefer to be by myself

I’ve been doing a bunch of thinking about our 200th episode and what got us started.  It all began with a love of using gaming technology.  Not just to play games, but to change the social interactions that people have at work.  The podcast originally was called “Imagine Playing Games at Work”.  There’s a great Farside cartoon.  Mike Lynch has a great blog post about the cartoon that inspired me to think about this.  Well not really inspired me to think about it, but puts things in perspective for a lot of people.

What got me thinking about it was the idea that I was spending way too much time playing Star Wars Galaxies – leveling up my crafting skills, and all to create virtual knowledge within the game.  If I could get the same benefit around leveling up my Java skills, or business analytics skills, I too could become a business Jedi!  The very behaviors which are rewarded in playing games, problem solving, creative thinking, and pattern recognitions, are the things that are valued at work.  How could I get these two radically different worlds to recognize the value the other had.  Michael Martine and I took off with this idea and created a BizTech project within IBM.  This was a way to get 20% of your time to work with a distributed team and build out that idea.

I created my first 3D model of a southpark Wookie, because we named our project – Project Wookie, in honor of Star Wars Galaxies and the idea of Dancing With Wookies as a way of leveling up your entertainment skills.  The whole world of 3D took on a value prop which I had never imagined.  The ability to share virtually concrete ideas thru 3D models helped push this even further.  Activeworlds, SecondLife, and There.com all became exciting as each took a different approach on how this technology could be used.

Well, before I get much deeper on this thought, I will just say, this and much more will be valid topics for Episode 200.  Come join us when we lock down the location for Beer!

Given that the wireless coverage in my house is so bad, the term faraday cage comes to mind, I really notice the hit that my network takes everytime that TimeMachine backs up my machine.  I’v e been monitoring it for some time and have been very frustrated to see that every hour I have between 200-300MB of data being backed up.  As such my network is almost always impacted by my TimeMachine.  At first I thought this was normal, but as I started leaving my machine on over night, I would notice that I was still having almost as much backed up every hour.  I decided that there must be some way to figure out what was changing, since I had excluded from my backup those files that I expected to change a lot. (Figuring that they would be backed up about 1 time a week when I do my SuperDuper backup.

I did a bit of searching on the net, and finally settled on trying out BackupLoupe by Soma-Zone software in Germany.  It seamed to do everything I needed, and then some.  I installed and immediately ran into a limitation of my setup.  I use a Drobo-FS to do my backups on TimeMachine.  As such, the Drobo Dashboard software manages when the TimeMachine partition is seen by the Mac.  BackupLoupe would not see the drive!  But Robby from Soma-Zone not only solved my problem, but held my hand over a series of very patient emails until I got it right.

So what does BackupLoupe do?  Well it not only shows you what has been backed up in each of your TimeMachine snapshots, it helps you manage this, by giving you access to the exclusion list while reviewing the data.  All of this is presented in an easy to understand and pleasing to view interface:

With this I was able to find the culprit (the Carbonite Backup Logs, also running each hour) and exclude them from my backups.

BackupLoupe is basically donationware, and they only ask $5 via Paypal as a donation.  You can say you donated and get the reminders to turn off, there doesn’t appear to be any actual registration.  But I strongly recommend that you support this software if you can use it.  Not everything has to come thru an App Store.

Michael R.

What’s cool?
Michael R – Off thinking about Duke Nukem Forever (Andy prefers Battlefield 3)
Michael M — Isle of Tune (Andy’s recommendation, Every Breath You Take)
Andy — Firefox 4 — (Michael adds the Lazarus awesomeness)

Obligatory 3-D Section
Hot Wheels Secret Race battle — lx5
Making of the Secret Race battle

Tech-no-tronic
Waves hands at Netflix — allerhed (which is fine if you are in the US…)
7 Kinect hacks — allerhed

Data Visualizations
Evolution of Web Design — lx5

Social Networkin’
Social Media Snakes & Ladders infographic — lx5
Crowdsourcing Mapmaking — Andy (including live realtime Pulse)
– iPhone Tracking and the visualisation app for OS X

Final tweet from the front lines — richrants
‘Alien’ found in Russia — allerhed
Social Engineering made too easy by plain text offenders — andy
Peepmail — andy
Rapleaf — michael

Music
Songs for Japan
Donate to Download

Final thoughts
AP  –  Eastern-Eggs & Great balls of lego!
MM — ‘twas brillig! / Jabberwocky / Peeperwocky

bonus links
the Jellybean Kinect lounge chair — allerhed
Duke Nuke’em Ships in June – michaelrowe01

This week, Michael Rowe is hiking in the Australian outback and has no Internet access (but we think he will enjoy catching up on his Kindle reading and photography). So, your hosts for this episode are the masterly Mr Michael Martine, and the dastardly Mr Andy Piper. Enjoy.

What’s (April) cool?
Andy P — own your own Apple Store! and Angry Birds, the movie.
Michael M — angry nerds and Willy Wonka Nerds

Tech-no-tronic
Paranga – tactile feedback for ebooks — heidigoseek
Lifting ketchup / catsup off a surface — lx5
iMobot the modular robot – allerhed
playing catch with a quadrocopter — allerhed
Kinect-style interaction on android tablets — heidigoseek (… and remember The Road Ahead – Bill Gates, 1995?)
World Backup Day — lx5

Data Visualizations
awesome cats — UncommonMan

The Gadget Spot / weekly discussion
Nintendo 3DS
Amazon cloud
Palm Pre / webOS – can HP be successful

… and some screenshots from Andy’s webOS device, for those interested in the look-and-feel

Final thoughts
Michael — Sensus & Better Place
Andy — Yalta on Lanyrd vs Instant / Realtime and history on the web

bonus links
April Fools!
Gmail Motion (now made into reality!)
Comic Sans Pro
YouTube’s 100th anniversary 1911 top viral videos
LinkedIn
XKCD in 3D

What’s cool?
Andy P — Farnham Twestival raised over £2k for charidee and speaking at ReLIVE 11 in September
Michael M – 1.001^23, 1.002^23, ready or not, here bee-dee-bee I come!
Michael R – Productive Discussions

Obligatory (Dead) 3-D Internet
Nintendo 3DS
…Amazon UK >:-( –  aaaargh

Tech-no-tronic
STEM education via game development — heidigoseek
Xbox Kinect on PS3 — allerhed
Math movie titles — uncommonman
Spherical television – Pufferfish — uncommonman
Gadget spot – PowerMonkey Explorer (not a Dell Vostro V130) with Andy’s unboxing video

Social Networkin’
flaxen waxen — epredator
radiation – lx5
swarming on the edge — uncommonman

the big question
how can one remain on the edge, when the gravitational pull draws one to the center?  coffee, dawgs, lawndry.  talk amongst yourselves.  :)

Hugh McLeod’s Ignore Everybody and Evil Plans

Final thoughts
MM —  iRobots in Japan and…  Forget.Me.Not
AP — Delayed.
MR — Gambaro

bonus links
Google Books deal blocked in US court – lx5
the iPad DOES MORE.
ROBOTS – Michael and Michael should borrow these costumes.

Kevin Brown who also featured in my previous eightbar post appears to be increasing his level of fame after appearing on Channel 4 news last night.

Kevin has done a lot of work with HCI (Human Computer Interfaces) and is leading the way in the Hursley Emerging Technology Services department. He has a huge interest and wealth of knowledge on the topic but the bleeding-edge HCI device catching people’s attention again at the moment is the brain reading headset from Emoviv Technologies. Kevin has been working with this device for quite some time already, having for example used it with hospital patients, and a wealth of other uses too including driving cars. This gives a good indicator to how far ahead of the curve our emerging tech team can be at times.

The Channel 4 news clip focuses on using the headset to drive cars and puts this in the context of Google’s self-drive car too, here’s the video:

The British Computer Society recently came to Hursley to interview some of the members of Emerging Technology Services about some of the work we’ve been doing recently. The results, as ever in ETS, are really interesting so here is the set of video interviews reposted for all you Eightbar subscribers out there.

To kick things off we have Bharat Bedi, IBM Master Inventor, talking about his work on the Universal Information Framework. This is an innovative idea that allows secure interactions that could benefit, for example, banks:

Another piece from Bharat Bedi but this time talking about his work on the Living Safe project which runs in Balzano, Italy to help older residents who live by themselves:

Now something a little different from Kevin Brown, IBM Senior Inventor, talking about his work using a mind-reading headset. Here he gets Brian Runciman from the BCS to drive a car with his brain and trains him to run a brain wave reading headset:

Next up we have Dominic Harries, IBM Emerging Technologies Specialist, talking about some of his work using a multi-user multi-touch surface. Here Dominic is demonstrating the use of a business application on the multi-touch table:

Last, but not least we have Helen Bowyer, Emerging Technologies Manager, talking about her work on Automatic Sign Language. Helen explains and demonstrates the Say It, Sign It (SiSi) project which uses an avatar to translate spoken English into sign language.

The original content can be found at http://www.bcs.org/content/conWebDoc/44430.

This is my mood (as identified from my facial expressions) over time while watching Never Mind the Buzzcocks.

The green areas are times where I looked happy.

This shows my mood while playing XBox Live. Badly.

The red areas are times where I looked cross.

I smile more while watching comedies than when getting shot in the head. Shocker, eh?

A couple of years ago, I played with the idea of capturing my TV viewing habits and making some visualisations from them. This is a sort of return to that idea in a way.

A webcam lives on the top of our TV, mainly for skype calls. I was thinking that when watching TV, we’re often more or less looking at the webcam. What could it capture?

What about keeping track of how much I smile while watching a comedy, as a way of measuring which comedies I find funnier?

This suggests that, overall, I might’ve found Mock the Week funnier. But, this shows my facial expressions while watching Mock the Week.

It seems that, unlike with Buzzcocks, I really enjoyed the beginning bit, then perhaps got a bit less enthusiastic after a bit.

What about The Daily Show with Jon Stewart?

I think the two neutral bits are breaks for adverts.

Or classifying facial expressions by mood and looking for the dominant mood while watching something more serious on TV?

This shows my facial expressions while catching a bit of Newsnight.

On the whole, my expression remained reasonably neutral whilst watching the news, but you can see where I visibly reacted to a few of the news items.

Or looking to see how I react to playing different games on the XBox?

This shows my facial expressions while playing Modern Warfare 3 last night.

Mostly “sad”, as I kept getting shot in the head. With occasional moments where something made me smile or laugh, presumably when something went well.

Compare that with what I looked like while playing Blur (a car racing game).

It seems that I looked a little more aggressive while driving than running around getting shot. For last night, at any rate.

Not just about watching TV

I’m using face recognition to tell my expressions apart from other people in the room. This means there is also a bunch of stuff I could look into around how my expressions change based on who else is in the room, and their expressions?

For example, looking at how much of the time I spend smiling when I’m the only one in the room, compared with when one or both of my kids are in the room.

To be fair, this isn’t a scientific comparison. There are lots of factors here – for example, when the girls are in the room, I’ll probably be doing a different activity (such as playing a game with them or reading a story) to what I would be doing when by myself (typically doing some work on my laptop, or reading). This could be showing how much I smile based on which activity I’m doing. But I thought it was a cute result, anyway.

Limitations

This isn’t sophisticated stuff.

The webcam is an old, cheap one that only has a maximum resolution of 640×480, and I’m sat at the other end of the room to it. I can’t capture fine facial detail here.

I’m not doing anything complicated with video feeds. I’m just sampling by taking photos at regular intervals. You could reasonably argue that the funniest joke in the world isn’t going to get me to sustain a broad smile for over a minute, so there is a lot being missed here.

And my y-axis is a little suspect. I’m using the percentage level of confidence that the classifier had in identifying the mood. I’m doing this on the assumption that the more confident the classifier was, the stronger or more pronounced my facial expression probably was.

Regardless of all of this, I think the idea is kind of interesting.

How does it work?

The media server under the TV runs Ubuntu, so I had a lot of options. My language-of-choice for quick hacks is Python, so I used pygame to capture stills from the webcam.

For the complicated facial stuff, I’m using web services from face.com.

They have a REST API for uploading a photo to, getting back a blob of JSON with information about faces detected in the photo. This includes a guess at the gender, a description of mood from the facial expression, whether the face is smiling, and even an estimated age (often not complimentary!).

I used a Python client library from github to build the requests, so getting this working took no time at all.

There is a face recognition REST API. You can train the system to recognise certain faces. I didn’t write any code to do this, as I don’t need to do it again, so I did this using the API sandbox on the face.com website. I gave it a dozen or so photos with my face in, which seemed to be more than enough for the system to be able to tell me apart from someone else in the room.

My monitoring code puts what it measures about me in one log, and what it measures about anyone else in a second “guest log”.

This is the result of one evening’s playing, so I’ve not really finished with this. I think there is more to do with it, but for what it’s worth, this is what I’ve come up with so far.

The script

####################################################
#  IMPORTS
####################################################

# imports for capturing a frame from the webcam
import pygame.camera
import pygame.image

# import for detecting faces in the photo
import face_client

# import for storing data
from pysqlite2 import dbapi2 as sqlite

# miscellaneous imports
from time import strftime, localtime, sleep
import os
import sys

####################################################
# CONSTANTS
####################################################

DB_FILE_PATH="/home/dale/dev/audiencemonitor/data/log.db"
FACE_COM_APIKEY="MY_API_KEY_HERE"
FACE_COM_APISECRET="MY_API_SECRET_HERE"
DALELANE_FACETAG="dalelane@dale.lane"
POLL_FREQUENCY_SECONDS=3

class AudienceMonitor():

    #
    # prepare the database where we store the results
    #
    def initialiseDB(self):
        self.connection = sqlite.connect(DB_FILE_PATH, detect_types=sqlite.PARSE_DECLTYPES|sqlite.PARSE_COLNAMES)
        cursor = self.connection.cursor()

        cursor.execute('SELECT name FROM sqlite_master WHERE type="table" AND NAME="facelog" ORDER BY name')
        if not cursor.fetchone():
            cursor.execute('CREATE TABLE facelog(ts timestamp unique default current_timestamp, isSmiling boolean, smilingConfidence int, mood text, moodConfidence int)')

        cursor.execute('SELECT name FROM sqlite_master WHERE type="table" AND NAME="guestlog" ORDER BY name')
        if not cursor.fetchone():
            cursor.execute('CREATE TABLE guestlog(ts timestamp unique default current_timestamp, isSmiling boolean, smilingConfidence int, mood text, moodConfidence int, agemin int, ageminConfidence int, agemax int, agemaxConfidence int, ageest int, ageestConfidence int, gender text, genderConfidence int)')

        self.connection.commit()

    #
    # initialise the camera
    #
    def prepareCamera(self):
        # prepare the webcam
        pygame.camera.init()
        self.camera = pygame.camera.Camera(pygame.camera.list_cameras()[0], (900, 675))
        self.camera.start()

    #
    # take a single frame and store in the path provided
    #
    def captureFrame(self, filepath):
        # save the picture
        image = self.camera.get_image()
        pygame.image.save(image, filepath)

    #
    # gets a string representing the current time to the nearest second
    #
    def getTimestampString(self):
        return strftime("%Y%m%d%H%M%S", localtime())

    #
    # get attribute from face detection response
    #
    def getFaceDetectionAttributeValue(self, face, attribute):
        value = None
        if attribute in face['attributes']:
            value = face['attributes'][attribute]['value']
        return value

    #
    # get confidence from face detection response
    #
    def getFaceDetectionAttributeConfidence(self, face, attribute):
        confidence = None
        if attribute in face['attributes']:
            confidence = face['attributes'][attribute]['confidence']
        return confidence

    #
    # detects faces in the photo at the specified path, and returns info
    #
    def faceDetection(self, photopath):
        client = face_client.FaceClient(FACE_COM_APIKEY, FACE_COM_APISECRET)
        response = client.faces_recognize(DALELANE_FACETAG, file_name=photopath)
        faces = response['photos'][0]['tags']
        for face in faces:
            userid = ""
            faceuseridinfo = face['uids']
            if len(faceuseridinfo) > 0:
                userid = faceuseridinfo[0]['uid']
            if userid == DALELANE_FACETAG:
                smiling = self.getFaceDetectionAttributeValue(face, "smiling")
                smilingConfidence = self.getFaceDetectionAttributeConfidence(face, "smiling")
                mood = self.getFaceDetectionAttributeValue(face, "mood")
                moodConfidence = self.getFaceDetectionAttributeConfidence(face, "mood")
                self.storeResults(smiling, smilingConfidence, mood, moodConfidence)
            else:
                smiling = self.getFaceDetectionAttributeValue(face, "smiling")
                smilingConfidence = self.getFaceDetectionAttributeConfidence(face, "smiling")
                mood = self.getFaceDetectionAttributeValue(face, "mood")
                moodConfidence = self.getFaceDetectionAttributeConfidence(face, "mood")
                agemin = self.getFaceDetectionAttributeValue(face, "age_min")
                ageminConfidence = self.getFaceDetectionAttributeConfidence(face, "age_min")
                agemax = self.getFaceDetectionAttributeValue(face, "age_max")
                agemaxConfidence = self.getFaceDetectionAttributeConfidence(face, "age_max")
                ageest = self.getFaceDetectionAttributeValue(face, "age_est")
                ageestConfidence = self.getFaceDetectionAttributeConfidence(face, "age_est")
                gender = self.getFaceDetectionAttributeValue(face, "gender")
                genderConfidence = self.getFaceDetectionAttributeConfidence(face, "gender")
                # if the face wasnt recognisable, it might've been me after all, so ignore
                if "tid" in face and face['recognizable'] == True:
                    self.storeGuestResults(smiling, smilingConfidence, mood, moodConfidence, agemin, ageminConfidence, agemax, agemaxConfidence, ageest, ageestConfidence, gender, genderConfidence)
                    print face['tid']

    #
    # stores face results in the DB
    #
    def storeGuestResults(self, smiling, smilingConfidence, mood, moodConfidence, agemin, ageminConfidence, agemax, agemaxConfidence, ageest, ageestConfidence, gender, genderConfidence):
        cursor = self.connection.cursor()
        cursor.execute('INSERT INTO guestlog(isSmiling, smilingConfidence, mood, moodConfidence, agemin, ageminConfidence, agemax, agemaxConfidence, ageest, ageestConfidence, gender, genderConfidence) values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
                        (smiling, smilingConfidence, mood, moodConfidence, agemin, ageminConfidence, agemax, agemaxConfidence, ageest, ageestConfidence, gender, genderConfidence))
        self.connection.commit()

    #
    # stores face results in the DB
    #
    def storeResults(self, smiling, smilingConfidence, mood, moodConfidence):
        cursor = self.connection.cursor()
        cursor.execute('INSERT INTO facelog(isSmiling, smilingConfidence, mood, moodConfidence) values(?, ?, ?, ?)',
                        (smiling, smilingConfidence, mood, moodConfidence))
        self.connection.commit()

monitor = AudienceMonitor()
monitor.initialiseDB()
monitor.prepareCamera()
while True:
    photopath = "data/photo" + monitor.getTimestampString() + ".bmp"
    monitor.captureFrame(photopath)
    try:
        faceresults = monitor.faceDetection(photopath)
    except:
        print "Unexpected error:", sys.exc_info()[0]
    os.remove(photopath)
    sleep(POLL_FREQUENCY_SECONDS)

Behind-the-scenes documentaries, like Doctor Who Confidential, matter. They matter because they show viewers, in particular children still deciding what to do with their lives, that it takes more to produce a high-class TV programme than just a few actors who become famous. It shows what other creative and/or technical jobs there are in television.

A couple of weekends ago, we went to the Doctor Who Official Convention (#dwcuk) in Cardiff. While one of the three main panels featured the three stars, Matt Smith, Karen Gillan and Arthur Darvill (along with executive producers Stephen Moffat and Caroline Skinner), most of the other scheduled events were focused on how Doctor Who is made.

At the very start of the day, we went to see Danny Hargreaves blow things up talk about the Special Effects on Doctor Who. In his Q&A session (after making it snow indoors), the first question asked was “How did you get into special effects work?” and, between questions like how he blew up the Torchwood Hub and how he makes the Doctor’s hands and head fiery during a regeneration, a later question was “When did you realise you wanted to work in special effects?”. Attendees were interested not just in the fictional stories and characters but in how the programme is made and the interesting careers they might not otherwise have come across.

Throughout the day, I heard audience members ask how to become costume and prosthetics designers and how to become script writers. Danny described how his team designs and creates the effects, assess the risks of blowing things up, and who they work with to make it all happen. He also explained how he came to be a trainee in the nascent world of special effects before studying Mechanical Engineering so that he could build the devices they need for Doctor Who (and the other shows he’s worked on, like Coronation Street). Directors of photography, set designers, executive producers, writers, and directors went on to talk about what their own jobs entailed day-to-day and how it all comes together to make an episode of Doctor Who.

These discussions continued the story that used to be told after each new episode of Doctor Who by Doctor Who Confidential on BBC3. Doctor Who Confidential started in 2005 with the return of Doctor Who. As well as talking about some interesting perspective on making that night’s episode of Doctor Who, it featured interviews with, and ‘day-in-the-life’ documentaries about, the actors (including showing the less glamorous side of shivering in tents and quilted coats between takes), the casting directors, the producers, the writers, the choreographers, the costume designers, the special effects supervisors, the monster designers, the prosthetics experts, the directors, the assistant directors, and many, many others. It also held competitions for children to write a mini episode and then see the process of making it, which would’ve been an amazing experience!

Yes, it took a slightly odd turn in the last series when it turned a bit Top Gear by showing Karen Gillan having a driving lesson and Arthur Darvill swimming with sharks; possibly a misguided attempt to increase its popularity before it got canned anyway to cut costs.

I think it’s a real shame to lose Doctor Who Confidential and its insights into the skill, hard work, and opportunities in TV and film production.

Cool photo of Danny in the snow by Tony Whitmore.

The graph of our total gas usage per year doesn’t decrease quite so impressively as our electricity graph, which I blogged about halving over five years. Because the numbers were getting ridiculously big and difficult to compare at a glance, I’ve re-created the electricity graph here in terms of our average daily electricity usage instead of our annual usage (click the graph to see a larger version):

If you compare it with the average daily gas usage graph below, you can see (just from the scales of the y-axes) that we use much more gas than electricity (except in 2007, which was an anomalous year because we didn’t have a gas fire during the winter so we used a electric halogen heater instead):

Our gas usage has come down overall since 2005 (from 11280 kWh in 2005 to 8660 kWh in 2011; or 31 kWh per day to 24 kWh per day on average) but not so dramatically as our electricity usage has. Between 2005 and 2011, we reduced our electricity usage by about a half  and our gas usage by about a quarter.

Gas, in our house, is used only for heating rooms and water. So if I were to chart the average outside temperatures of each year, they’d probably track reasonably closely to our gas usage. In 2005 (when we used an average of 31 kWh per day), we still had our old back boiler (with a lovely 1970s gas fire attached) which our central heating installer reckoned was about 50% efficient. In 2006 (26 kWh per day), we replaced it with a new condensing boiler (apparently 95% efficient) but didn’t replace the gas fire until mid-2007 (the dodgy year that doesn’t really count). In 2006, we also had the living-room (our most heated room) extended so it had a much better insulated outside wall, door, and window. These changes could explain the pattern of reducing gas usage year by year up till then.

In 2009, January saw sub-zero temperatures and it snowed in November and December. I think that must be the reason why our usage for the whole year shot back up again, despite the new boiler, to 31 kWh per day. In 2010 (21 kWh per day), it was again very cold and snowy in January; I think the slight dip in gas usage that year compared with both 2008 (25 kWh per day) and 2011 (24 kWh per day) was down to a problem with the gas fire that meant we used the electric halogen heater again during the coldest month. In 2011 it snowed in January but was fairly mild for the rest of the year.

I think 2008, 2010, and 2011 probably represent ‘typical’ years of heating our house with its new boiler and gas fire. Like I concluded about reducing our electricity usage, I think our gas usage went down mostly by getting some better insulation and a more efficient boiler but we did also reduce the default temperature of our heating thermostat to about 17 degrees C (instead of 20 degrees C) a couple of years ago too (we increase it when we need to but it stays low if we don’t), which I think has made some difference but it’s hard to tell when our heating usage is so closely tied to the outside temperature. Also, we don’t currently have any way of separating out our water heating from our central heating, and our gas fire from the boiler.

Of course, what really matters overall is the total amount of energy we use (that is, the gas and electricity numbers combined). So I’ve made a graph of that too. Now we’re talking numbers like 48 kWh per day in 2005 to 33 kWh per day in 2011.

Overall, that means we reduced our total energy usage by about one-third over seven years.

Thanks again to @andysc for helping create the graph from meter readings on irregular dates.

We IBM employees are encouraged, indeed incented, to be innovative and to invent.  This is particularly poignant for people like myself working on the leading edge of the latest technologies.  I work in IBM emerging technologies which is all about taking the latest available technology to our customers.  We do this in a number of different ways but that's a blog post in itself.  Innovation is often confused for or used interchangeably with invention but they are different, invention for IBM means patents, patenting and the patent process.  That is, if I come up with something inventive I'm very much encouraged to protect that idea using patents and there are processes and help available to allow me to do that.

Ian Hughes/Epredator

As part of the reorganisation of the Eightbar site recently I’ve been catching up with some of the honored past Eightbar members. We say past in the loosest sense of course, Eightbar was set up with the principle that “Once you’re Eightbar, you’re always Eightbar”. Here, I manage to muscle in on some of Ian Hughes’ (a.k.a epredator) time as he’s kindly answered some questions for us. What follows is a 10 question interview style post where I talk to Ian about life after IBM – in more than 140 characters. I think it’s a really interesting read, enjoy…!

Ian, you worked for IBM for a long time (somewhere around 20 years!) before making the big decision to leave and form your own start-up at Feeding Edge nearly 3 years ago!

1. What have you found are the main things keeping you busy now?

Just as when I was at IBM my work life is very varied. Living and working with technology and social changes, and being a bit of a polymath I find myself mixing a lot of skills.

Sometimes I am coding or combining code, usually on open source platforms but often in Unity3d. Building some game elements for a startup. Other times I am on the conference circuit helping
people to see the future by showing examples of how various things have changed already and how they link together to form a disruptive future. i.e. carrying on as an evangelist.

Much of this is still related to virtual worlds because they form a social and technical glue that still surprises many people only just getting to grips with Twitter and Facebook.

2. We’ve seen your continued rise to stardom on the ITV programme The Cool Stuff Collective, how did that come about?

Stardom is a very strong word It was an ambition I had tucked away to do some more TV work. Like many things though it was serendipity that brought that about.

As I still blog many of my ideas and things about interesting advances many of my friends still read that. A good friend and IBMer Scotty (Kevin Scott / @starbase37) had told his friend John Marley / @marleyman007 who runs a TV production company Archie Productions about all the stuff I was talking about. Games, 3d printing, virtual worlds etc. So we got connected and had a meeting about a new show John was looking to start.

The aim of the meeting was really a friendly catchup and for me to give John a list of things that he could put on his show. Somewhere in the conversation he said “and then you will come on set and explain that to camera and the other presenter?” Which I still thought he meant he wanted me to be tech advisor for the kids show. Then it clicked and I realised I was being thrown in at the deep end. It was one of the few shows ITV/CITV has commissioned over the past few years.

So really because I have always shared what I know, used the web and social media to explain and offer a kind of open source advise I ended up with a character and role on the show. Which we have done 3 series of too!

Cue Showreels TV Showreel

3. You must enjoy being the CSC resident g33k and teaching the viewers, what do you learn from them?

It has been the most fun and rewarding thing I have done. The third series in particular we moved from a studio and just the crew to being on location with schools in a Top Gear style. Whilst we were making a show for a mass audience it became even more important to be able to reach kids directly. I learned, and re-learned that the willingness to go with the flow on some ideas because they just are cool is still a magical thing. The things I say on the show are the same things I say in boardrooms and at conferences. The kids put many adults to shame though in not worrying straight away about ROI or marketing blurb. They get the idea and then fly with it.

It was also great to be able to reclaim geek/g33k. In a few schools the kids who were the tech geeks were suddenly allowed to be cool too. After all there was a bloke off the telly they could talk to.

We always had questions at the end of my future tech slot and I often didn’t get to know what they were up front, they were their questions and they were always taking me by surprise with their new angles or just the depth of understanding they showed. Once again putting many adults to shame.

4. Your time on Eightbar was mainly filled with Virtual Worlds work, what’s going down with the 3D Internet now, has it progressed as you thought?

It’s interesting as in many ways parts of the metaverse are now so mainstream, yet still not so much in the “business” world as you may have expected. We know that people tend to have to evolve through things, hence the struggling to understand the power of connection in social media is still a struggle for many decision makers in business. In a time of global recession with restricted travel it seems that the obvious use for communication and understanding via virtual environments is still not being exploited. Much of this is due to people being risk averse when they think their jobs are on the line. I find that many of the things we do and talk about are still reaching an audience who then say wow I didn’t think of it like that.

When they are used in their various forms they have a huge impact. Imperial College have some of the best examples, even with just a simple Opensim environment to help people plan a particular event it showed up real world procedures needed fixing after the first 5 minutes which saved more than money.

Lots of companies have floundered who where virtual world providers, but equally lots of their code is now open source. At the same time though lots of the games industry has been turned on its head by the arrival of minecraft. Which is a “game” but that uses co creation tools live in the environment. It has done a lot to help the games industry (who also did not understand virtual worlds of this sort) to look and say “oh! thats what its all about”.

So none of it has gone away. It hit the usual Gartner trough of dissillusionment after the confused hype and now is ploughing up the right slope.

Regular business will get hit with a minecraft moment though. A game changer in the same way open source software hit the IT industry, or Amazon hit retail. It’s just about being prepared to go with it when it arrives.

Another great development has been the ability to self build game tech environments with products like unity3d (a huge nod to Rob Smart for spotting unity3d way back too!) and have socket servers like photon and smartfoxserver.

I should also mention gamification, a horrible word, another thing for people to misunderstand, yet it covers the principles of applying both gaming and game technology into places it has not been before. It is often used in a lazy fashion slapping badges on things and giving out points, however at its heart the elements of playing with identity and expression online with a virtual environment in a business context provide way more benefit.

5. What has the past 3 years done for 3d printing, another of your interest areas?

3d printing has gone from strength to strength. It is appearing in more places and often more people have seen something about it when I talk about it. It is linked to the virtual worlds work as when you consider that a virtual environment is often about distributing digital assets from one place to another, you bolt a 3d printer on the end of that and you get digital design and distribution of physical product and the world changes.

The increase in open source builds like the RepRap make the hobby end of this accessible (around £400 of bits to build one). Makerbot provide some very cheap, but clever printers too that were featured heavily at CES 2012 (Consumer Elecrtonics Show) note the Consumer in that ! Services that print for you, like Shapeways, initially funded by Phillips, have grown and moved to New York.

It is still something that when someone has never seen it they think it is witchcraft, somewhat like google used to seem to people That magic is nice to share, but then applying the extrapolation of the change to the entire world economy and manufacturing business as it moves on then scares and excites in equal measure.

6. What would you like to see Eightbar doing more/less of after the departure of Andy Piper from Hursley recently?

When we all set up eightbar it was an antidote to the west coast US tech bloggers getting all the kudos. We’re doing some great things over here too Just as tech blogging has evolved I would love to see eightbar carrying on as a mini brand and a voice of that same attitude wherever it needs to be.

7. Looking back at IBM, any regrets about leaving? Things you miss?

I miss all the people, well nearly all Though in reality much of the work was with people all over the world having a base of people in the same timezone and same place eating lunch in the same canteen provides an anchor. As does having to battle the same corporate resilience to change, or political short sightedness. There are still a great many sparky, slightly subversive but for the right reasons, renegade thought leaders under the radar at IBM.

Oh and the regular pay

8. What’s been the best thing about moving on?

Diversity of experiences and freedom to explore them. Like the TV work, it was just because of being open minded and master of my own calendar. I like to link everything, let one piece of work and ideas flow with another. That is tricky in a billable utilisation environment when you are not in control of the finances and the workload. It is why big corporations will keep getting side swiped by very small fast moving organisations with huge world connectivity at their finger tips.

I have also had to learn a lot about the various forms and processes needed to run even the smallest Ltd company. It’s an odd and archaic system, but they are the rules It has also been fun picking various ideas and developing them getting people with the money to get interested. It gets all very Dragon’s den.

Freedom also allows me to try and pick things based on if I think they are beneficial in some way, not purely just because they are there. I have always prided myself on trying to act honourably in everything and with positive principles. So now it is up to me to stick to that and help others try and do the same.

9. Your personal life and work-life balance must have adjusted, what does a day in the life of epredator look like now you’re self-employed?

Aha! I called myself self employed once and my accountant was quick to point out I am not This is part of what I was saying about companies and rules. As Feeding Edge is a limited company it is a legal entity in its own right that I happen to be a director of. At the same time there is a person on its official payroll, an employee… me So as many twist and turns in business language as in any piece of tech

My day is much more thinly sliced than ever. I get up check a few streams of information, spot anything urgent, then do the school run, back home for 45 minute workout on UFC trainer on the Kinect, do some calls afterwards whilst cooling down. Most of the day is spent talking to the US and or my other biz partners around the gaming startup we have, building some code, pitching how bizarre the idea is. This is usually interspersed with some contacts from previous conferences getting in touch or some BCS animation and Games Dev SG business. Several times a month I pop along to a convention or meeting to talk about Tech and usual with Cool Stuff Collective as a backdrop. So the cycle continues.

Then there are the ad hoc conversations around other possible TV shows, or helping other startup businesses who are focussed using new tech with some connections or ideas.

Evenings are mix of cooking for the family, putting the kids(predlets) to bed, some gaming, heading to a Choi Kwang Do class or late night calls with US west coast for an interview or in Second Life.

However there is not start or end to a working day, a tweet on the way back form the school run may lead to something as much as a scheduled Skype call at 2pm. The emphasis is still on talking and sharing online.

10. Finally, give us a plug for Feeding Edge, who might I be if I were your customer and what might you be able to do for me?

Feeding Edge is a vehicle for people to get help from me, consulting or hands on development. As I say I am taking a bite out of technology so you don’t have to. All the years of experience with corporate tech and now several years out in the wild having to use what I talk about gives me a view on the world that many people don’t have time to consider, in person, in writing, on the TV, on stage, in the lab. I cover how technology feels and changes your life as much as the more obvious version x with version y tech.

In conferences I am usually the one put there to shake everybody up. So if you need a jolt of inspiration and a view of the future. well thats Feeding Edge and epredator. Cue show reels again

Well that’s it from Ian again for now. It’s really good to hear him talking in a wider context again, reading about the mix of drawing inspiration from such a wide variety of sources is really refreshing. It’s certainly reminded me to go “heads up” more often than I generally manage to do, so easy is it to keep too narrow a view on your immediate work tasks.

Thanks Ian, it’s been a pleasure – as always!

I learnt something interesting today: between 2007 and 2011, we halved the amount of electricity we use in our house:

In 2007, we used 6783 kWh of electricity (for electricity, a kilowatt hour is the same thing as a ‘unit’ on your bill). In 2011, by contrast, we used 3332 kWh (or ‘units’). 2007 was slightly on the high side (compared with 2006) because we had no gas fire in the living-room during the winter of 2006-7 so we’d used an electric oil heater during the coldest weeks (we don’t have central heating in that room) ¹.

That’s an average of 19 kWh per day in 2007 compared with 9 kWh per day in 2011. Which is quite a difference. So what changed?

In early 2008, I got a plug-in Maplin meter (similar to this one) and one of the very early Current Cost monitors, which display in real-time how much electricity is being used in your whole house:

Aside from the fun of seeing the display numbers shoot up when we switched the kettle on, it informed us more usefully that when we went to bed at night or out to work, our house was still using about 350 Watts (which is 3066 kWh per year)² of electricity. That’s when the house is pretty much doing nothing. Nothing, that is, apart from powering:

• Fridge
• Freezer
• Boiler (gas combi boiler with an electricity supply)
• Hob controls and clock
• Microwave clock
• Infrared outside light sensor
• Print/file server (basically a PC)
• Wireless access point
• Firewall and Internet router
• DAB clock radio
• ADSL modem
• MythTV box (homemade digital video recorder; basically another PC)

And that’s the thing, this ‘baseline’ often makes a lot of difference to how much electricity a house uses overall. 3066 kWh per year was 56% of 2007′s total electricity usage.

The first six items on that list draw less than 100 Watts (876 kWh per year) altogether. They’re the things that we can’t really switch off. But there were clearly things that we could do something about.

Over the next couple of years, we reduced our baseline by about 100 Watts by getting rid of some of the excessive computer kit, buying more efficient versions when we replaced the old print/file server and MythTV box, and replaced most of our lightbulbs with energy-efficient equivalents. We also, importantly, changed our habits a bit and just got more careful about switching lights off when we weren’t using them (which wouldn’t affect the baseline but does affect the overall energy usage), and switching off, say, the stereo amplifier when we’re not using it.

That brought our baseline down to about 230 Watts (2015 kWh per year), which is a lot better, though it’s still relatively high considering that the ‘essentials’ (eg fridge and freezer) contribute less than half of that.

And that’s about where we are now. We tended to make changes in fits and starts but none of it has been that arduous. I don’t think we’re living much differently; just more efficiently.

At the London Green Hackathon a few weeks ago, the small team that had coalesced around our table (Alex, Alex, Andy, and me) had got to about 10pm on Saturday night without a good idea for a hack, in this case a piece of cool software relevant to the theme of sustainability. We were thinking about creating a UK version of the US-based Good Guide app using on their API to which we had access. The Good Guide rates products according to their social, environmental, and health impacts; the company makes this data available in an API, a format that programmers can use to write applications. Good Guide uses this API itself to produce a mobile app which consumers can use to scan barcodes of products to get information about them before purchase.

The problem is that the 60,000 products listed in the Good Guide are US brands. We guessed that some would be common to the UK though. We wondered if it would be possible to match the Good Guide list against the Amazon.co.uk product list so that we could look up the Good Guide information about those products at least. Unfortunately, when we (Andy) tried this, we discovered that Amazon uses non-standard product IDs in its site so it wasn’t possible to match the two product lists.

The equivalent of the Good Guide in the UK is The Good Shopping Guide, of which we had an old copy handy. The Good Shopping Guide is published each year as a paperback book which, while a nicely laid out read, isn’t that practical for carrying with you to refer to when shopping. We discovered that The Ethical Company (who produce the Good Shopping Guide) have also released an iPhone app of the book’s content but it hasn’t received especially good reviews; a viewing of the video tour of the app seems to reveal why.

By this point it was getting on for midnight and the two coders in our team, Andy and Alex, had got distracted hacking a Kindle. Alex and I, therefore, decided to design the mobile app that we would’ve written had we (a) had access to the Good Shopping Guide API and (b) been able to write the code needed to develop the app.

While we didn’t have an actual software or hardware hack to present back at the end of the hackathon weekend, we were able to present our mockups which we called our ‘UX hack’ (a reference to the apparently poor user experience (UX) of the official Good Shopping Guide mobile app). Here are the mockups themselves, along with a summary of the various ideas our team had discussed throughout the first day of the hackathon: