The state of Open Source projects in Ireland vs other countries, based on GitHub activity

Last weekend I was working away on a personal side-project of mine and I suddenly realised I follow almost no Irish projects or developers on GitHub. I then started writing this post which was going to be an attack on our lack of people who work on OSS projects outside of work.

But rather than go with gut, I decided to see if I could back it up with data. I decided GitHub relative activity could give us a strong metric of where Ireland fits in to the Open Source world. Of course there are tons of other OSS project hosting sites like Sourceforge, BitBucket, Assembla, Gitorious and even self-hosting but GitHub should do as a rough measure.

I assumed someone had done some simple analysis like this before but all I found were “unusual” visualisations. Then I discovered that GitHub posts all their public commit data to Google BigQuery. This is a big online DB that you can query in a simple SQL console. I lashed together some simple queries and the data basically proved me completely wrong.

SELECT count(*) as commits, repository_owner, actor_attributes_location FROM [githubarchive:github.timeline] where actor_attributes_location CONTAINS 'Ireland' group by actor_attributes_location, repository_owner order by commits DESC LIMIT 100000000 

So in total, I can see 9381 repository committers that mention “Ireland” in their location and have done at least one commit.

That drops to 5872 if we exclude people who have only ever done one commit.

select commits, repository_owner, actor_attributes_location from (SELECT count(*) as commits, repository_owner, actor_attributes_location FROM [githubarchive:github.timeline] where actor_attributes_location CONTAINS 'Ireland' group by actor_attributes_location, repository_owner order by commits DESC) where commits > 1

And 1799 if we exclude those who have done 5 commits or fewer.

Note that these are not repository owners, just people who have done commits.

If we then filter it down by people who have committed to their own repos we get 970. So with a population of 4.6m, 970 people have setup a GitHub account with a public repository and committed at least once to it.

SELECT count(*) as commits, repository_owner, actor_attributes_location FROM [githubarchive:github.timeline] where actor_attributes_location CONTAINS 'Ireland' and actor=repository_owner group by actor_attributes_location, repository_owner order by commits DESC

At this point I was totally depressed. 970 repos. And many of these may be company ones or forks of other repos or school exercises. But then I decided to compare to some other countries.

Let’s start with the UK (Population: 63m): 7643 repos
Using UK, United Kingdom, Great Britain, England, Wales, Scotland, Northern Ireland: 9929 repos
Gives us 724 vs 970 using Ireland as the baseline

Finland (Population: 5.4m): 1332 repos
Using Suomi, Finland: 1333 repos
Gives us 1135 vs 970 using Ireland as the baseline

Australia (Population: 21.5m): 4375 repos
Using Australia, Oz: 4375 repos
Gives us 936 vs 970 using Ireland as the baseline

New Zealand (Population: 4.5m): 1048 repos
Using New Zealand, NZ: 1181 repos
Gives us: 1207 vs 970 using Ireland as the baseline

Canada (Population: 33.9m): 4526 repos
Gives us: 614 vs 970 using Ireland as the baseline

Finally, the spiritual home of OSS and the actual home of GitHub:

USA (Population: 315.5m): 6146
Using USA, US, United States: 7913
Gives us: 115 vs 970 using Ireland as the baseline

(My guess is this number is way too low and Americans just use city/state as they consider the country implied :-) )

So I was wrong. Ireland isn’t particularly bad. Really, we’re all in or around the same ballpark except for the US. So this isn’t just an Irish thing. People working in the tech industry worldwide just don’t seem that interested in writing code in their spare time and making it freely available to others.

Thoughts? Corrections to my SQL? Anyone want to build a better query for the US?

Visualisation of Irish Property Price Register Data using Interactive Map from Google Fusion Tables

UPDATE:
Second attempt at this:

I just grabbed the three CSV files from the Residential Property Price Register site (why is a Captcha needed for that?). I merged some address columns in Excel and added “Ireland” to all of the addresses. I then just imported into Google Fusion Tables and told it to Geo-Code the data.

Note that zero data validation has been done. Everything on this map may be completely wrong. I’m sure some of the more obscure addresses (or non-unique ones) are missing.

Aha! There are daily limits on Geo-coding so I’ll have to do this over a few days before the full map of points appears. But this is a decent start:

UPDATE 2:

2011 Fusion Table. 2011 Map (this will populate properly over next few days too):

2012 Fusion Table. 2012 Map (this will populate properly over next few days too):

UPDATE 3:
Replacement 2010 Fusion Table and Map with Euro symbol correctly encoded:

UPDATE 4: Of course I should have pointed out earlier that the Fusion tables themselves are very useful. You can filter on location, date, price etc without having to fill-out a silly captcha.

Anyone interested in one aggregated Table/Map with all the data from all years in it?

UPDATE 5: As of 16:09 on Oct 1st, the 2012 and 2011 data is 100% geo-coded and 2010 is at 63%. However, based on Brendan’s comments below, we may be missing a lot of data points because Google can’t geo-code them. Once the process is complete, I’ll see if we can extract the missing ones and manually adjust the addresses so that they work. There are also plenty of errors from what I can see.

UPDATE 6: As of 9am Oct 2nd, the final data from 2010 is now geo-coded. Short follow-up post to come on how Google could do far more intelligent and less error-prone geo-coding on Irish addresses without post-codes.

My mind is blown – Met Eireann’s radar image converted to #OpenData

Brendan Cunningham has just left the most exciting comment this blog has ever received. He has thrown some serious smarts at the Met Eireann radar image of rain and converted in to actual numbers you can use!

That vector data with rainfall values is now up on Google Fusion Tables and is updated every 15 minutes so now it can be integrated with GIS and other location based services.

As he points out, with national coverage like this, you can watch for rainfall trends. Heck, you could even do predictions. He mentions that previous flood data may be available from the OPW  and a “Benefitting Lands” layer from the OSi.

I pointed out to Brendan that this data may be perfect for Open Weather Map which has a radar layer for the US and Canada already. Hopefully there are no rights issues with the data (which I assume the Irish taxpayer has paid for). His follow-up comment seems to indicate that all is ok as long as it’s not for profit.

It’s projects like this that give me hope for Ireland. Where the State is unwilling, unable or simply lacking in knowledge to release #OpenData, we can step in. Motivated, knowledgeable people with a community spirit can make the data we have already paid for, do more for the country.

I’m working on a small project in this area at the moment and have just received all of the hardware. Once it’s ready, I’ll publish the PoC here. I hope it will be of benefit to a lot of people.

 

Roku + Netflix + iPlayer now available in Ireland. Anyone using Roksbox or roConnect?

The Verge is reporting that the Roku LT and Roku 2 XS are now shipping in UK and Ireland. They both come with Netflix and BBC iPlayer support. (Oddly their UK site still says “Coming Soon” for the LT).

The LT is particularly interesting as it is only £49 despite the lack of wired Ethernet or 1080p. But both models suffer the major and strange limitation that they cannot stream your home content, even over UPNP.

However I have found two free apps, Roksbox and roConnect which create a private Roku channel for you which gives you access to your home media. Neither of them are UPNP either, they require you to install a web-server on a home machine. Having said that, it looks like a five minute job.

Anyone have experience with either app?

I have to admit I’m tempted by the Roku. £49 is very very cheap for a media streamer that does 720p.

I got the Patriot PBO Core from Amazon UK this week to replace a dead XBOX+XBMC and I’ll review it over the weekend. It is a fine piece of kit for local streaming up to 1080p and does SMB shares rather than the UPNP horror. I’m not sure why I mentioned 1080p since it is connected to a 12 year old 14″ CRT Philips ‘portable’ TV.

But it doesn’t do Netflix except via TVersity or PlayOn. And both of those are currently broken for Netflix Ireland+UK due to the silly incompatible way Netflix has implemented everything over here.

One open question is whether you need a global iPlayer subscription to use it in Ireland on the Roku. I assume so.

 

A Rural Irish School’s App Inventor Pilot. The Story So Far.

Last May I started playing around with the idea of doing some sessions on Google App Inventor in my kids’ Primary School. I’d had great success trying it out with three of my own kids aged 12 to 7.

If you are not familiar with App Inventor, it is a way of building mobile phone applications for Google Android phones using a simple graphical blocks approach. It is highly suited to an educational environment and has seen a lot of uptake since its launch in summer 2010. It has recently been taken over by MIT and I expect great things from them in the coming year.

I am a member of the school’s Parent’s Association and I pitched the idea of me doing some after-school sessions with maybe 5th and 6th class in the Autumn. They loved it and so I contacted the headmistress and the teacher mainly responsible for computers in the school. Due to the summer holidays, nothing much happened until September except for me realising that after-school requires a very long Garda approval process whilst during-school doesn’t. So we switched the focus to that.

A note on the school itself. It is a very successful rural school just outside Bandon, Co Cork with approx 25-30 kids in each year. The mix of kids (boys and girls) is everything from farmer’s children to those who live in Bandon town itself. Socio-economically, it ranges from the very well off to those on social welfare. Every class has an interactive white-board and there are 15 laptops available in addition to the teacher’s own. Most are PCs but there are a couple of Mac Books. However, there is no formal teaching of “computers”. More on this topic later.

Over the summer I made contact with some relevant people and was delighted to get the loan of 15 HTC Magic phones to use in the pilot. However do not under-estimate the amount of time needed to check that 15 phones work ok with App Inventor, have the correct settings turned on and are charged!

In the new term I met with some of the teachers and the new headmistress and they were very enthused. We put together a plan that would see me do a 30-minute session a week with each of 5th and 6th class from October until Christmas. The children would work in teams of two with a laptop and phone for each pair.

My big concern with the pilot was to ensure that “no kid gets left behind”. I knew those children who have PCs and even Smartphones at home would have no problems. I was much more interested in the ones who had neither and who had zero technical skills. I was also interested to see if App Inventor would work well for those who had resource teachers in certain subjects and even those who were on the autistic spectrum.

Basically, I had this idealised vision that our little pilot could help a bunch of kids discover the joy that comes from being creative in a technology sphere and maybe even give them job options in 8 years time that they had never considered possible. I know, I know.

I used the Google Apps for Schools system to create 30 GMail accounts for the kids which they would need to login to App Inventor. I took the first standard tutorial (HelloPurr) and re-wrote it to be much more simplistic. I made sure to get all the domains we needed allowed through the educational firewall. I installed Java and App Inventor on some of the laptops and the 5th Class teacher did the remainder in advance.

I headed in on the first day to introduce them all to App Inventor. The response was amazing. Every kid in both classes was totally pumped by the idea. They all immediately wanted to create Angry Birds, Zombie games and 3D games. Everyone seemed to be a Cut The Rope and Doodle Jump fanatic. It all looked good.

I put together a “curriculum” which would see us do two tutorials and then move to the kids coming up with ideas for their own Apps and then building them. The aim was to have a competition at Christmas for the best App.

Things didn’t quite work out as planned………..

The first few weeks were hell. Both myself and the kids got very frustrated. And it was all down to the most basic problems that you never think about as a technical person but which can stop any normal person in their tracks.

In week one we didn’t even manage to log every team into App Inventor. The problems included:

  1. Very few understood the difference between the address bar and Google Search
  2. Many had issues just getting the email address and password in correctly
  3. A lot of time was being wasted asking the children to type in long URLs to get to images etc and they would type them wrong or mis-capitalise things
  4. The Captchas stumped most of them
  5. App Inventor had huge issues with re-directing to blank pages and requiring page refreshes
  6. We finally realised that App Inventor is incompatible with Internet Explorer but that was all we had on every machine
  7. 15 laptops logging into the same site on a school rotuer caused freeze-ups due to lack of bandwidth
  8. Logout/Login between 5th and 6th class was a nightmare of bad re-directs, cookies and browser cache
  9. Kids don’t listen :-)

So we got Google Chrome installed on all of the laptops and things improved in week 2 (but not the blank screen issue).

Instead of us getting HelloPurr done in 2 weeks, it took us until late November. Each week I would have to jump from pair to pair, dealing with problems ranging from kids forgetting their email address, to Chrome problems, to phone connection problems, to mis-understandings. It was chaos. And none of it was their fault, it was mine for not realising how little most children know about the nuts and bolts of using software. Most of them know how to get to a web-site in IE8, that’s all.

The Blocks Editor was a particular pain point and still is. It looks like we’ll have to switch to Firefox to get it to launch reliably.

But the day we got HelloPurr running on 15 phones in each class was really special. All of them were delighted with themselves. They were beaming. I also pointed out to them that their use of a phone emulator in the previous weeks put them in a very small group of programmers globally.

In the last couple of weeks we have switched to doing the MoleMash tutorial. The reason for this is that it can form the basis for a wide range of games. Due to the previous chaos and the fact that we have most of the nitty gritty problems sorted, I am doing this tutorial as a classroom style lesson. I do each step live on the white-board and they copy me. I don’t go to the next step until everyone says they have successfully carried out the previous one. The progress has been far better and on the 21st Dec, they all got the first revision of MoleMash running in an emulator.

One other advantage of this approach is that the teachers say they can follow things a lot better themselves. They were competely lost as I battled with Internet Explorer, the Blocks Editor and the Emulator.

The last two sessions were also an hour with each class instead of 30 minutes. I think we’ll stick with this in the new year. We lose 10-15 mins of every session with login/logout problems and transferring 15 laptops between the classes. We had much more than 2x progress with the longer sessions.

The Google tutorials themselves are totally unsuitable for children of that age as they are far too information-dense and assume too much knowledge. A lot more hand-holding is needed.

With the Google shutdown and the delay before MIT is ready, I have created a private install of App Inventor for the school so we can continue. I expect them all to get the full MoleMash game running on real phones before the end of January. Then I think we’ll do some variations on it so they can see how easy it is to modify. Then we need to explain some of the other key blocks and then maybe let them try their own Apps.

Whilst the sessions are exhausting I enjoy doing them. Particularly when the 6th class teacher told me that she sees kids completely engaged during those sessions who she has been unable to motivate in anything else. She also told me that certain kids who are struggling in school in general seem to shine doing App Inventor. Maybe I wasn’t being so idealistic?

One big problem we have is that I don’t scale. Committing to a day a week for months was unrealistic and I had to re-schedule a bunch of sessions. I always knew we’d have to do “train the trainer” but I don’t know if the school has other parents with the skills, interest and time to take part. I hope so. I’ll be putting the call out soon.

We will still have our App competition with a prize, it’ll just be months later than I expected.

The general lack of computing skills in Irish schools is pretty shocking and it tells me that it’s time to wind down certain subjects and add “computers” as a subject at Primary level. Can I suggest moving all religious education to Sunday Schools like other religions do? Parents with the skills, interest and time could run those religion sessions. That would free up a huge amount of time to let schools do their real job.

Overall? A big success, loved by both kids and teachers. I just need to be more realistic with the timescales and the skillsets.

 

The Condom Train Returns

After some more reading on the lack of Paid Apps in Ireland, it appears that all the blame may not be on Google after all. From what I can see, Mobile Operators in each country have to sign up to the Open Handset Alliance (OHA) to get “access” to the Android Market.

How much do you bet that none of the Irish operators have done so?

We know that Three UK SIMs work in Vodafone Ireland HTC Desires. I’m pretty confident that they’ll work in all other Irish Android handsets since the operators are just box-shifting rather than getting custom versions done.

Those Three SIMs are free.

In the 1970s, there were a series of famous train journeys to the North where motivated modern people went to buy tons of condoms and distribute them down South where they were banned by the local yokels.

So let’s organise a follow-up.

Swarms of desperate Android owners and their friends grabbing fistfuls of free 3UK SIMs in Belfast and Newry, bringing them back over the border and distributing them to the hungry-for-apps hordes?

You know it makes sense.

“What do we want?” “Paid Apps” “When do we want them?” “Now!”

The revolution is upon us.

 

Dear Vodafone/Meteor/O2, please give Google a hard slap from Irish Android Owners

The first Google Android phone went on sale in Ireland at the start of November 2009. It is now May 2010. And we still don’t have Paid Apps in Ireland. All of the networks are losing Android phone sales because of this and all of those people who have Android phone are pissed off and wishing they’d bought bloody iPhones.

What the hell is the hold up Google?

As I have posted here many times, I can stick a PAYG Three UK SIM in my phone when in Ireland and see all of the paid apps. I can then buy them in Google Checkout using my Irish Credit Card.

There is even an app out there to trick the phones into thinking there are not in Ireland to achieve the same thing.

So the problem is not technical.

Where is the problem? Taxation? Surely you have that sorted, given the billions you run through the Irish operations here?

Not setup to get your 30% cut? 30% of SFA is still SFA. The rev-share you’ll get in Ireland on App sales would barely cover petty cash for paper-clips in Mountain View.

I would guess that minimal revenue is the reason until you look at the list of enabled countries. It’s tiny. 

  • Australia
  • Austria
  • Canada
  • France
  • Germany
  • Italy
  • Japan
  • Netherlands
  • New Zealand
  • Spain
  • Switzerland
  • United Kingdom
  • United States

And I don’t think I have seen that change in more than 6 months.

Wait a second. New Zealand? Are you kidding me?

Sort this.

Now.