UPDATE:
Second attempt at this:
I just grabbed the three CSV files from the Residential Property Price Register site (why is a Captcha needed for that?). I merged some address columns in Excel and added “Ireland” to all of the addresses. I then just imported into Google Fusion Tables and told it to Geo-Code the data.
Note that zero data validation has been done. Everything on this map may be completely wrong. I’m sure some of the more obscure addresses (or non-unique ones) are missing.
Aha! There are daily limits on Geo-coding so I’ll have to do this over a few days before the full map of points appears. But this is a decent start:
UPDATE 2:
2011 Fusion Table. 2011 Map (this will populate properly over next few days too):
2012 Fusion Table. 2012 Map (this will populate properly over next few days too):
UPDATE 3:
Replacement 2010 Fusion Table and Map with Euro symbol correctly encoded:
UPDATE 4: Of course I should have pointed out earlier that the Fusion tables themselves are very useful. You can filter on location, date, price etc without having to fill-out a silly captcha.
Anyone interested in one aggregated Table/Map with all the data from all years in it?
UPDATE 5: As of 16:09 on Oct 1st, the 2012 and 2011 data is 100% geo-coded and 2010 is at 63%. However, based on Brendan’s comments below, we may be missing a lot of data points because Google can’t geo-code them. Once the process is complete, I’ll see if we can extract the missing ones and manually adjust the addresses so that they work. There are also plenty of errors from what I can see.
UPDATE 6: As of 9am Oct 2nd, the final data from 2010 is now geo-coded. Short follow-up post to come on how Google could do far more intelligent and less error-prone geo-coding on Irish addresses without post-codes.
Hi Conor,
Would it be a good idea to compare Google Fusion’s geocoder against MapQuest’s geocoder to check who returns the better results?
MapQuest has a daily limit of 5000 requets and the accuracy is also very good in my opinion.
What is the Fusion daily limit?
Of course, if only we had postcodes…
Brendan
Do not get me started on postcodes
I had thought the Fusion Table limit was 2,500 per day but it seems to be getting thorough the data pretty quickly this morning (up to 70% on one Table). Limits also seem to be per table rather than per account which is cool. The three tables are approx 20k, 17k and 14k rows respectively.
I’ll try and have a go at Mapquest tonight.
Currently at 50% for 2010, 63% for 2011 and 74% for 2012 so hopefully will be done by tomorrow.
Of course I have noticed few markers in Australia and the US!
I tried using MapQuest on a data sample from each table… I used the first 100 rows from 2010, 2011 and 2012.
Results were not great to be honest…
- 107 geocoded (of these 83 matched to address level, 24 matched to nearest town or parent thoroughfare)
- 193 not matched at all!
I used some neat processes with FME (www.fmeireland.com) but the MapQuest address data just wasn’t up to scratch. I was using their “Open” Data API, rather than their commercial API but I’m not sure what the difference would be.
Here’s a good example from the 2010 table: 134 Ashewood Walk, Summerhill Lane, Portlaoise.
This was not matched with either the Google or MapQuest APIs:
http://open.mapquestapi.com/geocoding/v1/address?location=134%20Ashewood%20Walk,%20Summerhill%20Lane,%20Portlaoise,Ireland&callback=renderGeocode&outFormat=xml
https://maps.google.ie/maps?q=134+Ashewood+Walk,+Summerhill+Lane,+Portlaoise,+Ireland&t=m&z=16
However, I started removing the first address field and rechecking data. For this example, Google got a match for nearby Summerhill Lane but MapQuest again failed to get a match! Of course, this meant settling for a non-address geocode that places the house within a mile or so of the location:
http://open.mapquestapi.com/geocoding/v1/address?location=Summerhill%20Lane,%20Portlaoise,Ireland&callback=renderGeocode&outFormat=xml
https://maps.google.ie/maps?q=Summerhill+Lane,+Portlaoise,+Ireland&t=m&z=16
So while the method used was good for filtering through results, it proved that MapQuest’s open data geocoder API just isn’t at the races for this type of thing (ahemmm, non postcode geocoding!). I use MapQuest for it’s free routing API in Ireland when I know the source Lat and Lng to assist http://MapAlerter.com, but geocoding is a different story unfortunately.
If anyone uses FME and wants to see my method then let me know. It’s a good way to build your own geocoder and to interact with web APIs like MapQuest and Google as well as do hundreds of other checks (“Is this property in Ireland or Australia!).
Brendan
Brendan, do you know if there is a way to specify each part of an address to Google Maps? Its raw parsing of full address strings appears to be totally illogical.
Here are two wrongly placed examples from the maps above:
11 The Rise, Friars Hill, Wicklow Town, Wicklow, Ireland – Fusion Tables geo-codes this to South Africa! Google Maps offers various US addresses.
Friars Hill, Wicklow Town, Wicklow, Ireland – Google Maps gets it close in the correct town.
Friarshill, Wicklow Town, Wicklow, Ireland – Closer
11 The Rise, Friarshill, Wicklow Town, Wicklow, Ireland – Perfect! So one space changes the continent!
The other one was:
47 The Bay, Merrion Road, Dublin 4, Dublin, Ireland – Fusion Tables puts this in the middle of Dublin Bay. Google Maps makes a half-decent stab at Tara Towers on the Merrion Road.
Merrion Road, Dublin 4, Dublin, Ireland – Gives a decent rough approximation in Google Maps.
The Bay, Merrion Road, Dublin 4, Dublin, Ireland – Google Maps goes nautical again!
Even if we could say country=X, state=Y, city=z, we’d be in the right ballpark rather than having these gross parsing errors that Fusion Tables is making.
I also wonder why Google Maps seems to be better than the Geo-Coder at guessing. Is it a different text parser?
I have to say I am having a laugh at the single space moving a result to a different continent…
Here’s what I think is a solution…but it is not Fusion-friendly.
I had an application a few years ago that carried out geocoding using Google’s API at the point of registration. The idea was for users to add their town/city and county so that they wouldn’t have to manually add this on a web map. All was going well until someone in Waterford showed up in Waterford California. So that was that!
But curiousity got the better of me and I figured out that Google Maps has multiple geocoders. There is an API for maps.google.de, maps.google.co.uk, etc, etc. And this is where I think the problem exists for Fusion… they are using maps.google.com for all their geocoding instead of a regional approach. This is just a hunch, but I imagine it to be the case as there doesn’t seem to be a http://www.google.co.uk/fusiontables, for instance.
Not sure if you saw my results using MapQuests Open Geocoder, but the results weren’t great… http://www.fmeireland.com/blog/32-analysing-property-prices-fme-ireland
I wonder if whitespaces are a problem too?
Finally, your blog is VERY much liked by Google… for this response I typed in “google maps geocoding fusion ireland” and this post showed up!
Thanks Brendan. Pity the parser won’t let you specify the sub-fields.
e.g.
?country=IE®ion=dublin&city=dublin&detail=47+The+Bay,+Merrion+Road
The Goog has always been good to my blogs for SEO
That FME piece you wrote is great! I think I’ll do a wrap-up post of every site/app/info created off the back of the Property Register Data.
Pingback: From tiny CSV acorns, many #OpenData oaks can grow. Now where is Ireland’s CIO/CTO? | Cross Dominant
OSM popped into my head this morning after reading about MapBox.
I just tried the Wicklow example on http://nominatim.openstreetmap.org – Total waste of time.
The OSM data for Wicklow is terrible. They seem to have all the streets but almost none of them have names. I’m surprised somewhere of that size, close to Dublin is so badly mapped. Not one geek in Wicklow willing to spend an hour editing? I was able to tidy up parts of Bandon with minimal effort and I’m not even from here.
Getting the government interested in OpenData is an even bigger challenge if we can’t even point to enough tech individuals doing it.