# tianjara.net | Andrew Harvey's Blog

### Entries tagged "dev".

9th October 2012

I've recently pushed out a new srtm3-stylesheets repository which contains shell scripts for working with NASA SRTM DEM data, gdaldem based stylesheets for shaded relief maps, Mapnik stylesheets for contours and a TileStache configuration for sandwich those styles together into a single map.

This was spurred on by the fact that I simply needed a map which showed hills. I've used Andy Allan's OpenCycleMap in the past which has color relief and contours. Unfortunately it is closed source.

So I put my head down and hacked together repeatable scripts to get the source data up and running and some basic stylesheets to produce a usable and pleasant looking map. All released as free and open source software under the CC0 license.

I want to avoid adding things like streets etc, such maps could be built as separated layers based upon this style and sandwiched together, for example, with the TileStache sandwich provider.

I hope to build upon the lessons learnt here to produce a map like the Stamen Terrain map, except with the source code released under a free and open source license. Perhaps just with hill and slope shading applied to landuse with other map features placed on top.

I've rendered NSW (only server resource prevent worldwide!) as a slippy map here.

Tags: dev, geo, osm.
30th September 2012

I've just released some scripts and stylesheets for the Geoscience Australia Australian Bathymetry and Topography Grid data.

Here are some samples of the final outputs. (also available as a layer on my demo slippy map)

These images based on data © Commonwealth of Australia (Geoscience Australia) 2009, which was released under the Creative Commons Attribution 2.5 Australia Licence, http://creativecommons.org/licenses/by/2.5/au/.

The code I used to produce these images is CC0, so as far as I am concerned you are free to recreate them under whichever license is compatible with the upstream data. If you just wish to use these images, I license the ones you see here (so only if you grab them from here, rather than re-rendering them) under the Creative Commons Attribution 3.0 Australia license, http://creativecommons.org/licenses/by/3.0/au/.

Tags: dev, geo.
11th December 2011

## Map Labelling Suburbs and Cities

Yesterday I pushed a bunch of changes to my OSM/Aerial Imagery Hybrid Style, see the (demo). I spent a bit of time on suburb and city labels as they are a really important feature of a map aimed at non-experts.

I'm finding making a non-trivial style like this has defiantly made me realise the difficulties of such a task.

There are a lot of improvements that can be made (either by modifying or making new components) to the OSM Mapping conventions, osm2pgsql/imposm, mapnik library, carto language stack. That said, the current form is still great and you can still make great maps. But,

I would like to be able to but this needs
have labels for large bays at lower zooms than small bays need bays mapped as closed ways covering their area, rather than a point in the centre
define a linear function for the size of icons. i.e. at z10 the icon is 10px, at z20 the icon is 20px, now linearly interpolate all sizes for zooms in between either build this functionality into the carto language, or make another higher level macro like language which you can code this in which is then compiled into carto
render spread text inside a riverbank needs functionality in the mapnik rendering engine

## Static OSM Tiles

I'm still yet to find a tile server which is fast and works well with lighttpd (nginx would probably suffice too). As an experiment I decided to pre-render a bunch of tiles for my hybrid style sheet. This tile layer doesn't need to be minutely updated, anyone who needs that can use the normal mapnik layer. Also static tiles server straight from the webserver should be pretty fast (maybe on memcached tiles would be faster) and I wanted my tiles to be fast.

Next up how can I generate these static tiles? There is the popular generate_tiles.py, but that won't render meta tiles, seems like such a waste to render every tile with a buffer of 128px when I would render a 5 by 5 meta tile of the same buffer for only 36% of the total pixels rendered. The larger the meta tiles the larger the latency, but if I'm pre-rendering them all than latency doesn't matter any more.

So I wrote a C++ program as my replacement for generate_tiles.py. I also programmed it to render from a list of meta tiles rather than a bbox. This means for my demo I can only render high zooms where there is nearmap coverage. This is where https://github.com/andrewharvey/OSMTileListFromGeometry/ came in, which pulls nearmap coverage areas from an osm2pgsql database, and generates a list of meta tiles.

Using this method I rendered up to and including zoom 17, composed of 22444 5 by 5 meta tiles or 561100 regular tiles in a time of,

real    198m17.021s
user    103m49.985s
sys     37m36.117s

with disk usage,

<1M   0-8
2.0M  9
6.8M  10
26M   11
98M   12
13M   13
36M   14
114M  15
411M  16
1.5G  17
========
2.2G  total


(Updated with results when using "png" rather than "png256")

real    228m17.082s
user    160m24.725s
sys     31m42.653s

with disk usage,

<1M   0-8
3.1M  9
8.6M  10
28M   11
121M  12
34M   13
83M   14
218M  15
663M  16
2.0G  17
========
3.1G total


I think that time could still be sped up with,

• tuning of the mapnik stylesheet queries
• tuning of the postgres database
• using larger meta tiles
• using a machine with more that 512MB of RAM
• not running minutely updates in the background

## Git Rebase

As I use git more and more I'm slowing learning more of the features it has to offer (and thanks to the free hosting by github). One such feature which I think is awesome is git rebase. As an example the other day I did git commit -ammend instead of git commit --amend (I knew there was a duplicate character somewhere but because I seem to have a tint of dyslexia I confused the duplicate -- with mm).

Of course this resulted in committing all files which had changed with a message of "mend" as a new commit. I did this twice, and only noticed after I had already made a bunch of correct commits afterwards. With git rebase I could pop some commits of the commit stack, remove the two "mend" commits fix the commit which I should have been amended to and pop my other commits back on top of the stack.

Thanks to http://stackoverflow.com/a/180085.

## Next Up

• build a daily OSM planet extract by pushing minute-replicate files into the osmosis psql simple schema, then dumping to an OSM file
• use this OSM file as a basis for monav and an OSRM service for fosm data
• run a name finder service for fosm data
• more updates to the hybrid map style
• render the hybrid style tiles as oblique images for nearmap multiview
Tags: dev, geo, osm.
12th September 2011

Just a quick update to say that I have exhausted all my options to build a latest .osm file from fosm.org minute-replicate osc files + a base .osm file from osm.org. Currently fosm.org is down, but I still want to edit on top of the latest data so I can submit my changesets when it comes back online. This is proving to be more difficult that I thought. Although really it is a good thing as even though I've tried to ensure I have everything I need to rebuild fosm.org if it were to go down for good, I'm still not 100% sure as I haven't tried, well now I get to try.

My first method was to use osmconvert with the aid of the workflow given in https://github.com/rrankin/osmconvert/blob/e1fbb7319f92f338e0023d110a3098f5a979fd64/update_osm.sh. It was quite fast and it almost worked, but due to limitation of osmconvert it seems that it expects the objects to be sorted in the order given by the object IDs. We can either patch osmconvert to work around this, or sort the osc files...

Putting the first method on hold, I gave installing the rails port that runs the API at osm.org a go with the idea that I can load the changesets or osc files directly into the API or DB. I got it installed and got to the step of loading in my starting OSM .osm file from before the FOSM fork. http://wiki.openstreetmap.org/wiki/The_Rails_Port#Populating_the_database even gives the command line to run, but osmosis didn't like this. I believe the issue is fixed with later versions of osmosis though.

This leads me to require a later version of osmosis. I don't really want to install osmosis from source using the upstream project's method as it will pull in a bunch of 3rd party libraries using maven, so the only real option is to work to upgrade the debian package for osmosis. This isn't easy either as I don't know that much about ant/maven/ivy. I could spend I whole weekend just trying to update the debian package of osmosis and still get nowhere, all the time I'm just getting further away from my original goal which was to make some FOSM changesets from my latest trip while my memory is still fresh...

Tags: debian, dev, osm.
20th February 2011

As part of my quest to georeference the old NSW Parish Maps, I ran into the ESRI World file format...

# The Format

I relied on lot on http://en.wikipedia.org/wiki/World_file as a reference when figuring out how to make sense of world files. I remade the diagram from http://en.wikipedia.org/wiki/File:WorldFileParametersSchemas.gif, into two views: pixel centric, and graticule centric (svg versions here).

...and a difference case, where the graticules are rotated in the other direction,

For the purposes of my java program (which I explain below), I define theta as the angle the east/west pointing graticules (I call them lat graticules as they are shown at regular lines of latitude) make with the horizontal, and phi as the angle the north/south pointing graticules (I call them long graticules as they are shown at regular lines of longitude) make with the vertical.

Keep in mind that the image coordinate system and projected coordinate system are different (assuming we are using some kind of UTM projection).

# Writing to a Wld File

Some of the parish maps have graticules shown and a reference origin for the easting and northing values on the graticules. If we can extract this information we should be able to georeference the raster maps. Actually I'm not sure what projection is used... but I think using a zone of universal transverse mercator should be okay. Also I assume that the eastings and northings on the map are in chains.

The first step is extracting the graticules from the raster map to vectors. I do this by loading the image into Inkscape and tracing the graticules as line segments, with an svg path id for the segment something like "w220", for example to indicate west 220. After I have this svg file I run it through pmap-svggraticules2csv.pl which extracts these vector graticules from the svg file and saves them into a csv file.

[caption id="attachment_1275" align="aligncenter" width="421" caption="Example of vector graticles drawn over the raster map. Base map is Public Domain."][/caption]

From the csv file I then can use my Java program graticules2wld to find a best fit world file (which is really just an affine transformation matrix) to georeference this raster image via a best fit approach.

An alternative is to use pmapgrid2gcps.pl to extract ground control points (GCPs) from the svg file by finding the intersection points of the graticules. You can then pass these gcps to GDAL, to either warp the image or use gcps2wld.py (from the Debian package python-gdal) to make a best fit world file from the gcps.

I've made a debian package for the graticules2wld program. The package was really hard to make, although in the end I finally did get it working. I ended up using jh_makepkg on just the source (i.e. using no external buildfiles, just the source code). If you want to make the debian package yourself you should be able to grab this directory, then under graticules2wld-0.1 run dpkg-buildpackage. If you are able to help me so that I'm not duplicating my code in this deb-source directory in the source tree, please help me.

# The Next Step...

Half the point of using the world file, is so I can load the original image into JOSM and apply the affine transformation matrix (from the world file) to show the raster as a backdrop without having to warp the image unnecessarily. So my next step is to get JOSM to be able to open raster images with a world file and correctly place it as a backdrop in the editor window.

Tags: dev, geo.
14th February 2011

For a while I used to think that all there was to XML was <blah attribute="value">inner</blah>, but of course there is much more. I'm now digging into the real stuff like XPath, XSLT and XML Schemas.

I've come across a data set of bus stops (as well as live info on where buses are, and their status). The bus stop data set (http://nswbusdata.info/ptipslivedata/getptipslivedata?filename=stopdescriptions.zip, no longer active so I'm hosting my original copies at http://tianjara.net/data/nsw-buses/ for preservation) is in an XML format,

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<stop longitude="151.17832" latitude="-33.81852" tsndescription="Osborne Rd nr Ronald Av" TSN="206699"/>
<stop longitude="151.17359" latitude="-33.8082" tsndescription="Ralston St nr Murray St" TSN="2066138"/>
<stop longitude="151.17764" latitude="-33.82054" tsndescription="Second Av nr Osborne Rd" TSN="206698"/>
<stop longitude="151.17629" latitude="-33.81926" tsndescription="Fourth Av nr Second Av" TSN="206697"/>
...

Although because of the license, I cannot use this data in OpenStreetMap, I was still interested in converting it into an a .osm file. The perfect job for XSLT!

It turned out to be quite a simple task with a neat solution. My XSLT stylesheet used to do the translation:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/>

<xsl:template match="/StopDescriptionList">
<osm version='0.6' generator='XSLT'>
<xsl:apply-templates select="stop"/>
</osm>
</xsl:template>

<xsl:template match="stop">
<xsl:variable name="count">
<xsl:number/>
</xsl:variable>

<node id='-{count}' lat="{@latitude}" lon="{@longitude}"> <tag k='ref:tsn' v='{@TSN}' /> <tag k='fixme' v='{@tsndescription}' /> </node> </xsl:template> </xsl:stylesheet> Then it was a just a simple, xsltproc -o busses.osm busses-stylesheet.xslt stopdescription.xml The data is CC BY-NC-ND 3.0, but they sneak in some additional terms in the fine print, which in addition to the NC-ND would further lead to incompatibilities with the OSM license, and would under my definition of free data, make this data set non-free. For interest the first three additional terms are, 1. You must not use the Data in any way that could create false or misleading outcomes or interpretations, or bring the RTA into ridicule or disrepute. You must not use the Data in conjunction with the promotion of alcohol or unsafe road practices. 2. You must ensure that the Data used is current, and provide details as to the date and time of sourcing the Data from the RTA in all reproductions of the Data (including in any software applications incorporating the Data). 3. In all reproductions of the Data (including in any software applications incorporating the Data), the following disclaimer must be provided: “The accuracy or suitability of the Data is not verified and it is provided on an “as is” basis.” Tags: dev, osm. 28th October 2010 Some eye candy from my experiments of combining Nearmap terrain tiles with OpenStreetMap tiles. These images (which are hyperlinked to full sized ones) are derived works from OpenStreetMap.org default mapnik tiles (which is based on data from OpenStreetMap contributors licensed CC BY-SA 2.0), and Nearmap.com shaded relief terrain maps licensed under the Nearmap Community License. [caption id="attachment_1198" align="aligncenter" width="600" caption="Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License."][/caption] [caption id="attachment_1199" align="aligncenter" width="600" caption="Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License."][/caption] [caption id="attachment_1200" align="aligncenter" width="600" caption="Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License."][/caption] [caption id="attachment_1201" align="aligncenter" width="600" caption="Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License."][/caption] [caption id="attachment_1202" align="aligncenter" width="600" caption="Nearmap Terrain Maps with OpenStreetMap Mapnik Map. Based on images (c) OpenStreetMap contributors, CC-BY-SA, and (c) Nearmap, Nearmap Community License."][/caption] My approach is based on http://wiki.openstreetmap.org/wiki/TopOSM/Details. I used two mapnik stylesheets, one for everything minus the labels, and one with just the labels. From here I used this bash script, [sourcecode language="bash"] for f in nearmap-dem///* do f=echof|sed 's/[^\/]*\///' #get rid of the nearmap-dem part d=dirname $f mkdir -p "grayshaded/$d"

# ...alternatively, extract the Lightness channel...

convert "nearmap-dem/$f" -separate -channel Lightness -colorspace gray "grayshaded/$f" lastdir=dirname &quot;grayshaded/$f&quot; lastfile=basename &quot;grayshaded/$f&quot; .png rm -f "$lastdir/$lastfile-0.png" "$lastdir/$lastfile-2.png" mv "$lastdir/$lastfile-1.png" "$lastdir/$lastfile.png" done [/sourcecode]

to convert a directory of Nearmap Terrain tiles into something suitable to be passed in as the hillshade layer to the combine script at http://wiki.openstreetmap.org/wiki/TopOSM/Details#Combining_images_into_a_final_composite, and then I used combine script to merge the layers together for each tile.

Update: The Mapnik stylesheets I used for labels and nolabels are at http://gist.github.com/653184

Tags: dev, nearmap, osm.
26th July 2010

I have just discovered that my ADSL router supports UPnP, and that this provides an interface to access 3 important bits of information from the router (external IP address, total bytes sent and total bytes received). I had previously been scraping the router's web interface to grab the external IP. As for the bytes sent and received, I didn't even know the router had a method of reporting these.

My first instinct was to look for a Perl library for UPnP, I found two. One in the Ubuntu repositories (and in CPAN) http://search.cpan.org/perldoc?Net::UPnP::GW::Gateway and another which appears to be in neither, http://perlupnp.sourceforge.net/.

I tried out the first one and after some fiddling get it working (though I haven't yet been able to eliminate the 2 seconds it spends blocked, ie. not executing on the CPU but still not complete).

Next I found a great program that allows you to place an arbitrary command's output in the Gnome Panel, http://code.google.com/p/compa/. Which resulted in,

The Perl script I use to provide the output to the Compa applet is at http://github.com/andrewharvey/perlmisc/blob/master/upnp_router_inoutbytes_to_compa.pl

Tags: computing, dev.
11th December 2009

Ideally I would like to write my own music player because I don't really like any that are currently available (Amarok 1.4, Amarok 2, Songbird, Rhythmbox, Banshee, Exaile). I like features from each but none seem to fit all my needs. All the time I keep rethinking what I should do and I still cannot decide. Anyway this is what my ideal music player would be like...

• Backend Database
• The backend metadata would be stored in an external Postgresql database, with the option for using sqlite for people who don't want to set up and run postgresql.
• The schema should be good and documented, so that a user can read and write into the database. If not at least give an interface to allow this.
• Full playback information. I want my music player to store the timestamps of every time a given song has been played. I want history too, for instance the times of when the song rating was changed.
• Collection Manger
• I want the music player to be the library not just the librarian. I want to give it a file (say an MP3), along with details such as song title, artist, etc. and I want it to take that file and store it on the hard disk in a nice file structure (like iTunes does). Amarok 1.4 attempts to do this but its really hard, because initially it will just add the file to your playlist and not move it across to your collection, and even then if you change the details say the artist it will not correct this in the folder structure used to store that file.
• Tagging songs. Amarok does this well.
• Web scraper
• Acoustic Analysis
• Surely there are algorithms to guess the BPM (beats per minute) of a song. I want that integrated into the music player.
• I need a moodbar so I can navigate a song, and to gather contextual information on how the style of the music varies over the song.
• I don't know much about acoustics, but there must be other algorithm which give meaningful measures of audio. These should be used to group songs and find similar ones.
• This must be done locally, I don't want to send things to web services (MusicBrainz, http://echonest.com/).
• I want a concept of a "Library" rather than a Playlist. Amarok only has playlists, but 99.9% of the time I want a list of all my songs.
• Statistics

Now for the solution. I could try everything from writing my own music player from scratch that implements that all (but I gave up on that after I could not decide what programming language to use C, C++, Java, Perl, Python, what GUI widget toolkit to use Qt, GTK+, wxWidgets, graphics api for nice graphs Cairo, raw OpenGL, OpenGL behind Clutter, R's graph drawing, Processing, or some other CPAN Perl module for drawing nice graphs. I can mix a few but the core app needs one programming language and it needs a core GUI toolkit for the GUI. There is too much choice and I don't have enough experience to know before hand what is best and what I will find easiest and simplest to use.)

I could try to capture playback statistics by looping last.fm and audioscrobber.com to localhost and capturing the data that Amarok sends. Or I could just write a script for Amarok which captures playback, but this only solves part of the problem and then I'm stuck using a certain application. Alternatively I could just take an existing program and fork it to suit my needs.

There should be more to come on this as I start experimenting.

Tags: computing, dev.
11th December 2009

I've just uploaded to GitHub a script to pause Amarok 1.4 playback when the screensaver/screenlock starts and up pause again when closed/unlocked. It addresses the issue I was having with the script at http://nxsy.org/getting-amarok-to-pause-when-the-screen-locks-using-python-of-course where the script would start Amarok if it was not running and it would restart playback on screensaver end/unlock regardless of whether it was playing when the screensaver started.

You could start the script on start-up or plug it into Amarok's script engine to only be active when Amarok is active.

(Oh and in the future I'll try to avoid posts that just duplicate item's from other RSS/Atom feeds that don't add much extra value.)

Tags: computing, dev, sh.