GDAL 1.8 with MrSID

Since moving to Debian I no longer had the MrSID reader for GDAL. It's non free software so I could do without it, but some data I have was only provided in MrSID format, and I need to get it out of the format. I used the binaries from http://developer.lizardtech.com/ (http://www.lizardtech.com/developer/members/download.php?dl=Unified_DSDK_8.0_linux.x86-64.gcc41.tgz), along with http://download.osgeo.org/gdal/gdal-1.8.0.tar.gz. I used this ./configure line (which fixes the "skipping incompatible /lib/Release/libltidsdk.a when searching for -lltidsdk" error). Binaries are put in apps.

./configure --with-spatialite --with-poppler --with-libkml --with-mrsid=/home/pathto/Unified_DSDK_8.0_linux.x86-64.gcc41/Raster_DSDK --with-jp2mrsid --libdir=/home/pathto/Unified_DSDK_8.0_linux.x86-64.gcc41/Raster_DSDK/lib

BOM Geofabric Dataset

A while back the Bureau of Meteorology made available the Australian Hydrological Geospatial Fabric dataset licensed for use under Creative Commons 3.0 license. It is available at ftp://ftp.bom.gov.au/anon/home/geofabric/. I was unable to read the files in the supplied formats, so upon my request the BOM made available the files in ESRI Shapefile format, if you would like the files in SHP file format you should make a request to AHGF@bom.gov.au. (or give me a public FTP server I can upload the dataset to).

Once I had the SHP files, I used ogr2osm to convert them to OSM files, which I could then load into JOSM.

[sourcecode language="bash"]

!/bin/sh

for f in SH_*SHP/SH/.shp do echo "$f"

# clip the source shp files
# EPSG:4326 is WGS84 LL
# we convert to EPSG:4326 as this allows us to specify the clip bbox in WGS84
dir=`dirname "$f"`
mkdir -p "01-trim/$dir"
ogr2ogr -t_srs EPSG:4326 -clipdst minlon minlat maxlon maxlat  "01-trim/$f" "$f"

# run ogr2osm with output file specified
base=`basename "$f"`
mkdir -p "02-osm/$dir/"
ogr2osm.py -o "02-osm/$dir/$base.osm" "01-trim/$f"

done [/sourcecode]

Apart from the learning experience this exercise seems to have been futile, as the data is far inferior to the existing data in OpenStreetMap.

In the past few months I've been contributing to the OpenStreetMap database. From the day I started I've had privacy concerns on my mind. If a criminal goes around committing crimes, leaving behind a signature and always travelling x km from home to commit the crime, you get a nice circle around their home. So I'm interested to see what kind of pattern I'm leaving behind. Someone else has probably got an existing solution out there, but I decided to experiment and put together something myself.

I already had all my osmChange files (i.e. the ones from http://www.openstreetmap.org/api/0.6/changeset/${changeset_id}/download). I pull these daily from http://api.openstreetmap.org/api/0.6/changesets?display_name=aharvey find the changeset ids and download the osmChange files for all my new changesets. I do this as I try to keep a local copy of things I upload into the cloud, as a backup in case the service I uploaded to ever decides to cut my access. I know I should really be saving a copy of what I upload, rather than uploading then asking the cloud server for a copy, but JOSM uploads a full osm file, not an osmChange file. Scripts here.

Anyway, from these osmChange files I pull out the coordinates of all the nodes I created or modified, script here.

From this list of points I could use any heat map program. gheat which is based on http://blog.corunet.com/the-definitive-heatmap/ looks good. The problem is I ran out of time setting it up– it wasn't in the Debian repository. Instead I used http://www.sethoscope.net/heatmap/, which I combined with a wrapper script to render to tiles in the OSM Slippy Map format. However it is really slow– perhaps I should try to speed it up.

Now that I had the transparent heat map tiles, the next step was to get these on top of an OpenStreetMap map. Given they are in the standard format, it should be really easy to add this layer to any software that lets you view OSM tiles. Support for overlays like this in libchamplain is on my todo list, in the meantime I gave Polymaps a try, HTML with JavaScript embedded here.

[caption id="attachment_1246" align="aligncenter" width="600" caption="Heat map image of my created or modified nodes in OpenStreetMap"][/caption]

The results don't come as much of a surprise, I already knew which areas I edit and how much. This map is also biased towards nodes, and doesn't take into account editing an existing way or relation. One surprise was some cosmetic work along creeks shows up as nice lines (tracing a creek adds a lot more nodes than adding a single toilet node)...

Postscript

While I was working on this I once again needed a bbox string to test with. This has come up a lot in the past and I didn't know any easily way to use a GUI to position the map, and then just copy and paste the bounding box of that view. So I made a modification to one of the libchamplain demos to return a bounding box from the current view.

After spending way to much time trying to edit some EXIF metadata in some JPEG photographs, I'm posting my method here for future reference.

The first thing I needed to do was geoencode/geotag the images with a GPS location tag. I did this with the perl module Image::ExifTool::Location (script here).

The second thing I needed to do was examine what metadata was in the JPEG and reduct some of it. I used exifprobe and exiftool -v to examine the metadata. I ended up using this command on each image:

exiftool -overwrite_original -scanForXMP \
 -MakerNotes:SerialNumber='0' -MakerNotes:OwnerName='' -MakerNotes:InternalSerialNumber="0" -XMP:SerialNumber= \
 -XMP:OwnerName="Andrew Harvey <andrew.harvey4@gmail.com>" \
 -XMP-cc:AttributionName='Andrew Harvey' -XMP-cc:License='http://creativecommons.org/licenses/by/3.0/' -XMP-cc:attributionURL="http://www.flickr.com/photos/andrewharvey4/"\
"file_name.jpg"

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html was a good reference to find out about the MakerNotes tags. Also it took me a little experimenting and research before finding out that "MakerNotes tags may be edited, but not created or deleted individually." This is why the SerialNumber tag is set to zero, and not removed. I tried to remove the camera serial numbers, but who knows, Canon probably secretly embed the serial number into the image pixel values as well...

http://wiki.creativecommons.org/XMP also provided me with some hints on how best to embed these JPEGs with at least some form of machine readable tagging as CC BY licensed.