Just as a preamble: it seems obvious to me that this has almost certainly been done by someone, somewhere. Almost everything has been, and if anyone could point me to the appropriate hole in the interchoobs I will sing their praises.
OK, so here goes.
Background.
I have a task that can be done by brute force - if a specific map region has changed at a given scale, upload new geoTIFF at that scale, then rebuild the tile pyramid for that scale. (I don't create the geoTIFFs... the client does).
The brute force approach is easy, and is the method for scales/zoom levels where the cost of re-tiling would not be prohibitive.
There is one scale - 1:20K - where it is prohibitive, however. At scales under 1:20K, the mapset concentrates on key areas, which can be treated separately... and none of them take more than half an hour to tile from scratch. At scales over 1:20k, the tiling is taking place at a sufficiently broad zoom that it's a 5 minute job.
However at 1:20k, the coverage expands - totalling 400-odd geoTIFFs weighing in at 10GB in total filesize.
The city in question wraps around a really large bay, so tiling the bounding box of the constituent geoTIFFs (which is what will happen if they are vrt-ed together) increases the workload by 75% (mostly because of the bay, but also because of the shape of the parts of the city that are farthest from the bay). All of that will be tiles full of nodata.
The 'edge problem' for the tiles means that if one geoTIFF changes, it can't be retiled by itself and the tiles 'sticky-taped' into the pyramid - because when you tile them individually you wind up with nodata areas (the edges) that wipe out the non-nodata areas of the corresponding edges of the next geoTIFF...
To illustrate, I've kludged together a couple of images of tiles generated from adjacent geoTIFFs in isolation - both of the images below are what's in 12/3695/1584.png - i.e., z=12, x=3695, y=2511 (or y=1584 TMS) for the respective geoTIFFs (obviously another 2 geotiffs furnish the bottom part of the merged image). If all of the 12/3695/2511.png were merged, only the nodata from each of the pngs would be affected.
When this task was first done, the geoTIFFs were mosaiced together (dunno what mechanism was used - I would build a GDAL vrt), and tiled in one shot. Talk around here is that it took the best part of a week (before my time).
That's not an option now, simply because the aim of this exercise is to enable updating of the tile pyramid in a timely manner - a day's worth of re-tiling isn't anybody's definition of 'timely'.
I've considered the following process, and rejected it as probably inefficient:
It seems to me that this should be functionally identical to vrt-ing together the affected geoTIFF and its neighbours, then tiling an exact (k×256)×(m×256), k,m integers, subset of that vrt, where the subset starts at the top corner of the xyz ref to the left of, and above, the upper-left corner of the affected geoTIFF, and ends at the xyz ref below and to the right of the bottom-right corner of the affected geoTIFF.
Given the the co-ordinates of the extent of each geoTIFF (trivial to find), this should be doable (and should really be the easiest approach... but I have had a go at it for the last 2 days without anything remotely like success.
It also crossed my mind that it ought to be possible to 'merge' the files somehow - tiles that occupy the same xyz will have the same 'y' name, and will be in the same 'x' directory for a given zoom level; they will have non-overlapping data areas and merging them should work a treat.
So I was envisaging a third alternative as follows:
Copy all of the pyramids for every geoTIFF into one directory tree, and merge any tile files that have the same xyz.
Anyhow... this probably reads as slightly 'stream-of-consciousness-ish'; I have racked my brain for decent search terms to plug into NSa's Little helper (Google), to no avail thus far.
أكثر...
OK, so here goes.
Background.
I have a task that can be done by brute force - if a specific map region has changed at a given scale, upload new geoTIFF at that scale, then rebuild the tile pyramid for that scale. (I don't create the geoTIFFs... the client does).
The brute force approach is easy, and is the method for scales/zoom levels where the cost of re-tiling would not be prohibitive.
There is one scale - 1:20K - where it is prohibitive, however. At scales under 1:20K, the mapset concentrates on key areas, which can be treated separately... and none of them take more than half an hour to tile from scratch. At scales over 1:20k, the tiling is taking place at a sufficiently broad zoom that it's a 5 minute job.
However at 1:20k, the coverage expands - totalling 400-odd geoTIFFs weighing in at 10GB in total filesize.
The city in question wraps around a really large bay, so tiling the bounding box of the constituent geoTIFFs (which is what will happen if they are vrt-ed together) increases the workload by 75% (mostly because of the bay, but also because of the shape of the parts of the city that are farthest from the bay). All of that will be tiles full of nodata.
The 'edge problem' for the tiles means that if one geoTIFF changes, it can't be retiled by itself and the tiles 'sticky-taped' into the pyramid - because when you tile them individually you wind up with nodata areas (the edges) that wipe out the non-nodata areas of the corresponding edges of the next geoTIFF...
To illustrate, I've kludged together a couple of images of tiles generated from adjacent geoTIFFs in isolation - both of the images below are what's in 12/3695/1584.png - i.e., z=12, x=3695, y=2511 (or y=1584 TMS) for the respective geoTIFFs (obviously another 2 geotiffs furnish the bottom part of the merged image). If all of the 12/3695/2511.png were merged, only the nodata from each of the pngs would be affected.
When this task was first done, the geoTIFFs were mosaiced together (dunno what mechanism was used - I would build a GDAL vrt), and tiled in one shot. Talk around here is that it took the best part of a week (before my time).
That's not an option now, simply because the aim of this exercise is to enable updating of the tile pyramid in a timely manner - a day's worth of re-tiling isn't anybody's definition of 'timely'.
I've considered the following process, and rejected it as probably inefficient:
- identify the 'edge' tiles for each geoTIFF and store the file detailsfor each edge tile;
- if a given geoTIFF is revised, find the affected edge tiles from the saved details; and
- when tiling the new geoTIFF, vrt it with its affected neighbours, but only save tiles 1-2 rows/columns past the edge of the changed geoTIFF.
It seems to me that this should be functionally identical to vrt-ing together the affected geoTIFF and its neighbours, then tiling an exact (k×256)×(m×256), k,m integers, subset of that vrt, where the subset starts at the top corner of the xyz ref to the left of, and above, the upper-left corner of the affected geoTIFF, and ends at the xyz ref below and to the right of the bottom-right corner of the affected geoTIFF.
Given the the co-ordinates of the extent of each geoTIFF (trivial to find), this should be doable (and should really be the easiest approach... but I have had a go at it for the last 2 days without anything remotely like success.
It also crossed my mind that it ought to be possible to 'merge' the files somehow - tiles that occupy the same xyz will have the same 'y' name, and will be in the same 'x' directory for a given zoom level; they will have non-overlapping data areas and merging them should work a treat.
So I was envisaging a third alternative as follows:
- keep a copy of tile pyramids for each geoTIFF;
- if a geoTIFF is to be amended, delete its old pyramid and create a new one
Copy all of the pyramids for every geoTIFF into one directory tree, and merge any tile files that have the same xyz.
Anyhow... this probably reads as slightly 'stream-of-consciousness-ish'; I have racked my brain for decent search terms to plug into NSa's Little helper (Google), to no avail thus far.
أكثر...