Re: MBTiles TileSet storage

Lynn Deffenbaugh

Another resurrected message with some MBTiles discussion.  The issue with creating the numbered directories has been fixed.  It does still require an empty directory to point to, even though files should not be placed there any more.

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32

-------- Forwarded Message --------
Subject: Re: [aprsisce] MBTiles TileSet storage
Date: Fri, 2 Feb 2018 10:28:05 -0500
From: 'Lynn W Deffenbaugh (Mr)' kj4erj@... [aprsisce] <aprsisce@...>
Reply-To: aprsisce@...
To: aprsisce@...

Ok, to be clear.  It only creates the directories, but doesn't put any actual map tile images in them, correct?  At least, that's what my test here did.

Note:  Everywhere that I reference <TileServer> in the following discussion, it's actually the <OSM.*> element that is used which comes from the <TileServer>'s corresponding elements when a new Configure / Map / Tile Set is selected for display.

Currently APRSIS32 will still require the <TileServer>'s <Path> to exist and it will still create the numbered directories.  But as long as the downloaded tile is successfully stored in the .mbtiles file, it will not be stored in the number directory tree.  Eventually <TileServer>s will support an empty <Path> with a non-empty <MBTile> to prevent this behavior.

Also, on the <Path> inside the <MBTile>, it should point the whole way to the .mbtiles file.  There is no assumption of the name of the file vs the name attribute of the <MBTile> element.  So that added block should have had no effect on your test.

Here's the current sequence of events when a tile is needed for display:

The <MBTile> databases are checked in the order they appear in the XML and only if enabled.  If the tile is found and is a valid image, it is used.

The <TileServer>'s <MBTiles> file is checked if it is configured and it exists.  If the tile is found and is a valid image, it is used.

The <TileServer>'s <Path> is checked for a tile file.  If it is found and is a valid image, it is used.

This same three-step logic is used to look for tiles one or two levels further out which will be stretched for display.  Even if a tile is found and stretched, the actual tile will still be downloaded as follows.  APRSISMO currently doesn't stretch tiles.

If the actual tile is not found, it will be downloaded according to the <TileServer>'s <Server><Port> and <URLPrefix>.  If the download is not successful, all ends here.  Oh, and the appropriate numbered directory is created at this point in anticipation of a successful download.  This will eventually be removed for <MBTiles>-configured servers, but read on.

If a tile is successfully downloaded, it is stored in the <TileServer>'s <MBTiles> database.  If the database insertion fails, or if an <MBTiles> database is not configured, the downloaded tile is stored in the appropriate numbered directory that was previously created.

Once a tile is successfully downloaded, Perfetches are queued for the 2 tiles at -1 and -2 zooms and the 4 tiles at zoom +1.  Remember, each tile has 4 tiles "below" it at the next closer zoom level.  As each of these is processed, they create the appropriate numbered directories and are subsequently stored as described above.  Prefetches do NOT queue additional prefetches, only tiles downloaded for display needs queue prefetches.

Now that I've resurrected that knowledge, I wonder why I ended up with numbered directories from 1 through 15 when I was only displaying at zoom 14?  12 through 15 would make sense, but not 1 through 12.  Gotta look into that one...

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32

On 2/1/2018 11:46 PM, Rob Giuliano kb8rco@... [aprsisce] wrote:
Tried again.


<MBTile Name="OSMTiles">

Under <TileServer Name="Original">.

Now I can close and open the app and it will use the .MBTiles.
Still generates numbered directories, then seem to add them to the database.  Leaves them there.

I think it might have deleted (cleaned up) the number folders once
Robert Giuliano

From: "Rob Giuliano kb8rco@... [aprsisce]" <aprsisce@...>
To: "aprsisce@..." <aprsisce@...>
Sent: Thursday, February 1, 2018 11:31 PM
Subject: Re: [aprsisce] MBTiles TileSet storage

Okay, I gave this a try:
1. Copied my XML from a running system to a new directory
2. Created a shortcut to the EXE (save space and ensure using the same copy)
3. Edit XML to use MBTiles using the default set by changing
4. Same under TileServer[0] 

On startup, it asks for the location of the APRSIS32.osm file.
The MBTile file went from 4k to 1.5 MB, HOWEVER, all the numbered directories and png files are also showing up. 

I tried deleting the numbered directories and APRSIS32.osm.  Asks for location of APRS32.osm again.

I tried deleting the just the numbered directories.  Stops asking, but creates the numbered directories.

It seem to use the .mbtiles files after it starts, but it downloads to numbered directories.  Moving the directories, they aren't recreated, but new zooms and map areas make new directories.

Is it suppose to?
Robert Giuliano

From: "'Lynn W Deffenbaugh (Mr)' kj4erj@... [aprsisce]" <aprsisce@...>
To: APRSISCE Group <aprsisce@...>
Sent: Thursday, February 1, 2018 6:59 PM
Subject: [aprsisce] MBTiles TileSet storage

So, you're wondering about the <MBTiles> and <OSM.MBTiles> elements are
doing in the APRSIS32.XML file? They are the beginnings of support for
using MBTiles (SQLite) databases to store the map tile images instead of
directory trees of individual files.  You can set this up yourself if
you'd like to test it with the following procedure.

0) Make a copy of your working APRSIS32 instance directory so that if
you bork it up, you still have a good working copy!

1) Download an empty MBTiles database from* - It's called Empty.mbtiles and
is only 4,096 bytes.

2) Copy that file to a new name appropriate to the tile set that you'll
be placing into it.  It is easiest to put it in the default execution
directory alongside your APRSIS32.XML file.

3) Edit your APRSIS32.XML file and locate the actual <TileServer> that
you want to test with.  If you want, you can copy an entire
<TileServer>...</TileServer> block to a new entry and change the Name
value within the <TileServer Name="xxxx"> element.

4) In that block, add the name that you called the copy of
Empty.mbtiles, including the .mbtiles extension and the full path unless
it is in your default execution directory.  Yes, with full paths
multiple instances of APRSIS32 can share a single .mbtiles file.  It is
important that each instance is drawing from the same tile server or
thing can get really confusing really quickly.

5) If you are testing with the currently selected TileSet, then you also
need to put the same value in the <OSM.MBTiles> element.  Or you can
just fire up APRSIS32, switch to a different tile set, and then switch
back to the one that is using the <MBTiles> entry.

6) Done.

But what does it actually DO and how can I tell if it is working?  Well,
that's a bit trickier.

The easiest way I now of is to make a new copy of a <TileServer> entry
and point its <Path> to a newly-created empty directory and specify the
<MBTiles> entry as well pointing to a suitably-named copy of the
Empty.mbtiles file.  Then when you select that new TileServer in a
running APRSIS32, you should see the map tiles get downloaded, but
nothing should appear in the directory tree.  And the .mbtiles file
should be growing in size as new tiles are downloaded. Zooming and
panning around to fetch lots of tiles should show the .mbtiles file
continuing to grow.

Then, when you want to see what you've got, open the new .mbtiles file
up in the SQLite database browser mentioned earlier and execute the
following SQL command:

SELECT zoom_level, count(*), min(tile_column), max(tile_column), min(tile_row), max(tile_row) from tiles group by zoom_level order by zoom_level;

Or you can just restart APRSIS32 and look in the sqlite trace log to see
if it has any zoom level ranges in the .mbtiles file.  That output will
look something like:

WinMain:2018-02-01T21:44:21.183 Test.mbtiles z[1] 2 Tiles x:1-1 y:0-1
WinMain:2018-02-01T21:44:21.183 Test.mbtiles z[2] 2 Tiles x:3-3 y:1-2
WinMain:2018-02-01T21:44:21.183 Test.mbtiles z[3] 3 Tiles x:7-7 y:2-4
WinMain:2018-02-01T21:44:21.183 Test.mbtiles z[4] 20 Tiles x:0-15 y:1-8
WinMain:2018-02-01T21:44:21.184 Test.mbtiles z[5] 60 Tiles x:0-13 y:6-15
WinMain:2018-02-01T21:44:21.184 Test.mbtiles z[6] 206 Tiles x:0-26 y:14-32
WinMain:2018-02-01T21:44:21.185 Test.mbtiles z[7] 137 Tiles x:16-37 y:41-54
WinMain:2018-02-01T21:44:21.185 Test.mbtiles z[8] 360 Tiles x:36-78 y:86-108
WinMain:2018-02-01T21:44:21.187 Test.mbtiles z[9] 535 Tiles x:77-153 y:176-217
WinMain:2018-02-01T21:44:21.188 Test.mbtiles z[10] 591 Tiles x:158-305 y:355-431
WinMain:2018-02-01T21:44:21.190 Test.mbtiles z[11] 498 Tiles x:323-608 y:750-860
WinMain:2018-02-01T21:44:21.191 Test.mbtiles z[12] 494 Tiles x:652-1213 y:1502-1718
WinMain:2018-02-01T21:44:21.192 Test.mbtiles z[13] 426 Tiles x:1564-2423 y:3008-3434
WinMain:2018-02-01T21:44:21.193 Test.mbtiles z[14] 138 Tiles x:4197-4844 y:6018-6866
WinMain:2018-02-01T21:44:21.193 Test.mbtiles z[15] 123 Tiles x:8333-9684 y:11516-12816
WinMain:2018-02-01T21:44:21.200 Test.mbtiles z[16] 73 Tiles x:16771-16781 y:25623-25629

This says that my Test.mbtiles file has tiles from zooms 1 through 16
with a concentration of them (300-500/level) in zooms 8 through 13.  The
x and y ranges show the actual OSM tile numbers that are cached in that
level.  Note that this is only the min and max, it does not mean that
every tile in the entire x/y rectangle is actually present in the file.

If any of you are brave enough to give this a shot, let me know how it
goes or especially if it goes bust!

And as with the overlays, I'll be incorporating this into the Configure
/ Map / TileSets / New... user interface.  If you use the SQLite
database browser and look at the contents of the "metadata" table,
you'll notice that it includes TileServer definition information.

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32

PS.  Note that not all .mbtiles files are compatible with APRSIS32 tile
storage.  Some .mbtiles files have been optimized to remove redundant
tile images.  These .mbtiles files can be used as overlay tilesets, but
not as main tile storage as their internal table format is different. 
As long as the SQLite database browser shows only 2 tables called
"metadata" and "tiles", it should be good.

For instance, MBTiles files from mapbox have the "tiles" as a view
rather than an explicit table.  These files can be used as overlay
MBTiles, but not as TileSet .mbtiles storage.

and of course, vector tiles are something completely different!

Notice that the satellite views .mbtiles file (available for pay) is
196GB in size.  Yes, 196 GIGAbytes!


Posted by: "Lynn W Deffenbaugh (Mr)" <KJ4ERJ@...>


Join to automatically receive all group messages.