Re: MBTiles?


Rob Giuliano
 

Solid State drives (including SDs and thumb drives) have always had a read/write limitation concern.  Most list something in the range of 10,000 to 100,000 read/write cycles (or at least that was an old range).

When we got new laptops (Lenovos) at work, they came with solid state drives (great for battery life!).  We were concerned about the life expectancy.  We compile software which creates lots of temporary files.  We also do a lot of testing which generates a lot of data files.

Taking 50,000 as an average and 200 work days a year, that is only 250 read/writes per year.  Of course that is for a given "area" of the device.  BUT, since the drives are only 250 GB (vs. the old spinning drives which were 512 - 1000 MB), our laptop drives are typically very full.  That means the small space that is available is seeing all the "read/writes cycles".

These newer drives are obviously going beyond that.  I have many coworkers with their laptops at > 95% capacity for at least 3 years now, and haven't heard of any crashing yet. 

I suspect newer USB drives and uSD cards are in that same category (or at least hope so!)
 
Robert Giuliano
KB8RCO



From: "'Fred Hillhouse' fmhillhouse@... [aprsisce]"
To: aprsisce@...
Sent: Friday, March 31, 2017 12:44 PM
Subject: RE: [aprsisce] MBTiles?

 
I used a 120GB USB hard drive for 5 years or so in the heat and cold. So far it has been the best storage medium. It has been the most reliable drive and it is still working. It does stay at home now after paying its dues.
 
My 128GB uSD lasted three months which came to a $1/day usage fee before failure. My 64 GB uSD has last 2 years now.
 
Most of the hard drive failures still allowed access to the majority of files while all of the uSD cards failed completely. My USB flash drives fail the same as the uSD.
 
I guess the real point of all this is back up your data. Losing a card on the phone usually means you lost all media (pictures, video) as well. I have as many apps as possible save the data to my uSD card.
 
Best regards,
Fred N7FMH
 
 
From: aprsisce@... [mailto:aprsisce@...]
Sent: Friday, March 31, 2017 9:34 AM
To: aprsisce@...
Subject: Re: [aprsisce] MBTiles?
 
 
Having just replaced a failing uSD card in my phone (3 years-ish old), the failure mode was that the card become non-writable.   When Android tried to write to it, the card spontaneously dismounted causing ALL of my map tiles to disappear (I was making changes to APRSISMO at the time and really thought I had borked everything until I found the dismounted SD card).

I'm suspecting that the single file really isn't any more prone to failure modes than the file system on the card containing lots of tiles.  In either case, you're hosed.

I will look into keeping a "safe" copy of the MBTiles file, especially the active one if it is enabled for writing.  (Yes, I'm thinking to provide for Read-Only MBTiles tile sets that never fetch tiles, but only display what you already have in the file.)  (And yes, I'm also considering using an MBTiles file for what's available within it and using other TileSets for displaying other areas.)  (And yes, I just put the code in APRSIS32 that stores freshly downloaded tiles inside an MBTile database.)  (And yes, this is the last parenthetical!).

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32


On 3/31/2017 9:24 AM, 'Fred Hillhouse' fmhillhouse@... [aprsisce] wrote:
I am thinking more along the mobile/portable application where vibration and temperature variation might cause a failure and there is no access to the back up since it is home. This is not unsurmountable since uSD cards are small.
 
Best regards,
Fred N7FMH
 
 
 
 
From: aprsisce@... [mailto:aprsisce@...]
Sent: Thursday, March 30, 2017 7:12 PM
To: aprsisce@...
Subject: RE: [aprsisce] MBTiles?
 
 
Yes, but even if you keep a bankup (like the -safe xml) the space could be smaller than the separate tile files.  At least on FAT32 disks.
Robert Giuliano
KB8RCO
 
 
My only real concern is if the single file becomes corrupt then the whole thing is lost. Whereas in the other case, only a single tile may be gone.
 
It could be argued that there are fewer disk accesses so there would be less chance of failure.
 
Best regards,
Fred N7FMH
 
 
 
From: aprsisce@... [mailto:aprsisce@...]
Sent: Wednesday, March 29, 2017 11:26 PM
To: aprsisce@...
Subject: Re: [aprsisce] MBTiles?
 
 
The real issue here is that we still format uSD (thumb drives, etc) in a vey inefficient FAT32 format.  Place the same files on NTFS and the two numbers (actual vs. on disk) match much closer.  Unfortunately it is the most "cross platform" file system out there.
 
Therefore, I like the idea of a database style, single file map interface.
 
Robert Giuliano
KB8RCO
 
 

From: "'Lynn W Deffenbaugh (Mr)' kj4erj@... [aprsisce]" <aprsisce@...>
To: APRSISCE Group <aprsisce@...>; APRSISMO@...
Sent: Wednesday, March 29, 2017 3:51 PM
Subject: [aprsisce] MBTiles?
 
 
Greetings all of you mappers out there.

I recent had occasion to copy all of my accumulated map tiles off of one
(failing) uSD card onto another. And I was SHOCKED to see how
inefficiently these tiles are stored on a 64GB uSD card! I've got a
total of 118,405 tiles across 4,624 folders whose size is 667MB, not bad
until you notice that they're occupying 14.4GB (yes GIGAbytes) on disk!
My zoom 20 is 7,925 files of 8.58MB occupying 989MB on disk. And my
zoom 18 is 22,001 files of 50.9MB occupying 2.67GB on the card. I've
got to do something to make this better. (Note that this is on an uSD
card under Android. The waste on an NTFS-formatted Windows volume is
nowhere near that bad.)

I've been considering for some time storing so-called "meta-tiles" the
way an OSM tile server does. It stores 8x8 squares of tiles in a single
meta-tile file with a hashed name that serves to distribute the files
across a file system rather than the strict z/x/y structure of the
actual tile names. The tile server knows how to get the correct image
to serve a single tile from an OSM-formatted URL, but they're not
actually stored on the server in that structure. But I really didn't
like this because it would make my cached tiles completely non-portable.

Enter MBTiles. This is the MapBox spec for tile storage and transport.
It amounts to an SQLite database storing blobs of image data inside a
single file rather than the current directory tree of individual files.
There are tools that can generate MBTiles-formatted files from various
data sources, including from the OSM Overpass API. I'm experimenting
with Maperitive right now generating an MBTiles file for all of Palm Bay
(my hometown) from zooms 0 through 18. I'm using Maperitive's internal
renderer from an Overpass downloaded data file with the web map tiles
disabled. Zoom 18 is taking FOREVER and will end up generating 41,989
tiles, most of which I'll never need because they're out in the
never-never swamp that surrounds the city.

But, enough on that. The question is: do any of you have any experience
or knowledge of MBTiles and do you have any
suggestions/comments/criticism of incorporating that as a tile store in
my APRS clients (yes, both on APRSISCE/32 and APRSISMO).

My plan would be to support importing externally-created MBTiles files
as well as storing all web-downloaded tiles locally in an MBTiles
database rather than as individual files in a folder hierarchy. I would
also hope to provide tools to migrate a current TileSet into an MBTiles
database and vice-versa, exploding an MBTiles database into a folder
hierarchy.

You can read about this at the following URLs:

Specification: https://github.com/mapbox/mbtiles-spec

Maperitive Home: http://maperitive.net/

Maperitive documentation: http://maperitive.net/docs/default.html

Tutorial using Maperitive: http://mapit-gis.com/offline-maps/

MapTiler MBTiles viewer (and more): https://www.maptiler.com/download/

Oh, and even though the storage waste on NTFS isn't as bad as an (u)SD
card, I'm planning this for APRSISCE/32 in the name of map tile
portability. It's a whole lot easier to copy a single file than a whole
tree structure of individual tiles.

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32
 
 

Avast logo
This email has been checked for viruses by Avast antivirus software.
www.avast.com
 
 

Avast logo
This email has been checked for viruses by Avast antivirus software.
www.avast.com
 
 



Avast logo
This email has been checked for viruses by Avast antivirus software.
www.avast.com



Join APRSISCE@groups.io to automatically receive all group messages.