Certainly as you say, individual user patterns are not generally predictable. Sometimes aggregate patterns can be. The "sea of tiles" is the natural design and works great in normal cases. It seems the way to teach it in any case.
Where the filesystem issue comes in would be, for example, the nominal 1 meter per pixel Google Earth, which in plate carrée or like form with 400x400 pixel tiles consists of 253,701,184 tiles at that one ground sample distance. That is a lot for "ls" and a lot for most file systems to enjoy quickly accessing in a single directory. A pyramidal reduced resolution dataset hierarchy will require 4/3rds of this in total, or 338,268,246 tiles. Finer details, such as 50cm advanced satellite and 10cm aerial/drone images scale the portions covered by 4x to 100x. So in the limit one will find the edge of what any OS designer expects "sane" developers to expect. :-) On Wed, Dec 20, 2017 at 3:33 PM, Pablo Rozas Larraondo < p.rozas.larrao...@gmail.com> wrote: > Hi Michael, > > Thanks for your comments, I totally agree with them. File systems will > struggle with the explosion of files resulting from the tile operation. As > you point out, other formats, such as geoTIFF, HDF5 or NetCDF define the > tiling or chunking process internally at the file level. > > The reason for creating the tiles as individual files in the article was > because this is ultimately intended to be stored on the cloud as objects > (this will be covered in the 3rd article). As far as I know, cloud object > stores (ie AWS S3, Google Cloud Storage) do not have a limitation in the > number of objects stored in a bucket (If someone has more information about > this, please share). That is why I proposed to split the tiles as separate > files in the article. > > I also find the caching considerations quite amusing. It is a complex > matter and, in my experience, cache optimisations are quite dependent on > the user access patterns, which are normally hard to predict. > > Cheers, > Pablo > > > > On Wednesday, December 20, 2017 at 2:24:01 AM UTC+1, Michael Jones wrote: >> >> Thank you, Pablo. Very helpful to have this kind of step by step example >> for Go developers. >> >> I have some familiarity in this area and I'd say the practical issues in >> large-scale, high-throughput operation tend to relate to the native >> filesystem. Too many small files overwhelm them and can make directory >> lookups slow. Too many directory levels leads to slow filesystem traversal. >> Sometimes it can help to dice the big image into small independent tiles >> and store those tiles as a mosaic in one's own file type. This is the >> nature of TILED vs ROW storage in the TIFF format. The next level of tuning >> is about leverage the operating system's cache of data read from disk in a >> productive way. You can have our own cache in RAM, of course, but the OS >> likely has that same data cached. There are cases where memory mapping the >> small tile files does what you would want. >> >> There are also dynamic considerations. It may well be that a client >> accessing tile [i][j] will soon want one of the eight surrounding tiles. >> over time, it may be that a direction of browsing through tile-space can be >> established and this can encourage read-ahead, though the benefit is not >> always assured; maybe the accesses are structured and maybe they are not. >> >> Some high-throughput servers in the era of smart web clients (aka Google >> Maps / leaflet ./ etc.) refuse to build custom images and only supply tiles >> in response to a request--leaving tile assembly to the client. >> >> Just some thoughts. None of them would help make what you've done any >> clearer or more helpful to the reader. >> >> Best, >> Michael >> >> >> On Tue, Dec 19, 2017 at 3:37 PM, Pablo Rozas Larraondo < >> p.rozas....@gmail.com> wrote: >> >>> Thank you Thomas for the link to the vips library. I didn't know about >>> it and now I want to read more about its design and internals. >>> >>> The objective of the article was to set a baseline using the Go image >>> library and play with several factors to see how it affects performance. In >>> this first article, I wasn't really trying to come up with the fastest >>> possible image server but to point a few basic techniques that can improve >>> access speed and reduce memory consumption. These techniques should be >>> applicable to any image library, so similar relative performance gains can >>> be achieved with any language or library. >>> >>> The next part, which I'm currently writing, proposes the snappy >>> compression as a way of improving access speed to the data. >>> >>> Cheers, >>> Pablo >>> >>> On Tuesday, December 19, 2017 at 10:28:48 AM UTC+1, Thomas Bruyelle >>> wrote: >>>> >>>> Interesting and nice pieces of code. I wonder if the performances can >>>> be compared to something like `vips` (https://jcupitt.github.io/libvips >>>> ). >>>> >>>> Le lundi 18 décembre 2017 22:51:49 UTC+1, Pablo Rozas Larraondo a >>>> écrit : >>>>> >>>>> Hi, >>>>> >>>>> For those interested on serving or using satellite imagery, I've just >>>>> published the first of a three part series on this subject using Go: >>>>> >>>>> https://medium.com/@p.rozas.larraondo/divide-compress-and-co >>>>> nquer-building-an-earth-data-server-in-go-part-1-d82eee2eceb1 >>>>> >>>>> Any feedback or comment that you might have would be greatly >>>>> appreciated! >>>>> >>>>> Thanks, >>>>> Pablo >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Michael T. Jones >> michae...@gmail.com >> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Michael T. Jones michael.jo...@gmail.com -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
