Sean
I'm not familiar with Cloudfront specifics, but reading
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html,
my understanding is that while Cloudfront will not attempt at caching
files larger than 50 GB, it is able to forward HTTP Get Range requests
smaller than 50 GB to the origin. And you would have to design a
particularly unfriendly file for GDAL to have to download chunks of > 50
GB !
Otherwise yes the strategy of using a VRT mosaicing tiles < 50 GB each
should work fine if you want to benefit from Cloudfront caching. The
easiest to tile your raster is to use gdal_retile, but be careful to
make sure each tile fit into RAM (as gdal_retile reads & writes each
tiles in a single time). To create large tiles, you'll rather have to
use gdal_translate -srcwin in a loop iterating over raster coordinates.
Even
Le 09/09/2024 à 16:06, Sean Turner via gdal-dev a écrit :
Hi all,
Curious if anyone is doing something similar.
We've got our data in an S3 Bucket that is fronted by Cloudfront. From
there we can use the GDAL `GDAL_HTTP_HEADER_FILE` environment variable
to point to a file containing Cloudfront signed cookies (which
control access to files within Cloudfront).
Cloudfront supports a maximum file size of 50gb. Trying to add a
raster layer that is greater than 50gb immediately results in an error.
Is anyone doing something similar and have a work around? AWS
Documentation mentions RANGE requests where a range of incremental
50gb are requested from Cloudfront and downloaded simultaneously. Is
this supported by GDAL (QGIS)?
Otherwise also considering using a VRT?
https://gdal.org/en/latest/drivers/raster/vrt.html
Open to any and all suggestions!
Cheers. Also, sorry I haven't done one of these mailing lists before,
apologies if this submission is in the wrong place.
All the best,
Sean
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
