Hi Andrea,
After digging through this more, my understanding now is that all granules
are harvested because:
1) The CoverageStore post handler instantiates an ImageMosaicReader [1]
2) An ImageMosaicReader automatically harvests granules when it is created.
Example test showing this behavior in [2]
Only after the ImageMosaicReader is instantiated (and all granules
harvested) is the harvest method explicitly called with the filename I
provide as data in the POST call.
For my use case, I have successfully applied the following workaround:
1) Use the Indexer wildcard to identify a single granule on disk
2) Update the Indexer XML file, replacing the wildcard with the granule
file name
3) Create the ImageMosaic and associated coverages
4) Revert the wildcard in the Indexer XML file to its original value, and
make a PUT call to update the ImageMosaic
5) At some later time, harvest granules by with a POST request like in my
original email
It seems like this is the best solution I have for now. But to avoid this,
maybe it would be reasonable to have some mechanism for constructing an
ImageMosaicReader without triggering a harvest?
Thanks,
Dan
[1]
https://github.com/geoserver/geoserver/blob/e4817074ef05fe96f446e8f3ca7449c6c4f95b65/src/restconfig/src/main/java/org/geoserver/rest/catalog/CoverageStoreFileController.java#L95
[2]
https://github.com/geotools/geotools/blob/c97f23e6575349db5e092ac1ec86bd52901e03a5/modules/plugin/coverage-multidim/netcdf/src/test/java/org/geotools/coverage/io/netcdf/NetCDFMosaicReaderTest.java#L162
On Thu, Mar 22, 2018 at 8:34 AM, Daniel Baston <dbas...@gmail.com> wrote:
> Hi Andrea,
>
> Thank you for the reply. After the curl command above, every file that
> matches the wildcard in the XML Indexer file (a) is listed in the PostGIS
> granule index table, and (b) has an index file created in NETCDF_DATA_DIR.
> Files that do not match the wildcard are not processed.
>
> If I pass a filename that does not exist, I get a "Failed to locate the
> input file" error response, but only after all files matching the wildcard
> are processed.
>
> I verified that this is still happening in 2.13.0. Looking at the code, I
> didn't see anything obvious, but I will try to attach a debugger and follow
> it through.
>
> Thanks,
> Dan
>
> On Thu, Mar 22, 2018 at 5:23 AM, Andrea Aime <andrea.a...@geo-solutions.it
> > wrote:
>
>> Hi Daniel,
>> the underlying mechanism does not have a notion of single file, but one
>> of "directory + file filter".
>> In theory the filter should make the code scroll over the directory
>> contents, and skip all files but
>> the ones that is due to be harvested.
>>
>> If you have evidence it's actually opening other files (instead of just
>> listing them and then ignoring them), then yes, I'd say it's a bug
>>
>> Cheers
>> Andrea
>>
>>
>> On Wed, Mar 21, 2018 at 1:51 PM, Daniel Baston <dbas...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I have a script that configures some ImageMosaics in GeoServer through
>>> the REST API. The granules are stored in netCDF format, and there are a lot
>>> of them. Because granule harvesting takes a lot of time, I'd like to create
>>> all of my ImageMosaics with only a single granule, and then harvest
>>> granules after all the ImageMosaics have been created. However, when I try
>>> to harvest a single granule with the REST API, GeoServer instead harvests
>>> the entire folder. Is this a bug, or am I misunderstanding how this should
>>> work?
>>>
>>> I am following the steps in the GeoSolutions training [1]. First,
>>> I create the ImageMosaic with a PUT call to this URL:
>>>
>>> http://172.17.0.4:8080/geoserver/rest/workspaces/mar07/cover
>>> agestores/composite_forecast/file.imagemosaic?configure=none
>>>
>>> That works as expected. Then, I use the following CURL command to try
>>> and harvest a single file:
>>>
>>> curl -v -u admin:geoserver -XPOST -H "Content-Type: text/plain" -d
>>> "file:///opt/data/mar07/composite/composite_1mo_201706_trgt201707.nc" "
>>> http://172.17.0.4:8080/geoserver/rest/workspaces/mar07/cove
>>> ragestores/composite_forecast/external.imagemosaic"
>>>
>>> This takes several minutes and results in all files in
>>> /opt/data/mar07/composite
>>> being harvested.
>>>
>>> Am I missing something here? Any advice would be appreciated.
>>>
>>> Thanks,
>>> Dan
>>>
>>> [1] https://geoserver.geo-solutions.it/edu/en/multidim/rest/index.html
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Geoserver-users mailing list
>>>
>>> Please make sure you read the following two resources before posting to
>>> this list:
>>> - Earning your support instead of buying it, but Ian Turton:
>>> http://www.ianturton.com/talks/foss4g.html#/
>>> - The GeoServer user list posting guidelines:
>>> http://geoserver.org/comm/userlist-guidelines.html
>>>
>>> If you want to request a feature or an improvement, also see this:
>>> https://github.com/geoserver/geoserver/wiki/Successfully-req
>>> uesting-and-integrating-new-features-and-improvements-in-GeoServer
>>>
>>>
>>> Geoserver-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>>
>>>
>>
>>
>> --
>>
>> Regards,
>>
>> Andrea Aime
>>
>> ==
>> GeoServer Professional Services from the experts! Visit
>> http://goo.gl/it488V for more information.
>> ==
>>
>> Ing. Andrea Aime
>> @geowolf
>> Technical Lead
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> <https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>
>> 55054 Massarosa
>> <https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>
>> (LU)
>> phone: +39 0584 962313 <+39%200584%20962313>
>> fax: +39 0584 1660272 <+39%200584%20166%200272>
>> mob: +39 339 8844549 <+39%20339%20884%204549>
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>>
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>> principi dettati dal D.Lgs. 196/2003.
>>
>> The information in this message and/or attachments, is intended solely
>> for the attention and use of the named addressee(s) and may be confidential
>> or proprietary in nature or covered by the provisions of privacy act
>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>> copying, distribution, or either dissemination, either whole or partial, is
>> strictly forbidden except previous formal approval of the named
>> addressee(s). If you are not the intended recipient, please contact
>> immediately the sender by telephone, fax or e-mail and delete the
>> information in this message that has been received in error. The sender
>> does not give any warranty or accept liability as the content, accuracy or
>> completeness of sent messages and accepts no responsibility for changes
>> made after they were sent or for other risks which arise as a result of
>> e-mail transmission, viruses, etc.
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this
list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html
If you want to request a feature or an improvement, also see this:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
