On Thu, 03 Sep 2020, Steffen Möller wrote:
> I looked at datasets.datalad.org. I could well imagine to use your > technology for other (larger) databases like Pfam or UniProt or PDB. For > cute little genomes my initial reaction was that I felt overwhelmed. > Your pointer will certainly help to define what we want. Many thanks! FWIW, a few more notes since you seems to be interested ;): we do have an elderly https://github.com/datalad/datalad-crawler/ "origin of it all" but now just an extension to datalad which allows for efficient "updates" and crawling of external resources. See e.g. this asciinema/script: https://www.datalad.org/for/data-consumers But in many use cases a straight "datalad addurls" command (http://docs.datalad.org/en/stable/generated/man/datalad-addurls.html part of handbook with an example: http://handbook.datalad.org/en/latest/usecases/HCP_dataset.html?highlight=addurls#dataset-creation-with-datalad-addurls) could be sufficient to "quickly" (depending on bandwidth and/or either you use --fast option) populate a datalad dataset with files specified in a spreadsheet/structured records. So if you have some kind of .json or .csv/.tsv with records -- you could try it quickly. addurls also automagically adds columns as git-annex metadata per each file so someone could "toy around" (so far I underused the feature) with "git annex views": https://git-annex.branchable.com/git-annex-view/ or later to facilitate metadata extraction/aggregation/search. A sample dataset (original cause for addurls to be written) is available from http://datasets.datalad.org/?dir=/labs/openneurolab/metasearch if you decide to explore (that data is open so no authorization for access would be needed). The problem you might encounter in your cases is (not that great) scalability of git/git-annex to contain hundreds of files in a single repo. So you might like splitting them into subdatasets (git submodules) or providing custom views as I had mentioned before. addurls makes it easy by establishing a subdataset whenever it encounters // (instead of /) for path separation in the provided filename. PS I shut up now ;) Sorry for the flood of info. We are just very excited for DataLad even though we had been working on it for over 6 years and should be sick of it and git-annex by now ;-) but we do not! -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 WWW: http://www.linkedin.com/in/yarik
