Please pardon me in advance -- came out long again. TL;DR summary:
- would be great to merry getData and datalad to benefit from knowledge getData possess ATM on data sources, but then gain (many) benefits from git/git-annex/datalad, especially while thinking about "research process", data management, reproducibility etc - I shared some initial prototype for the "dh-annex" helper I started years back but might be over-complicated/under-engineered, and definitely not yet complete NB BTW https://wiki.debian.org/getData still points to SVN which is gone? "and here we go..." <preaching> On Fri, 09 Oct 2020, Steffen Möller wrote: > I added datalad-crawler to > https://docs.google.com/spreadsheets/d/1tApLhVqxRZ2VOuMH_aPUgFENQJfbLlB_PFH_Ah_q7hM/edit#gid=401910682 > . thanks! requested write access to contribute (no worries -- I will not be as vocal there ;)) > The problem I still have with a datalad-only solution is that it > alienates the folks that have always done it by themselves, i.e. > fetching the databases from upstream, unpacking and indexing it all for > all the tools that possibly ask for it. What I see is > * some automated routine that prepares all the downloads/indexes somewhere > * whoever wants to redo/improve that process themselves please copy > that automated routine > * redistribution of the such prepared files and folders with datalad. sure -- could be done from "generic" to specific ("datalad"). But as for Debian users, they do not want to learn any generic or specific. They know and want "apt install" so it is the question of "what would be the best". (I also like apt-file command to often find a file which might be in some package but I do not know which...) Sure it could be also accomplished with non-datalad "backend" to actually fetch the data BUT you would need to re-implement what git-annex does for us already (checksumming, redundant access urls, etc), and might quickly get into trouble of unfetchable data so you would need to keep copies, but become efficient in how you manage them to not duplicate across versions of the same dataset when those start to appear, so you would arrive to annex'es .git/annex/objects keystore... so - yes, could be done, but might be actually more work in the long run IMHO and AFAIK from experience of working with more "live" (not just a dump at the end of the project, but being cooked as more data comes in, analyzed/reanalyzed etc) > Somewhere somehow we need to link this to the packages that are > installed/installable. I don't think we should have any redistribution > with datalad without that automated processing. For me, strongly biased, > the automation comes via getData. If getData could account for aforementioned possible gotchas and provide resilient solution, sure -- why not? ;) What is great about having getData is it is a great resource to indeed feed datalad (there is also datalad addurls which is an "alternative" to datalad-crawler extension; we will merry them eventually) to establish datalad datasets and thus possibly provide redundant availability to that data if hosting it somewhere (or nowhere and just keeping it until original host disappears or changes data) and then publish ;) (I do the same... see also note about containers below and take it along with the news that docker hub soon will start pruning elderly images). And datalad is also in debian already. git format of debian source packages have been supported for a while. so in principle you could just wrap pre-created "generic" git/git-annex (datalad) datasets to become debian source packages and just provide debhelpers for post-install/uninstall and be done, gain resilience through redundant availability and exact versioning, etc. but damn me -- why I haven't done it already if it all so "easy"? ;) I think I was over-engineering starting with a too complex project to start with, and thus not finishing at all :-/ And then just started to use datalad directly too much. But I found that thing I started to work on https://github.com/yarikoptic/dh-annex which has the script https://github.com/yarikoptic/dh-annex/blob/master/tools/generate_sample_repo which I believe was my CI for the https://github.com/yarikoptic/dh-annex/blob/master/tools/dh_annex ;) Back to the topic of "why may be datalad": And then there is the whole question of empowering 'git knowledgeable' users... any installed datalad dataset then could become a subdataset within "study" dataset; throw some containers inside (could also be a debian package -- have a look at my collection of singularity containers for neuroimaging which is also datalad dataset: https://github.com/ReproNim/containers/), use datalad container-run (datalad-container and singularity-container are in debian), and make the whole "research" project if not fully auto-reproducible but with a thorough history and exact versioning of all ingredients. Have a look e.g. at http://handbook.datalad.org/en/latest/basics/basics-yoda.html for more information. </preaching> <sorry> I will try to not pollute the list with <preaching> much more </sorry> > Concerning Sepp, my hunch is that tipp and the data is not needed for > immediate aims and I would skip that for now, leaving your comment on > the excel sheet. coolio. -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 WWW: http://www.linkedin.com/in/yarik
