While crafting an initial reply to Jun, I just stumbled across https://genexa.ch/sars2-bioinformatics-resources/ in the Nanopore Covid-19 forum which is offering readily usable Covid-19 resources for a set of tools that we already have in our distribution.
For one, Yaroslav may have an idea about the kind of technology they should be using to redistribute their downloads. :o) While I am writing these lines, I think I would not want to use any of these .zip files/.tarballs if I do not exactly know how these were created. My initial reaction was that I would never use this. And then I looked a the descriptions and thought like .. uh .. something as trivial as a wget "some search URL"|clustalw -if - or so, I mean, who would want to redo that - not many. But no way you use that without knowing exactly what has been done. @Yaroslav, how would you prepare that genexa site, instead? Could there be some "mouseover"-event to inspect the provenance? Any other reasonable way to github-like have a web presentation for a set of resources that also informs on the provenance? If you have not spotted this - I am basically declaring a partial defeat. I do not need to redo what is trivial if its provenance is inspectable and I _could_ redo it. Best, Steffen On 10.10.20 02:08, Yaroslav Halchenko wrote: > On Fri, 09 Oct 2020, Steffen Möller wrote: >>> On Fri, 09 Oct 2020, Steffen Möller wrote: >>>> I added datalad-crawler to >>>> https://docs.google.com/spreadsheets/d/1tApLhVqxRZ2VOuMH_aPUgFENQJfbLlB_PFH_Ah_q7hM/edit#gid=401910682 >>>> . >>> thanks! requested write access to contribute (no worries -- I will not >>> be as vocal there ;)) >> Jun has proven to be exceptionally fast. > yeap, I already fixed the typo ;) > >>>> * redistribution of the such prepared files and folders with datalad. >>> sure -- could be done from "generic" to specific ("datalad"). But as >>> for Debian users, they do not want to learn any generic or >>> specific. They know and want "apt install" so it is the question of >>> "what would be the best". (I also like apt-file command to often find a >>> file which might be in some package but I do not know which...) >> Yip. Using that, too. For the moment I do not see how to mix a >> getData-installed setup and a datalad-provided one. That may be trivial >> but I don't see it, not knowing enough. > If "getData" (sorry, didn't look) can output just a list of URLs to > download and any metadata to decide on filenames (besides the ones from > the URLs, and as contained in the archives, if desired) it could be as > easy as > > datalad create blah && cd blah && getData --url-records blah | datalad > addurls - '{url}' '{_url_basename}' > > to get yourself a datalad dataset with all the files. If there are > tarballs , subsequent call to datalad add-archive-content would add > extracted files. At some point we should "merry" datalad-crawler and > "datalad addurls" to make it even easier to crawl/update. Actually if > `getData` returns such records, should be easy to just create getData > specific crawler pipeline which would do all needed, and be > efficient for updates. > >>> If getData could account for aforementioned possible gotchas and provide >>> resilient solution, sure -- why not? ;) >> Well. We should talk to diverse upstreams if they would offer their >> (raw) data via git-annex in some way. > that is the beauty of git-annex which sparked the whole datalad effort: > git-annex can access data from a wide range of sources, including plain > urls , and allow for custom "external special remotes". This way > we added support for getting files from tarballs (we first add tarball > to git annex, and individual files are obtained via datalad-archives > git-annex special remote), or from portals with odd authentication > schemes (we have datalad special remote), so we would first ask user > credentials, and authenticate on their behalf. E.g. in case of some it > would be interaction with their auth server to first get a token to then > use for access to S3. > > The point is -- no need to talk to any human/upstream! Just alert > them whenever their data is actually not accessible or broken -- I ran > into a good number of cases of broken archives etc. They post data, > nobody cares to download and thus they do not even know that it is junk > they host. > >> You discuss transport-reliability. I don't care too >> much, will just manually (?) reinitiate what has failed and check into what >> datalad then redisitributes only after getData was successful. > I am talking more about upstream changing or moving the data, > portals going offline (I had a use case for > https://github.com/datalad/nih--videocast whenever government shutdown > and their original portal went down, but videos still were accessible > from original urls which otherwise nobody could get to ;)). > >> Maybe that got lost a bit. We have talked a lot about the complexity of >> workflows. But maintaining the reference datasets has some complexity to >> it, too. I lack immediate examples, but it is not unthinkable that the >> same search tool is used from different tools but each downstream tool >> uses it with different parameters which may need to prepare different >> indexes for. > yes. that is what Michael Hanke (with whom we started DataLad) had in > mind while developing python3-whoosh based search -- ability to > establish individual index depending on the purpose... in my simple > human life, I am looking more for 'google-like' approach -- simple > query, if desired -- more specialized, and internal implementation > should take care about optimizing (well - current default search backend > in datalad is not dumb simple and not optimizing -- a very simple loop > through the records ;)) > >> Or - different genome sizes may be suggestive for different hash >> sizes/min index lengths whatever and possibly downstream tools need to >> know about these parameters? > sorry, I have little to no clue in bioinformatics, so might misinterpret > what has sizes we are talking about ;) > >> I really like all the meta information that datalad can pass with its >> data, but we fall short of everything if we only discuss redistribution. > EXACTLY! that is why datalad is not just for re-distribution, although > born to fill that niche. > >>> And datalad is also in debian already. git format of debian source >>> packages have been supported for a while. so in principle you could just >>> wrap pre-created "generic" git/git-annex (datalad) datasets to >>> become debian source packages and just provide debhelpers for >>> post-install/uninstall >> Here I disagree (yet). We get in trouble if we just remove some TB of >> data and hours of indexing because of a deleted package on a server when >> the clients want to keep using those indexes. I would separate data and >> package management. > ;-) use-cases vary! Nothing and nobody can prevent someone to remove > TBs of data in general. So it is more about avoiding unintended > actions. By default 'datalad remove' is quite slow since it first > verifies that data is still available in the original location and could > be re-obtained if desired -- takes time -- gives time for a 2nd > thought ;) Other prevention mechanisms could be implemented to ask user > a confirmation to uninstall something as massive. > >> Again, RL stronly suggest me keep doing something else for the immediate >> future. > understood. RL is indeed a ... RL ;) > >> Do you have neurological data from COVID-19 patients? Many are > nope :-/ in research environments, unlikely any useful dataset would > come out, since they would not have means to recruit relevant > population, and many are still on quarantine (like ours) and not > scanning. Hospitals would be the main source of good relevant data, > e.g. google lead me to > http://www.ajnr.org/content/early/2020/09/10/ajnr.A6717 > I have emailed corresponding author, let's see... > > But I feel that "very very unlikely" any clinical neuroimaging > data would be shared. Eh, forgot to mention authors our other > project https://open-brain-consent.readthedocs.io/ which could have come > handy (or could be in the future). > > The best candidate would be https://www.ukbiobank.ac.uk/ when they > collect more of longitudinal neuroimaging data on participants who also > went through COVID. But that database is not free to access, and > process is somewhat tedious, but the database is quite rich. > >> discussing depressions etc. And would there possibly be OMICS data from >> the same individuals? Have not checked the literature, yet. > me neither ;) > >> Should be >> available, somewhere. We could then use such data as a joint exercise >> and come up with a not completely unreasonable small project, say (Just >> something from the hip) to rank patients for their Ca++ transporter >> expressions (in the blood which may also affect neurons, either directly >> or via the blood) and for look associated phenotypes on the EEG. When >> we get through this together, we should then also know about how getData >> or datalad can help us. > Sounds like indeed a good project to pursue. I will my ears/eyes open > on either any neuro-related data pops up somewhere, but I have stated my > concerns above ;) >
