On Thu, Jul 4, 2019 at 7:45 PM Ankur Sinha <sanjay.an...@gmail.com> wrote:
> On Thu, Jul 04, 2019 12:41:26 +0800, Mohd Izhar Firdaus Ismail wrote: > > Hi all, > > Hello! > > Long time no see!! > yeah its been a while ^^ > > > I'm wondering if there are people who are working on data engineering is > > interested in working on a SIG focusing on DE. > > > > Currently my idea on this SIG would be: > > 1 - packaging data engineering related softwares into Fedora, and make > them > > easy to install, covering from workflow tools (eg: airflow, luigi), data > > processing engines (eg: apache spark, flink), visualization tools > (superset, > > redash) and make life easier for that. I'm not sure how much these tools > can > > fit into fedora packaging guidelines (lots of bundled jars, and users > expects > > upstream binaries, esp on engines such as spark/flink), which is > something to > > brainstorm on. > > 2 - ambassador related activities around promoting fedora as a platform > for > > data engineers to use. > > > > If interested, i'm on this telegram group: > https://t.me/fedora_dataengineering > > While I'm not directly in data engineering/science, this is now a very > very important component of the general scientific pipeline. So, I'd be > interested to help out, and I'm sure the rest of the NeuroFedora team > would be too. It'll probably also be beneficial to rope in the sci-tech > folks, and any other folks that were part of the data science/machine > learning SIGS (not sure if the SIGs are active). I also expect the > Astronomy SIG would be interested in some tools, since they also do deal > with rather large amounts of data. If we can bring more of these SIGs > together to help each other, we'll get a lot more work done :) > > https://fedoraproject.org/wiki/Category:SciTech_SIG > https://fedoraproject.org/wiki/SIGs/ML > https://fedoraproject.org/wiki/Category:Astronomy_SIG?rd=Astronomy_SIG > https://docs.fedoraproject.org/en-US/neurofedora/overview/ > > yeah i agree .. my exp dealing with the tools highlights one tricky problem we might face -> avoiding statically / bundled dependencies , while on the same time getting all the tools to work .. most of them locks to specific dependency version, and wont work on other.. and at the moment i'm not sure whats the right approach to this, so really need input from others. My current workflow is to install each tool and all their dependencies in their own virtualenvs or vendor directory. > We're meeting at Flock to discuss how we can leverage the excellent > resources that the Fedora community provides to enable Free Science. > Please do join the discussion too: > > https://pagure.io/flock/issue/220 > > Unfortunately i dont think i can reach Flock, flying to budapest at the moment is beyond my budget and time availability. But if you can bring the topic on how to handle such dependency conflict / bundled dependencies, that'll be really helpful. -- > Thanks, > Regards, > Ankur Sinha "FranciscoD" (He / Him / His) | > https://fedoraproject.org/wiki/User:Ankursinha > Time zone: Europe/London > _______________________________________________ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > -- Mohd Izhar Firdaus Bin Ismail / KageSenshi Fedora Malaysia Contributor & Ambassador http://blog.kagesenshi.org 92C2 B295 B40B B3DC 6866 5011 5BD2 584A 8A5D 7331
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
