Re: perl streaming framework

2020-07-14 Thread Shlomi Fish
Hi Warren! Please reply to list. On Tue, 14 Jul 2020 10:34:57 +0800 Warren Pang wrote: > Hi > > Does perl have a stream computing framework? > I know Java/python have many, such as spark, flink, beam etc. > But I am looking for a perl alternative, since most of our team members > have been usi

Re: perl streaming framework

2020-07-14 Thread Fernando Santagata
…or maybe something like Pandas https://pandas.pydata.org/ ? BTW, it would be nice to build something like that for Raku. (Shameless plug: I'm working on a Raku interface to the GNU Scientific Library (see: https://modules.raku.org/search/?q=Math%3A%3ALibgsl). The next modules in my queue are all

Re: perl streaming framework

2020-07-14 Thread Warren Pang
Hello We are already using perl PDL and something similar. The streaming framework is mainly used for real time data analysis. Tyler from Apache Beam project has wrote a great book about streaming system: http://streamingsystems.net/ So I have interest to know if there is the perl implementation f

NativeLibs not loading on Module testing

2020-07-14 Thread Richard Hainsworth
I'm adding a Module to the ecosystem, and the Build system failed, see https://travis-ci.com/github/Raku/ecosystem/builds/175326153 It doesn't seem that it is my Module that's failing, but NativeLibs. see line 165. As for my Module (raku-pod-render) it passes all tests, including Test-Meta lo

Re: perl streaming framework

2020-07-14 Thread Richard Hainsworth
There was a talk by Steven Lembark about using the lazy gathers and concurrency constructs in Raku at the last Perl conference that might be of interest. Steven (if I remember correctly) was using these tools to analyse vast quantities of data. The language constructs exist in Raku, and the

impact of compiling pod doc

2020-07-14 Thread Marcel Timmerman
Hi, I was wondering if pod documentation has a large impact on compiling modules. I assume on small files it would be ignorable but I have some modules which have grown big, one has 7153 lines of which the biggest part is pod doc. Would it be useful to split such a file in two. one for code a

Re: perl streaming framework

2020-07-14 Thread Shlomi Fish
Hi Warren, On Tue, 14 Jul 2020 16:54:49 +0800 Warren Pang wrote: > Hello > > We are already using perl PDL and something similar. > The streaming framework is mainly used for real time data analysis. > Tyler from Apache Beam project has wrote a great book about streaming > system: > http://stre

Re: perl streaming framework

2020-07-14 Thread Brad Gilbert
If you really want a streaming framework for Perl, the mailing list for Raku users might not be the best place to ask. (Raku used to be known as Perl6, and we haven't done anything to change the name of this mailing list.) Raku has a very similar syntax to Perl. (It used to be called Perl6 after a

Re: perl streaming framework

2020-07-14 Thread Aureliano Guedes
I already mentioned it here. Since Raku supports truly well functional programming. To data analysis might be better something like https://www.tidyverse.org/ implemented in R to approximate a natural language. On Tue, Jul 14, 2020 at 11:03 AM Brad Gilbert wrote: > If you really want a strea

Re: impact of compiling pod doc

2020-07-14 Thread JJ Merelo
It's going to be pretty much the same. If it's installed, it's going to be precompiled anyway. The overhead added by reading the file might even make it slower. Of course, you can always give it a try and measure. Measuring might always surprise you :-) El mar., 14 jul. 2020 a las 14:20, Marcel Ti

Re: NativeLibs not loading on Module testing

2020-07-14 Thread JJ Merelo
I've submitted now a PR to NativeLibs that fixes that issue. El mar., 14 jul. 2020 a las 11:03, Richard Hainsworth (< rnhainswo...@gmail.com>) escribió: > I'm adding a Module to the ecosystem, and the Build system failed, see > > https://travis-ci.com/github/Raku/ecosystem/builds/175326153 > > It

Re: perl streaming framework

2020-07-14 Thread Ralph Mellor
I second Brad's comment -- perlmonks.org is a great place to ask. Use the following link for a quick start. The user interface is something out of the dark ages but you'll generally get outstandingly good replies, especially if you explicitly add something like "Thank you for considering my questi

Re: impact of compiling pod doc

2020-07-14 Thread Marcel Timmerman
Thank you for answer JJ. It is quite a work to split such a large file up so I thought I'd better ask before I dive in head first. Later I might think it over. I have this documentation on the github pages too so I could separate the pod doc and keep them somewhere else where they won't get pr

Re: perl streaming framework

2020-07-14 Thread William Michels via perl6-users
Steven Lembark's talk (using Raku to analyze 123GB of BLAST-formatted AA sequences). Definitely worth watching: https://www.youtube.com/watch?v=rgCk5w2o-GY On Tue, Jul 14, 2020 at 2:16 AM Richard Hainsworth wrote: > > There was a talk by Steven Lembark about using the lazy gathers and > concurr

Re: perl streaming framework

2020-07-14 Thread Warren Pang
Thank you all very much. I have tried to search "perl discussion" and it brings me to perl6's list page. Yes for data analysis we primarily use classic perl5, which is smart enough especially the regex matching. There are "spark streaming", "flink streaming", "storm streaming", and a lot of others

Re: perl streaming framework

2020-07-14 Thread William Michels via perl6-users
Hi Warren, The topic you raise is an interesting one. When I've previously searched for 'streaming' and 'Python' I often see articles on Iterables such as the following: [1] "Data streaming in Python: generators, iterators, iterables" https://rare-technologies.com/data-streaming-in-python-generat

Re: perl streaming framework

2020-07-14 Thread Aureliano Guedes
I'm really sad about the conceptions of programming with non-functional languages. Truly better an R as a reference than a OO in data analysis. On Wed, Jul 15, 2020 at 12:13 AM William Michels wrote: > Hi Warren, > > The topic you raise is an interesting one. When I've previously > searched for

Re: perl streaming framework

2020-07-14 Thread Warren Pang
Thanks for the info @Wiliam. We are probably seeking for a streaming framework which is Engineering-oriented. The documentation you provided are mostly Algorithm-oriented. For instance, as an algorithm engineer, I can implement the algorithm Logistic regression by hand. But I won't do this, instea