One more point I forgot to mention. Flink SQL supports Hive UDF's[1]. I haven't tested it, but the datasketch hive package should just work out of the box.
Seth [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html On Mon, Apr 27, 2020 at 2:27 PM Seth Wiesman <sjwies...@gmail.com> wrote: > Hi Lee, > > I really like this project, I used it with Flink a few years ago when it > was still Yahoo DataSketches. The projects clearly complement each other. > As Arvid mentioned, the Flink community is trying to foster an ecosystem > larger than what is in the main Flink repository. The reason is that the > project has grown to such a scale that it cannot reasonably maintain > everything. To encourage that sort of growth, Flink is extensively > pluggable which means that components do not need to live within the main > repository to be treated first-class. > > I'd like to outline somethings the DataSketch community could do to > integrate with Flink. > > 1) Create a page on the flink packages website. > > The flink community hosts a website call flink packages to increase the > visibility of ecosystem projects with the flink user base[1]. Datasketches > are usable from Flink today so I'd encourage you to create a page right > away. > > 2) Implement TypeInformation for DataSketches > > TypeInformation is Flink's internal type system and is used as a factory > for creating serializing for different types. These serializers are what > Flink uses when shuffling data around the cluster and when storing records > in state backends as state. Providing type information instances for the > different sketch types, which would just be wrappers around existing > serializers in the data sketch codebase. This should be relatively > straightforward. There is no DataStream aggregation API in the way you are > describing so this is the *only* step you would need to take to provide > first-class support for Flink DataStream API[2][3]. > > 3) Implement sketch UDFs > > Along with its Java API, Flink also offers a relational API and UDFs. The > community could provide UDFs for datasketches like Hive. To do so only > requires implementing the aggregation function interface[4]. Flink SQL > offers the concept of modules, which are a collection of SQL UDFs that can > easily be loaded in the system[5]. A DataSketch SQL module would provide a > simple way for users to get started and expose these UDFs as if they were > native to Flink. > > I hope this helps, I look forward to watching the DataSketch community > grow! > > Seth > > [1] https://flink-packages.org/ > [2] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/types_serialization.html > [3] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_api.html > [4] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/functions/udfs.html#aggregation-functions > [5] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/modules.html > > > On Mon, Apr 27, 2020 at 12:57 PM Flavio Pompermaier <pomperma...@okkam.it> > wrote: > >> If this can encourage Lee I'm one of the Flink users that already use >> datasketches and I found it an amazing library. >> When I was trying it out (lat year) I tried to stimulate some >> discussion[1] >> but at that time it was probably too early.. >> I really hope that now things are mature for both communities! >> >> [1] >> >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-and-sketches-td26852.html >> >> Best, >> Flavio >> >> On Mon, Apr 27, 2020 at 7:37 PM leerho <lee...@gmail.com> wrote: >> >> > Hi Arvid, >> > >> > Note: I am dual listing this thread on both dev lists for better >> tracking. >> > >> > 1. I'm curious on how you would estimate the effort to port >> datasketches >> > > to Flink? It already has a Java API, but how difficult would it be >> to >> > > subdivide the tasks into parallel chunks of work? Since it's >> already >> > > ported >> > > on Pig, I think we could use this port as a baseline >> > >> > >> > Most systems (including systems like Druid, Hive, Pig, Spark, >> PostgreSQL, >> > Databases, Streaming Platforms, Map-Reduce Platforms, etc) have some >> sort >> > of aggregation API, which allows users to plug in custom aggregation >> > functions. Typical API functions found in these APIs are Initialize(), >> > Update() (or Add()), Merge(), and getResult(). How these are named and >> > operate vary considerably from system to system. These APIs are >> sometimes >> > called User Defined Functions (UDFs) or User Defined Aggregation >> Functions >> > (UDAFs). >> > >> > DataSketches is a library of Sketching (streaming) aggregation >> functions, >> > each of which perform specific types of aggregation. For example, >> counting >> > unique items, determining quantiles and histograms of unknown >> > distributions, identifying most frequent items (heavy hitters) from a >> > stream, etc. The advantage of using DataSketches is that they are >> > extremely fast, small in size, and have well defined error properties >> > defined by published scientific papers that define the underlying >> > mathematics. >> > >> > The task of porting DataSketches is usually developing a thin wrapping >> > layer that translates the specific UDAF API of Flink to the equivalent >> API >> > methods of the targeted sketches in the library. This is best done by >> > someone with deep knowledge of the UDAF code of the targeted system. >> We >> > are certainly available answer questions about the DataSketches APIs. >> > Although we did write the UDAF layers for Hive and Pig, we did that as >> a >> > proof of concept and example on how to write such layers. We are a >> small >> > team and are not in a position to support these integration layers for >> > every system out there. >> > >> > 2. Do you have any idea who is usually driving the adoptions? >> > >> > >> > To start, you only need to write the UDAF layer for the sketches that >> you >> > think would be in most demand by your users. The big 4 categories are >> > distinct (unique) counting, quantiles, frequent-items, and sampling. >> This >> > is a natural way of subdividing the task: choose the sketches you want >> to >> > adapt and in what order. Each sketch is independent so it can be >> adapted >> > whenever it is needed. >> > >> > Please let us know if you have any further questions :) >> > >> > Lee. >> > >> > >> > >> > >> > On Mon, Apr 27, 2020 at 2:11 AM Arvid Heise <ar...@ververica.com> >> wrote: >> > >> > > Hi Lee, >> > > >> > > I must admit that I also heard of data sketches for the first time >> (there >> > > are really many Apache projects). >> > > >> > > Datasketches sounds really exciting. As a (former) data engineer, I >> can >> > > 100% say that this is something that (end-)users want and need and it >> > would >> > > make so much sense to have it in Flink from the get-go. >> > > Flink, however, is a quite old project already, which grew at a strong >> > pace >> > > leading to some 150 modules in the core. We are currently in the >> process >> > to >> > > restructure that and reduce the number of things in the core, such >> that >> > > build times and stability improve. >> > > >> > > To counter that we created Flink packages [1], which includes >> everything >> > > new that we deem to not be essential. I'd propose to incorporate a >> Flink >> > > datasketch package there. If it seems like it's becoming essential, we >> > can >> > > still move it to core at a later point. >> > > >> > > As I have seen on the page, there are already plenty of adoptions. >> That >> > > leaves a few questions to me. >> > > >> > > 1. I'm curious on how you would estimate the effort to port >> > datasketches >> > > to Flink? It already has a Java API, but how difficult would it be >> to >> > > subdivide the tasks into parallel chunks of work? Since it's >> already >> > > ported >> > > on Pig, I think we could use this port as a baseline. >> > > 2. Do you have any idea who is usually driving the adoptions? >> > > >> > > >> > > [1] https://flink-packages.org/ >> > > >> > > On Sun, Apr 26, 2020 at 8:07 AM leerho <lee...@gmail.com> wrote: >> > > >> > > > Hello All, >> > > > >> > > > I am a committer on DataSketches.apache.org >> > > > <http://datasketches.apache.org/> and just learning about Flink, >> > Since >> > > > Flink is designed for stateful stream processing I would think it >> would >> > > > make sense to have the DataSketches library integrated into its >> core so >> > > all >> > > > users of Flink could take advantage of these advanced streaming >> > > > algorithms. If there is interest in the Flink community for this >> > > > capability, please contact us at d...@datasketches.apache.org or on >> our >> > > > datasketches-dev Slack channel. >> > > > Cheers, >> > > > Lee. >> > > > >> > > >> > > >> > > -- >> > > >> > > Arvid Heise | Senior Java Developer >> > > >> > > <https://www.ververica.com/> >> > > >> > > Follow us @VervericaData >> > > >> > > -- >> > > >> > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink >> > > Conference >> > > >> > > Stream Processing | Event Driven | Real Time >> > > >> > > -- >> > > >> > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> > > >> > > -- >> > > Ververica GmbH >> > > Registered at Amtsgericht Charlottenburg: HRB 158244 B >> > > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, >> Ji >> > > (Toni) Cheng >> > > >> >
