Re: Need help to join Apache Flink community on.Slack

I need the invite as well . kdq...@gmail.com On Sun, 5 Jun 2022 at 9:32 PM, sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > >> Hi Santhosh, >> >> just invited y

Apache Flink + Apache arrow flight

Hi Have anyone done any POC around using arrow or arrow flight with flink ? Is there any use case available around these 2 APIs being used ? I am thinking to use arrow as default data format , so the actions and transformations would be fast where data shuffling is required . Any insights or pointe

Re: Parquet files in streaming mode

I would suggest taking a look at CheckpointRollingPolicy. You need to extend it and override the default behviors in your FileSink. HTH. Thanks Deepak On Mon, Dec 27, 2021 at 8:13 PM Mathieu D wrote: > Hello, > > We’re trying to use a Parquet file sink to output files in s3. > > When running i

Re: Observability tools on top of Flink

I am interested in learning this fact as well as I need to put in observability in a flink pipeline. On Wed, 22 Sep 2021 at 8:40 PM, Dan Hill wrote: > Hi! > > I saw a recent Medium article >

Observability around Flink Pipeline/stateful functions

@d...@spark.apache.org @user I am looking for an example around the observability framework for Apache Flink pipelines. This could be message tracing across multiple flink pipelines or query on the past state of a message that was processed by any flink pipeline. If anyone has done similar work a

Re: Realtime Data processing from HBase

I would suggest another approach here. 1.Write a job that reads from hbase , checkpoints and pushes the data to broker such as Kafka. 2.Flink streaming job would be the second job to read for kafka and process data. With the separation of the concern as above , maintaining it would be simpler. Th

unsubscribe

Re: Best coding practises guide while programming using flink apis

g > > > > 在 2019年9月22日，下午2:12，Deepak Sharma 写道： > > Hi All > I guess we need to put some examples in the documentation around best > coding practises , concurrency , non blocking IO and design patterns while > writing Apache Flink pipelines. > Is there any such guid

Best coding practises guide while programming using flink apis

Hi All I guess we need to put some examples in the documentation around best coding practises , concurrency , non blocking IO and design patterns while writing Apache Flink pipelines. Is there any such guide available ? E.g. when and how to use the GOF design patterns . Any code snippet can be put

Contribute as freelancer on a flink project from scratch

Hi All I am looking to contribute on a flink project (May be a enterprise use case) Please do let me know if anyone got such opportunities. Thanks Deepak -- Thanks Deepak www.bigdatabig.com www.keosha.net

Re: serialization issue in streaming job run with scala Future

Does TaxiRide or TaxiRideFare implements Serializable? On Mon, Sep 16, 2019 at 3:47 PM Debasish Ghosh wrote: > Hello - > > The following piece of code is an example of a connected data streams .. > > val rides: DataStream[TaxiRide] = > readStream(inTaxiRide) > .filter { ride ⇒ ride.getIsSt

Re: May be useful: our reference document for "Understanding State in Flink"

I want to volunteer for maintaining or adding to this kind of document. Please do let me know if i can. Thanks Deepak On Wed, Apr 24, 2019 at 6:33 AM Deepak Sharma wrote: > > > On Wed, Apr 24, 2019 at 5:14 AM Till Rohrmann > wrote: > >> Thanks for sharing this resour

Re: May be useful: our reference document for "Understanding State in Flink"

On Wed, Apr 24, 2019 at 5:14 AM Till Rohrmann wrote: > Thanks for sharing this resource with the community Oytun. It looks really > helpful. > > I'm pulling in David and Fabian who work a lot on documentation. Maybe > it's interesting for them to take a look at. The community had once the > idea

Re: Error with Cassandra

can you try with this? 3.3.0 On Mon, Aug 13, 2018 at 1:52 PM yuvraj singh <19yuvrajsing...@gmail.com> wrote: > Hi , > > I am getting > > java.lang.NoClassDefFoundError: Could not initialize class > com.datastax.driver.core.Cluster > > please help with it . > > my pom is > > > > 1.5.0 >

Re: Real time streaming as a microservice

Is it on github Mich ? I would love to use the flink and spark edition and add some use cases from my side. Thanks Deepak On Sun, Jul 15, 2018, 13:38 Mich Talebzadeh wrote: > Hi all, > > I have now managed to deploy both ZooKeeper and Kafka as microservices > using docker images. > > The idea c

Any remote opportunity to work on Flink project?

Hi Flink Users, Sorry to spam your inbox and GM all. I am looking for opportunity to work on Flink project , specifically if its Flink ML over streaming Please do let me know if anyone is looking for freelancers around any of their Flink projects. -- Thanks Deepak

Re: Multiple hdfs

Wouldnt 2 core-site and hdfs-site xmls need to be provided in this case then ? Thanks Deepak On Tue, May 22, 2018, 19:34 Raul Valdoleiros < raul.valdoleiros.olive...@gmail.com> wrote: > Hi Kien, > > Thanks for you reply. > > Your goal is to store the checkpoints in one hdfs cluster and the data

co-location groups vs slot sharing groups

Hi, I have a few questions regarding slot sharing and co-location: The slotSharingGroup(String name) function sets the slot sharing group, is there a function to set the co-location group? Does setting a colocation group exclude tasks of other groups from the co-located group's task slots? Than

Sample project on real time ingestion of more than 1Billion events at a time

Hi All Can anyone point me to sample code that can scale upto real time ingestion of events in millions or billion ? Also i need reference to which analytic engine to store the transformed events for real time querying . Thanks a lot in advance. --Deepak

Re: Use Flink in Oozie Workflow

I don't think there is custom flink action. You can use sshAction instead. Thanks Deepak On Jan 8, 2017 17:31, "Malte Schwarzer" wrote: > Hi all, > > does Flink currently support the integration into a Apache Oozie ( > https://oozie.apache.org/ ) workflow? Is there a Flink custom action for > t

Re: Serializing NULLs

Hi Matt Can you please share the code like how you are mapping null to long? Thanks Deepak On Tue, Dec 20, 2016 at 9:25 PM, Matt wrote: > Here is the back trace: https://gist.github.com/ > 56af4818bcf5dee6b97c248fd9233c67 > > In the meanwhile I've solved the issue by creating a POJO class where

Available for freelancing

Hi I am spark developer with good exposure to spark streaming/core/sql. I have just started on flink and keen to take any part time role as developer , in case anyone needs it. Please do write me back , if anyone has got any such opportunity. -- Thanks Deepak

Re: SQL for Flink

v/table_api.html. > > https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND% > 20component%20%3D%20%22Table%20API%20%26%20SQL%22%20ORDER% > 20BY%20priority%20DESC > > Greg > > On Wed, Sep 14, 2016 at 12:27 PM, Deepak Sharma &

Re: SQL for Flink

+1 Yes.I agree to having SQL for Flink. I can take up some tasks as well once this starts. Thanks Deepak On Wed, Sep 14, 2016 at 3:47 PM, Radu Tudoran wrote: > Hi, > > > > As a follow up to multiple discussions that happened during Flink Forward > about how SQL should be supported by Flink, I w

Re: problem of sharing TCP connection when transferring data

Thanks Ufuk. Sure I would pick some starter issues and get to these low level issues. -Deepak On 23 May 2016 11:21 pm, "Ufuk Celebi" wrote: > On Mon, May 23, 2016 at 7:30 PM, Deepak Sharma > wrote: > > Would it be possible to get involved on this issues and start >

Re: problem of sharing TCP connection when transferring data

I am not Flink master or regular user of FLink , but would like to start contributing to Flink. Would it be possible to get involved on this issues and start contributing to Flink community? Thanks Deepak On Mon, May 23, 2016 at 10:49 PM, Ufuk Celebi wrote: > On Mon, May 23, 2016 at 6:55 PM, wa

Re: keyBy on a collection of Pojos

Can you try serializing your POJO ? Thanks Deepak On Mon, May 23, 2016 at 8:42 PM, Flavio Pompermaier wrote: > Sorry Rami, you're right :) > Unfortunattely I've never used Flink streaming so I cannot be helpful > there.. > Myabe is it something related to the default triggering policy of the >

Re: Where to put live model and business logic in Hadoop/Flink BigData system

I see the flow to be as below: LogStash->Log Stream->Flink ->Kafka->Live Model | Mongo/HBASE The Live Model will again be Flink streaming data sets from Kakfa. There you analyze the incoming stream for the certain value and

Processing millions of messages in milliseconds real time -- Architecture guide required

Hi all, I am looking for an architecture to ingest 10 mils of messages in the real time streaming mode. If anyone has worked on similar kind of architecture , can you please point me to any documentation around the same like what should be the architecture , which all components/big data ecosystem

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu Would you be able to provide the use case here in ElasticSearch with Flink? Thanks Deepak On Sat, Dec 5, 2015 at 1:25 AM, Madhukar Thota wrote: > Sure. I can submit the pull request. > > On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels > wrote: > >> Hi Madhu, >> >> Great. Do you wa

Any role for volunteering

Hi All Sorry for spamming your inbox. I am really keen to work on a big data project full time(preferably remote from India) , if not I am open to volunteering as well. Please do let me know if there is any such opportunity available -- Thanks Deepak