Re: Time extracting in flink

imestamp field (maybe an array) in your record with > printing a timestamp on in by each processing. > > > Andy Hoang mailto:a...@parcelperform.com>> > 于2019年7月22日周一 下午4:49写道： > Hi guys, > > I’m trying to write elk log for flink, this help us to store/calculate

Time extracting in flink

Hi guys, I’m trying to write elk log for flink, this help us to store/calculate processing time of a group of operators for business auditing. I read about process_function and Debugging Windows & Event Time in docs. They’re focus on “keyed” events and monitoring using web/metric, where I want

Akka tcp address on AWS EMR between job/task managers

Hi guys, I want to use some client lib that integrated with akka system Flink aslo already created akka system job manager and those task manager can use akka by its akka.tcp addess My architecture stucture currently is: EMR (flink 1.8) 1 core node 1 master node Flink job on Yarn I have on task

CoFlatMapFunction vs BroadcastProcessFunction

Hi guys, I read about http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Dynamic-Rule-Evaluation-in-Flink-td21125.html#a21241 http://apache-flink-user-maili

Weird behavior with CoFlatMapFunction

Hi guys, I want to merge 2 diffrent stream, one is config stream and the other is the value json, to check again that config. Its seem like the CoFlatMapFunction should be used. Here my sample: val filterStream: ConnectedStreams[ControlEvent, JsValue]=(specificControlStream).connect(eventS

Json manipulating in flink

Hi guys, I’m just curious what json lib you’re doing for data transformation between operations. I’m trying to pick one so I’m looking for experiences about personal choices, pro, cons of them. From what I read, it is spray, argonaut, circe, upickle. I’m trying upickle, its easy indeed but I’m

Re: Scalaj vs akka as http client for Asyncio Flink

> occur in production. > > Cheers, > Till > > On Fri, Apr 12, 2019 at 11:29 AM Andy Hoang <mailto:a...@parcelperform.com>> wrote: > Hi Till, > Unfortunately I have to wait for the cluster to upgrade to 1.8 to use that > feature: https://issues.apache.org/jir

Organize env using files

I have 3 different files for env: test, staging and production. Each of those has different parameters like: kafka host, endpoint urls, redis connection host… I read about `https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#register-the-parameters-globally`

Re: Scalaj vs akka as http client for Asyncio Flink

; User code is first instantiated on the client where you create the job > topology (basically where you call new RichAsyncHttpClient). The code is then > serialized and shipped to the cluster where it is actually executed. > > Cheers, > Till > > On Fri, Apr 12, 2019 at 5

Re: Scalaj vs akka as http client for Asyncio Flink

er` which you initialize in the > `RichAsyncFunction#open` method. That way you only create the > `ActorMaterialier` on the `TaskManager` where the operator is executed and > solve the problem of serializability and you make it much more efficient > because you don't create a

Scalaj vs akka as http client for Asyncio Flink

Hi guys, I’m try to decide which http client to go with Flink, currently I tested with scalaj and akka http client and both work ok with our current dev environment. For scalaj its is pretty straight forward since its is just calling an http request with its timeout. For akka http client its a

local dev using kafka consumer by docker got wrong cluster node id

So I want to run flink in my local. Kafka docker and its zookeeper has been work great for local dev of other projects, I want to try this kafka with new flink project in local. I have problem of first, the connect from my kafka consumer source is created but then it try to connect with a differ

Kafka consumer do not commit offset at checkpoint

Hi all, I posted a bug here but its seem is my configuration problem: https://issues.apache.org/jira/browse/FLINK-11335 so I resend this to mailing list My env: AWS EMR 5.20: hadoop, flink plugin flink: 1.62/1.70 run under yarn-cluster K