Documentation of mesos-appmaster-job.sh

Hello All, I cannot find any documentation or help about how to use $flin_home/bin/mesos-appmaster-job.sh. Anybody help? Thanks! Jacky Yin

Facebook: Save Wilpattu One Srilanka's Most Loved Place

Hello there, I just signed the petition "Facebook: Save Wilpattu One Srilanka's Most Loved Place" and wanted to see if you could help by adding your name. Our goal is to reach 7,500 signatures and we need more support. You can read more and sign the petition here: http://chng.it/ZxfxKQcqqk Than

Re: [DISCUSS] Introduction of a Table API Java Expression DSL

Hi Timo, I'm +1 on the proposal. I like the idea to provide a Java DSL which is more friendly than string-based approach in programming. My concern is if/when we can drop the string-based expression parser. If it takes a very long time, we have to paid more development cost on the three Table API

Re: Async Function Not Generating Backpressure

Hey Andrey and Ken, Sorry about the late reply. I might not have been clear in my question The performance of writing to Cassandra is the same in both cases, only that the source rate was higher in the case of the async function is present. Something is "buffering" and not propagating backpressure

Re: [PROGRESS UPDATE] [DISCUSS] Flink-Hive Integration and Catalogs

Thanks, Shaoxuan! I've sent a Chinese version to user-zh at the same time yesterday. >From feedbacks we received so far, supporting multiple older hive versions is definitely one of our focuses next. *More feedbacks are welcome from our community!* On Tue, Mar 19, 2019 at 8:44 PM Shaoxuan Wang

Re: [VOTE] Release 1.8.0, release candidate #3

-1, observed stably failure on streaming bucketing end-to-end test case in two different environments (Linux/MacOS) when running with both shaded hadoop-2.8.3 jar file

Re: [VOTE] Release 1.8.0, release candidate #3

Thanks for the quick fix Aljoscha! The FLINK-11971 has been merged. Cheers, Jincheng Piotr Nowojski 于2019年3月21日周四 上午12:29写道： > -1 from my side due to performance regression found in the master branch > since Jan 29th. > > In 10% JVM forks it w

Re: [VOTE] Release 1.8.0, release candidate #3

-1 from my side due to performance regression found in the master branch since Jan 29th. In 10% JVM forks it was causing huge performance drop in some of the benchmarks (up to 30-50% reduced throughput), which could mean that one out of 10 task managers could be affected by it. Today we have m

state and Id of job in flatMap

Hello, I created a process for an anomaly detection with a flatMap. I need to know the end of each job at the level of the flatMap to be able to flush a buffer in the output collector. I saw that it is possible to get status of job by using ExecutionEnvironment, but i don't know how i can impleme

Re: Async Function Not Generating Backpressure

Hi Seed, Sorry for confusion, I see now it is separate. Back pressure should still be created because internal async queue has capacity but not sure about reporting problem, Ken and Till probably have better idea. As for consumption speed up, async operator creates another thread to collect the r

[no subject]

user@flink.apache.org -- *Cheers * *Puneet Kinra* *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com * *e-mail :puneet.ki...@customercentria.com *

Streaming Protobuf into Parquet file not working with StreamingFileSink

Hi, I'm trying to stream events in Prorobuf format into a parquet file. I looked into both streaming-file options: BucketingSink & StreamingFileSink. I first tried using the newer *StreamingFileSink* with the *forBulkFormat *API. I noticed there's currently support only for the Avro format with th

Re: Flink CEP pattern design question

I think what you ask for is something like timing out greedy[1] quantifier, which is not supported. As a rather dirty workaround you could try sth like: Pattern .begin[Event]("start") .where(_._.getName == "a") .oneOrMore.opt

[REMINDER] Flink Forward San Francisco in a few days

Hi everyone, *Flink Forward San Francisco 2019 will take place in a few days on April 1st and 2nd.* If you haven't done so already and are planning to attend, you should register soon at: -> https://sf-2019.flink-forward.org/register Don't forget to use the 25% discount code *MailingList* for ma

Re: Status.JVM.Memory.Heap.Used metric shows only a few megabytes but profiling shows gigabytes (as expected)

Thanks, next time I'll check in in jira better. On Tue, Mar 19, 2019 at 6:27 PM Chesnay Schepler wrote: > Known issue, fixed in 1.7.3/1.8.0: > https://issues.apache.org/jira/browse/FLINK-11183 > > On 19.03.2019 15:03, gerardg wrote: > > Hi, > > > > Before Flink 1.7.0 we were getting correct valu

Flink CEP pattern design question

Currently I'm designing a CEP pattern to satisfy our business needs. Basically, there's two events let's call it a and b. Both a and b can have zero or multiple entries in the log. For input {a,b1,b2}, I want to get the output of {a,b1,b2} For input {b1,b2}, I want to get the output of {b1,b2} afte

Re: How to split tuple2 returned by UDAF into two columns in a result table

Hi Dongwon, Couldn't you just return a tuple from the aggregation function and extract the fields from the nested tuple using a value access function [1]? table table2 = table1 .window(Slide.over("3.rows").every("1.rows").on("time").as("w")) .groupBy("w, name") .select("name, my

Re: [VOTE] Release 1.8.0, release candidate #3

Thanks Jincheng! It would be very good to fix those but as you said, I would say they are not blockers. > On 20. Mar 2019, at 09:47, Kurt Young wrote: > > +1 (non-binding) > > Checked items: > - checked checksums and GPG files > - verified that the source archives do not contains any binaries

Re: Flink performance drops when async checkpoint is slow

Hi Paul! One issue could be that state in Flnk 1.5.x state is asynchronous, but timers are synchronous. Timers are asynchronous starting from Flink 1.6. Best, Stephan On Fri, Mar 1, 2019 at 4:03 AM zhijiang wrote: > Hi Paul, > > Thanks for your feedback. If the at-least-once mode still causes

Re: Schema Evolution on Dynamic Schema

Hi, I think this would work. However, you should be aware that all keys are kept forever in state (unless you configure idle state retention time [1]). This includes previous versions of keys. Also note that we are not guaranteeing savepoint compatibility across Flink versions yet. If the state o

Re: [VOTE] Release 1.8.0, release candidate #3

+1 (non-binding) Checked items: - checked checksums and GPG files - verified that the source archives do not contains any binaries - checked that all POM files point to the same version - build from source successfully Best, Kurt On Wed, Mar 20, 2019 at 2:12 PM jincheng sun wrote: > Hi Aljosc

Re: End-to-end exactly-once semantics in simple streaming app

Hi Andrey, thanks for your feedback. I am not sure if I understand 100% correctly, but using the flink state to store my stuff (in addition to the oracle database) is not an option, because to my knowledge flink state does not allow arbitrary lookup queries, which I need to do, however. Also, g

Re: Set partition number of Flink DataSet

Hi, I'm sorry but I'm only familiar with the high-level design but not with the implementation details and concrete roadmap for the involved components. I think that FLINK-10288 [1] and FLINK-10429 [2] are related to partition handling. Best, Fabian [1] https://issues.apache.org/jira/browse/FLIN

Re: flink on yarn log rolling

Hi Shengnan, If you have the ssh permission, you could take a look at the container working directories to check if the taskmanager local log4j.properties is as expected. Best, Paul Lam > 在 2019年3月20日，15:30，Shengnan YU 写道： > > Hi all: > I'd like to enable log rolling for flink on yarn. I tri

Re: flink on yarn log rolling

Could you share the content of log4j.properties file you modified? And which version of Flink are you using? Best Guowei Shengnan YU 于2019年3月20日周三 下午3:30写道： > Hi all: > I'd like to enable log rolling for flink on yarn. I tried to modify > log4j.properties and logback.xml in flink/conf however t

flink on yarn log rolling

Hi all: I'd like to enable log rolling for flink on yarn. I tried to modify log4j.properties and logback.xml in flink/conf however there still one "taskmanager.log" in yarn container log. Any idea with that? Thank you very much!