Re: Providing Custom Serializer for Generic Type

Hi Andrea: Why not make your MyClass POJO? [1] If it is a POJO, then flink will use PojoTypeInfo and PojoSerializer that have a good implementation already. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/types_serialization.html#rules-for-pojo-types Best, JingsongLee ---

Providing Custom Serializer for Generic Type

Dear community, in my job, I run with a custom event type *MyClass* which is a sort of "generic event" that I handle all along my streaming flow both as an event (DataStream[MyClass]) and as a managed state. I see that Flink warns me about generic serialization of *MyClass* INFO [run-main-0] (Typ

Re: Queryable state and TTL

Hi Yu, Our sink is actually Kafka hence we cannot query it properly, from there we distribute it to different consumers. We keep info in our state such as entry time, some accumulated data etc' , this data is not kept elsewhere hence we need to query our state. Best regards Avi On Thu, Jul 4, 20

Re: Queryable state and TTL

Thanks for the ping Andrey. For me the general answer is yes, but TBH it will probably not be added in the foreseeable future due to lack of committer bandwidth (not only QueryableState with TTL but all about QueryableState module) as per Aljoscha pointed out in another thread [1]. Although we co

[DISCUSS] Make window state queryable

Hi folks, Currently, the queryable state is not widely used in production. IMO, there are two key reasons caused this result. 1) the client of the queryable state is hard to use. Because it requires users to know the address of TaskManager and the port of the proxy. Actually, most business users w

Re:Load database table with many columns

Hi, Soheil Pourbafrani For the current implementation of JDBCInputFormat, it cannot automatically infer the column types. As far as I know, there also is no other way to do this. If you're going to implement such an input format, the inference work needs to be done by yourself. Because it re

Re:RE: Re:Re: File Naming Pattern from HadoopOutputFormat

Hi, Andreas I'm glad you have had a solution. If you're interested in option 2 I talked about, you can follow up on the progress of the issue (https://issues.apache.org/jira/browse/FLINK-12573) that Yitzchak said by watching it. Best, Haibo At 2019-07-03 21:11:44, "Hailu, Andreas" wrote:

Migrating existing application to Flink

Hi all, How would one go about migrating a stateful streaming application that doesn't use Flink to one that does? My question is specifically how to load state for the first time? I can set the source operator (Kafka in my case) to start from a desired point in time but I've no idea how I'd go a

Re: Source Kafka and Sink Hive managed tables via Flink Job

BTW, I'm adding user@ mailing list since this is a user question and should be asked there. dev@ mailing list is only for discussions of Flink development. Please see https://flink.apache.org/community.html#mailing-lists On Wed, Jul 3, 2019 at 12:34 PM Bowen Li wrote: > Hi Youssef, > > You nee

Load database table with many columns

Hi, I use the following sample code to load data from a database into Flink DataSet: DataSet dbData = env.createInput( JDBCInputFormat.buildJDBCInputFormat() .setDrivername("org.apache.derby.jdbc.EmbeddedDriver") .setDBUrl("jdbc:derby:memory:per

Watermarks and Kafka

Hello! We currently have a job which reads from Kafka and uses punctuated watermarks based on the messages we read. We currently keep track of the watermarks for each partition to emit a consensus watermark, taking the smallest of all partitions. We ran into an issue because we are not storing

Re: Queryable state and TTL

Hi Avi, It is on the road map but I am not aware about plans of any contributor to work on it for the next releases. I think the community will firstly work on the event time support for TTL. I will loop Yu in, maybe he has some plans to work on TTL for the queryable state. Best, Andrey On Wed,

Re: Could not load the native RocksDB library

Hi Samya, Additionally to Haibo's answer: Have you tried the previous 1.7 version of Flink? The Rocksdb version was upgraded in 1.8 version. Best, Andrey On Wed, Jul 3, 2019 at 5:21 AM Haibo Sun wrote: > Hi, Samya.Patro > > I guess this may be a setup problem. What OS and what version of JDK

Re: Flink Kafka ordered offset commit & unordered processing

Hi, > Will Flink able to recover under this scenario? I’m not sure exactly what you mean. Flink will be able to restore the state to the last successful checkpoint, and it well could be that the some records after this initial “stuck record” were processed and emitted down the stream. In th

Re: [ANNOUNCE] Apache Flink 1.8.1 released

I've also tweeted about it from @ApacheFlink https://twitter.com/ApacheFlink/status/1146407762106040321 Dawid Wysakowicz 于2019年7月3日周三 下午4:24写道： > Congrats to everyone involved and thank you Jincheng for being the release > manager! > On 03/07/2019 08:38, JingsongLee wrote: > > Thanks jincheng fo

Queryable state and TTL

Hi, Adding queryable state to state with ttl is not supported at 1.8.0 (throwing java.lang.IllegalArgumentException: Queryable state is currently not supported with TTL) I saw in previous mailing thread

RE: Re:Re: File Naming Pattern from HadoopOutputFormat

Hi Haibo, Yitzchak, thanks for getting back to me. The pattern I chose to use which worked was to extend the HadoopOutputFormat class, override the open() method, and modify the "mapreduce.output.basename" configuration property to match my desired file naming structure. // ah From: Haibo Sun

Re: Flink error handling

We ended up using side outputs for now and basically implementing our own map/flatMap that internally uses a ProcessFunction. Sent from my iPhone > On Jul 3, 2019, at 6:02 AM, Halfon, Roey wrote: > > Hi, > Do you have any progress with that? > > -Original Message- > From: Steven Nelso

Re: Flink Application JAR

>> At the moment, for every code change in any part of the project, we need to deploy and test the whole project code This is exactly what we do and want to do. I rather have my build/test CI tell me if I broke something than keep a mental model of where a change may have impact. It also makes

Re: Job recovers from an old dangling CheckPoint in case of Job Cluster based Flink pipeline

I guess using a session cluster rather then a job cluster will decouple the job from the container and may be the only option as of today? On Sat, Jun 29, 2019, 9:34 AM Vishal Santoshi wrote: > So there a re 2 scenerios > > 1. If JM goes down ( exits ) and k8s re launches the Job Cluster ( the J

RE: Flink Application JAR

Hi, Thanks for answering, Yes, Of course I would want to test it. But after testing, I want to be able to upload only the changed/related jars and not all the others that weren't changed at all.. At the moment, for every code change in any part of the project, we need to deploy and test the whole

Re: Flink Application JAR

I think any jars in $FLINK_HOME/lib gets picked up in the class path. As for dynamic update, wouldn't you want to run it through your tests/CI before deploying? FWIW we also use a fat jar. Its large but also since we just build a docker container anyway it doesn't really matter. Tim On Wed, J

RE: Flink error handling

Hi, Do you have any progress with that? -Original Message- From: Steven Nelson Sent: Tuesday, June 18, 2019 7:02 PM To: user@flink.apache.org Subject: Flink error handling Hello! We are internally having a debate on how best to handle exceptions within our operators. Some advocate fo

Flink Application JAR

Hi All, We have an application which contains few modules (one for the main application with the "driver" and few other for the operators implementation, configuration etc.). At the moment we are building fat jar contains all the modules and all the 3rd parties dependencies. As we continue deve

Re: [ANNOUNCE] Apache Flink 1.8.1 released

Congrats to everyone involved and thank you Jincheng for being the release manager! On 03/07/2019 08:38, JingsongLee wrote: > Thanks jincheng for your great job. > > Best, JingsongLee > > -- > From:Congxian Qiu > Send Tim