Re: Parallelism with onTimer() in connectedStream

Any idea, guys? Can timers work correctly with parallelism? May be it's my fault it works such way -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Parallelism with onTimer() in connectedStream

Some logs and comments: --- user key: Test_1 __Co-Keyed-Process -> Sink: Sink (2/3)#0 // get user and store in 2 thread (process1: user stream) --- user key: Test_2 __Co-Keyed-Process -> Sink: Sink (3/3)#0 // get user and store in 3 thread (process1: user stream) --- user key: Test_3 __Co-Keyed

Parallelism with onTimer() in connectedStream

Hello! I have a ConnectedStream consists of two DataStreams: - user (dataStream) - event (dataStream) It allows me to use storages of that DataStreams to manage information of user (user info depends on event of this user) and events: - users (store) - events (store) So: - user info is changed b

Re: Proper way to get DataStream

Arvid Heise-4, Ok, this is clear for me now. Good answer. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: DataStream from kafka topic

Thank you all very much! The problem is solved using ConfluentRegistryAvroDeserializationSchema.forGeneric(schema, "http://xxx.xx.xxx.xx:8081";) method. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Proper-way-to-get-DataStream-lt-GenericRecord-gt-td42640.html

Re: Proper way to get DataStream

Hi, @Arvid Heise-4, @Matthias I'm very appreciate for your attention, guys. And sorry for my late reply. Yes, Arvid, you are right, the second way in fact works. I coppied schema from Schema Registry using it's API and created the .avsc format file. And thanks again for explaining me why the firs

Proper way to get DataStream

Hi, I'm trying to solve a task with getting data from topic. This topic keeps avro format data. I wrote next code: public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); Schema schema

Re: DataStream from kafka topic

I tried this: 1. Schema (found in stackoverflow) class GenericRecordSchema implements KafkaDeserializationSchema { private String registryUrl; private transient KafkaAvroDeserializer deserializer; public GenericRecordSchema(String registryUrl) { this.registryUrl = registryUr

DataStream from kafka topic

Hi everyone. How can I get entry in GenericRecord format from kafka topic using SchemaRegistry? I read this: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html But can't to build it in my code... Is there some tutorials or examples to deserialise data using schema.

Re: Hadoop is not in the classpath/dependencies

I downloaded the lib (last version) from here: https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/ and put it in the flink_home/lib directory. It helped. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Hadoop is not in the classpath/dependencies

I have the same problem ... -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Fault Tolerance with RocksDBStateBackend

Ok, thank you, Guowei Ma -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Fault Tolerance with RocksDBStateBackend

Hi everyone, I want to build a flink cluster with 3 machines. What if I choose RocksDBStateBackend with next settings: #== # Fault tolerance and checkpointing #=

Re: Questions with State Processor Api

Please, someone help me to understand is State Processor Api a solve or not for a task. I want to add to state 'Events' some target actions of user and remove them if cancel action is received. Every X period I need to check this state if it's time to make some communication with user. If yes, so

Re: No saving data using rocksdb

Roman, thank you for your attention. It looks like you are absolutely right. Thank you very much for helping. Before submitting a job I do next steps: 1. ./bin/start-cluster.sh 2. ./bin/taskmanager.sh start And in my code there is these line: env.setStateBackend(new RocksDBStateBackend("file:///

Re: No saving data using rocksdb

Hey, Roman I use every time the same key. And I get the correct value in StateManager every time the processElement() method executes. But then I stop the job and submit it again. And first execution processElement() get me null in state store. The key wasn't change. So, I'am in confuse Thank

No saving data using rocksdb

I have following piece of configuration in flink.yaml: Key Value high-availability zookeeper high-availability.storageDir file:///home/flink/flink-ha-data high-avai

Questions with State Processor Api

Hi, folk Using State Processor Api can I: 1. get full state of flink-application with rocksdb backend in cluster mode (as I realised it's checkpoins or savepoints)? 2. update it? 3. get this state from other flink-application (other jar)? 4. query it with sql (Table API & SQL) to get data I need?

Questions with State Processor Api

Hi, folk Using State Processor Api can I: 1. get full state of flink-application with rocksdb backend in cluster mode (as I realised it's checkpoins or savepoints)? 2. update it? 3. get this state from other flink-application (other jar)? 4. query it with sql (Table API & SQL) to get data I need?

Re: KeyedProcessFunction

I missed in documentation: A KeyedProcessFunction is always a RichFunction. Therefore, access to the RuntimeContext is always available and setup and teardown methods can be implemented. See RichFunction.open(org.apache.flink.configuration.Configuration) and RichFunction.close(). https://ci.apach

KeyedProcessFunction

Hello, I'm learning State Processor API: https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html There is example in this page with StatefulFunctionWithTime extends KeyedProcessFunction. And here we can see method open() we need implement to initialize state. But

Standard method to generate watermark forBoundedOutOfOrderness

Hello, everyone. I'm learning Flink but still not sure if I realise the topic of watermark mechanism. That is a simple common example of pipeline with event-time mode: I want to use

Re: Flink CEP: can't process PatternStream

Hello, David. Yes, I’m using 1.12. And my code is now working. Thank you very much for your comment. Yuri L. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Cep application with Flink

Hello, *Jörn Franke*. Thank you for reply. If I correctly realise your answer, the watermark Flink mechanism should help me sort events in order I need. So I should dig deeper in that issue. For example, I read three topics, make joins and after get two events by the same user in this order: