Hi flink community,
Is there any test framework that we can use to test flink jobs performance?
We would like to automate process for regression tests during flink version
upgrade and job performance tests when rolling out new changes to prod.
Any suggestions would be appreciated!
Thank you
Bes
One thing I noticed is that if I set drain = true, the job could be stopped
correctly. Maybe that's because I'm using a Parquet file sink which is a
bulk-encoded format and only writes to disk during checkpoints?
Thomas
On Sat, Jun 5, 2021 at 10:06 AM Thomas Wang wrote:
> Hi Yun,
>
> Thanks for
Small correction, in T4 and T5 I mean Job2, not Job 1 (as job 1 was save
pointed).
Thank you,
Alex
> On Jun 4, 2021, at 3:07 PM, Alexander Filipchik wrote:
>
>
> Looked through the logs and didn't see anything fishy that indicated an
> exception during checkpointing.
> To make it clearer,
Hi Yun,
Thanks for the tips. Yes, I do see some exceptions as copied below. I'm not
quite sure what they mean though. Any hints?
Thanks.
Thomas
```
2021-06-05 10:02:51
java.util.concurrent.ExecutionException:
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
Could no
Ohthat won't work for me either. I needed to use MapState.
Perhaps I should describe my problem. I am using a KeyedState process
function, but the workload that it is processing is not distributing well
across the cluster. I have four task managers, but the way my data is keyed
in this opera
Hi,
I tried google and found one possible related issue in [1]. Could you try to
remove the config
properties.setProperty(ProducerConfig.RETRIES_CONFIG, "5");
and try again ?
Best,
Yun
[1]
https://stackoverflow.com/questions/55192852/transactional-producer-vs-just-idempotent-producer-java
Hi Thomas,
For querying the savepoint status, a get request could be issued to
/jobs/:jobid/savepoints/:savepointtriggerid [1] to get the status and position
of the savepoint. But if the job is running with some kind of per-job mode and
JobMaster is gone after the stop-with-savepoint, the reque
Hi Marco,
I think yes, the operator state could be used in batch mode. Since there
is no checkpoint in batch mode, the operator state would serve as a kind
of ordinary in-memory storage.
Best,
Yun
--
Sender:Marco Villalobos
Date:20
Hi Chirag,
If be able to produce the exception, could you first add some logs to print
the value of valueState, valueState.value(), inEvent and
inEvent.getPriceDelta() ?
I think either object being null would cause NullPointerException here.
For the second exception, I found a similar issue[1],
Hi,
I am getting multiple exceptions while trying to use RocksDB as astate backend.
I have 2 Task Managers with 2 taskslots and 4 cores each.
Below is our setup:
Kafka(Topic with 2 partitions) ---> FlinkKafkaConsumer(2Parallelism) >
KeyedProcessFunction(4 Parallelism) > FlinkKafka
Does that work in the DataStream API in Batch Execution Mode?
On Sat, Jun 5, 2021 at 12:04 AM JING ZHANG wrote:
> Hi,
> please use `CheckpointedFunction`, you could initialize your operator
> state in `initializeState` method by using
> context.getOperatorStateStore().***
>
> Best regards,
> JIN
Hi,
After some investigation, I found a simpler way to satisfy the demand, use
cdc connector[1] which supports reading database snapshots and continues to
read binlogs in the database without deploying kafka and debezium.
Best regards,
JING ZHANG
[1] https://github.com/ververica/flink-cdc-connect
Hi,
please use `CheckpointedFunction`, you could initialize your operator state
in `initializeState` method by using context.getOperatorStateStore().***
Best regards,
JING ZHANG
Marco Villalobos 于2021年6月5日周六 下午1:55写道：
> Is it possible to use OperatorState, when NOT implementing a source or
> s
13 matches
Mail list logo
