[ANNOUNCE] Apache Flink 1.17.2 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.17.2, which is the second bugfix release for the Apache Flink 1.17 series. Apache Flink® Is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink h

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congratulations! Unlike other data-lakes, Paimon might be the first one to act as a stream-first (not batch-first) data-lake. Best Yun Tang From: Xianxun Ye Sent: Tuesday, March 28, 2023 10:52 To: d...@flink.apache.org Cc: Yu Li ; user ; user-zh ; d

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Thanks Yuanfei for driving the frocksdb release! Best Yun Tang From: Yuan Mei Sent: Tuesday, January 31, 2023 15:09 To: Jing Ge Cc: Yanfei Lei ; d...@flink.apache.org ; user ; user...@flink.apache.org Subject: Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0

Re: Flink CEP Incremental Checkpoint Issue

hould care about the checkpoint size too much. Instead, we should care more about the output results. Best Yun Tang From: Martijn Visser Sent: Wednesday, October 19, 2022 22:03 To: Puneet Duggal Cc: user Subject: Re: Flink CEP Incremental Checkpoint Issue Hi,

Re: ExecutionMode in ExecutionConfig

va/org/apache/flink/streaming/examples/wordcount/WordCount.java#L98 Best Yun Tang From: zhanghao.c...@outlook.com Sent: Thursday, September 15, 2022 0:03 To: Hailu, Andreas ; user@flink.apache.org Subject: Re: ExecutionMode in ExecutionConfig It's added in

Re: [FEEDBACK] Metadata Platforms / Catalogs / Lineage integration

acks rich information on >FlinkJobListener just as Feng mentioned, which has been supported well by >Spark, to send data lineage to external systems. [1] https://feature-requests.datahubproject.io/p/flink-integration Best Yun Tang From: wangqinghuan <

Re: NegativeArraySizeException trying to take a savepoint

/iterator.cc#L239-L245 Best Yun Tang From: Martijn Visser Sent: Monday, June 13, 2022 21:47 To: Mike Barborak Cc: user@flink.apache.org Subject: Re: NegativeArraySizeException trying to take a savepoint Hi Mike, It would be worthwhile to check if this still occurs

Re: RocksDB's state size discrepancy with what's seen with state processor API

ververica/frocksdb/blob/8608d75d85f8e1b3b64b73a4fb6d19baec61ba5c/java/src/main/java/org/rocksdb/DBOptions.java#L520 Best Yun Tang From: Alexis Sarda-Espinosa Sent: Tuesday, May 3, 2022 8:47 To: Peter Brucia Cc: user@flink.apache.org Subject: RE: RocksDB

Re: [Discuss] Creating an Apache Flink slack workspace

set up our own slack workspace. Best Yun Tang From: Jingsong Li Sent: Thursday, May 12, 2022 10:49 To: Xintong Song Cc: dev ; user Subject: Re: [Discuss] Creating an Apache Flink slack workspace Hi all, Regarding using ASF slack. I share the problems I saw in

Re: Failed to restore from ck, because of KryoException

compatibility problem, we also suggest you to use customized serializers for your customized class for better performance. Best Yun Tang From: Liting Liu (litiliu) Sent: Friday, May 6, 2022 10:20 To: user@flink.apache.org Subject: Failed to restore from ck, because of

Re: RocksDB efficiency and keyby

ttps://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-memory-partitioned-index-filters Best Yun Tang From: Yaroslav Tkachenko Sent: Thursday, April 21, 2022 0:44 To: Trystan Cc: user Subject: Re: RocksDB effici

Re: RocksDB state not cleaned up

really useful for Flink as this could push data to the last level, which leads to increase the read amplification. [1] https://javadoc.io/doc/org.rocksdb/rocksdbjni/6.20.3/org/rocksdb/RocksDB.html Best Yun Tang From: Alexis Sarda-Espinosa Sent: Friday, April 8

Re: flink pipeline handles very small amount of messages in a second (only 500)

whether the sink back-pressured the source to impact the throughput of source. Last but not least, did your source already have 100% CPU usage, which means your source operator has already reached to its highest throughput. Best Yun Tang From: Sigalit Eliazov

Re: RocksDB metrics for effective memory consumption

usage [1]. Please note that all RocksDB instances within same slot would share the same block cache, they will report same usage. [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-metrics-block-cache-usage Best Yun

Re: Incremental checkpointing & RocksDB Serialization

-docs-release-1.13/docs/ops/monitoring/back_pressure/ [2] https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/custom_serialization/ Best, Yun Tang From: Vidya Sagar Mula Sent: Sunday, March 6, 2022 4:16 To: Yun Tang Cc

Re: Incremental checkpointing & RocksDB Serialization

o vs > Avro) >From our experience, kryo is not a good choice in most cases. Best Yun Tang From: Vidya Sagar Mula Sent: Friday, March 4, 2022 17:00 To: user Subject: Incremental checkpointing & RocksDB Serialization Hi, I have a cluster that c

Re: Pods are OOMKilled with RocksDB backend after a few checkpoints

/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-metrics-block-cache-usage Best Yun Tang From: Alexandre Montecucco Sent: Friday, February 25, 2022 20:14 To: user Subject: Pods are OOMKilled with RocksDB backend after a few checkpoints

Re: How to prevent check pointing of timers ?

Hi Alex, I think the better solution is to know what the problem you have ever met when restoring the timers? Flink does not support to remove state (including timer state) currently. Best Yun Tang From: Alex Drobinsky Sent: Monday, February 7, 2022 21:09 To

Re: Inaccurate checkpoint trigger time

[1] https://github.com/apache/flink/blob/90e850301e672fc0da293abc55eb446f7ec68ffa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L540 [2] https://issues.apache.org/jira/browse/FLINK-17073 Best Yun Tang From: Robert Me

Re: Reading performance - Kafka VS FileSystem

Hi Jasmin, >From my knowledge, it seems no big company would adopt pure file system source >as the main data source of Flink. We would in general choose a message queue, >e.g Kafka, as the data source. Best Yun Tang From: Jasmin Redžepović Sent:

Re: Failure Restart Strategy leads to error

Hi Siddhesh, The root cause is that the configuration of group.id is missing for the Flink program. The configuration of restart strategy has no relationship with this. I think you should pay your attention to kafka related configurations. Best Yun Tang From

Re: Is Scala the best language for Flink?

- ASF JIRA<https://issues.apache.org/jira/browse/FLINK-14105> As the consensus among our community(please link dedicated thread if there is) we keep in mind that flink-runtime will be eventually scala-free. It is because of ... issues.apache.org  Best Yun Tang ___

Re: Question about MapState size

total-sst-files-size[1] to know the total sst files on disks of each rocksDB. [1] https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/config/#state-backend-rocksdb-metrics-total-sst-files-size Best Yun Tang From: Abdul Rahman Sent: Satur

Re: Apache Flink - Can AllWindowedStream be parallel ?

Hi Singh, All the output operator transformed by AllWindowedStream would be SingleOutputStreamOperator, which cannot be parallel. [1] https://github.com/apache/flink/blob/master/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/AllWindowedStream.scala Best Yun Tang

Re: [DISCUSS] Deprecate MapR FS

+1 for dropping the MapR Fs. Best Yun Tang From: Till Rohrmann Sent: Wednesday, January 5, 2022 18:33 To: Martijn Visser Cc: David Morávek ; dev ; Seth Wiesman ; User Subject: Re: [DISCUSS] Deprecate MapR FS +1 for dropping the MapR FS. Cheers, Till On Wed

Re: Operator state in New Source API

/#using-operator-state Best, Yun Tang From: Krzysztof Chmielewski Sent: Thursday, December 23, 2021 6:32 To: user Subject: Operator state in New Source API Hi, Is it possible to use managed operator state like MapState in an implementation of new unified source

Re: [ANNOUNCE] Open source of remote shuffle project for Flink batch data processing

Great news! Thanks for all the guys who contributed in this project. Best Yun Tang On 2021/11/30 16:30:52 Till Rohrmann wrote: > Great news, Yingjie. Thanks a lot for sharing this information with the > community and kudos to all the contributors of the external shuffle service > :-) &g

Re: Will Flink loss some old Keyed State when changing the parallelism

Hi Yang, Flink keeps the max key groups the same no matter how parallelism changes, and use this to avoid state data lost [1] [1] https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html Best Yun Tang On 2021/11/26 10:07:29 Nicolaus Weidner wrote: > Hi, > > to res

Re: savepoint.readKeyedState hangs on org.rocksdb.RocksDB.disposeInternal

each time. [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-log-dir [2] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-log-level Best Yun Tang

Re: [ANNOUNCE] Apache Flink 1.13.3 released

Thanks for Chesnay & Martijn and everyone who made this release happen. Best Yun Tang From: JING ZHANG Sent: Friday, October 22, 2021 10:17 To: dev Cc: Martijn Visser ; Jingsong Li ; Chesnay Schepler ; user Subject: Re: [ANNOUNCE] Apache Flink 1.13.3 rele

Re: Reset of transient variables in state to default values.

, it will pick the registered kryo serializer only for checkpoint/restore. Since java-based state backend would not deep copy key-values for performance reasons, it might be changed unexpectedly if user misused, which might make the field reset to default value. Best, Yun Tang

Re: Reset of transient variables in state to default values.

, check whether could read the transient field back if using FileSystemStateBackend. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/serialization/types_serialization/#flinks-typeinformation-class Best Yun Tang From: Alex Drobinsky Sent:

Re: Checkpoint size increasing even i enable increasemental checkpoint

ksDB's own mechanism to remove useless files, which is triggered by internal compaction. You should not care too much on the checkpointed data size as your job consuming more and more records, moreover the increasing size is actually quite small (from 1.32GB to 1.34GB).

Re: Cleaning old incremental checkpoint files

/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L99 [2] https://issues.apache.org/jira/browse/FLINK-24149 Best Yun Tang From: Robin Cassan Sent: Tuesday, September 7, 2021 20:17 To: Yun Tang Cc: Robert Metzger ; user Subject: Re: Cleaning old

Re: State processor API very slow reading a keyed state with RocksDB

e new read option which disable fillCache [2] to speedup bulk scan in the future to help improve the performance. Best Yun Tang [1] https://github.com/jvm-profiling-tools/async-profiler [2] https://javadoc.io/doc/org.rocksdb/rocksdbjni/6.20.3/org/rocksdb/ReadOptions.html#setFillCache(bo

Re: Cleaning old incremental checkpoint files

sible solution to provide the ability to re-upload all files under some specific configured option so that we could let new job get decoupled with older checkpoints. Do you think that could resolve your case? Best Yun Tang From: Robin Cassan Sent: Wednesday, Sept

Re: JobManager Resident memory Always Increasing

. [1] https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/ [2] https://gist.github.com/thomasdarimont/79b3cef01e5786210309 Best Yun Tang From: Pranjul Ahuja Sent: Monday, August 16, 2021 13:10 To: user@flink.apache.org Subject: JobManager Resident

Re: Inspecting SST state of rocksdb

state names. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/state_processor_api/#state-processor-api [2] https://github.com/apache/flink/blob/0d2b945729df8f0a0149d02ca24633ae52a1ef21/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L99 Bes

[ANNOUNCE] Apache Flink 1.13.2 released

available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12350218&styleName=&projectId=12315522 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Yun Tang

Re: Best Practice of Using HashSet State

. Best Yun Tang From: Jerome Li Sent: Friday, August 6, 2021 7:57 To: user@flink.apache.org Subject: Best Practice of Using HashSet State Hi, I am new to Flink and state backend. I find Flink does provide ValueState, ListState, and MapState. But it does not

Re: Bloom Filter - RocksDB - LinkageError Classloading

Hi Sandeep, If you set the flink-statebackend-rocksdb as provided scope, it should not include the org.rocksdb classes, have you ever checked your application jar package directly just as what I described? Best Yun Tang From: Sandeep khanzode Sent: Friday

Re: Bloom Filter - RocksDB - LinkageError Classloading

has different flink version with servers. Best, Yun Tang From: Stephan Ewen Sent: Wednesday, August 4, 2021 19:10 To: Yun Tang Cc: Sandeep khanzode ; user Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading @Yun Tang Does it make sense to add RocksDB

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

] https://github.com/facebook/rocksdb/tree/master/java/jmh Best, Yun Tang From: Piotr Nowojski Sent: Thursday, August 5, 2021 2:01 To: Yuval Itzchakov Cc: Yun Tang ; Nico Kruber ; user@flink.apache.org ; dev Subject: Re: [ANNOUNCE] RocksDB Version Upgrade and

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

b.com/ververica/frocksdb/pull/19 [13] https://github.com/facebook/rocksdb/pull/5441/ [14] https://github.com/facebook/rocksdb/pull/2283 Best, Yun Tang On Wed, Aug 4, 2021 at 2:36 PM Yuval Itzchakov mailto:yuva...@gmail.com>> wrote: We are heavy users of RocksDB and have had several issues wi

Re: Bloom Filter - RocksDB - LinkageError Classloading

citly added the dependency of org.rocksdb:rocksdbjni in your pom). Best Yun Tang From: Sandeep khanzode Sent: Wednesday, August 4, 2021 11:54 To: user Subject: Bloom Filter - RocksDB - LinkageError Classloading Hello, I tried to add the bloom filter functionali

Re: Flink k8 HA mode + checkpoint management

point/CheckpointCoordinator.java#L1226 Best Yun Tang From: Manong Karl Sent: Wednesday, August 4, 2021 9:17 To: Harsh Shah Cc: user@flink.apache.org Subject: Re: Flink k8 HA mode + checkpoint management Can You please share your configs? I'm using native kubernetes with

Re: as-variable configuration for state ac

Hi Mason, I think this request is reasonable and you could create a JIRA ticket so that we could resolve it later. Best, Yun Tang From: Mason Chen Sent: Tuesday, July 27, 2021 15:15 To: Yun Tang Cc: Mason Chen ; user@flink.apache.org Subject: Re: as

Re: as-variable configuration for state ac

Hi Mason, In rocksDB, one state is corresponding to a column family and we could aggregate all RocksDB native metrics per column family. If my understanding is right, are you hoping that all state latency metrics for a particular state could be aggregated per state level? Best Yun Tang

Re: Exception in snapshotState suppresses subsequent checkpoints

https://github.com/apache/flink/commit/b332ce40d88be84d9cf896f446c7c6e26dbc8b6a#diff-54e9ce3b15d6badcc9376ab144df066eb46c4e516d6ee31ef8eb38e2d4359042 Best Yun Tang From: Matthias Pohl Sent: Thursday, July 1, 2021 16:41 To: tao xiao Cc: Yun Tang ; user ; Roman Khachatryan Su

Re: Exception in snapshotState suppresses subsequent checkpoints

ther your tasks containing 'FromElementsFunctionT' has ever been completed. Best Yun Tang From: tao xiao Sent: Saturday, June 26, 2021 16:40 To: user Subject: Re: Exception in snapshotState suppresses subsequent checkpoints Btw here is the checkpoint relat

Re: Metric for JVM Overhaed

JVM. Some tools provided by memory allocator such jemalloc or tcmalloc, could help find how much the memory usage via OS malloc. Even though, there still exists some memory used by mmap or on local stack, which is not so easy to detect. Best Yun Tang From: Guowei

Re: High Flink checkpoint Size

-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L158-L159 Best, Yun Tang From: Vijayendra Yadav Sent: Wednesday, June 23, 2021 11:02 To: user Subject: High Flink checkpoint Size Hi Team, I

Re: PoJo to Avro Serialization throw KryoException: java.lang.UnsupportedOperationException

/flink/api/common/typeinfo/TypeInformation.java#L208 Best Yun Tang From: Rommel Holmes Sent: Wednesday, June 23, 2021 13:43 To: user Subject: PoJo to Avro Serialization throw KryoException: java.lang.UnsupportedOperationException My Unit test was running OK under

Re: How to set state.backend.rocksdb.latency-track-enabled

Hi Chen-Che, The PR-16177 [1] is the documentation for state access latency tracking, thought it has not been merged, you could still refer it for more details. [1] https://github.com/apache/flink/pull/16177 Best Yun Tang From: Chen-Che Huang Sent: Friday

Re: RocksDB CPU resource usage

thread stack. [1] https://github.com/jvm-profiling-tools/async-profiler Best Yun Tang From: Robert Metzger Sent: Thursday, June 17, 2021 14:11 To: Padarn Wilson Cc: JING ZHANG ; user Subject: Re: RocksDB CPU resource usage If you are able to execute your job loc

Re: Discard checkpoint files through a single recursive call

Hi Jiang, Please take a look at FLINK-17860 and FLINK-13856 for previous discussion of this problem. [1] https://issues.apache.org/jira/browse/FLINK-17860 [2] https://issues.apache.org/jira/browse/FLINK-13856 Best Yun Tang From: Guowei Ma Sent: Wednesday

Re: Question about State TTL and Interval Join

much state size will be. Moreover, the OOM should not be related to RocksDB as it used off-heap native memory, and you might need some work to dig what occupied the JVM memory during checkpoints. Best Yun Tang From: McBride, Chris Sent: Saturday, June 5, 2021 3:

Re: Dynamic configuration of Flink checkpoint interval

-alignment-timeout Best Yun Tang From: Senhong Liu Sent: Monday, May 31, 2021 10:33 To: JING ZHANG Cc: Kai Fu ; user Subject: Re: Dynamic configuration of Flink checkpoint interval Hi all, In fact, a pretty similar JIRA has been created, which is https

Re: Error restarting job from Savepoint

customized serializer for those classes. Another better solution is to ensure the class backwards compatibility with customized serializer or leverage apache avro. You could refer to [1] for more details. [1] https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html Best Yun

Re: [ANNOUNCE] Apache Flink 1.13.0 released

Thanks for Dawid and Guowei's great work, and thanks for everyone involved for this release. Best Yun Tang From: Xintong Song Sent: Thursday, May 6, 2021 12:08 To: user ; dev Subject: Re: [ANNOUNCE] Apache Flink 1.13.0 released Thanks Dawid & Guow

Re: savepoint command in code

/c688bf3c83e72155ccf5d04fe397b7c0a1274fd1/flink-tests/src/test/java/org/apache/flink/test/checkpointing/SavepointITCase.java#L438 Best Yun Tang From: Abdullah bin Omar Sent: Tuesday, May 4, 2021 11:50 To: user@flink.apache.org Subject: savepoint command in code Hello, I am trying

Re: Checkpoint error - "The job has failed"

Hi Dan, You could refer to the "Fix Versions" in FLINK-16753 [1] and know that this bug is resolved after 1.11.3 not 1.11.1. [1] https://issues.apache.org/jira/browse/FLINK-16753 Best Yun Tang From: Dan Hill Sent: Tuesday, April 27, 2021 7:50 To: Yu

Re: Checkpoint error - "The job has failed"

Hi Dan, I think you might use older version of Flink and this problem has been resolved by FLINK-16753 [1] after Flink-1.10.3. [1] https://issues.apache.org/jira/browse/FLINK-16753 Best Yun Tang From: Robert Metzger Sent: Monday, April 26, 2021 14:46 To: Dan

Re: Measuring the Size of State, Savepoint Size vs. Restore time

FLINK-19013 [2] https://issues.apache.org/jira/browse/FLINK-17012 Best Yun Tang From: Guowei Ma Sent: Thursday, April 1, 2021 11:57 To: Kevin Lam Cc: user ; Yun Tang Subject: Re: Measuring the Size of State, Savepoint Size vs. Restore time Hi, Kevin If you us

Re: State size increasing exponentially in Flink v1.9

Hi, If using RocksDB state backend, why it would occur `CopyOnWriteStateMap`? CopyOnWriteStateMap should only exist in heap based state-backend. Best Yun Tang From: Chesnay Schepler Sent: Friday, March 26, 2021 18:45 To: Almeida, Julius ; user@flink.apache.org

Re: EOFException on attempt to scale up job with RocksDB state backend

Hi Alexey, Flink would only write once for checkpointed files. Could you try to write checkpointed files as block blob format and see whether the problem still existed? Best Yun Tang From: Alexey Trenikhun Sent: Thursday, March 18, 2021 13:54 To: Yun Tang

Re: EOFException on attempt to scale up job with RocksDB state backend

ee whether this behavior still existed? [1] https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Block_Blob_with_Compaction_Support_and_Configuration Best Yun Tang From: Alexey Trenikhun Sent: Thursday, March 18, 2021 12:00 To: Yun Tang ; Tzu-Li (G

Re: EOFException on attempt to scale up job with RocksDB state backend

Yun Tang From: Alexey Trenikhun Sent: Thursday, March 18, 2021 0:45 To: Yun Tang ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Yun, I've copied 77e77928-cb26-4543

Re: Saved State in FSstate Backend

ld be sent to JM directly). [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#directory-structure [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#state-backend-fs-memory-threshold Bes

Re: EOFException on attempt to scale up job with RocksDB state backend

eckpointing-snapshot-compressions/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression Best Yun Tang From: Alexey Trenikhun Sent: Wednesday, March 17, 2021 14:25 To: Yun Tang ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException

Re: EOFException on attempt to scale up job with RocksDB state backend

Hi Alexey, Thanks for your reply, could you also share logs during normal restoring just as I wrote in previous thread so that I could compare. Best Yun Tang From: Alexey Trenikhun Sent: Wednesday, March 17, 2021 13:55 To: Yun Tang ; Tzu-Li (Gordon) Tai ; user

Re: EOFException on attempt to scale up job with RocksDB state backend

From: Alexey Trenikhun Sent: Tuesday, March 16, 2021 15:10 To: Yun Tang ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Also restore from same savepoint without change in parallelism works fine

Re: Prefix Seek RocksDB

Hi Rex, Prefix seek iterator has not ever been used in Flink when seeking. I hope you could first read more details about this from RocksDB wiki as prefix extractor could impact the performance. Best Yun Tang From: Rex Fenley Sent: Wednesday, March 17, 2021 2

Re: Prefix Seek RocksDB

Hi Rex, You could configure prefix seek via RocksDB's column family options [1]. Be careful to use correct prefix extractor. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#passing-options-factory-to-rocksdb Best Fro

Re: Question about session_aggregate.merging-window-set.rocksdb_estimate-num-keys

[1] https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/version_set.cc#L919-L924 Best Yun Tang From: Vishal Santoshi Sent: Monday, March 15, 2021 5:48 To: user Subject: Re: Question about session_aggregate.merging-window-set.r

Re: EOFException on attempt to scale up job with RocksDB state backend

Hi, Can you scale the job at the same version from 1.12.1 to 1.12.1? Best Yun Tang From: Alexey Trenikhun Sent: Tuesday, March 16, 2021 4:46 To: Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB

Re: failure checkpoint counts

Hi Abdullah, The "Connection refused" exception should have no direct relationship with checkpoint, I think you could check whether the socket source has worked well in your job. Best Yun Tang From: Abdullah bin Omar Sent: Tuesday, March 9, 2021 0:1

Re: Is Ververica Connector Redis open source?

Hi Yik, As far as I know, the source code of ververica connector is not public, and you could refer to [1] for open-source implementation. [1] https://github.com/apache/bahir-flink/tree/master/flink-connector-redis Best Yun Tang From: Yik San Chan Sent

Re: Performance issues when RocksDB block cache is full

noticed anything different on RocksDB metrics? Perhaps you might meet problem of flushing write buffer too early [1] and partitioned index [2] might help. [1] https://issues.apache.org/jira/browse/FLINK-19238 [2] https://issues.apache.org/jira/browse/FLINK-20496 Best Yun Tang

Re: question on ValueState

/StateTable.java Best Yun Tang From: Colletta, Edward Sent: Sunday, February 7, 2021 19:53 To: user@flink.apache.org Subject: question on ValueState Using FsStateBackend. I was under the impression that ValueState.value will serialize an object which is stored

Re: Memory usage increases on every job restart resulting in eventual OOMKill

]. The problem both went away once they adopt to use JeMalloc. [1] https://issues.apache.org/jira/browse/FLINK-19125 [2] https://issues.apache.org/jira/browse/FLINK-18712 Best Yun Tang From: Lasse Nedergaard Sent: Wednesday, February 3, 2021 14:07 To: Xintong Song

Re: rocksdb block cache usage

Hi, If you have enabled managed memory, and since all rocksDB instances share the same block cache within one slot, all flink_taskmanager_job_task_operator_window_contents_rocksdb_block_cache_pinned_usage in the same slot would report the same value. Best Yun Tang

Re: Problem with overridden hashCode/equals in keys in Flink 1.11.3 when checkpointing with RocksDB

/custom_serialization.html Best Yun Tang From: Robert Metzger Sent: Thursday, January 21, 2021 22:49 To: David Haglund Cc: user@flink.apache.org Subject: Re: Problem with overridden hashCode/equals in keys in Flink 1.11.3 when checkpointing with RocksDB Hey David, this is a good

Re: Setting different timeouts for savepoints and checkpoints

Hi Timo and Rex, Actually, there existed several existing issues: FLINK-9465 [1] targets for CLI option while FLINK-10360 [2] targets for REST API. [1] https://issues.apache.org/jira/browse/FLINK-9465 [2] https://issues.apache.org/jira/browse/FLINK-10360 Best Yun Tang

Re: Restoring from checkpoint with different parallism

Hi Rex, I think doc [1] should have given some descriptions. Rescaling from previous checkpoint is still supported in current Flink version. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint Best Yun Tang

Re: Queryable state on task managers that are not running the job

r, no queryable state could be queried. [1] https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/yarn.html#per-job-cluster-mode Best Yun Tang From: Martin Boyanov Sent: Monday, December 21, 2020 19:04 To: user@flink.apache.org Su

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Thanks Dian and Robert for driving this release and thanks everyone who makes this great work possible ! Best Yun Tang From: Wei Zhong Sent: Thursday, December 10, 2020 20:32 To: d...@flink.apache.org Cc: user ; annou...@apache.org Subject: Re: [ANNOUNCE

Re: Problem when restoring from savepoint with missing state & POJO modification

-processing-api/src/main/java/org/apache/flink/state/api/WritableSavepoint.java#L103 [2] https://github.com/apache/flink/blob/168124f99c75e873adc81437c700f85f703e2248/flink-libraries/flink-state-processing-api/src/main/java/org/apache/flink/state/api/output/StatePathExtractor.java#L54 Best Yun Tang

Re: Flink cli Stop command exception

Hi Suchithra, Have you ever checked job manager log to see whether the savepoint is triggered and why the savepoint failed to complete. Best Yun Tang From: V N, Suchithra (Nokia - IN/Bangalore) Sent: Wednesday, December 9, 2020 23:45 To: user@flink.apache.org

Re: Recommendation about RocksDB Metrics ?

trics.total-sst-files-size BTW, state.backend.rocksdb.metrics.column-family-as-variable is not rocksDB internal metrics but to expose column family as variable so that we could classify different state status. Best Yun Tang From: Steven Wu Sent: Wednesday,

Re: Problem when restoring from savepoint with missing state & POJO modification

Hi Bastien, Flink supports to register state via state descriptor when calling runtimeContext.getState(). However, once the state is registered, it cannot be removed anymore. And when you restore from savepoint, the previous state is registered again [1]. Flink does not to drop state directly a

Re: Flink 1.9Version State TTL parameter configuration it does not work

what's RocksDB doing. [1] https://github.com/jvm-profiling-tools/async-profiler Best Yun Tang From: Andrey Zagrebin Sent: Friday, December 4, 2020 17:49 To: user Subject: Re: Flink 1.9Version State TTL parameter configuration it does not work Hi

Re: Running Flink job as a rest

k-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java Best Yun Tang From: dhurandar S Sent: Thursday, December 3, 2020 5:31 To: Flink Dev ; user Subject: Running Flink job as a rest Can Flink job

Re: Questions regarding DDL and savepoints

/flink/flink-docs-release-1.11/ops/cli.html#restore-a-savepoint Best Yun Tang From: Kevin Kwon Sent: Thursday, December 3, 2020 8:31 To: user@flink.apache.org Subject: Questions regarding DDL and savepoints I have a question regarding DDLs if they are considered

Re: State Processor API SQL State

0faa8454db36f0/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/stream/StreamingJoinOperator.java#L83 Best Yun Tang From: Dominik Wosiński Sent: Tuesday, December 1, 2020 21:05 To: dev Subject: State Processor AP

Re: fromCollection() and savepoints

/c354f7bd679b9fa8c1e0d75feb3827ccca7f317b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FromElementsFunction.java#L123 Best Yun Tang From: Tomasz Dudziak Sent: Wednesday, November 25, 2020 18:22 To: user@flink.apache.org Subject: fromCollection

Re: Job Manager is taking very long time to finalize the Checkpointing.

/flink-docs-master/monitoring/checkpoint_monitoring.html#monitoring Best Yun Tang From: Slim Bouguerra Sent: Thursday, November 19, 2020 7:56 To: Yun Tang Cc: user@flink.apache.org Subject: Re: Job Manager is taking very long time to finalize the Checkpointing. Hi Y

Re: What happens when a job is rescaled

/RocksDBIncrementalRestoreOperation.java#L274 [2] https://issues.apache.org/jira/browse/FLINK-17288 Best Yun Tang From: Richard Deurwaarder Sent: Saturday, November 14, 2020 0:14 To: user Subject: What happens when a job is rescaled Hello, I have a question about what

Re: Job Manager is taking very long time to finalize the Checkpointing.

rease. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#state-backend-fs-memory-threshold Best Yun Tang From: Slim Bouguerra Sent: Wednesday, November 18, 2020 6:16 To: user@flink.apache.org Subject: Job Manager is taking very long time to fi

Re: Flink checkpointing state

Hi Added Yang Wang who mainly develops this feature, I think he could provide more information. Best Yun Tang From: Boris Lublinsky Sent: Tuesday, October 27, 2020 22:57 To: Yun Tang Cc: user Subject: Re: Flink checkpointing state Thanks Yun, This refers to

  1   2   3   4   >