Re: Redis as a State Backend

ce mechanism, beyond what can be implemented natively in the heap-state backend. It can do a full snapshot of a database (costly, we need incremental, ...) or a text-based append log (changelog state backend). All data needs to fit in memory. In text form. No compression (you can of course com

Re: Redis as a State Backend

When it comes to decoupling the state store from Flink, I suggest taking a look at FlinkNDB, which is an experimental state backend for Flink that puts the state into an external distributed database. There's a Flink Forward talk [1] and a master's thesis [2] available.

Re: Redis as a State Backend

Thanks Zakelly and Junrui. I was actually exploring RocksDB as a state backend and I thought maybe Redis could offer more features as a state backend. For e.g. maybe state sharing between operators, geo-red of state, partitioning etc. I understand these are not native use cases for Flink, but

Re: Redis as a State Backend

tter performance compared with Redis I > guess. > > > Best, > Zakelly > > On Tue, Jan 30, 2024 at 2:15 PM Chirag Dewan via user < > user@flink.apache.org> wrote: > >> Hi, >> >> I was looking at the FLIP-254: Redis Streams Connector and I was &g

Re: Redis as a State Backend

erformance compared with Redis I guess. Best, Zakelly On Tue, Jan 30, 2024 at 2:15 PM Chirag Dewan via user wrote: > Hi, > > I was looking at the FLIP-254: Redis Streams Connector and I was > wondering if Flink ever considered Redis as a state backend? And if yes, > why was

Re: Redis as a State Backend

Hi Chirag, Indeed, the possibility of using Redis as a state backend for Flink has been considered in the past. You can find a detailed discussion about this topic in the JIRA issue FLINK-3035[1] as well as in the comments section of this PR[2]. The outcome of these discussions was that Redis is

Redis as a State Backend

Hi, I was looking at the FLIP-254: Redis Streams Connector and I was wondering if Flink ever considered Redis as a state backend? And if yes, why was it discarded compared to RocksDB?  If someone can point me towards any deep dives on why RocksDB is a better fit as a state backend, it would be

Re: RocksDB State Backend GET returns null intermittently

zer, as this is one of the >> main differences between RocksDBStateBackend and HashMapStateBackend >> (HashMapStateBackend does not perform serialization and deserialization). >> >> On Wed, Jun 21, 2023 at 3:44 PM Prabhu Joseph >> wrote: >> >>> Hi, >>> >>&

Re: RocksDB State Backend GET returns null intermittently

and HashMapStateBackend > (HashMapStateBackend does not perform serialization and deserialization). > > On Wed, Jun 21, 2023 at 3:44 PM Prabhu Joseph > wrote: > >> Hi, >> >> RocksDB State Backend GET call on a key that was PUT into the state like >> 100 ms earli

Re: RocksDB State Backend GET returns null intermittently

one of the main differences between RocksDBStateBackend and HashMapStateBackend (HashMapStateBackend does not perform serialization and deserialization). On Wed, Jun 21, 2023 at 3:44 PM Prabhu Joseph wrote: > Hi, > > RocksDB State Backend GET call on a key that was PUT into the state like

RocksDB State Backend GET returns null intermittently

Hi, RocksDB State Backend GET call on a key that was PUT into the state like 100 ms earlier but is not returned intermittently. The issue never happened with the HashDB State backend. We are trying to increase block cache size, write buffer size, and enable bloom filter as per the doc: - https

Re: Could not restore keyed state backend for KeyedProcessOperator

.x. Last time was during a node upgrade, but that was not >>>>> the case this time. >>>>> >>>>> I've not been able to reproduce this issue. I've checked that I can >>>>> kill the taskmanager and jobmanager (using kubectl delete po

Re: Could not restore keyed state backend for KeyedProcessOperator

node upgrade, but that was not >>>> the case this time. >>>> >>>> I've not been able to reproduce this issue. I've checked that I can >>>> kill the taskmanager and jobmanager (using kubectl delete pod), and the job >>>> restores as expect

Re: Could not restore keyed state backend for KeyedProcessOperator

(using kubectl delete pod), and the job >>> restores as expected. >>> >>> The job is running with kubernetes high availability, rocksdb and >>> incremental checkpointing. >>> >>> Any tips are highly appreciated. >>> >>> Thanks, &g

Re: Could not restore keyed state backend for KeyedProcessOperator

using kubectl delete pod), and the job >> restores as expected. >> >> The job is running with kubernetes high availability, rocksdb and >> incremental checkpointing. >> >> Any tips are highly appreciated. >> >> Thanks, >> Lars >> >> Caused by

Re: Could not restore keyed state backend for KeyedProcessOperator

tal checkpointing. > > Any tips are highly appreciated. > > Thanks, > Lars > > Caused by: org.apache.flink.util.FlinkException: Could not restore keyed > state backend for > KeyedProcessOperator_bf374b55

Could not restore keyed state backend for KeyedProcessOperator

. Any tips are highly appreciated. Thanks, Lars Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_bf374b554824ef28e76619f4fa153430_(2/2) from any of the 1 provided rest

Re: Setting S3 as State Backend in SQL Client

;> Hi, >> >> If I remember correctly, set operations supports only a limited set of >> configurations. >> >> Most of them are table options that are listed on table configuration [1] >> plus some pipeline options. >> >> State backend options are not likely on

Re: Setting S3 as State Backend in SQL Client

e table options that are listed on table configuration [1] > plus some pipeline options. > > State backend options are not likely one of them. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/config/ > > Best, > Paul Lam > > 2022年3

Re: Setting S3 as State Backend in SQL Client

Hi, If I remember correctly, set operations supports only a limited set of configurations. Most of them are table options that are listed on table configuration [1] plus some pipeline options. State backend options are not likely one of them. [1] https://nightlies.apache.org/flink/flink

Re: Setting S3 as State Backend in SQL Client

Just tried editing flink-conf.yaml and it seems SQL Client does not respect that also. Is this an intended behavior? On Tue, Mar 15, 2022 at 7:14 PM dz902 wrote: > Hi, > > I'm using Flink 1.14 and was unable to set S3 as state backend. I tried > combination of: > > SET st

Setting S3 as State Backend in SQL Client

Hi, I'm using Flink 1.14 and was unable to set S3 as state backend. I tried combination of: SET state.backend='filesystem'; SET state.checkpoints.dir='s3://xxx/checkpoints/'; SET state.backend.fs.checkpointdir='s3://xxx/checkpoints/'; SET state.checkpoint-sto

RE: Re: Job requiring a lot of memory despite using rocksdb state backend

ase the > amount of memory you provided to RocksDB? > > On 05/02/2022 07:12, Salva Alcántara wrote: > > I have a job running on Flink 1.14.3 (Java 11) that uses rocksdb as > > the state backend. The problem is that the job requires an amount of > > memory pretty sim

Re: Job requiring a lot of memory despite using rocksdb state backend

What where the errors you received that caused you to increase the amount of memory you provided to RocksDB? On 05/02/2022 07:12, Salva Alcántara wrote: I have a job running on Flink 1.14.3 (Java 11) that uses rocksdb as the state backend. The problem is that the job requires an amount of

Job requiring a lot of memory despite using rocksdb state backend

I have a job running on Flink 1.14.3 (Java 11) that uses rocksdb as the state backend. The problem is that the job requires an amount of memory pretty similar to the overall state size. Indeed, for making it stable (and capable of taking snapshots) this is what I'm using: - 4 TMs with 30

Re: Some question of RocksDB state backend on ARM os

our reply. > > Will this feature be released in version 1.14? > > Best, > > Hui > > *发件人:* Robert Metzger [mailto:rmetz...@apache.org] > *发送时间:* 2021年7月20日 19:45 > *收件人:* Wanghui (HiCampus) > *抄送:* user@flink.apache.org > *主题:* Re: Some question of RocksDB state

RE: Some question of RocksDB state backend on ARM os

Hi Robert： Thank you for your reply. Will this feature be released in version 1.14? Best, Hui 发件人: Robert Metzger [mailto:rmetz...@apache.org] 发送时间: 2021年7月20日 19:45 收件人: Wanghui (HiCampus) 抄送: user@flink.apache.org 主题: Re: Some question of RocksDB state backend on ARM os The RocksDB

Re: Some question of RocksDB state backend on ARM os

commenting on https://issues.apache.org/jira/browse/FLINK-13598 to raise awareness. On Tue, Jul 20, 2021 at 5:26 AM Wanghui (HiCampus) wrote: > Hi all: > >When I use RocksDB as state backend on an aarch64 system, the > following error occurs: > > 1. Does the aarch64

Some question of RocksDB state backend on ARM os

Hi all: When I use RocksDB as state backend on an aarch64 system, the following error occurs: 1. Does the aarch64 system not support rocksdb? 2. If not, is there a support plan for later versions of flink? Caused by: java.lang.Exception: Exception while creating

Question about setting up Task-local recovery with a RocksDB state backend

Hello, I've been going through the documentation for task-local recovery and came across this section which discusses that with incremental check

Re: EOFException on attempt to scale up job with RocksDB state backend

Tang ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend I Yun, I've changed configuration to use block blobs, however due to another issue [1], I can't make savepoint, I hope eventually job will able to proces

Re: EOFException on attempt to scale up job with RocksDB state backend

March 18, 2021 5:08 AM To: Alexey Trenikhun ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Alexey, Flink would only write once for checkpointed files. Could you try to write checkpointed files as block blob format

Re: EOFException on attempt to scale up job with RocksDB state backend

; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Yun, How underlying storage explains fact that without re-scale I can restore from savepoint? Does Flink write file once or many times, if many times, then potentially

Re: EOFException on attempt to scale up job with RocksDB state backend

ed in [1] or without compaction? Thanks, Alexey From: Yun Tang Sent: Wednesday, March 17, 2021 9:31 PM To: Alexey Trenikhun ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Alexey,

Re: EOFException on attempt to scale up job with RocksDB state backend

ordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Yun, Azure web UI shows size of all files created by Flink as 128Mib * X (128, 256, 640), see screenshot attached. In my understanding this is because Flink creates them as Page Blob

Re: EOFException on attempt to scale up job with RocksDB state backend

Yun Tang From: Alexey Trenikhun Sent: Thursday, March 18, 2021 0:45 To: Yun Tang ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Yun, I've copied 77e77928-cb26-4543

Re: EOFException on attempt to scale up job with RocksDB state backend

, not sure does it have effect on savepoint or not) Thanks, Alexey From: Yun Tang Sent: Wednesday, March 17, 2021 12:33 AM To: Alexey Trenikhun ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state b

Re: EOFException on attempt to scale up job with RocksDB state backend

on attempt to scale up job with RocksDB state backend Attached. From: Yun Tang Sent: Tuesday, March 16, 2021 11:13 PM To: Alexey Trenikhun ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state back

Re: EOFException on attempt to scale up job with RocksDB state backend

@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Hi Yun, I'm attaching shorter version of log, looks like full version didn't come through Thanks, Alexey From: Yun Tang Sent: Tuesday, March 16, 2021

Re: EOFException on attempt to scale up job with RocksDB state backend

From: Alexey Trenikhun Sent: Tuesday, March 16, 2021 15:10 To: Yun Tang ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backend Also restore from same savepoint without change in parallelism works fine

Re: EOFException on attempt to scale up job with RocksDB state backend

state backend No, I believe original exception was from 1.12.1 to 1.12.1 Thanks, Alexey From: Yun Tang Sent: Monday, March 15, 2021 8:07:07 PM To: Alexey Trenikhun ; Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job

Re: EOFException on attempt to scale up job with RocksDB state backend

RocksDB state backend Hi, Can you scale the job at the same version from 1.12.1 to 1.12.1? Best Yun Tang From: Alexey Trenikhun Sent: Tuesday, March 16, 2021 4:46 To: Tzu-Li (Gordon) Tai ; user@flink.apache.org Subject: Re: EOFException on attempt to scale up job

Re: EOFException on attempt to scale up job with RocksDB state backend

state backend Savepoint was taken with 1.12.1, I've tried to scale up using same version and 1.12.2 From: Tzu-Li (Gordon) Tai Sent: Monday, March 15, 2021 12:06 AM To: user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB

Re: EOFException on attempt to scale up job with RocksDB state backend

Savepoint was taken with 1.12.1, I've tried to scale up using same version and 1.12.2 From: Tzu-Li (Gordon) Tai Sent: Monday, March 15, 2021 12:06 AM To: user@flink.apache.org Subject: Re: EOFException on attempt to scale up job with RocksDB state backen

Re: EOFException on attempt to scale up job with RocksDB state backend

Hi, Could you provide info on the Flink version used? Cheers, Gordon -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

EOFException on attempt to scale up job with RocksDB state backend

org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_8131c39062c4256ee506e2382c4a7bfd_(3/8) from any of the 1 provided restore

Re: OOM error for heap state backend.

used for other tasks ? >>> >>> >>> I don't quite understand this question. I believe the recommendation in >>> docs has the same reason: use larger state objects so that the Java object >>> overhead pays off. >>> RocksDB keeps state in mem

Re: OOM error for heap state backend.

hat the Java object >> overhead pays off. >> RocksDB keeps state in memory and on disk in the serialized form. >> Therefore it usually has a smaller footprint. >> Other jobs in the same task manager can potentially use other state >> backend depending on their state requireme

Re: OOM error for heap state backend.

son: use larger state objects so that the Java object > overhead pays off. > RocksDB keeps state in memory and on disk in the serialized form. > Therefore it usually has a smaller footprint. > Other jobs in the same task manager can potentially use other state > backend depending on th

Re: OOM error for heap state backend.

pays off. RocksDB keeps state in memory and on disk in the serialized form. Therefore it usually has a smaller footprint. Other jobs in the same task manager can potentially use other state backend depending on their state requirements. All tasks in the same task manager share the JVM heap as the task

Re: OOM error for heap state backend.

t;- Memory leak >>>- Not enough GC threads >>>- Concurrent GC starts too late >>>- ... >>> >>> I would suggest taking a look at the GC logs. >>> >>> Thank you~ >>> >>> Xintong Song >>> >>>

Re: OOM error for heap state backend.

> > On Fri, Aug 21, 2020 at 10:34 PM Vishwas Siravara > wrote: > >> Hi guys, >> I use flink version 1.7.2 >> I have a stateful streaming job which uses a keyed process function. I >> use heap state backend. Although I set TM heap size to 16 GB, I get OOM >>

Re: OOM error for heap state backend.

, Aug 21, 2020 at 10:34 PM Vishwas Siravara wrote: > Hi guys, > I use flink version 1.7.2 > I have a stateful streaming job which uses a keyed process function. I use > heap state backend. Although I set TM heap size to 16 GB, I get OOM error > when the state size is around 2.5 GB(f

OOM error for heap state backend.

Hi guys, I use flink version 1.7.2 I have a stateful streaming job which uses a keyed process function. I use heap state backend. Although I set TM heap size to 16 GB, I get OOM error when the state size is around 2.5 GB(from dashboard I get the state size). I have set taskmanager.memory.fraction

Re: State backend considerations

Hi Nick, Both questions are hard to answer given that it depends on your hardware, access patterns (read/update), record size/structure, parallelism, and probably a ton of other parameters. The usual approach is to simply evaluate it in your setting. Since it's a matter of configuration, you can

State backend considerations

Hi guys, I have a few questions on state backends. Is there a guideline on how big the state has to be where it makes sense to use RocksDB rather than FsStatebackend ? Is there an analysis on latency for a full checkpoint for FsSateBackend based on increase in state size ? Best, Nick.

Re: Pulsar as a state backend

- Work on state integration is also in progress and expected sometime > in May. >- Using Pulsar as a Flink state backend is something that the team >will also be working on once the off-loader (mentioned above) is complete >so this should be expected sometime in June. > >

Re: Pulsar as a state backend

currently in progress and expected to be out sometime in May. - Work on state integration is also in progress and expected sometime in May. - Using Pulsar as a Flink state backend is something that the team will also be working on once the off-loader (mentioned above) is complete so

Pulsar as a state backend

Hello, I recently browse this post : https://flink.apache.org/2019/05/03/pulsar-flink.html and mainly : *Finally, an alternative way of integrating the technologies could include using Pulsar as a state backend with Flink. Since Pulsar has a layered architecture (Streams and Segmented Streams

Re: Flink cluster on k8s with rocksdb state backend

:31 PM, dhanesh arole wrote: > > Hello all, > > I am trying to provision a Flink cluster on k8s. Some of the jobs in our > existing cluster use RocksDB state backend. I wanted to take a look at the > Flink helm chart or deployment manifests that provision task managers with

Flink cluster on k8s with rocksdb state backend

Hello all, I am trying to provision a Flink cluster on k8s. Some of the jobs in our existing cluster use RocksDB state backend. I wanted to take a look at the Flink helm chart or deployment manifests that provision task managers with dynamic PV and how they manage it. We are running on kops

Re: Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline

t; Yun Tang > > -- > *From:* Kaymak, Tobias > *Sent:* Tuesday, August 13, 2019 21:20 > *To:* user@flink.apache.org > *Subject:* Flink 1.8: Using the RocksDB state backend causes > "NoSuchMethodError" when trying to stop a pipeline > >

Re: Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline

ay, August 13, 2019 21:20 To: user@flink.apache.org Subject: Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline Hi, I am using Apache Beam 2.14.0 with Flink 1.8.0 and I have included the RocksDb dependency in my projects pom.xml as w

Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline

Hi, I am using Apache Beam 2.14.0 with Flink 1.8.0 and I have included the RocksDb dependency in my projects pom.xml as well as baked it into the Dockerfile like this: FROM flink:1.8.0-scala_2.11 ADD --chown=flink:flink http://central.maven.org/maven2/org/apache/flink/flink-statebackend-rocksdb_

Re: HDFS checkpoints for rocksDB state backend:

Hi, Andrea If you are running flink cluster on Yarn, the jar `flink-shaded-hadoop2-uber-1.6.4.jar` should exist in the lib dir of the flink client, so that it could be uploaded to the Yarn Distributed Cache and then be available on JM and TM. And if you are running flink standalone cluster, the j

Re: HDFS checkpoints for rocksDB state backend:

HI Qiu, my jar does not contain the class `org.apache.hadoop.hdfs.protocol.HdfsConstants*`, *but I do expect it is contained within `flink-shaded-hadoop2-uber-1.6.4.jar` which is located in Flink cluster libs. Il giorno gio 27 giu 2019 alle ore 04:03 Congxian Qiu < qcx978132...@gmail.com> ha scrit

Re: HDFS checkpoints for rocksDB state backend:

Hi Andrea As the NoClassDefFoundError, could you please verify that there exist `org.apache.hadoop.hdfs.protocol.HdfsConstants*` *in your jar. Or could you use Arthas[1] to check if there exists the class when running the job? [1] https://github.com/alibaba/arthas Best, Congxian Andrea Spina

HDFS checkpoints for rocksDB state backend:

Dear community, I'm trying to use HDFS checkpoints in flink-1.6.4 with the following configuration state.backend: rocksdb state.checkpoints.dir: hdfs:// rbl1.stage.certilogo.radicalbit.io:8020/flink/checkpoint state.savepoints.dir: hdfs:// rbl1.stage.certilogo.radicalbit.io:8020/flink/savepoints

Re: Unable to set S3 like object storage for state backend.

Hi Ken, Thanks for reaching out, I created a compliant bucket with name aip-featuretoolkit. I now get the exception "Unable to execute HTTP request: aip-featuretoolkit.SU73ECSG1P1d.***.COM: Name or service not known" from org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.class l

Re: Unable to set S3 like object storage for state backend.

Hi Vishwas, It might be that you’ve got a legacy bucket name (“aip_featuretoolkit”), as AWS no longer allows bucket names to contain an underscore . I’m guessing that the Hadoop S3 code is trying to treat your path as a va

Unable to set S3 like object storage for state backend.

Hi, I am using flink version 1.7.2 , I am trying to use S3 like object storage EMC ECS( https://www.emc.com/techpubs/ecs/ecs_s3_supported_features-1.htm) . I am using the flink-s3-fs-hadoop-1.7.2.jar file as a dependency for s3 filesystem and I have placed it under the lib folder and is available

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Unfortunately the audit logs for hdfs were not enabled. We will enable them and post he results when the problem happens again. Nonetheless, we don't have ay other process using hadoop besides flink - Best Regards, Pedro Chaves -- Sent from: http://apache-flink-user-mailing-list-archive.2336

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hi Pedro From the previous given log, I found that checkpoint 65912 has been expired then, the raise the IOException. When some checkpoint expired, the checkpoint dir will be deleted(CheckpointCoordinator#549 on release-1.6 branch), and the unfinished task will still write to the previous files,

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

The issue happened again. /AsynchronousException{java.lang.Exception: Could not materialize checkpoint 47400 for operator ENRICH (1/4).} at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153) at org.

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello Andrey, The audit log doesn't have anything that would point to it being deleted. The only thing worth mentioning is the following line. /2019-05-15 10:01:39,082 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK* blk_1248714854_174974084 is COMMITTED but not COMPLETE(numNodes=

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello, Thanks for the help. I've attached the logs. Our cluster has 2 job managers (HA) and 4 task managers. logs.tgz Regards, Pedro - Best Regards, Pedro Chaves -- Sent from: http://apache-flink

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hi Pedro Could you please share the audit log for file `/flink/data/checkpoints/76f7b4f5c679e8f2d822c9c3c73faf5d/chk-65912/68776faf-b687-403b-ba0c-17419f8684dc`, seems this did not exist cause this problem (maybe this file was created and deleted for some reason) Best, Congxian Andrey Zagrebin

Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

The stack trace shows that the state is being restored which has probably already happened after job restart. I am wondering why it has been restarted after some time of running. Could you share full job/task manager logs? On Thu, May 16, 2019 at 6:26 AM anaray wrote: > Thank You Andrey. Arity o

Re: Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hi, could you also post job master logs? and ideally full task manager logs. This failure can be caused by some other previous failure. Best, Andrey On Wed, May 15, 2019 at 2:48 PM PedroMrChaves wrote: > Hello, > > Every once in a while our checkpoints fail with the following exception: > > /A

Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

Thank You Andrey. Arity of the job has not changed. Here issue is that job will run for sometime (with checkpoint enabled) and then after some time will get into above exception. The job keeps restarting afterwards. One thing that I want point out here is that we have a custom *serialization sche

Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

Hi, I am not sure that FLINK-8836 is related to the failure in the stack trace. You say you are using Flink in production, does it mean it always worked and has started to fail recently? >From the stack trace, it looks like the arity of some Tup

Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello, Every once in a while our checkpoints fail with the following exception: /AsynchronousException{java.lang.Exception: Could not materialize checkpoint 65912 for operator AGGREGATION-FILTER (2/2).} at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler

flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

options. Please advice. java.lang.IllegalStateException: Could not initialize operator state backend. at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initOperatorState(AbstractStreamOperator.java:301) at

Re: [EXTERNAL] Re: Looking for help in configuring Swift as State Backend

.apache.flink.runtime.client.JobExecutionException: Could > not set up JobManager > >at > org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:176) > >at > org.apache.flink.runtime.dispatcher.Dis

Re: [EXTERNAL] Re: Looking for help in configuring Swift as State Backend

4) ... 7 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not instantiate configured state backend at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGr

Re: Looking for help in configuring Swift as State Backend

kkalmukk...@comcast.com> wrote: > Hi, > > > > I am looking for some help in configuring the Swift Fs as State Backend. I > am unable to configure it, let me know if anyone has prior done this or > knowledge to help me > > Do we still need to run an HDFS to use this feature ? > > > > Thanks, > > Shakir >

Looking for help in configuring Swift as State Backend

Hi, I am looking for some help in configuring the Swift Fs as State Backend. I am unable to configure it, let me know if anyone has prior done this or knowledge to help me Do we still need to run an HDFS to use this feature ? Thanks, Shakir

Re: getting an error when configuring state backend to hdfs

[1] > > [1] https://flink.apache.org/downloads.html#latest-stable-release-v171 > > > Best > Yun Tang > -- > *From:* Avi Levi > *Sent:* Thursday, December 20, 2018 2:11 > *To:* Steven Nelson > *Cc:* Chesnay Schepler; user@flink.apache.o

Re: getting an error when configuring state backend to hdfs

error when configuring state backend to hdfs when I try running from my IDE (intellij) I am getting this exception Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Could not retrieve JobResult. at org.apache.flink.runtime.minicluster.MiniCluste

Re: getting an error when configuring state backend to hdfs

when I try running from my IDE (intellij) I am getting this exception Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Could not retrieve JobResult. at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:643) at org.apache.flink.stre

Re: getting an error when configuring state backend to hdfs

What image are you using? Sent from my iPhone > On Dec 19, 2018, at 9:44 AM, Avi Levi wrote: > > Hi Chesnay, > What do you mean? I am creating a fat jar with all dependencies (using sbt > assembly). which jar I should place in the /lib directory ? > >> On Wed, Dec 19, 2018 at 4:44 PM Chesnay

Re: getting an error when configuring state backend to hdfs

flink-hadoop-fs should be in /lib On 19.12.2018 16:44, Avi Levi wrote: Hi Chesnay, What do you mean? I am creating a fat jar with all dependencies (using sbt assembly). which jar I should place in the /lib directory ? On Wed, Dec 19, 2018 at 4:44 PM Chesnay Schepler

Re: getting an error when configuring state backend to hdfs

Hi Chesnay, What do you mean? I am creating a fat jar with all dependencies (using sbt assembly). which jar I should place in the /lib directory ? On Wed, Dec 19, 2018 at 4:44 PM Chesnay Schepler wrote: > Are you including the filesystems in your jar? Filesystem jars must be > placed in the /lib

Re: getting an error when configuring state backend to hdfs

Are you including the filesystems in your jar? Filesystem jars must be placed in the /lib directory of the flink distribution. On 19.12.2018 15:03, Avi Levi wrote: Hi, I am trying to set the backend state to hdfs /val stateUri = "hdfs/path_to_dir"/ /val backend: RocksDBStateBackend = new RocksD

getting an error when configuring state backend to hdfs

Hi, I am trying to set the backend state to hdfs *val stateUri = "hdfs/path_to_dir"* *val backend: RocksDBStateBackend = new RocksDBStateBackend(stateUri, true)* *env.setStateBackend(backend)* I am running with flink 1.7.0 with the following dependencies (tried them with different combinations)

Re: Question over Incremental Snapshot vs Full Snapshot in rocksDb state backend

Thanks a lot Andrey . Your answer to 1st question clarifies well for both the questions. Appreciate your help in clearing doubt Regards, Chandan On Thu, Oct 25, 2018 at 6:10 PM Andrey Zagrebin wrote: > Hi Chandan, > > > 1. Why did we took 2 different approaches using different RocksDB apis ? >

Re: RocksDB State Backend Exception

Hi Andrey, Thank you for the explanation. I think you are right. It is either kStaleFile or kNoSpace. We found the cause of the issue, even though we still don't know how to explain it. We set the java.io.tmpdir to an EBS-backed drive instead of the default and the exception started happening. Th

Re: Question over Incremental Snapshot vs Full Snapshot in rocksDb state backend

Hi Chandan, > 1. Why did we took 2 different approaches using different RocksDB apis ? > We could have used Checkpoint api of RocksDB for fullSnapshot as well . The reason here is partially historical. Full snapshot in RocksDB backend was implemented before incremental and rescaling for incremen

Re: RocksDB State Backend Exception

Hi Ning, The problem here first of all is that RocksDB java JNI client diverged from RocksDB cpp code in status.h, as mentioned in the Flink issue you refer to. Flink 1.6 uses RocksDB 5.7.5 java client. The JNI code there misses these status subcodes: kNoSpace = 4, kDeadlock = 5, kStaleFile = 6

RocksDB State Backend Exception

Hi, We are doing some performance testing on a 12 node cluster with 8 task slots per TM. Every 15 minutes or so, the job would run into the following exception. java.lang.IllegalArgumentException: Illegal value provided for SubCode. at org.rocksdb.Status$SubCode.getSubCode(Status.java:109

  1   2   3   >