Re: Error when trying to setup and run wordcount example on dataproc

There is already a follow up discussion on Beam ML [1]. [1] https://lists.apache.org/x/thread.html/r25910bf9ef5bc4b5f3c3d1c99071cb6d43103898a054dcd2a14d8da8@%3Cuser.beam.apache.org%3E Best, D. On Mon, Jul 12, 2021 at 9:27 AM Chesnay Schepler wrote: > I would recommend reaching out to the Beam

Re: Kafka data sources, multiple interval joins and backfilling

Hi Dan, unfortunately Flink currently provides no source level synchronization, except for Kinesis [1], so it's easy to run into large states, when processing historical data. There is an on-going effort, to provide a generic watermark-based alignment of FLIP-27 sources [2], that will most likely

Re: Flink TaskManager container got restarted by K8S very frequently

If you run `kubectl describe pod ...` on the affected pod, you should see a reason why the previous pod has terminated (eg. OOM killed by Kubernetes). Best, D. On Thu, Jul 22, 2021 at 9:30 AM Fabian Paul wrote: > CC user ML > >

Re: Flink user subscription request

Hi Samir, to unsubscribe please send an empty-body / empty-subject email to user-unsubscr...@flink.apache.org. You can see a community page [1] in docs for more details. [1] https://flink.apache.org/community.html Best, D. On Fri, Jul 23, 2021 at 9:01 AM Samir Vasani wrote: > Hi, > > This is

Re: Unable to read state Witten by Beam application with Flink runner using Flink's State Processor API

This thread is duplicated on the dev mailing list [1]. [1] https://lists.apache.org/x/thread.html/r87fa8153137a4968f6a4f6b47c97c4d892664d864c51a79574821165@%3Cdev.flink.apache.org%3E Best, D. On Tue, Jul 27, 2021 at 5:38 PM Kathula, Sandeep wrote: > Hi, > > We have a simple Beam applicati

Re: Obtain JobManager Web Interface URL

Hi Andreas, you should be able to discover running JobManagers using YarnClient (looking at the API, "ApplicationReport#getTrackingUrl()" [1] may be what you're looking for). I don't think it would be easy to have a generic support for this in Flink without making assumptions about the underlying

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

Hi Rion, As you probably already know, for dynamic indices, you can simply implement your own ElasticsearchSinkFunction

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

rouping by > clusters and dynamically handling indices)? > > I’d be happy to give a shot at making the appropriate changes to the sink > as well, although I’m far from an Elastic expert. If you point me in the > right direction, I may be able to help out. > > Thanks much! > > Ri

Re: JobManager Resident memory Always Increasing

Hi Pranjul, which deployment mode are you using? - For session cluster, I'd say it's possible that memory grows with # of jobs. - For application mode, there is actual user-code executed, so if you're using some native libraries in your job driver, that may be another reason for the growing memor

Re: Problems with reading ORC files with S3 filesystem

Hi Piotr, unfortunately this is a known long-standing issue [1]. The problem is that ORC format is not using Flink's filesystem abstraction for actual reading of the underlying file, so you have to adjust your Hadoop config accordingly. There is also a corresponding SO question [2] covering this.

Re: Upgrading from Flink on YARN 1.9 to 1.11

Hi Andreas, Per-job and session deployment modes should not be affected by this FLIP. Application mode is just a new deployment mode (where job driver runs embedded within JM), that co-exists with these two. >From information you've provided, I'd say your actual problem is this exception: ``` C

Re: s3 access denied with flink-s3-fs-presto

Hi Vamshi, >From your configuration I'm guessing that you're using Amazon S3 (not any implementation such as Minio). Two comments: - *s3.endpoint* should not contain bucket (this is included in your s3 path, eg. *s3:///*) - "*s3.path.style.access*: true" is only correct for 3rd party implementati

Re: Exploring Flink for a HTTP delivery service.

Hi Prasanna, here are some quick thoughts 1) Batching is an aggregation operation.But what I have seen in the > examples of windowing is that they get the count/max/min operation in the > particular window. So could the batching be implemented via a windowing > mechanism ? > I guess a custom fu

Re: NullPointerException in StateTable.put()

My intuition is that you have a non-deterministic shuffle key. If you perform any "per-key" operation, you need to make sure that the same key always end up in the same partition. To simplify this, it means that the key needs to have a consistent *hashCode* and *equals* across different JVMs. Usua

Re: NullPointerException in StateTable.put()

ry is used. So this could be the problem then. > > On Mon, Aug 16, 2021 at 5:33 PM David Morávek wrote: > >> Hi László, >> >> My intuition is that you have a non-deterministic shuffle key. If you >> perform any "per-key" operation, you need to make sure tha

Re: RabbitMQ 3.9+ Native Streams

Hi Rob, there is currently no on-going effort for this topic, I think this would be a really great contribution though. This seems to be pushing RabbitMQ towards new usages ;) Best, D. On Mon, Aug 16, 2021 at 8:16 PM Rob Englander wrote: > I'm wondering if there's any work underway to develop

Re: NullPointerException in StateTable.put()

Hi Laszlo, Please use reply-all for mailing list replies. This may help others finding their answer in the future ;) > sb.append(DeviceDetail.class.getName()).append('@').append(Integer.toHexString(System.identityHashCode(this))).append('['); This part will again make your key non-deterministi

Re: RabbitMQ 3.9+ Native Streams

.apache.org/contributing/how-to-contribute.html Best, D. On Tue, Aug 17, 2021 at 1:17 AM Rob Englander wrote: > I will definitely consider the contribution idea :) > > > On Mon, Aug 16, 2021 at 3:16 PM David Morávek wrote: > >> Hi Rob, >> >> there is currently no

Re: Upgrading from Flink on YARN 1.9 to 1.11

ted and sent you a signup link to our firm’s secure > document-sharing app called Lockbox. In the repository, I’ve uploaded both > our full client and YARN app logs (named half_failure-client_log and > half_failure-yarn-log, respectively) in a directory named Flink support > logs/Flink 1.11/1.

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

ose elements > to the sink, which would create the tenant-specific Elastic connection from > the ConfigurationT element and handle caching it and then just grab the > element and send it on it's way? > > Those are really the only bits I'm stuck on at the moment: > > 1. The

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

t; I can see the underlying sink getting initialized, the open call being > made, etc. however for some reason it looks like there's an issue related > to the context during the invoke call namely* "The runtime context has > not been initialized". *I had assumed this would be

Re: NullPointerException when using KubernetesHaServicesFactory

Hi Jonas, this exception is raised because "kubernetes.cluster-id" [1] is not set. I'd also recommend setting "kubernetes.namespace" option, unless you're using "default" namespace. I've filled FLINK-23961 [2] so we provide more descriptive warning for this issue next time ;) [1] https://ci.apac

Re: NullPointerException when using KubernetesHaServicesFactory

t;>> When bashing into the image: >>> >>> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls >>> flink-s3-fs-presto-1.12.5.jar >>> >>> Any idea? >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

f anything that might be similar to this even > within other sinks? Just trying to think of something to model this after. > Once I get things started, I'll spin up a JIRA issue for it and go from > there. > > Thanks so much for your help! > > Rion > > On Tue,

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

to me, I'd be appreciative. I'll >> likely need some assistance at a few points to ensure things look as >> expected, but I'm happy to help with this contribution. >> >> Rion >> >> On Wed, Aug 25, 2021 at 11:37 AM David Morávek wrote: >> >

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

a project of mine is working > like a charm, so I'm hoping it's something that others will be able to > leverage for their own needs. > > Rion > > On Thu, Aug 26, 2021 at 11:45 AM David Morávek wrote: > >> Hi Rion, >> >> persona

Re: Unable to read a (text)file in FileProcessing.PROCESS_CONTINUOS mode

Hi Samir, I'd second Roman here. Can you please verify that the directory you're trying to read from is accessible in all of your docker containers (TMs / JMs)? Ideally if you could list the directory inside the docker running container and post the output in this thread. Also please check that t

Re: How to mount PVC volumes using Flink Native Kubernetes ?

OT: Beware that even if you manage to solve this, EBS is replicated network storage, therefore rocksdb performance will be affected significantly. Best, D. On Fri 10. 9. 2021 at 16:19, tao xiao wrote: > The use case we have is to store the RocksDB sst files in EBS. The EC2 > instance type (m5)

Re: Flink-Zookeeper Security

Hi Beata, you need to upgrade zookeeper to 3.5+ on Flink side [1] and set JVM properties the same way you'd do with vanilla ZK client. Please refer to the following thread [2] for more details. It would be great if you could provide feedback for future reference, whether this approach has worked f

Re: RocksDB state not cleaned up

Hi Tao, my intuition is that the compaction of SST files is not triggering. By default, it's only triggered by the size ratios of different levels [1] and the TTL mechanism has no effect on it. Some reasoning from Stephan: It's very likely to have large files in higher levels that haven't been >

Re: Flink Kafka Partition Discovery Never Forgets Deleted Partitions

Hi Constantinos, The partition discovery doesn't support topic / partition removal as this may easily lead to a data loss (partition removal is not even supported by Kafka for the same reason) Dynamically adding and removing partitions as part of a business logic is just not how Kafka is designed

Re: flink : Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextShutdownAware

Hi Ragini, I think you actually have the opposite problem that your classpath contains slf4j binding for log4j 1.2, which is no longer supported. Can you try getting rid of the slf4j-log4j12 dependency? Best, D. On Tue, Sep 14, 2021 at 1:51 PM Ragini Manjaiah wrote: > when I try to run flink .

Re: flink : Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextShutdownAware

.0.0,) > > shade > > > > > > > >

Re: Flink Kafka Partition Discovery Never Forgets Deleted Partitions

Hi Jan, Notion of completeness is just one part of the problem. The second part is that once you remove the Kafka topic, you are no longer able to replay the data in case of failure. So you basically need a following workflow to ensure correctness: 1) Wait until there are no more elements in the

Re: Streaming SQL support for redis streaming connector

Hi Osada, in theory building a Redis table from "CDC stream" should definitely be doable. Unfortunately Flink currently doesn't have any official Redis Sink for the Table API and there is currently no on-going effort for adding it, so it would need to be implemented first. The resulting syntax wou

Re: Flink Kafka Partition Discovery Never Forgets Deleted Partitions

e. how is it useful for Flink > to continue remembering the partition of an already-deleted topic? (the > checkpoint is invalid regardless, right?) > > > > On Tue, Sep 14, 2021 at 5:20 PM Jan Lukavský wrote: > >> On 9/14/21 3:57 PM, David Morávek wrote: >> >> Hi Ja

Re: Beam with Flink runner - Issues when writing to S3 in Parquet Format

Hi Sandeep, Jan has already provided pretty good guidelines for getting more context on the issue ;) Because this is not for the first time, I would like to raise awareness, that it's not OK to send a user related question to four Apache mailing list (that I know of). Namely: - user@flink.apache

Re: Flink Kafka Partition Discovery Never Forgets Deleted Partitions

externally (for > whatever reason). > > I think I will drop this now. > > On Tue, Sep 14, 2021 at 5:50 PM David Morávek wrote: > >> We are basically describing the same thing with Fabian, just a different >> wording. >> >> The problem is that if you dele

Re: KafkaSource builder and checkpointing with parallelism > kafka partitions

If we are shutting down any sources of unbounded jobs that run on Flink versions without FLIP-147 (available in 1.14) [1], that Matthias has mentioned, than it's IMO a bug, because it effectively breaks checkpointing. Fabian, can you please verify whether this is an intended behavior? In the meant

Re: RocksDB state not cleaned up

> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/state/state.html#cleanup-during-rocksdb-compaction >> >> >> On Tue, Sep 14, 2021 at 5:38 PM David Morávek wrote: >> >>> Hi Tao, >>> >>> my intuition is t

Re: hdfs lease issues on flink retry

Hi, AttemptId needs to be an integer (take a look at TaskAttemptID class for more details). As for your prior question, any random id should solve the issue. Best, D. On Mon 20. 9. 2021 at 22:32, Shah, Siddharth wrote: > Hi Matthias, > > > > Thanks for suggesting a workaround, but our jobs fail

Re: hdfs lease issues on flink retry

neering] > *Sent:* Thursday, September 23, 2021 9:46 AM > *To:* 'David Morávek' ; Matthias Pohl < > matth...@ververica.com> > *Cc:* Hailu, Andreas [Engineering] ; > user@flink.apache.org; Erai, Rahul [Engineering] < > rahul.e...@ny.email.gs.com>; Chittajallu, Rajiv [

Re: EKs FlinkK8sOperator for 1.20

Hi Dhiru, What is the actual issue / failure that you've encountered when trying to deploy the operator into EKS cluster? In general, if you're running into any specific EKS issues with the operator, I'd say the best approach would be reaching out to its authors / community around it, as we have

Re: [DISCUSS] Creating an external connector repository

We are mostly talking about the freedom this would bring to the connector authors, but we still don't have answers for the important topics: - How exactly are we going to maintain the high quality standard of the connectors? - How would the connector release cycle to look like? Is this going to af

Re: Flink 1.14.0 reactive mode cannot rescale

Hi ChangZhuo, this seems to be a current limitation of the unaligned checkpoints [1], are you using any broadcasted streams in your application? [1] https://issues.apache.org/jira/browse/FLINK-22815 Best, D. On Tue, Oct 19, 2021 at 3:58 AM ChangZhuo Chen (陳昌倬) wrote: > Hi, > > We found that F

Re: Flink ignoring latest checkpoint on restart?

Hi Matt, this seems interesting, I'm aware of some possible inconsistency issues with unstable connections [1], but I have to yet find out if this could be related. I'll do some research on this and will get back to you. In the meantime, can you see anything relevant in the zookeeper logs? Also w

Re: Flink 1.14.0 reactive mode cannot rescale

Fixed in context of FLINK-22815 just means that the feature set described in this issue have been delivered. In this case it means that unaligned checkpoints have been disabled. D. On Tue, Oct 19, 2021 at 2:22 PM ChangZhuo Chen (陳昌倬) wrote: > On Tue, Oct 19, 2021 at 11:51:44AM +0200, Da

Re: Possibility of supporting Reactive mode for native Kubernetes application mode

Hi Fuyao, this is a great question ;) 1) First let's be clear on what the reactive mode actually is. Reactive Mode is related to how the Flink makes use of the newly available resources. It greedily uses all of the resources that are available in your Flink cluster (if new task manager joins in,

Re: New blog post published - Sort-Based Blocking Shuffle Implementation in Flink

Thanks Daisy and Kevin for a great write up! ;) Especially the 2nd part was really interesting, I really like the idea of the single spill file with a custom scheduling of read requests. Best, D. On Mon, Nov 1, 2021 at 10:01 AM Daisy Tsang wrote: > Hey everyone, we have a new two-part post publ

Re: [External] : Re: Possibility of supporting Reactive mode for native Kubernetes application mode

reporters/#prometheus > > [2] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/#enabling-and-configuring-checkpointing > > > > Best, > > Fuyao > > > > > > > > *From: *David Morávek >

Re: unsubscribe

Hi Peter, to unsubscribe, please send an email to user-unsubscr...@flink.apache.org Best, D. On Fri, Nov 5, 2021 at 9:28 AM Peter Schrott wrote: > unsubscribe >

Re: how to expose the current in-flight async i/o requests as metrics?

Hi Dongwon, There are currently no metrics for the async work-queue size (you should be able to see the queue stats with debug logs enabled though [1]). As far as I can tell from looking at the code, the async operator is able to checkpoint even if the work-queue is exhausted. Arvid can you pleas

Re: Elasticsearch6 connector in flink stand alone

Hi Ravi, I'm moving this thread to the user@flink mailing list, which is designed for these type of questions. For your issue, I don't think it's related to the elasticsearch integration. It seems like there is something wrong with your log4j setup. Either you have a conflicting log4j jars on the

Re: Restarting a job with drain flag set to true

Hi Pedro, draining basically means that all of the sources will finish and progress their watermark to end of the global window, which will fire all of the triggers as a result. In other words, it will trigger the _ON_TIME_ results from all of the unfinished windows, even though they might not hav

Re: Restarting a job with drain flag set to true

this for pipeline upgrades. Or am I missing something? > > Thanks, > > Pedro > > > On Mon, 8 Nov 2021 at 15:58, David Morávek wrote: > >> Hi Pedro, >> >> draining basically means that all of the sources will finish and progress >> their waterma

Re: how to expose the current in-flight async i/o requests as metrics?

alculator-2.0-SNAPSHOT.jar:?] >> >> at >> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244) >> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?] >> >> at java.lang.Thread.run(Thread.java:829) [?:?] >> > > Can

Re: FLink Accessing two hdfs cluster

Hi chenqizhu, this exception doesn't seem to come from Flink, but rather from a YARN container bootstrap. When YARN container starts up, it needs to download resources from HDFS (your job archives / configuration / distributed cache / ...) which are necessary for startup of the user application (

Re: Issues in Batch Jobs Submission for a Session Cluster

Hi Jay, It's hard to say what going on here. My best guess is that you're running out of memory for your process (eg. hitting ulimit). Can you please start with checking the ulimits memory usage of your container? For the cleanup, right now it may happen in some failover scenarios that we don't c

Re: Question about relationship between operator instances and keys

Hi haocheng, in short it works as follows: - Each parallel instance of an operator is responsible for one to N key groups. - Each parallel instance belongs to a slot, which is tied with a single thread (slot may actually introduce multiple subtasks) - # of keygroups for each operator = max parall

Re: Buffering when connecting streams

Hi Alexis, I'm not sure what "watermark" step refers to in you graph, but in general I'd say your intuition is correct. For the "buffering" part, each sub-task needs to send data via data exchange (last operator in chain) has an output buffer and the operator that consumes this data (maybe on dif

Re: Buffering when connecting streams

from open > to close) triggered on second 17, and my windows are evaluated every > minute, so it wasn’t a race condition. > > > > Regards, > > Alexis. > > > > *From:* David Morávek > *Sent:* Donnerstag, 2. Dezember 2021 14:52 > *To:* Alexis Sarda-Espinosa

Re: Unable to create new native thread error

Hi Ilan, we are aware of multiple issues when web-submission can result in classloader / thread local leaks, which could potentially result in the behavior you're describing. We're working on addressing them. FLINK-25022 [1]: The most critical one leaking thread locals. FLINK-25027 [2]: Is only a

Re: GCS/Object Storage Rate Limiting

Hi Kevin, this happens only when the pipeline is started up from savepoint / retained checkpoint right? Guessing from the "path" you've shared it seems like a RockDB based retained checkpoint. In this case all task managers need to pull state files from the object storage in order to restore. This

Re: Buffering when connecting streams

I haven’t been able to come up with a good way of > ensuring that all data from the side stream for a given minute is processed > by processElement2 before all data for the same (windowed) minute reaches > processElement1, even when considering watermarks. > > > > Regards, >

Re: Broadcast and watermark

Hi Sweta, the output timestamp seems reasonable to me. I guess you're concerned about watermarks you're seeing, is that correct? final Instant min = Instant.ofEpochMilli(Long.MIN_VALUE); final Instant max = Instant.ofEpochMilli(Long.MAX_VALUE); System.out.printf("Min: %s, Max: %s%n", min, max);

Re: Broadcast and watermark

One more thought, if you're "broadcasting" the output of the KafkaSource, it may as well be the case that some partition is empty? Best, D. On Thu, Dec 2, 2021 at 5:11 PM David Morávek wrote: > Hi Sweta, > > the output timestamp seems reasonable to me. I guess

Re: Buffering when connecting streams

tion I > can’t tell Flink to "halt" processElement1 and switch to the other stream > depending on watermarks. I could look into TwoInputStreamOperator if you > think that’s the best approach. > > > > Regards, > > Alexis. > > > > *From:* David Moráve

Re: Watermark behavior when connecting streams

Hi Alexis, please take a look at AbstractStreamOperator [1] for details how the watermark is calculate for TwoInputOperator. It uses pretty much the same approach as for with the single input one (it simply takes a minimum). For watermark re-assignment, we ignore input watermark unless it's Long.

Re: Watermark behavior when connecting streams

t; > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#consecutive-windowed-operations > > > > Regards, > > Alexis. > > > > *From:* David Morávek > *Sent:* Donnerstag, 2. Dezember 2021 17:26 > *To:* Alexis Sarda-Espinosa

Re: custom metrics in elasticsearch ActionRequestFailureHandler

Hi Lars, quickly looking at the ES connector code, I think you're right and there is no way to do that :( In general I'd say that being able to expose metrics is a valid request. I can imagine having some kind of `RichActionRequestFailureHandler` with `{get|set}RuntimeContext` methods. More or l

Re: Buffering when connecting streams

framework calls the operator’s processElement1 continuously (even for > several minutes) before calling processElement2 a single time? How does the > framework decide when to switch the stream processing when the streams are > connected? > > > > Regards, > > Alexis. > &

Re: Unable to create new native thread error

> > > Another question, I see that the most critical issue (FLINK-25022) is in > progress and should be released on with version 1.13.4 , do you know when > this version is planned to be released? > > > > Thanks again, > > Ilan. > > > > *From: *David

Re: GCS/Object Storage Rate Limiting

27;re trying to restore >> with (especially with RocksDB). >> > > Can you elaborate on this a bit? We aren't changing the parallelism when > restoring. > > On Thu, Dec 2, 2021 at 10:48 AM David Morávek wrote: > >> Hi Kevin, >> >> this happens

[DISCUSS] Strong read-after-write consistency of Flink FileSystems

Hi Everyone, as outlined in FLIP-194 discussion [1], for the future directions of Flink HA services, I'd like to verify my thoughts around guarantees of the distributed filesystems used with Flink. Currently some of the services (*JobGraphStore*, *CompletedCheckpointStore*) are implemented using

Re: [DISCUSS] Strong read-after-write consistency of Flink FileSystems

ke Store Gen2? That's currently > supported and also used by Flink users [1]. There's also MapR FS, but I > doubt if that is still used. > > Best regards, > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > On Mon

Re: [DISCUSS] Deprecate MapR FS

+1, agreed with Seth's reasoning. There has been no real activity in MapR FS module for years [1], so the eventual users should be good with using the jars from the older Flink versions for quite some time [1] https://github.com/apache/flink/commits/master/flink-filesystems/flink-mapr-fs Best, D.

Re: Flink 1.13.3, k8s HA - ResourceManager was revoked leadership

Hi Alexey, please be aware that the json-based logs in the mail may not make it pass the spam filter (at least for gmail they did not) :( K8s based leader election is based on optimistic locking of the underlying config-map (~ periodically updating the lease annotation of the config-map). If JM f

Re: FW: EXT: Re: Issues in Batch Jobs Submission for a Session Cluster

9 AM Ghiya, Jay (GE Healthcare) < jay.gh...@ge.com> wrote: > Hi @David Morávek , > > > > PFA details regarding memory config in the configmap we have set and > corresponding usage details in terms of cpu,mem and jvm when the issue > happens. > > > > Credits: @R,

Re: Unable to create new native thread error

net.PlainSocketImpl.socketConnect(Native Method)\n', > > > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)\n', > > > > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)\n', > > > > java.n

Re: Unable to create new native thread error

are > dedicated only to the cluster. > The configuration of the docker are pulled from the hosts so same number > of threads is configured on the task and job managers. > > > > Kind regards, > > Ilan > > > > *From: *David Morávek > *Date: *Monday, 13 December 2021

Re: [DISCUSS] Strong read-after-write consistency of Flink FileSystems

Any other thoughts on the topic? If there are no concerns, I'd continue with creating a FLIP for changing the "written" contract of the Flink FileSystems to reflect this. Best, D. On Wed, Dec 8, 2021 at 5:53 PM David Morávek wrote: > Hi Martijn, > > I simply wasn&#

Re: How do I determine which hardware device and software has log4j zero-day security vulnerability?

Hi Turritopsis, I fail to see any relation to Apache Flink. Can you please elaborate on how Flink fits into it? Best, D. On Thu, Dec 16, 2021 at 3:52 PM Turritopsis Dohrnii Teo En Ming < ceo.teo.en.m...@gmail.com> wrote: > Subject: How do I determine which hardware device and software has > log

Re: question on jar compatibility - log4j related

Hi Eddie, the APIs should be binary compatible across patch releases, so there is no need to re-compile your artifacts Best, D. On Sun 19. 12. 2021 at 16:42, Colletta, Edward wrote: > If have jar files built using flink version 11.2 in dependencies, and I > upgrade my cluster to 11.6, is it sa

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

Hello Hans, it's DNS ;) You need to make sure, that "repo.maven.apache.org" can be resolved from your docker container (you can use tools such as host, dig, nslookup to verify that). This is may be tricky to debug, unless you're familiar with networking. A good place to start might be checking the

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

gt; 2021-12-21 10:06:39 (4.42 MB/s) - ‘flink-table-api-java-1.13.1.pom’ > saved [5394/5394] > > Only setting /root/.m2/settings.xml correctly helps > And that is what I don't understand as the proxy configuration will not be > the same for everyone. > > Regards Hans

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

p in just a few steps*. We will walk > you through the necessary commands and show how to validate that everything > is running correctly. > > Which is in fact not exactly true > > The compose command failse. > > I think that instructions need to be added to fix this. > >

Re: How to know if Job nodes are registered in cluster?

Hi John, there is usually no need to run multiple JM, if you're able to start a new one quickly after failure (eg. when you're running on kubernetes). There is always only single active leader and other JMs effectively do nothing besides competing for the leadership. Zookeeper based HA uses the De

Re: Avoiding Dynamic Classloading for User Code

Hi Lior, can you please provide details about the steps (I'm not sure what load jar / execute with the API means)? are you submitting the job using the REST API or Flink CLI? I assume you're using a session cluster. also what is the concern here? do you run into any class-loading related issues?

Re: Class loader

I assume this is a duplicate of the previous thread [1] [1] https://lists.apache.org/thread/16kxytrqycolfwfmr5tv0g6bq9m2wvof Best, D. On Tue, Dec 21, 2021 at 3:53 PM Lior Liviev wrote: > Hello, I wanted to know if I have my user code Jar in Flink, and I'm > running it 3 times, will the class l

Re: Class loader

tually no, it's a separate question > ------ > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 5:45 PM > *To:* Lior Liviev > *Cc:* user@flink.apache.org > *Subject:* Re: Class loader > > > *CAUTION*: external source > I assume t

Re: Avoiding Dynamic Classloading for User Code

hat's why I'm thinking of putting the jar in > flink/lib > -- > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 5:43 PM > *To:* Lior Liviev > *Cc:* user@flink.apache.org > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > &g

Re: Class loader

now if > it will resolve my oom metaspace problem > > Get Outlook for iOS <https://aka.ms/o0ukef> > -- > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 5:59:05 PM > *To:* Lior Liviev ; user > *Subject:* Re: Class loader > >

Re: Avoiding Dynamic Classloading for User Code

ook for iOS <https://aka.ms/o0ukef> > ---------- > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 6:08:51 PM > *To:* Lior Liviev ; user > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Please always

Re: [DISCUSS] Changing the minimal supported version of Hadoop

CC user@f.a.o Is anyone aware of something that blocks us from doing the upgrade? D. On Tue, Dec 21, 2021 at 5:50 PM David Morávek wrote: > Hi Martijn, > > from person experience, most Hadoop users are lagging behind the release > lines by a lot, because upgrading a Hadoop cl

Re: Avoiding Dynamic Classloading for User Code

t; anything else? > > Get Outlook for iOS <https://aka.ms/o0ukef> > ---------- > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 6:39:10 PM > *To:* Lior Liviev > *Cc:* user > *Subject:* Re: Avoiding Dynamic Classloading for User Code >

Re: [DISCUSS] Changing the minimal supported version of Hadoop

Agreed, if we drop the CI for lower versions, there is actually no point of having safeguards as we can't really test for them. Maybe one more thought (it's more of a feeling), I feel that users running really old Hadoop versions are usually slower to adopt (they most likely use what the current H

Re: CVE-2021-44228 - Log4j2 vulnerability

Hi Debraj, we're currently not planning another emergency release as this CVE is not as critical for Flink users as the previous one. However, this patch will be included in all upcoming patch & minor releases. The patch release for the 1.14.x branch is already in progress [1] (it may be bit delay

Re: Avoiding Dynamic Classloading for User Code

your REST API to >load it? >2. If I have my JAR in the folder AND I load same JAR via REST API, >will I run into problems? (class loading strategy is set to parent-first) > > ------ > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 6:

Re: Avoiding Dynamic Classloading for User Code

to >load it? >2. If I have my JAR in the folder AND I load same JAR via REST API, > will I run into problems? (class loading strategy is set to parent-first) > > -- > *From:* David Morávek > *Sent:* Tuesday, December 21, 2021 6:53 PM > *To

Re: Avoiding Dynamic Classloading for User Code

nd I'm using avro 1.10 > ---------- > *From:* David Morávek > *Sent:* Thursday, December 23, 2021 12:37 PM > *To:* Lior Liviev ; user > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > I guess I'

  1   2   >