Hello flinkers,
We will activate namenode HDFS high availability in our cluster, and I
want to know if there is additional configuration for flink ?
We actually use YARN for launching our flink application, and hdfs
filesystem to store the state backend
Thanks
Thomas
Ok, we have all this configuration set up, so it will be fine :-)Thanks for getting response ! Thomas
d Exception) althoug it is included in the jar file of the
installation.
I forked the chef receipt and updated to flink 0.9.1 but the same issue.
Is there another simple script to install flink with hadoop on an ec2
cluster and working s3n filesystem?
Freelancer
on Behalf of Fraunhofer IES
te",Template.class.getResource("/conf/core-site.xml").toString());
GlobalConfiguration.includeConfiguration(configuration);
Any Idea why the class is not included in classpath? Is there another script to
setup flink on ec2 cluster?
When will flink 0.10 be released?
.java:57)
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:146)
... 21 more
--
Viele Grüße
Thomas Götzinger
Freiberuflicher Informatiker
Glockenstraße 2a
D-66882 Hütschenhausen OT Spesbach
Mobil: +49 (0)176 82180714
Privat: +49 (0) 6371 954050
mailto:m...@simplydevelop.de &l
But the default wordcount example in which flink is accessing hadoop runs?
Or is that something different?
Am 09.11.2015 11:54 schrieb "Maximilian Michels" :
> Hi Thomas,
>
> It appears Flink couldn't pick up the Hadoop configuration. Did you
> set the environment v
Hi Brian
Can you give me short summary how to achieve this.
Am 14.12.2015 23:20 schrieb "Brian Chhun" :
> For anyone else looking, I was able to use the s3a filesystem which can
> use IAM role based authentication as provided by the underlying AWS client
> library.
>
> Thanks,
> Brian
>
> On Thu,
cal:2181
> recovery.zookeeper.path.root: /whatever
> state.backend: filesystem
> state.backend.fs.checkpointdir: s3a:///YOUR-BUCKET/checkpoints
> recovery.zookeeper.storageDir: s3a:///YOUR-BUCKET/recovery
>
> That's all I had to do in the Flink side. obvs in the AWS side, I had my
> IAM role setu
x27;s better to use the Sinkclass or HBaseOutputFormat, for better performance
and cleaner code ? (Or equivalent ?)
Thanks,
B.R / Cordialement
Thomas Lamirault
attempts, with the last pending window.
How can I do that ? A simple restart of the application is enough, or do I have
to "clean" the recovery.zookeeper.path.root ?
Thanks !
Thomas Lamirault
es, a lot :-)
Thomas
De : Ufuk Celebi [u...@apache.org]
Envoyé : jeudi 18 février 2016 19:19
À : user@flink.apache.org
Objet : Re: Flink HA
On Thu, Feb 18, 2016 at 6:59 PM, Thomas Lamirault
wrote:
> We are trying flink in HA mode.
Great to hear!
> We s
more
If I run the application in not-HA mode, there is no problem.
What can cause this kind of error ?
Thanks
Thomas
De : Thomas Lamirault
[thomas.lamira...@ericsson.com]Envoyé : vendredi 19 février 2016 09:39À :
user@flink.apache.orgObjet : RE:Flink
?
Thanks
Thomas
De : Thomas Lamirault [thomas.lamira...@ericsson.com]
Envoyé : vendredi 19 février 2016 10:56
À : user@flink.apache.org
Objet : RE:Flink HA
After set this configuration, I have some exceptions :
java.lang.Exception: Could not restore
Hello everyone,
We are facing the same probleme now in our Flink applications, launch using
YARN.
Just want to know if there is any update about this exception ?
Thanks
Thomas
De : ni...@basj.es [ni...@basj.es] de la part de Niels Basjes [ni
Hi Max,
I will try these workaround.
Thanks
Thomas
De : Maximilian Michels [m...@apache.org]
Envoyé : mardi 15 mars 2016 16:51
À : user@flink.apache.org
Cc : Niels Basjes
Objet : Re: Flink job on secure Yarn fails after many hours
Hi Thomas,
Nils (CC
RMQSource seems buggy.
Does my reasoning make sense to you?
Best Regards,
Thomas Eckestad
Hi Andrey,
Thank you for your response. I created
https://issues.apache.org/jira/browse/FLINK-20244.
Best Regards,
Thomas
From: Andrey Zagrebin
Sent: Thursday, November 19, 2020 8:41
To: Thomas Eckestad
Cc: user@flink.apache.org
Subject: Re: Strange behaviour
://github.com/apache/flink-docker/blob/d3416e720377e9b4c07a2d0f4591965264ac74c5/Dockerfile-debian.template#L19
Thanks,
Thomas
On Tue, Mar 10, 2020 at 12:26 PM Andrey Zagrebin
wrote:
> Hi All,
>
> Thanks a lot for the feedback!
>
> *@Yangze Guo*
>
> - Regarding the flink_docker_utils#i
-->
On Mon, Mar 16, 2020 at 1:58 AM Andrey Zagrebin
wrote:
> Thanks for the further feedback Thomas and Yangze.
>
> > A generic, dynamic configuration mechanism based on environment variables
> is essential and it is already supported via envsubst and an environment
> varia
I met this issue three months ago. Finally, we got the conclusion that is
Prometheus push gateway can not handle high throughout metric data. But we
solved the issue via service discovery. We changed the Prometheus metric
reporter code, adding the registration logic, so the job can expose the ho
I’m wondering that why you use a beta feature for production. Why not push the
latest state into down sink like redis or hbase with Apache phoenix .
From: Annemarie Burger
Sent: Monday, May 18, 2020 11:19:23 PM
To: user@flink.apache.org
Subject: Re: Incremental
Hi,
Actually, seems like spark dynamic allocation saves more resources in that
case.
From: Arvid Heise
Sent: Monday, May 18, 2020 11:15:09 PM
To: Congxian Qiu
Cc: Sergii Mikhtoniuk ; user
Subject: Re: Process available data and stop with savepoint
Hi Sergii,
Hi guys,
I'm using hive to store kafka topic metadata as follows::
CREATE TABLE orders (
user_idBIGINT,
productSTRING,
order_time TIMESTAMP(3),
WATERMARK FOR order_time AS order_time - '5' SECONDS
) WITH (
'connector.type' = 'kafka',
Hi Jingsong,
Cool, Thanks for your reply.
Best wishes.
From: Jingsong Li
Sent: Tuesday, May 19, 2020 10:46
To: Thomas Huang
Cc: Flink
Subject: Re: Is it possible to change 'connector.startup-mode' option in the
flink job
Hi Thomas,
Good to hea
Hi Flink Community,
Currently, I'm using yarn-cluster mode to submit flink job on yarn, and I
haven't set high availability configuration (zookeeper), but set restart
strategy:
env.getConfig.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, 3000))
the attempt time is 10 and the wait
@Yun Tang<mailto:myas...@live.com>，Thanks.
From: Yun Tang
Sent: Monday, June 15, 2020 11:30
To: Thomas Huang ; Flink
Subject: Re: The Flink job recovered with wrong checkpoint state.
Hi Thomas
The answer is yes. Without high availability, once the job m
explains why it is
deprecated and why usage of it is discouraged, or?
Thanks,
Thomas
Thomas Eckestad
Systems Engineer
Road Perception
NIRA Dynamics AB
Wallenbergs gata 4
58330 Link?ping, Sweden
Mobile: +46 738 453 937
thomas.eckes...@niradynamics.se
www.niradynamics.se
OK, thank you for validating my thoughts =) I created
https://issues.apache.org/jira/browse/FLINK-21666#
Thanks,
Thomas
On 3 Mar 2021, at 22:02, Chesnay Schepler
mailto:ches...@apache.org>> wrote:
Your understanding of cancel vs stop(-with-savepoint) is correct.
I agree that we should
oint"}`.
I'm using Flink 1.11.2 Commit ID: DeadD0d0.
Any suggestions on how I can debug this?
Another question is, given the response "request-id", which endpoint should
I query to get the status of the request? Most importantly, where can I get
the expected savepoint location?
Thanks.
Thomas
t location?
Thanks.
Thomas
On Fri, Jun 4, 2021 at 2:31 AM Yun Gao wrote:
> Hi Thomas,
>
> I think you are right that the CLI is also using the same rest API
> underlying, and since
> the response of the rest API is ok and the savepoint is triggered
> successfully, I reckon
> t
Hi Yun,
Thanks for the tips. Yes, I do see some exceptions as copied below. I'm not
quite sure what they mean though. Any hints?
Thanks.
Thomas
```
2021-06-05 10:02:51
java.util.concurrent.ExecutionException:
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
One thing I noticed is that if I set drain = true, the job could be stopped
correctly. Maybe that's because I'm using a Parquet file sink which is a
bulk-encoded format and only writes to disk during checkpoints?
Thomas
On Sat, Jun 5, 2021 at 10:06 AM Thomas Wang wrote:
> Hi Yu
This is actually a very simple job that reads from Kafka and writes to S3
using the StreamingFileSink w/ Parquet format. I'm all using Flink's API
and nothing custom.
Thomas
On Sun, Jun 6, 2021 at 6:43 PM Yun Gao wrote:
> Hi Thoms,
>
> Very thanks for reporting the exceptio
d the job to run unstably, but
I'm not quite sure. Any other potential causes here? How should I debug
from here if resources are not the issue? Is there a way to detect memory
leaks?
Thanks in advance!
Thomas
Thanks everyone. I'm using Flink on EMR. I just updated to EMR 6.3 which
uses Flink 1.12.1. I will report back whether this resolves the issue.
Thomas
On Wed, Jun 9, 2021 at 11:15 PM Yun Gao wrote:
> Very thanks Kezhu for the catch, it also looks to me the same issue as
>
semantics?
Thanks.
Thomas
Hi,
We recently upgraded our Flink version from 1.11.2 to 1.12.1 and one of our
jobs that used to run ok, now sees the following error. This error doesn't
seem to be related to any user code. Can someone help me take a look?
Thanks.
Thomas
java.lang.NoSuchMethod
urn record;
}
Although the app works just fine, I feel the whole process is unnecessarily
convoluted. I would appreciate any guideline on this front. Am I doing this
job right?
Thanks.
Thomas
Sounds good. Thanks.
Thomas
On Wed, Jun 23, 2021 at 11:59 AM Seth Wiesman wrote:
> It will just work as long as you enable partition discovery.
>
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/#partition-discovery
>
> On Tue,
cause in this case and what's the
recommended way to avoid this going forward?
Thanks.
Thomas
this
job was using like 99% of the entire cluster. However in that case
shouldn't Yarn wait for containers to become available? I'm not quite sure
how Flink would behave in this case. Could someone provide some insights
here? Thanks.
Thomas
On Sun, Jun 27, 2021 at 4:24 PM Thomas Wan
I need it to connect to Pulsar and stream from Pulsar. I could not find any
code on how to connect to Pulsar. I've done the WordCount, but I need
sample code for how to connect to Pulsar.
Thomas J. Raef
Founder, WeWatchYourWebsite.com
http://wewatchyourwebsite.com
tr...@wewatchyourwebsit
Thanks, I'll check them out.
Thomas J. Raef
Founder, WeWatchYourWebsite.com
http://wewatchyourwebsite.com
tr...@wewatchyourwebsite.com
LinkedIn <https://www.linkedin.com/in/thomas-raef-74b93a14/>
Facebook <https://www.facebook.com/WeWatchYourWebsite>
On Mon, Jun 28, 2021
Thanks Piotr. This is helpful.
Thomas
On Mon, Jun 28, 2021 at 8:29 AM Piotr Nowojski wrote:
> Hi,
>
> You should still be able to get the Flink logs via:
>
> > yarn logs -applicationId application_1623861596410_0010
>
> And it should give you more answers about what has
Thanks Matthias. Could you advise how I can confirm this in my environment?
Thomas
On Tue, Jun 29, 2021 at 1:41 AM Matthias Pohl
wrote:
> Hi Rommel, Hi Thomas,
> Apache Parquet was bumped from 1.10.0 to 1.11.1 for Flink 1.12 in
> FLINK-19137 [1]. The error you're seeing l
Hello,
I am having trouble with a Flink job which is configured using a RocksDB state
backend.
Tl;dr: How can I debug the key serialization for RocksDB MapState for a
deployed Flink job?
Details:
When I “put” a key/value pair into a MapState, and then later try to “get”
using a key which has
FieldSerializer was not producing byte-wise equal
serializations.
Thanks for the prompt response!
Tom
From: Yuval Itzchakov
Date: Wednesday, June 30, 2021 at 12:56 PM
To: Thomas Breloff
Cc: user@flink.apache.org
Subject: Re: RocksDB MapState debugging key serialization
Here is what the
Is there any activity on FLIP-27 that would make it relevant for 1.10
release?
Thanks Gary for the update, it provides excellent visibility on current
activity and what we can expect with the release.
On Fri, Nov 1, 2019 at 1:52 PM Steven Wu wrote:
> Gary, FLIP-27 seems to get omitted in the
look and help figuring out the path forward?
Thanks,
Thomas
[1]
https://lists.apache.org/thread.html/c7336f2594c482de0580ecf2472a6b3e5a0fe2b7dd9258b159f2@%3Cdev.flink.apache.org%3E
[2] https://issues.apache.org/jira/browse/FLINK-12675
On Mon, Nov 4, 2019 at 5:48 AM Becket Qin wrote:
>
There is a related use case (not specific to HDFS) that I came across:
It would be nice if the jar upload endpoint could accept the URL of a jar
file as alternative to the jar file itself. Such URL could point to an
artifactory or distributed file system.
Thomas
On Mon, Nov 18, 2019 at 7:40 PM
related
https://issues.apache.org/jira/browse/FLINK-13792
Regards,
Julian.
On Mon, 25 Nov 2019 15:25:14 +0530 Caizhi Weng
wrote
Hi,
As far as I know, Flink currently doesn't have a built-in throttling function.
You can write your own user-defined function to achieve
,
"s3:GetObject",
]
Effect = "Allow"
Resource = "arn:aws:s3::://*"
},
Thomas
e:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
Could someone help me confirm this bug and maybe have it fixed some time?
Thanks.
Thomas
Thanks Ingo. Adding the following setting worked.
fs.s3a.aws.credentials.provider:
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Thomas
On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote:
> Hi Thomas,
>
> I think you might be looking for this:
> https://github.com/apach
Ingo,
It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
You must specify a value for roleArn and roleSessionName". I assume I would
also need to specify that through the configuration file. Could you suggest
the key for this configuration? Thanks.
Thomas
/...
fs.s3a.aws.credentials.provider.role.sessionName: ...
However, for some reason, I'm still getting the same error. Please help!
Thanks.
Thomas
On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang wrote:
> Ingo,
>
> It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
> You must
nance to arrive at a
working Flink setup. This may mean we prioritize usability over
developer convenience, if that's in the best interest of Flink as a
whole.
Thanks,
Thomas
On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler wrote:
>
> Generally, the issues are reproducibility and contr
tor ecosystem.
As for keeping connectors within Apache Flink: I prefer that as the
path forward for "essential" connectors like FileSource, KafkaSource,
... And we can still achieve a more flexible and faster release cycle.
Thanks,
Thomas
On Wed, Oct 20, 2021 at 3:32 AM Jark Wu wrote:
>
Thanks for making the release happen!
On Thu, Oct 21, 2021 at 5:54 AM Leonard Xu wrote:
>
> Thanks to Chesnay & Martijn and everyone who made this release happen.
>
>
> > 在 2021年10月21日，20:08，Martijn Visser 写道：
> >
> > Thank you Chesnay, Leonard and all contributors!
> >
> > On Thu, 21 Oct 2021 a
e and found a way to work around this
limitation? It might be possible to work with a configuration
singleton that initializes from a file in a well known location, but
that depends on the deployment environment and doesn't play nice with
testing.
Thanks,
Thomas
[1] https://lists.apache.
constructor and no alternative way to pass a dependency.
Thanks,
Thomas
On 2021/11/10 13:23:21 Seth Wiesman wrote:
> Yes I did, thanks for sending it back :) Copying my previous reply for the
> ML:
>
> Hey Thomas,
> >
> > You are correct that there is no way to inject
ialization so that it can be restored without
external dependency. Unfortunately that also cannot work, because
savepoint creation will first write the serializers before they are
used to write the records.
Thomas
On Thu, Nov 11, 2021 at 12:37 AM Arvid Heise wrote:
>
> Thanks for clarifyin
Hello community,
My team is developing an application using Pyflink. We are using the
Datastream API. Basically, we read from a kafka topic, do some maps, and
write on another kafka topic. One restriction about it is the first map,
that has to be serialized and with parallelism equals to one. This
ready used, then let it be in charge of the
task manager resource allocation? If so, what scenario was the native
k8s integration originally intended for?
Thanks,
Thomas
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#deployment-
n mileage (certainly applies to
[2]). We could combine the best of these ideas into a unified
implementation as part of Flink itself as starting point.
Cheers,
Thomas
[1] https://github.com/wangyang0918/flink-native-k8s-operator
[2] https://github.com/lyft/flinkk8soperator
[3] https://lis
ling with active resource management
then perhaps it is worth to support it through the operator.
Previously mentioned operators all rely on the standalone model.
Cheers,
Thomas
On Wed, Jan 12, 2022 at 3:21 AM Konstantin Knauf wrote:
>
> cc dev@
>
> Hi Thomas, Hi everyone,
>
> Tha
allocation of the cluster resources prior to taking down the previous
job during upgrade when the goal is optimization for availability.
Thanks,
Thomas
[1] https://github.com/lyft/flinkk8soperator
On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf wrote:
>
> Hi everyone,
>
> I wou
Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351075&projectId=12315522
We would like to thank all contributors of the Apache Flink community
who made this release possible!
Regards,
Martijn & Thomas
Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344315
We would like to thank all contributors of the Apache Flink community who
made this release possible!
Regards,
Thomas
+1 for trimming the size by default and offering the fat distribution as
alternative download
On Wed, Jan 23, 2019 at 8:35 AM Till Rohrmann wrote:
> Ufuk's proposal (having a lean default release and a user convenience
> tarball) sounds good to me. That way advanced users won't be bothered by a
7.1-bin-scala_2.11.tgz
(I have also tried 1.6.3 and 1.7.0, no difference in behavior).
* docker --version -> Docker version 1.13.1
* uname -a -> Linux SEOSTL0069.SEC.INTRA 4.20.4-200.fc29.x86_64 #1 SMP Wed Jan
23 16:11:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Thank you,
Thomas Eckestad
Hi again,
when removing Spring Boot from the application it works.
I would really like to mix Spring Boot and Flink. It does work with Spring Boot
when submitting jobs to a session cluster, as stated before.
/Thomas
From: Thomas Eckestad
Sent: Friday, February
between Flink bootstrap-code and the Flink job?
/Thomas
From: Chesnay Schepler
Sent: Sunday, February 10, 2019 10:30:54 AM
To: Thomas Eckestad; user@flink.apache.org
Subject: Re: Running single Flink job in a job cluster, problem starting
JobManager
I'm afra
Thanks everyone!
It is exciting to be part of such an enthusiastic community that works on
great tech. Looking forward to continue to help on the journey!
Thomas
On Tue, Feb 12, 2019 at 6:13 PM Kurt Young wrote:
> Congrats Thomas!
>
> Best,
> Kurt
>
>
> On Wed, Feb
th similar setups? Does this sound like a
viable thing to do or is it just not meant to be done this way?
Thanks,
Thomas
I'm taking an existing REST based microservice application and moving all
of the logic into Flink DataStreams.
Is there an easy way to get a request/response from a Flink DataStream so I
can 'call' into it from a REST service? For example, something similar to
this Kafka streams example that use
tefun-docs-stable/
> >>
> >> Regards,
> >> Roman
> >>
> >> On Fri, Apr 8, 2022 at 6:56 PM Austin Cawley-Edwards
> >> wrote:
> >> >
> >> > Hi Jason,
> >> >
> >> > No, there is no HTTP source
Thank you for reporting the issue. Mason has already identified the root
cause and the JIRA is now assigned to him:
https://issues.apache.org/jira/browse/FLINK-27479
Thomas
On Tue, May 3, 2022 at 4:02 AM Martijn Visser
wrote:
> I'm looping in @Thomas Weise since he has expertise
cible consistently? Can you
capture the debug log of the enumerator?
Thanks,
Thomas
On Wed, May 4, 2022 at 10:05 AM Martijn Visser wrote:
>
> Hi Kevin,
>
> I'm hoping that @Thomas Weise could help with the issue regarding the
> recovery from the savepoint.
>
> Best regards,
>
One more question: Are you changing the parallelism when resuming from
savepoint?
On Sun, May 8, 2022 at 4:05 PM Thomas Weise wrote:
>
> Hi Kevin,
>
> Unfortunately I did not find a way to test the savepoint scenario with
> the MiniCluster. Savepoints are not supported in th
Hi Mason,
I added mason6345 to the Flink confluence space, you should be able to
add a FLIP now.
Looking forward to the contribution!
Thomas
On Thu, Jun 30, 2022 at 9:25 AM Martijn Visser wrote:
>
> Hi Mason,
>
> I'm sure there's a PMC (*hint*) out there who can grant yo
ector, but it
looks like I cannot use Tuple as the key type here. Is there any way I can
work around this as the rest of the logic of my application is the same.
Thank you!
public class SimpleRecordKeySelector implements
KeySelector
Thomas
I didn't copy the exact error message, but basically the idea of the error
message is that I cannot use the abstract class Tuple and instead, I should
use Tuple1, Tuple2 and etc.
Thomas
On Sun, Jul 10, 2022 at 12:47 PM Hemanga Borah
wrote:
> What error do you see?
>
> On Sun, Jul
Hemanga, the issue is that the number of keys is unknown at the compile
time.
I ended up using yidan's suggestion and serialized all keys into a string.
Thanks for the suggestion.
Thomas
On Sun, Jul 10, 2022 at 7:05 PM yidan zhao wrote:
> You can use string, and serialize all keys to
on FLink 1.13.6.
Thanks,
Thomas
Thomas Eckestad
Systems Engineer
Development RSI
NIRA Dynamics AB
Wallenbergs gata 4
58330 Link?ping, Sweden
Mobile: +46 701 447 279
thomas.eckes...@niradynamics.se
www.niradynamics.se
namespace). So it could have
been done that way I guess.
I did not find any other option for migrating the state due to a
namespace/package name change. Performing text replace with sed does not work.
On 7 Feb 2023, at 12:03, Thomas Eckestad
mailto:thomas.eckes...@niradynamics.se>> wrot
Is the intention to bump the Flink major version and only support Java 17+?
If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser
wrote:
> Scala 2.12.7 doesn't compile on Java 17, see
> https://issues.apache.org/jira/b
meException: There is no
PredictOperation defined for org.apache.flink.ml.classification.SVM
which takes a DataSet[org.apache.flink.ml.common.LabeledVector] as
input.
Any idea ?
Thanks
Thomas
Hi,
No problem I'm going to create a JIRA.
Regards
Thomas
2016-10-17 21:34 GMT+02:00 Theodore Vasiloudis <
theodoros.vasilou...@gmail.com>:
> That is my bad, I must have been testing against a private branch when
> writing the guide, the SVM as it stands only has a pred
Hello,
I'm trying to assign a unique (and deterministic) ID to a globally sorted
DataSet.
Given a DataSet of String, I can compute the frequency of each label as
follows:
val env = ExecutionEnvironment.getExecutionEnvironment
val data =
env.fromCollection(List("a","b","c","a","a","d","a","a","
.partitionByRange(1)
.sortPartition(1, Order.DESCENDING)
Then I want the most frequent label to be ID 0 and so on *in a decreasing
order*. My idea was to use zipWithIndex.
val result = mapping.zipWithIndex
But this does not guarantee that the global order will be preserved right ?
What can I do to get such mapping ?
Thanks
Regards
Thomas
Hi,
instead of using the RequestHedgingRMFailoverProxyProvider you could try
to use the org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.
You can configure this in the YARN configs =>
yarn.client.failover-proxy-provider.
Kind regards,
Thomas
On 11/20/2017 01:53 PM, Nishu wr
Hi everybody,
I would like to test a kafka / flink process in scala. I would like to proceed
as in the integration testing documentation :
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html#integration-testing
with Kafka as source and sink.
For example, I have
msg.toUpperCase
}
outputStream.writeAsText("test.csv", WriteMode.OVERWRITE)
env.execute()
consumeFirstStringMessageFrom("output-topic") shouldEqual
SimpleFlinkKafkaTest.expectedString
}
}
}
}
```
The flink process si running but nothing happen. I try ot write into a text
file to see any output but there is nothing into the file.
Any idea ? Does anybody use this library to test a Flink Job using Kafka ?
Thanks in advance,
Thomas
Hi,
withRunningKafka launch a kafka broker. This is one of the advantage of this
library.
I test to consume / produce messages with kafka command line, and it seems
alright.
Thanks
De : Ted Yu [mailto:yuzhih...@gmail.com]
Envoyé : jeudi 19 avril 2018 15:28
À : Chauvet, Thomas
Objet : Re
needs to create a kafka topic for managing
heartbeat. In theory, If the topic does not exist it will be automaitcally
created
But my kafka server does not authorize this auto creation ... so i create
this topic mannually with this name :
`__flink-heartbeat.postgres_cdc_source`
i also add this dbzPr
thanks Hongshun for your response !
Le ven. 17 mai 2024 à 07:51, Hongshun Wang a
écrit :
> Hi Thomas,
>
> In debezium dos says: For the connector to detect and process events from
> a heartbeat table, you must add the table to the PostgreSQL publication
> specified by the p
Hello everyone!
any chance of this bug being fixed?
https://issues.apache.org/jira/browse/FLINK-35387
Thanks
Le ven. 17 mai 2024 à 07:51, Hongshun Wang a
écrit :
> Hi Thomas,
>
> In debezium dos says: For the connector to detect and process events from
> a heartbeat table, you
Please unsubscribe.
--
Thanks and Regards
Nikita Thomas
99 matches
Mail list logo
