I'm trying to use the topic discovery feature of the Kafka Connector. The
problem I'm having is that Kafka Consumers fail to start if there are no topics
matching the topic regex when they start up. Is this intended behavior? Is
there some other property I could set to just continue discovery un
I think you’re looking for Broadcast State. Here’s a detailed guide.
https://flink.apache.org/2019/06/26/broadcast-state.html
From: Soheil Pourbafrani [mailto:soheil.i...@gmail.com]
Sent: Friday, January 17, 2020 6:50 AM
To: user
Subject: EXT :Flink solution for having shared variable between ta
that points to the taskmanager service isn’t something I put in
Flink’s config files anywhere, and I don’t think containers should be referring
to themselves by that name.
From: Yang Wang [mailto:danrtsey...@gmail.com]
Sent: Sunday, December 22, 2019 7:15 PM
To: Martin, Nick J [US] (IS)
Cc: user
I'm running Flink 1.7.2 in a Docker swarm. Intermittently, new task managers
will fail to resolve their own host names when starting up. In the log I see
"no hostname could be resolved" messages coming from TaskManagerLocation. The
webUI on the jobmanager shows the taskmanagers as are associated
, October 18, 2019 6:42 AM
To: Martin, Nick J [US] (IS)
Cc: Ravi Bhushan Ratnakar ; user
Subject: Re: EXT :Re: Jar Uploads in High Availability (Flink 1.7.2)
Hi Martin,
Flink's web UI based job submission is not well suited to be run behind a load
balancer at the moment. The problem is that th
From: Ravi Bhushan Ratnakar [mailto:ravibhushanratna...@gmail.com]
Sent: Tuesday, October 15, 2019 10:35 PM
To: Martin, Nick J [US] (IS)
Cc: user
Subject: EXT :Re: Jar Uploads in High Availability (Flink 1.7.2)
Hi,
i was also experiencing with the similar behavior. I adopted following approach
I'm seeing that when I upload a jar through the rest API, it looks like only
the Jobmanager that received the upload request is aware of the newly uploaded
jar. That worked fine for me in older versions where all clients were
redirected to connect to the leader, but now that each Jobmanager acce
Jobmanager.sh takes an optional argument for the hostname to bind to, and
start-cluster uses it. If you leave it blank it, the script will use whatever
is in flink-conf.yaml (localhost is the default value that ships with flink).
The dockerized version of flink runs pretty much the way you’re tr
Env.ssh.opts is the literal argument string to ssh as you would enter it on the
command line. Take a look at TMSlaves() in config.sh to see exactly how it’s
being used.
From: John Smith [mailto:java.dev@gmail.com]
Sent: Tuesday, June 11, 2019 12:30 PM
To: user
Subject: EXT :How to config us
AFAIK, there are no granular permissions like that built into Flink. Limiting
access to the REST API seems like a good place to start. The web UI uses the
API, but controlling it there means you’re locking down all means of access.
The designers of the API were disciplined about what HTTP verbs
Seye, are you running Flink and Zookeeper in Docker? I’ve had problems with
Jobmanagers not resolving the hostnames for Zookeeper when starting a stack on
Docker.
From: Till Rohrmann [mailto:trohrm...@apache.org]
Sent: Monday, March 04, 2019 7:02 AM
To: Seye Jin
Cc: user
Subject: EXT :Re: Flin
Yeah, that’s expected/known. Watermarks for the empty partition don’t advance,
so the window in your window function never closes.
There’s a ticket open to fix it
(https://issues.apache.org/jira/browse/FLINK-5479) for the kafka connector, but
in general any time one parallel instance of a sourc
Does that mean that the full set of fs.s3a.<…> configs in core-site.xml will be
picked up from flink-conf.yaml by flink? Or only certain configs involved with
authentication?
From: Till Rohrmann [mailto:trohrm...@apache.org]
Sent: Wednesday, January 16, 2019 3:43 AM
To: Vinay Patil
Cc: Kostas K
I'm running on Flink 1.7.0 trying to use the StreamingFileSink with an S3A URI.
What I'm seeing is that whenever the RollingPolicy determines that it's time to
roll to a new part file, the whole Sink just hangs, and the in progress
MultiPart Upload never gets completed. I've looked at the traffi
Yeah, I figured that part out. I’ve tried to make it work with 2.7 and 2.8, and
it looks like the prebuilt jars have actually moved to Hadoop 3
From: Paul Lam [mailto:paullin3...@gmail.com]
Sent: Tuesday, December 18, 2018 7:08 PM
To: Martin, Nick
Cc: user@flink.apache.org
Subject: EXT :Re
I'm working on Flink 1.7.0 and I'm trying to use the built in S3 libraries like
readFile('s3://bucket/object') or StreamingFileSink. My storage provider is not
AWS, but they implement the same API. So I need to point the S3 client to a
different address. The Hadoop documentation shows that there
Is value(index-1) stored in Keyed State, or just a local variable inside the
operator?
-Original Message-
From: Nicholas Walton [mailto:nwal...@me.com]
Sent: Monday, July 16, 2018 1:33 PM
To: user@flink.apache.org
Subject: Parallelism and keyed streams
I have a stream of tuples , which
Regarding the ‘partiality’ you mention in option one, wouldn’t you have to give
that up anyway to maintain exactly once processing? Suppose input message A
results in asynchronous queries/futures B and C, and imagine the following
series of events:
1. Your function receives A
2. As
I have a job where I read data from Kafka, do some processing on it, and write
it to a database. When I read data out of Kafka, I put it into an object that
has a String field based on the Kafka message key. The possible values for the
message key are tightly constrained so there are fewer than
Have a set of stateful operators that rely on keyed state. There is substantial
skew between keys (i.e. there will be 100 messages on keys A and B, and 10
messages each on keys C-J), and key selection assignment is dictated by the
needs of my application such that I can't choose keys in a way th
20 matches
Mail list logo
