Any issure or plan on the task?
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Deactive-a-job-like-storm-tp13088p13123.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
Till and Stephan, thanks for your clarification.
@Till One more question, from what I have read about the checkpointing [1],
the list operations don't seem likely to be performed frequently, so
storing state backend on s3 shouldn't have any severe impact on flink
performance. Is this assumption ri
Hi,
Is there a way to retrieve the timestamps that Kafka associates with each
key-value pair within Flink? I would like to be able to use these as values
within my application flow, and defining them before or after Kafka is not
acceptable for the use case due to the latency involved in sending or
Since YARN must support people running multiple Flink clusters, the JobManager
web UI binds to an ephemeral port by default (to prevent port usage conflicts).
Also, the AM (and web UI) may be placed on any of the Yarn nodes. Therefore, if
you wanted to access it directly instead of through the Y
I've been using QueryableStateClient in Flink 1.2 successfully. I have now
upgraded to release-1.3.0-rc0 and QueryableStateClient now requires a
HighAvailabilityServices parameter. The documentation hasn't been updated on
using HighAvailabilityServices so I'm a bit lost on what exactly I should
Got the answer on #2, looks like that will work, still looking for suggestions
on #1.
Thanks
Ankit
From: "Jain, Ankit"
Date: Thursday, May 11, 2017 at 8:26 AM
To: Stephan Ewen , "user@flink.apache.org"
Subject: Re: High Availability on Yarn
Following up further on this.
1) We are usin
Small addition to Till's comment:
In the case where file:// points to a mounted distributed file system (NFS,
MapRFs, ...), then it actually works. The important thing is that the
filesystem where the checkpoints go is replicated (fault tolerant) and
accessible from all nodes.
On Thu, May 11, 201
I am running flink-1.1.4 on Cloudera distributed Hadoop (Yarn).
I am not able to get through JobManager webUI through
http://:8081.
I am able to get to it through
Yarn Running applications ---> application master. My flink-conf.yaml has
jobmanager.web.port: 8081.
Amy I missing something here?
- S
Following up further on this.
1) We are using a long running EMR cluster to submit jobs right now and
as you know EMR hasn’t made Yarn ResourceManager HA.
Is there any way we can use the information put in Zookeeper by Flink Job
Manager to bring the jobs back up on a new EMR cluster if R
Hi,
The same exact jar on the same machine is being deployed just fine in couple of
seconds using the web interface. On the other hand, if I used the command line,
I get:
org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: Couldn't retrieve the JobExecutio
Dominique:
Which hadoop release are you using ?
Please pastebin the classpath.
Cheers
On Thu, May 11, 2017 at 7:27 AM, Till Rohrmann wrote:
> Hi Dominique,
>
> I’m not exactly sure but this looks more like a Hadoop or a Hadoop
> configuration problem to me. Could it be that the Hadoop version
Hi Dominique,
I’m not exactly sure but this looks more like a Hadoop or a Hadoop
configuration problem to me. Could it be that the Hadoop version you’re
running does not support the specification of multiple KMS servers via
kms://ht...@lfrarxxx1.srv.company;lfrarXXX2.srv.company:16000/kms?
Cheers
Dear all,
i got some trouble during the start of Flink in a Yarn-Container based
on Cloudera. I have a start script like that:
sla:/applvg/home/flink/mvp $ cat run.sh
export FLINK_HOME_DIR=/applvg/home/flink/mvp/flink-1.2.0/
export FLINK_JAR_DIR=/applvg/home/flink/mvp/cache
export YARN_CONF_D
Hi to all,
we have a particular use case where we have a tabular dataset on HDFS (e.g.
a CSV) that we want to enrich filling some cells with the content returned
by a query to a reverse index (e.g. solr/elasticsearch).
Since we want to be able to make this process resilient and scalable we
thought
To properly implement stop we have to change some internal orchestration
structure. This is not a trivial task and so far nobody had found time to
work on it. Moreover, the individual sources have to be adapted as well.
Cheers,
Till
On Thu, May 11, 2017 at 4:54 AM, yunfan123
wrote:
> But why Fl
Hi Ayush,
you’re right that RocksDB is the recommend state backend because of the
above-mentioned reasons. In order to make the recovery properly work, you
have to configure a shared directory for the checkpoint data via
state.backend.fs.checkpointdir. You can basically configure any file system
w
Yes, that looks right.
> On 10. May 2017, at 14:56, yunfan123 wrote:
>
> In upstairs example, it seems I should clear the state in onTimer function in
> order to free resource like follows:
> public void onTimer(long l, OnTimerContext onTimerContext,
> Collector> collector) throws Exception {
>
Hello,
I had a few questions regarding checkpoint storage options using
RocksDBStateBackend. In the flink 1.2 documentation, it is the recommended
state
backend due to it's ability to store large states and asynchronous
snapshotting.
For high availabilty it seems HDFS is the recommended store for
Glad to hear it!
Overlay networks (as most container infras use) are tricky and we need to
add some code to make diagnostics of issues easier in those cases...
Stephan
On Wed, May 10, 2017 at 9:30 PM, David Brelloch wrote:
> Stephan,
>
> Thanks for pointing us in the right direction on the di
Hello,
I had a few questions regarding checkpoint storage options using
RocksDBStateBackend. In the flink 1.2 documentation, it is the recommended
state
backend due to it's ability to store large states and asynchronous
snapshotting.
For high availabilty it seems HDFS is the recommended store for
Great! I was just thinking that, in principle, a streaming sink is an
extension of a batch one. Am I wrong?
This would avoid a lot of code duplication and would improve the overall
maintainability..
On Thu, May 11, 2017 at 4:35 AM, wyphao.2007 wrote:
> Hi Flavio, I made a PR for this : https://g
21 matches
Mail list logo
