No, this is not currently supported, but we plan to include this in 1.5 .
On 07.11.2017 20:01, vijayakumar palaniappan wrote:
Hi,
Is there a REST endpoint for triggering savepoint without cancelling
the job?
--
Thanks,
-Vijay
Hi,
Is there a REST endpoint for triggering savepoint without cancelling the
job?
--
Thanks,
-Vijay
Hi all,
We recently implemented a feature in our streaming flink job in which we have a
AvroParquetWriter which we build every time the overridden “write” method from
org.apache.flink.streaming.connectors.fs.Writer gets called. We had to do this
because the schema of each record is potentially
Thanks Fabian.
I am seeing thia consistently and can definitely use some help. I have plenty
of graphana views I can share if that helps :)
Sent from Yahoo Mail on Android
On Tue, Nov 7, 2017 at 3:54 AM, Fabian Hueske wrote: Hi
Ashish,
Gordon (in CC) might be able to help you.
Cheers, Fab
I’ve used the following simple script to capture Flink metrics by running:
python -u ./statsd_server.py 9020 > statsd_server.log
>>> flink-conf.yaml
metrics.reporters: statsd_reporter
metrics.reporter.statsd_reporter.class:
org.apache.flink.metrics.statsd.StatsDReporter
metrics.reporter.
Hi Regina,
the user code is uploaded once to the `JobManager` and then downloaded from
each `TaskManager` once when it first receives the command to execute the
first task of your job.
As Chesnay said there is no fundamental limitation to the size of the Flink
job. However, it might be the case t
Hi Federico,
For your given input and pattern there should (and there are) only two
timeouted patterns:
5> Left(Map(start -> List(Event(100,2017-11-05T03:56:02
5> Left(Map(start -> List(Event(100,2017-11-05T06:00:02
It is because in your patterns say the next event after events with valu
Hi XiangWei,
it is actually not intended to get access to the ExecutionGraph, because it
is a runtime component which does not make much sense to exist outside of
the JobManager. The RequestJob message is only a hack to make the
ExecutionGraph accessible to another actor running in the same ActorS
Hi everyone,
data Artisans is running a second annual Apache Flink user survey [1] in
order to understand Flink usage and the needs of the community. This survey
will help to shape the Flink roadmap and make Flink the best that it can be
for users.
We'll publish a report with a summary of finding
Hi,
I am using flink 2.1.0 version and protobuf-java 2.6.1 version.
I am getting below exception for protobuf generated class. I have included
jar which is having that class.
Can you please help me to check it.
org.apache.beam.sdk.util.UserCodeException:
org.apache.flink.streaming.runtime.
I agree with Ufuk, it would be helpful to know what stateful operations are in
the jobs (including windowing).
> On 7. Nov 2017, at 14:53, Ufuk Celebi wrote:
>
> Do you use any windowing? If yes, could you please share that code? If
> there is no stateful operation at all, it's strange where th
Do you use any windowing? If yes, could you please share that code? If
there is no stateful operation at all, it's strange where the list
state instances are coming from.
On Tue, Nov 7, 2017 at 2:35 PM, ebru wrote:
> Hi Ufuk,
>
> We don’t explicitly define any state descriptor. We only use map an
hi Till,
Sorry,I've made a mistake,i used
*StandaloneClusterClient*#*getJobManagerGateway
*to get *ActorGateway *to communicate with *JobManager *instead of using
*JobMasterGateway*.
Below is the code i executed for getting ExecuteGraph of a Job.
val flinkConfig = new Configuration()
Hi Ufuk,
We don’t explicitly define any state descriptor. We only use map and filters
operator. We thought that gc handle clearing the flink’s internal states.
So how can we manage the memory if it is always increasing?
- Ebru
> On 7 Nov 2017, at 16:23, Ufuk Celebi wrote:
>
> Hey Ebru, the me
Have you tried removing the "cygdrive" portion from the path?
Something along the lines of
“state.backend.fs.checkpointdir: file:///Y:/flink-checkpoint-dir
state.checkpoints.dir: Y:/flink-checkpoints
state.backend: filesystem
high-availability.storageDir: file:///Y:/flink-recovery
"
On 07.11.
Dear Ufuk,
Sorry, I still can’t get that work, I have double checked all the nodes can
access the mapped drive Y.
Here is my state backend configuration, and it writes files to local C
drive.
“state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir
state.checkpoints.dir: /cygdrive
hmm. While there is /technically/ no guarantee that
notifyCheckpointComplete is called, it virtually always is,
especially in local setups.
Is it possible for you to share more code (or all of it)? (you can also
send it to me directly)
On 07.11.2017 11:58, Rinat wrote:
Yes, but *notifyCheckpo
Yes, but notifyCheckpointComplete callback doesn’t called on await completion,
I do the same, as in specified test template :
ActorGateway jobManager = (ActorGateway)
Await.result(cluster.leaderGateway().future(), DEADLINE.timeLeft());
Future savepointResultFuture = jobManager.a
Do you verify that savepointResult is a
JobManagerMessages.TriggerSavepointSuccess? It could also be
JobManagerMessages.TriggerSavepointFailure. (instanceof check)
On 02.11.2017 19:11, Rinat wrote:
Chesnay, thanks for your reply, it was very helpful, but I took logic
from this test template an
Hey Jordan,
yeah, that should just work. Check out the state backend configuration
here:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html
– Ufuk
On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan wrote:
> Dear Ufuk,
>
> Thank you for your reply.
>
> All the c
Dear Ufuk,
Thank you for your reply.
All the cluster Flink servers are able to access network drive, and it mapped
as drive Y in all nodes.
Do I need to provide more information?
Thanks,
Jordan
> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi wrote:
>
> As answered by David on SO, the files need to
As answered by David on SO, the files need to be accessible by all
nodes. In your setup this seems not to be the case, therefore it won't
work. You need a distributed file system (e.g. NFS or HDFS) or object
store (e.g. S3) that is accessible from all nodes.
– Ufuk
On Tue, Nov 7, 2017 at 3:34 AM
Hey Ebru,
let me pull in Aljoscha (CC'd) who might have an idea what's causing this.
Since multiple jobs are running, it will be hard to understand to
which job the state descriptors from the heap snapshot belong to.
- Is it possible to isolate the problem and reproduce the behaviour
with only a
Hey Frederico,
let me pull in Dawid (cc'd) who works on CEP. He can probably clarify
the expected behaviour here.
Best,
Ufuk
On Mon, Nov 6, 2017 at 12:06 PM, Federico D'Ambrosio
wrote:
> Hi everyone,
>
> I wanted to ask if FlinkCEP in the following scenario is working as it
> should, or I hav
Hi,
We are using Flink 1.3.1 in production, we have one job manager and 3
task managers in standalone mode. Recently, we've noticed that we have
memory related problems. We use docker container to serve Flink cluster.
We have 300 slots and 20 jobs are running with parallelism of 10. Also
the
Hi XiangWei,
how do you use the JobMasterGateway with the actor message RequestJob? The
JobMasterGateway is a Java interface and does not represent an ActorCell to
which you can send actor messages. Instead you should call
JobMasterGateway#requestArchivedExecutionGraph.
Cheers,
Till
On Tue, No
Hi XiangWei,
I don't think this is a public interface, but Till (in CC) might know
better.
Best,
Fabian
2017-11-06 3:27 GMT+01:00 XiangWei Huang :
> Hi Flink users,
> Flink Jobmanager throw a NotSerializableException when i used
> JobMasterGateway to get ExecutionGraph of a specific job with
>
Hi Ashish,
Gordon (in CC) might be able to help you.
Cheers, Fabian
2017-11-05 16:24 GMT+01:00 Ashish Pokharel :
> All,
>
> I am starting to notice a strange behavior in a particular streaming app.
> I initially thought it was a Producer issue as I was seeing timeout
> exceptions (records expir
28 matches
Mail list logo
