Re: Flink TCP custom source - secured server socket

ng my application in that paper: http://dx.doi.org/10.1145/3579142.3594292 Best regards, Martin Am 2. Juli 2023 09:13:14 MESZ schrieb "Jan Lukavský" : >Hi, > >in plain TCP socket, there is no 'state' you can return to when restoring from >checkpoint. All you can do is

Re: videos Flink Forward San Francisco 2022

+1 Am 10. Oktober 2022 14:39:28 MESZ schrieb "guenterh.lists" : >really very sad - as far as I know this happens for the first time, attitude >of new Ververica? > >Hopefully immerok may resume the open mentality of data artisans. > >Günter > >On 10.10.22 11:26, Martijn Visser wrote: >> Hi Günter

Re: videos Flink Forward San Francisco 2022

Hey, that's sad. Is it possible for future Flink Forwards to record again and publish all sessions? Best regards Martin Am 10. Oktober 2022 11:26:26 MESZ schrieb Martijn Visser : >Hi Günter, > >I've understood that only the keynotes were recorded and not the other >ses

Re: Flink 1.14.3: Can not access job information from a jobmanager UI

request the job overview from the leading jobmanager.   On 20/01/2022 13:15, Martin wrote:  

Flink 1.14.3: Can not access job information from a jobmanager UI

Hey, I upgraded today my Flink application to Flink 1.14.3. I run it in a HA-Standalone-K8 deployment with 2 JobManagers, so one active and one on standby.As its only a prototyp I make the UI, port 8081 of the JobManager pods, avaiable via NodePort. Already with older Flink version I sometimes got

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

TaskManager which is removed).Is this normal, or did I maybe have a problem with my configuration? BRMartin   Martin schrieb am 08.01.2022 21:35 (GMT +01:00): Hello David, > That is just to avoid falling back to Kryo serialization, which is less effective, but this IMO shouldn't break your app

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

eally think of anything right now, I'd probably need a minimal case to reproduce this locally to get more insight.   D.     On Fri, Jan 7, 2022 at 9:40 PM Martin <mar...@sonicdev.de> wrote: I changed my flink job having an explicit keyBy instead of reinterpretAsKeyedStream.Situatio

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

I changed my flink job having an explicit keyBy instead of reinterpretAsKeyedStream.Situation is still the same, so its no problem with combination of reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode. This time I was able to check the logs of the task managers and it seems to be a seri

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

didnt test if it still cause the issue, but maybe it already shows some misuage of the state API.   BR Martin public class MyRecord2ProcessFunction extends KeyedProcessFunction {     private final OutputTag sideOutputLateData;     private transient UUID applicationId;     private transient ValueState

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

Martin <mar...@sonicdev.de> wrote: Hello David, I have a test setup, where the input is all the time the same.After processing, I check all the output if each sequence number ist just used once. Another output field is a random UUID generated on startup of a Task (in the open-method of the (c)

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

uot;taskid":"68731ce4-5f11-11ec-a5cd-0255ac100047","recordSequenceNumber":5   The percentage is then the number of output records which uses a already given sequence number (for each key1) compared to all output records.   Right now I change the flink job so, that instead of

Re: reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

Hi, typo: "I run that job via Native Kubernetes deployment and use elastic scaling in reactive mode."-> I run it of course via standalone kubernetes deployment, to make elastic scaling possible. BRMartin   mar...@sonicdev.de schrieb am 06.01.2022 21:38 (GMT +01:00): Hi, I have a job where I do a

reinterpretAsKeyedStream and Elastic Scaling in Reactive Mode

Hi, I have a job where I do a keyBy'd process function (a), then on the downstream a normal process function (b) and and then I use reinterpretAsKeyedStream to have yet another keyBy'd process function (c). The last keyed process function use keyed state for a increasing sequence number. I run that

Kafka Connector Topic Discovery

I'm trying to use the topic discovery feature of the Kafka Connector. The problem I'm having is that Kafka Consumers fail to start if there are no topics matching the topic regex when they start up. Is this intended behavior? Is there some other property I could set to just continue discovery un

Re: Deterministic rescale for test

Hi Jaffe, Thanks for your reply, I will try to use a Custom Partioner. Den tor. 14. jan. 2021 kl. 19.39 skrev Jaffe, Julian < julianja...@activision.com>: > Martin, > > > > You can use `.partitionCustom` and provide a partitioner if you want to > control explicitly how

Deterministic rescale for test

nment env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) env.setParallelism(parallelism) val rawStream: DataStream[String] = env.readTextFile(s3Path).setParallelism(1) rawStream.rescale ... best regards -- Martin Frank Hansen

Queryable state on task managers that are not running the job

tion is to size the cluster appropriately so that there are no idle task managers. I was wondering if there was a better solution or if this could be handled better in the future? Thanks in advance. Kind regards, Martin

Re: test (harness and minicluster)

Hi Till, Thanks for your answer! I will try the approach you suggested, hope I can make it work. Best regards martin Den ons. 2. dec. 2020 kl. 17.03 skrev Till Rohrmann : > Hi Martin, > > In general, Flink's MiniCluster should be able to run every complete Flink > JobGraph. How

test (harness and minicluster)

, so does anyone know if it will be supported in the future? (Or if it is supported how should I use it?) Furthermore does anyone have some good ideas for a test-setup for processWindowFunctions? best regards Martin Frank Hansen Data Engineer

Re: valuestate(descriptor) using a custom caseclass

Another note, the case class in hand has about 40 fields in it, is there a maximum limit for the number of fields? best regards Den fre. 18. sep. 2020 kl. 13.05 skrev Martin Frank Hansen < m...@berlingskemedia.dk>: > Hi Dawid, > > Thanks for your reply, much appreciated. > &g

Re: valuestate(descriptor) using a custom caseclass

) => Util.getPageviewBasedMetricsGroup(e)) .timeWindow(Time.minutes(1)) .process(new MetricsProcessFunction) .addSink(target1Min) Could the problem be in the open-method? best regards Den fre. 18. sep. 2020 kl. 12.30 skrev Dawid Wysakowicz < dwysakow...@apache.org>: > Hi Mart

valuestate(descriptor) using a custom caseclass

s(1)) .process(new MetricsProcessFunction) .addSink(target1Min) // execute program executionEnv.execute("Count pageview-based metrics") } } -- Martin Frank Hansen Data Engineer Digital Service M: +45 25 57 14 18 E: m...@berlingskemedia.dk

Re: restoring from externalized incremental rocksdb checkpoint?

/projects/flink/flink-docs-release-1.11/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/state/checkpoints.html#directory-structure > Best, > Congxian > > > Jeffrey Martin 于2020年9月15日周二 下午1

restoring from externalized incremental rocksdb checkpoint?

there's no metadata file in HDFS in the various directories under "$checkpointsDir/snapshots/$jobID", Thanks, Jeff Martin

Re: Writing to SQL server

Well got it working. The varchars in the database were set too small. Thanks for your help! Den fre. 22. maj 2020 kl. 13.30 skrev Martin Frank Hansen < m...@berlingskemedia.dk>: > Arh ok thanks, no problem. > > My problem is now that nothing is sent, do I need to format it in an

Re: Writing to SQL server

57 skrev Flavio Pompermaier < pomperma...@okkam.it>: > No sorry, you're right. The JDBCOutputFormat should work..I get confused > with the Table API > > On Fri, May 22, 2020 at 11:51 AM Martin Frank Hansen < > m...@berlingskemedia.dk> wrote: > >> Hi again, >&g

Re: Writing to SQL server

Hi again, I am a bit confused as to why the generic jdbc connector would not work with sql-server? Can you explain a bit more? Den fre. 22. maj 2020 kl. 11.33 skrev Martin Frank Hansen < m...@berlingskemedia.dk>: > Hi Flavio, > > Thanks for your reply. I will try another way

Re: Writing to SQL server

supported from what I know..for this I opened a > PR[1] that I should rebase. > If someone is interested in I could do it > > [1] https://github.com/apache/flink/pull/12038 (FLINK-14101 ) > > On Fri, May 22, 2020 at 8:35 AM Timo Walther wrote: > >> Hi Martin, >>

Re: Writing to SQL server

Hi Flavio, Thanks for your reply. I will try another way then. Den fre. 22. maj 2020 kl. 11.31 skrev Flavio Pompermaier < pomperma...@okkam.it>: > I expect you to see some exception somewhere, that sql server dialect is > not supported yet. > > On Fri, May 22, 2020 at 11:

Writing to SQL server

Hi, I am trying to write input from Kafka to a SQL server on AWS, but I have difficulties. I get the following error could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[org.apache.flink.types.Row] [error] val dsRow = tableEnv.toAppen

Re: Help with table-factory for SQL

// "0.8", "0.9", "0.10", "0.11", and "universal" > .topic("...") // required: topic name from which the table is read > > > Best, > Leonard Xu > [1] > https://ci.apache.org/projects/flin

Re: Help with table-factory for SQL

6. maj 2020 kl. 04.57 skrev Jark Wu : > Hi, > > Could you share the SQL DDL and the full exception message? It might be > you are using the wrong `connector.version` or other options. > > Best, > Jark > > On Fri, 15 May 2020 at 20:14, Martin Frank Hansen > wro

Help with table-factory for SQL

TableFactory allows to create different table-related instances from string-based properties. All available factories are called for matching to the given set of properties and a corresponding factory class. Factories leverage Java’s Service Provider Interfaces (SPI) <https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html> for discovering. This means that every dependency and JAR file should contain a file org.apache.flink.table.factories.TableFactory in the META_INF/services resource directory that lists all available table factories that it provides. But how do I do that? I thought the sbt-file would take care of this. Any help is highly appreciated! Best regards Martin Frank Hansen

RE: EXT :Flink solution for having shared variable between task managers

I think you’re looking for Broadcast State. Here’s a detailed guide. https://flink.apache.org/2019/06/26/broadcast-state.html From: Soheil Pourbafrani [mailto:soheil.i...@gmail.com] Sent: Friday, January 17, 2020 6:50 AM To: user Subject: EXT :Flink solution for having shared variable between ta

RE: EXT :Re: Taskmanagers in Docker Fail to Resolve Own Hostnames and Won't Accept Tasks

that points to the taskmanager service isn’t something I put in Flink’s config files anywhere, and I don’t think containers should be referring to themselves by that name. From: Yang Wang [mailto:danrtsey...@gmail.com] Sent: Sunday, December 22, 2019 7:15 PM To: Martin, Nick J [US] (IS) Cc: user

Taskmanagers in Docker Fail to Resolve Own Hostnames and Won't Accept Tasks

I'm running Flink 1.7.2 in a Docker swarm. Intermittently, new task managers will fail to resolve their own host names when starting up. In the log I see "no hostname could be resolved" messages coming from TaskManagerLocation. The webUI on the jobmanager shows the taskmanagers as are associated

RE: EXT :Re: Jar Uploads in High Availability (Flink 1.7.2)

, October 18, 2019 6:42 AM To: Martin, Nick J [US] (IS) Cc: Ravi Bhushan Ratnakar ; user Subject: Re: EXT :Re: Jar Uploads in High Availability (Flink 1.7.2) Hi Martin, Flink's web UI based job submission is not well suited to be run behind a load balancer at the moment. The problem is that th

RE: EXT :Re: Jar Uploads in High Availability (Flink 1.7.2)

From: Ravi Bhushan Ratnakar [mailto:ravibhushanratna...@gmail.com] Sent: Tuesday, October 15, 2019 10:35 PM To: Martin, Nick J [US] (IS) Cc: user Subject: EXT :Re: Jar Uploads in High Availability (Flink 1.7.2) Hi, i was also experiencing with the similar behavior. I adopted following approach

Jar Uploads in High Availability (Flink 1.7.2)

I'm seeing that when I upload a jar through the rest API, it looks like only the Jobmanager that received the upload request is aware of the newly uploaded jar. That worked fine for me in older versions where all clients were redirected to connect to the leader, but now that each Jobmanager acce

RE: [EXTERNAL] Re: How to restart/recover on reboot?

Jobmanager.sh takes an optional argument for the hostname to bind to, and start-cluster uses it. If you leave it blank it, the script will use whatever is in flink-conf.yaml (localhost is the default value that ships with flink). The dockerized version of flink runs pretty much the way you’re tr

RE: EXT :How to config user for passwordless ssh?

Env.ssh.opts is the literal argument string to ssh as you would enter it on the command line. Take a look at TMSlaves() in config.sh to see exactly how it’s being used. From: John Smith [mailto:java.dev@gmail.com] Sent: Tuesday, June 11, 2019 12:30 PM To: user Subject: EXT :How to config us

RE: EXT :read only mode for Flink UI

AFAIK, there are no granular permissions like that built into Flink. Limiting access to the REST API seems like a good place to start. The web UI uses the API, but controlling it there means you’re locking down all means of access. The designers of the API were disciplined about what HTTP verbs

RE: EXT :Re: Flink 1.7.1 Inaccessible

Seye, are you running Flink and Zookeeper in Docker? I’ve had problems with Jobmanagers not resolving the hostnames for Zookeeper when starting a stack on Docker. From: Till Rohrmann [mailto:trohrm...@apache.org] Sent: Monday, March 04, 2019 7:02 AM To: Seye Jin Cc: user Subject: EXT :Re: Flin

RE: EXT :Re: How to debug difference between Kinesis and Kafka

Yeah, that’s expected/known. Watermarks for the empty partition don’t advance, so the window in your window function never closes. There’s a ticket open to fix it (https://issues.apache.org/jira/browse/FLINK-5479) for the kafka connector, but in general any time one parallel instance of a sourc

RE: EXT :Re: StreamingFileSink cannot get AWS S3 credentials

Does that mean that the full set of fs.s3a.<…> configs in core-site.xml will be picked up from flink-conf.yaml by flink? Or only certain configs involved with authentication? From: Till Rohrmann [mailto:trohrm...@apache.org] Sent: Wednesday, January 16, 2019 3:43 AM To: Vinay Patil Cc: Kostas K

S3 StreamingFileSink never completes multipart uploads

x27;ve looked at the traffic between Flink and the S3 endpoint, and I don't ever see the POST message that should close off a completed upload. Has anyone else run into something like that? Nick Martin -- Notice

RE: EXT :Re: Custom S3 endpoint

Yeah, I figured that part out. I’ve tried to make it work with 2.7 and 2.8, and it looks like the prebuilt jars have actually moved to Hadoop 3 From: Paul Lam [mailto:paullin3...@gmail.com] Sent: Tuesday, December 18, 2018 7:08 PM To: Martin, Nick Cc: user@flink.apache.org Subject: EXT :Re

Custom S3 endpoint

n here (https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/aws.html#hadoop-provided-s3-file-systems---manual-setup) list a set of dependencies that seems to be completely wrong. How can use the S3 sources/sinks with a custom

Re: Task managers run on separate nodes in a cluster

Thanks for the feedback Liu and Till. @Liu Yeah that would work but unfortunately we run other services on the cluster so it's not really an option. @Till Will have a look and see how much time I can dedicate to this. M On Mon, Sep 17, 2018 at 7:21 AM Till Rohrmann wrote: > Hi Martin,

Re: Task managers run on separate nodes in a cluster

s to add support for the fenzo balanced host attribute constraint? Thanks, M On Fri, Sep 14, 2018 at 5:46 PM Till Rohrmann wrote: > Hi Martin, > > Flink supports the mesos.constraints.hard.hostattribute to specify task > constraints based on agent attributes [1]. I think you coul

Re: Task managers run on separate nodes in a cluster

Thanks Vino! On Fri, Sep 14, 2018 at 3:37 AM vino yang wrote: > Hi Martin, > > Till has done most of the work of Flink on Mesos. Ping Till for you. > > Thanks, vino. > > Martin Eden 于2018年9月12日周三 下午11:21写道： > >> Hi all, >> >> We're using Flink

Task managers run on separate nodes in a cluster

Hi all, We're using Flink 1.3.2 with DCOS / Mesos. We have a 3 node cluster and are running the Flink DCOS package (Flink Mesos framework) configured with 3 Task Managers. Our goal is to run each of them on separate hosts for better load balancing but it seems the task managers end up running on

Re: 答复: Best way to find the current alive jobmanager with HA mode zookeeper

rievalService member in Flink >>> 1.5. >>> >>> >>> >>> I wonder if I can use RestClusterClient@v1.5 on my client side, to >>> retrieve the leader JM of Flink v1.4 Cluster. >>> >>> >>> >>> Thanks >>> >&

Re: Best way to find the current alive jobmanager with HA mode zookeeper

Hi, This is actually very relevant to us as well. We want to deploy Flink 1.3.2 on a 3 node DCOS cluster. In the case of Mesos/DCOS, Flink HA runs only one JobManager which gets restarted on another node by Marathon in case of failure and re-load it's state from Zookeeper. Yuan I am guessing you

RE: Parallelism and keyed streams

Is value(index-1) stored in Keyed State, or just a local variable inside the operator? -Original Message- From: Nicholas Walton [mailto:nwal...@me.com] Sent: Monday, July 16, 2018 1:33 PM To: user@flink.apache.org Subject: Parallelism and keyed streams I have a stream of tuples , which

RE: Regarding the use of RichAsyncFunction as a FlatMap

Regarding the ‘partiality’ you mention in option one, wouldn’t you have to give that up anyway to maintain exactly once processing? Suppose input message A results in asynchronous queries/futures B and C, and imagine the following series of events: 1. Your function receives A 2. As

String Interning

I have a job where I read data from Kafka, do some processing on it, and write it to a database. When I read data out of Kafka, I put it into an object that has a String field based on the Kafka message key. The possible values for the message key are tightly constrained so there are fewer than

RE: flink and akka HTTP

0.4 || \--- org.scala-lang:scala-library:2.11.6 -> 2.11.12 |+--- com.google.code.gson:gson:2.8.4 |\--- com.google.protobuf:protobuf-java:3.5.1 \--- com.here:here-data-ingestion-sdk-java-hazard-events:0.0.1-SNAPSHOT (*) There is no akka-remote Martin

RE: flink and akka HTTP

No, without including the flink runtime to my own jar, nothing works. Regards Martin -- Martin Gäckler Entwicklung Schwarmdienste Im Auftrag der e.solutions GmbH Despagstr. 4a 85055 Ingolstadt Germany Registered

RE: flink and akka HTTP

ading is enabled by default and we did not change any settings here. Cheers Martin -- Martin Gäckler Entwicklung Schwarmdienste Im Auftrag der e.solutions GmbH Despagstr. 4a 85055 Ingolstadt Germany Registered O

flink and akka HTTP

', version: '1.4.2' implementation group: 'org.apache.flink', name: 'flink-connector-kafka-0.11_2.11',version: '1.4.2' } Relocation of AKKA did not solve the problem, because AKKA did no longer find its serialization classes. Using an o

Re: Classloader and removal of native libraries

Hi, I'm reviving this thread because I am probably hitting a similar issue with loading a native library. However I am not able to resolve it with the suggestions here. I am using Flink 1.3.2 and the Jep library to call Cpython from a RichFlatMapFunction with a parallelism of 10. I am instantiati

Re: Ceph configuration for checkpoints?

Hello Julio, how we deploy Ceph: 1) make sure Ceph is mounted through all the nodes that make up your Flink cluster; ensure path(s) are the same for each node 2) simply reference the Ceph mountpoint(s) via file:// Martin On Tue, Feb 13, 2018 at 10:18 AM Piotr Nowojski wrote: > Hi, >

Re: flowable <-> flink integration

. FlinkJobWrapper __ |OriginalFlinkJob | | | | | | | | L___ | Send signal L_ Question is How to do that? And that's all. (simulations are the next step) Thank you in advance for your answer. Regards Martin On 18 January 2018 at 19:30, Maciek Próchniak wrote: > Hi Martin,

flowable <-> flink integration

round the main class to execute flowable rest call at the end. Can you provide me a hint how to implement this wrapper please? Thank you in advance for the answer. Regards Martin

Custom Partitioning for Keyed Streams

Have a set of stateful operators that rely on keyed state. There is substantial skew between keys (i.e. there will be 100 messages on keys A and B, and 10 messages each on keys C-J), and key selection assignment is dictated by the needs of my application such that I can't choose keys in a way th

Re: Flink on K8s job submission best practices

The above applies to Mesos/DCOS as well. So if someone would also share insights into automatic job deployment in that setup would very useful. Thanks. M On Fri, Dec 22, 2017 at 6:56 PM, Maximilian Bode < maximilian.b...@tngtech.com> wrote: > Hi everyone, > > We are beginning to run Flink on K8s

Queryable State Python

Hi, Our team is looking at replacing Redis with Flink's own queryable state mechanism. However our clients are using python. 1. Is there a python integration with the Flink queryable state mechanism? Cannot seem to be able to find one. 2. If not, is it on the roadmap? 3. Our current solution is

Re: Writing an Integration test for flink-metrics

Hi, Not merged in yet but this is an example pr that is mocking metrics and checking they are properly updated: https://github.com/apache/flink/pull/4725 On Fri, Oct 13, 2017 at 1:49 PM, Aljoscha Krettek wrote: > I think we could add this functionality to the (operator) test harnesses. > I.e. a

EASY Friday afternoon question: order of chained sink operator execution in a streaming task

Hi all, Just a quick one. I have a task that looks like this (as printed in the logs): 17-09-29 0703510695 INFO TaskManager.info: Received task Co-Flat Map -> Process -> (Sink: sink1, Sink: sink2, Sink: sink3) (2/2) After looking a bit at the code of the streaming task I suppose the sink operat

Re: Using latency markers

Any follow-up on this? Jira? PR? On Wed, Sep 13, 2017 at 11:30 AM, Tzu-Li (Gordon) Tai wrote: > Hi Aitozi, > > Yes, I think we haven’t really pin-pointed out the actual cause of the > problem, but if you have a fix for that and can provide a PR we can > definitely look at it! That would be helpf

Re: Quick checkpointing related question

any overhead to the checkpoint. > > Best, > Stefan > > > Am 08.09.2017 um 10:20 schrieb Martin Eden : > > > > Hi all, > > > > I have a Flink 1.3.1 job with a source that implements > CheckpointingFunction. > > > > As I understand it, the notifyC

Quick checkpointing related question

Hi all, I have a Flink 1.3.1 job with a source that implements CheckpointingFunction. As I understand it, the notifyCheckpointComplete callback is called when all the downstream operators in the DAG successfully finished their checkpoints. Since I am doing some work in this method, I would like

Re: dynamically partitioned stream

: KEY VALUE EV_TIME INGEST_TIME PROCESSING_TIME . . . C V33 33 43 A V32 32 42 B V31 31 41 Thanks, M On Thu, Sep 7, 2017 at 9:24 AM, Tony Wei wrote: > Hi Mar

Re: dynamically partitioned stream

l happen. Ideally t2 - t1 ~ 0 but for our use case it is fine. Is this the correctness that you are talking about? Do I have the right picture of what happens? Thanks M On Thu, Sep 7, 2017 at 3:11 AM, Tony Wei wrote: > Hi Martin, > > The performance is an issue, but in your case, yes,

Re: dynamically partitioned stream

Wei wrote: > Hi Martin, Aljoscha > > I think Aljoscha is right. My origin thought was to keep the state only > after a lambda function coming. > > Use Aljoscha's scenario as example, initially, all data will be discarded > because there is no any lambdas. When lambda f1 [D, E]

Re: dynamically partitioned stream

This might be a way forward but since side inputs are not there I will try and key the control stream by the keys in the first co flat map. I'll see how it goes. Thanks guys, M On Thu, Aug 31, 2017 at 5:16 PM, Tony Wei wrote: > Hi Martin, > > Yes, that is exactly what I thou

Re: dynamically partitioned stream

oscha Krettek wrote: > Hi Martin, > > In your original example, what does this syntax mean exactly: > > f1[A, B, C]1 > > Does it mean that f1 needs one A, one B and one C from the main stream? If > yes, which ones, because there are multiple As and Bs and

Re: dynamically partitioned stream

managed state as a the actual control stream message. Once they are we can pass them in. Any other thoughts? M On Thu, Aug 31, 2017 at 12:06 PM, Tony Wei wrote: > Hi Martin, > > About problem 2. How were those lambda functions created? Pre-defined > functions / operators or a

Re: dynamically partitioned stream

suggestions. Thanks, M On Thu, Aug 31, 2017 at 9:59 AM, Tony Wei wrote: > Hi Martin, > > Let me understand your question first. > You have two Stream: Data Stream and Control Stream and you want to select > data in Data Stream based on the key set got from Control Stream. &g

dynamically partitioned stream

Hi all, I am trying to implement the following using Flink: I have 2 input message streams: 1. Data Stream: KEY VALUE TIME . . . C V66 B V66 A V55 A V44 C V33 A V33 B V33 B V22 A V1

Re: metrics for Flink sinks

actual time-series databases that specialize in these things. > > > On 28.08.2017 19:08, Martin Eden wrote: > > Hi all, > > Just 3 quick questions both related to Flink metrics, especially around > sinks: > > 1. In the Flink UI Sources always have 0 input records / bytes a

metrics for Flink sinks

Hi all, Just 3 quick questions both related to Flink metrics, especially around sinks: 1. In the Flink UI Sources always have 0 input records / bytes and Sinks always have 0 output records / bytes? Why is it like that? 2. What is the best practice for instrumenting off the shelf Flink sinks? Cu

Re: Is that possible for flink to dynamically read and change configuration?

Hey Desheng, Some options that come to mind: - Cave man style: Stop and restart job with new config. - Poll scenario: You could build your own thread that periodically loads from the db into a per worker accessible cache. - Push scenario: have a config stream (based off of some queue) which you co

Re: AVRO Union type support in Flink

Hey Vishnu, For those of us on the list that are not very familiar with Flink and Avro can you give a pointed to the docs you are referring to and how you intend to use it? Just so we gain understanding as well. Thanks, Martin On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath < vishnu.viswa

Re: Window + Reduce produces more than 1 output per window

Hello Piotrek! Thanks for answering! Yes I have already changed the "TimeCharacteristic" to "ProcessingTime". I need it for the ".setWriteTimestampToKafka(true)" option as I use the timestamp in the Kafka consumer who reads this app's output. I have already changed the code a bit for using Ke

Window + Reduce produces more than 1 output per window

Hello everybody! First of all, thanks for reading :D I am currently working on my bachelor's final project which is a comparison between Spark Streaming and Flink. Now let's focus on the problem: - THE PROBLEM: the problem is that my program is writing to Kafka more than once every window (

Re: VertexUpdateFunction

Hi Ali, If you're talking about a Java Map and the constructor of your compute function, then your implementation is not really scalable. Cheers, Martin On 31.05.2017 14:19, rost...@informatik.uni-leipzig.de wrote: Hi Martin, thanks for your answer.. For the vertex degree, I passed

Re: VertexUpdateFunction

Hi Ali :) You could compute the degrees beforehand (e.g. using the Graph.[in|out|get]degrees()) methods and use the resulting dataset as a new vertex dataset. You can now run your vertex-centric computation and access the degrees as vertex value. Cheers, Martin On 29.05.2017 09:28, rost

Re: Writing a reliable Flink source for a NON-replay-able queue/protocol that supports message ACKs

. Can you point me to a repo with reliability tests for Flink i.e. where things are killed to see whether the system recovers etc? Thanks, M On Tue, Apr 25, 2017 at 9:23 AM, Kostas Kloudas wrote: > Hi Martin! > > For an example of a source that acknowledges received messages, you coul

Re: Disk I/O in Flink

Hi Robert, Any updates on the below for the community? Thanks, M On Tue, Apr 25, 2017 at 8:50 AM, Robert Schmidtke wrote: > Hi Ufuk, thanks for coming back to me on this. > > The records are 100 bytes in size, the benchmark being TeraSort, so that > should not be an issue. I have played around

Writing a reliable Flink source for a NON-replay-able queue/protocol that supports message ACKs

Hi everyone, Are there any examples of how to implement a reliable (zero data loss) Flink source reading from a system that is not replay-able but supports acknowledging messages? Or any pointers of how one can achieve this and how Flink can help? I imagine it should involve a write ahead log bu

Re: Support/connector for Neo4j?

Please let me know, if you need help with the connector or if you want to extend it. Cheers, Martin On 24.03.2017 16:07, alex.decastro wrote: Thanks Tim! I missed that one on Jira. :-) -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com

Re: unclear exception when writing to elasticsearch

Hej, I finally found out what the problem was. I had added another dependency that was necessary to run things on hops for some reason that broke things. When I remove it, it works fine. I talking to the hops guys about it to understand what's going on. Thanks for the help. Cheers Martin

Re: unclear exception when writing to elasticsearch

search versions and they are just not compatible anymore. In the Worst case, I will hack something together that just writes the data using HttpURLConnection pushing things to the rest interface. (If that works from within flink) cheers Martin On Wed, Mar 1, 2017 at 12:24 PM, Flavio Pompermaie

Re: unclear exception when writing to elasticsearch

me.tasks.StreamTask.invoke(StreamTask.java:256) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585) > at java.lang.Thread.run(Thread.java:745) On Wed, Mar 1, 2017 at 7:58 AM, Tzu-Li (Gordon) Tai wrote: > Hi Martin, > > You can do that by adding a dependency to t

Re: unclear exception when writing to elasticsearch

the execution since I'm running things on top of Hopsworks. cheers Martin On Tue, Feb 28, 2017 at 5:42 PM, Tzu-Li (Gordon) Tai wrote: > Hi! > > This could be a Elasticsearch server / client version conflict, or that > the uber jar of your code wasn’t built properly. > &

unclear exception when writing to elasticsearch

Hej, I'm trying to write to elastic search from a streaming application and I get a weird error message I that I can't decipher. Hopefully, someone here can help me. I'm trying to run the java example

Flink streaming connect and split streams

ap to basically have 2 Streams as output the model updates and the prediction results. I could write a wrapper class containing both output types but that would require me to separate them afterwards. This feels very clunky, is there a better way of dealing with this? cheers Martin

Re: Apache Flink 1.0.3 IllegalStateException Partiction State on chaining

pe it helps, Martin On Wed, Jun 29, 2016 at 3:29 PM Ufuk Celebi wrote: > OK, looks like you can easily give more memory to the network stack, > e.g. for 2 GB set > > taskmanager.network.numberOfBuffers = 65536 > taskmanager.network.bufferSizeInBytes = 32768 > > For the

  1   2   >