atermark strategy
> - and replaces it with something that is arbitrary (at this point
> it is hard to guess the correct max lateness that is a mixture of the
> events from multiple Kafka partitions)
>
>
>
> Concusion:
>
> The only way to make the event tim
Hi,
I have a 2 node cluster just for testing.
When I start the cluster and the job I see that the parallelism is 2 as
expected.
But only they are both on the same node.
When I stop the taskmanager on that node it switches to the other one.
But I expected both nodes to have a subtask.
See below.
and it tells all downstream operators
>- Idleness specification is only useful if a respective operator has
>another source of valid watermarks (i.e. after a union of two streams, one
>active/one idle ….). this is not your case
>
>
>
> I hope this clarifies you
LEAST_ONCE)
.build();
Op do 17 mrt. 2022 om 17:29 schreef Matthias Pohl :
> Could you share more details on what's not working? Is the
> ssl.trustore.location accessible from the Flink nodes?
>
> Matthias
>
> On Thu, Mar 17, 2022 at 4:00 PM HG wr
)
)
.setDeliverGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
.build();
Op do 17 mrt. 2022 om 17:29 schreef Matthias Pohl :
> Could you share more details on what's not working? Is the
> ssl.trustore.location accessible from the Flink nodes?
>
> Matthias
>
>
Hi all,
I am probably not the smartest but I cannot find how to set ssl-properties
for a Kafka Sink.
My assumption was that it would be just like the Kafka Consumer
KafkaSource source = KafkaSource.builder()
.setProperties(kafkaProps)
.setProperty("ssl.truststore.type", trustStoreT
- Coming back to your idleness configuration: it only means that the
>incoming stream becomes idle == timeless after a while … i.e. watermarks
>won’t make progress from this steam, and it tells all downstream operators
>- Idleness specification is only useful if a respective
Hi,
I read from a Kafka topic events that are in JSON format
These event contain a handling time (aka event time) in epoch milliseconds,
a transaction_id and a large nested JSON structure.
I need to group the events by transaction_id, order them by handling time
and calculate the differences in ha
Hi
Can we use zookeeper that is already running for Kafka to service Flink?
Or is that not recommended?
Regards Hans
return -1;
}
}
}
Op ma 7 mrt. 2022 om 03:05 schreef yidan zhao :
> Collect the elements to a list, then sort, then collect out.
>
> HG 于2022年3月3日周四 22:13写道：
>
>> Hi,
>> I have need to sort the input of the ProcesWindowFunction by one of the
>> fie
ts to a list, then sort, then collect out.
>
> HG 于2022年3月3日周四 22:13写道：
>
>> Hi,
>> I have need to sort the input of the ProcesWindowFunction by one of the
>> fields of the Tuple4 that is in the Iterator.
>>
>> Any advice as to what the best way is?
ializationSchema that implements a Deserialization schema.
> You even should be able to copy the implementation from the flink included
> JSONKeyValueDeserializationSchema and change KafkaDeserializationSchema to
> DeserializationSchema.
> >
> > If you will have issues with implementing t
As per the documentation ,
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/
A kafka sink can be defined as further below
But in fact it fails with
* error: cannot find symbol
.setDeliverGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
^ symbol: meth
otalElapsed);
}
}
}
Op do 3 mrt. 2022 om 15:12 schreef HG :
> Hi,
> I have need to sort the input of the ProcesWindowFunction by one of the
> fields of the Tuple4 that is in the Iterator.
>
> static class MyProcessWindowFunction extends
> ProcessWi
Hi,
I have need to sort the input of the ProcesWindowFunction by one of the
fields of the Tuple4 that is in the Iterator.
static class MyProcessWindowFunction extends
ProcessWindowFunction, String, String,
TimeWindow> {
@Override
public void process(String key, Context context,
It
s 😊
>
>
>
> Thias
>
>
>
>
>
> *From:* HG
> *Sent:* Montag, 28. Februar 2022 16:25
> *To:* user
> *Subject:* processwindowfunction output Iterator
>
>
>
> Hi,
>
>
>
>
>
> Can processwindowfunction output an Iterator?
>
> I need to sort
Hi,
Can processwindowfunction output an Iterator?
I need to sort and subtract timestamps from keyed events and then output
them all with added elapsed times.
Regards Hans
")
> .build())
> .as("eventTime", "handlingTime", "transactionId",
> "originalEvent");
>
> Sorry I'm not an expert in Table / SQL and might miss or overlook something.
>
>
>
> Bes
Hello,
I have to convert the table to Datastream and try to do it with
toAppendStream (just saw that it is deprecated )
But I have not been able to do the conversion as yet. (See the attached
code).
Also my final Sink should be Kafka and the format ObjectNode/JSON.
So I need a serializer eventuall
Well I thought that in order to do the same with only the datastream api I
would need to use MapPartitionFunction.
Op do 17 feb. 2022 om 10:41 schreef Francesco Guardiani <
france...@ververica.com>:
> Why do you need MapPartitionFunction?
>
> On Wed, Feb 16, 2022 at 7:
expressions.Expression-
>
> On Wed, Feb 16, 2022 at 2:35 PM HG wrote:
>
>> Hello all
>>
>> I need to calculate the difference in time between ordered rows per
>> transactionId. All events should arrive within the timeframe set by the
>> out-of-orderness (
Hello all
I need to calculate the difference in time between ordered rows per
transactionId. All events should arrive within the timeframe set by the
out-of-orderness ( a couple of minutes). Events outside this time should be
ignored.
In SQL this would be :
select transactionId , handlingTime ,
ot;handlingTime").sum().over($("w")));
Op di 15 feb. 2022 om 15:50 schreef Chesnay Schepler :
> Aren't you missing a groupBy() between window() and select()?
>
> On 15/02/2022 15:45, HG wrote:
>
> Hi all,
>
> When I execute the code :
>
>
, 15:50 Chesnay Schepler wrote:
> Aren't you missing a groupBy() between window() and select()?
>
> On 15/02/2022 15:45, HG wrote:
>
> Hi all,
>
> When I execute the code :
>
> Table tupled3DsTable = tableEnv.fromDataStream(tuple3ds, Schema.newBuilder()
&
Hi all,
When I execute the code :
Table tupled3DsTable = tableEnv.fromDataStream(tuple3ds, Schema.newBuilder()
.column("f0","TIMESTAMP_LTZ(3)") // Only
TIMESTAMP_LTZ(0) to TIMESTAMP_LTZ(3) allowed
.column("f1","STRING")
.column("f2","STR
gVerwerkingsTijden
>
> What is esl.job.cag.verwerkingstijden.CagVerwerkingsTijden? Sounds like a
> bad import? Also, have you checked you have Flink deps aligned?
>
> On Mon, Feb 14, 2022 at 3:17 PM HG wrote:
>
>>
>> Hi,
>>
>> When I do :
>>
>&g
le of days nothing is produced), then you should set an
> idleness which after that, a watermark is produced.
>
> Idleness is
>
> On Fri, Feb 11, 2022 at 2:53 PM HG wrote:
>
>> Hi,
>>
>> I am getting a headache when thinking about watermarks and timestam
lang.String)
> > location: class esl.job.cag.verwerkingstijden.CagVerwerkingsTijden
>
> What is esl.job.cag.verwerkingstijden.CagVerwerkingsTijden? Sounds like a
> bad import? Also, have you checked you have Flink deps aligned?
>
> On Mon, Feb 14, 2022 at 3:17 PM HG wrote:
>
>>
>
Hi,
When I do :
Table counts = t
.groupBy($("transactionId"))
.select($("transactionId"),
$("handlingTime").sum().as("summedhandlingTime"));
The code below fails with :
cannot find symbol
.select($("transactionId"),
$("handlingTime").sum()
Hi,
I am getting a headache when thinking about watermarks and timestamps.
My application reads events from Kafka (they are in json format) as a
Datastream
Events can be keyed by a transactionId and have a event timestamp
(handlingTime)
All events belonging to a single transactionId will arrive
gy.EARLIEST))
.build();
Op wo 9 feb. 2022 om 09:46 schreef HG :
> Sorry to have bothered everyone.
>
> This is the obvious solution:
>
> .setDeserializer(KafkaRecordDeserializationSchema.of(new
> JSONKeyValueDeserializationSchema(false)))
>
>
>
dDeserializationSchema.html#of-org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema-
>
>
> Regards,
> Roman
>
> On Tue, Feb 8, 2022 at 5:43 PM HG wrote:
> >
> > Hi all,
> >
> > When I build this code:
> >
> &
Schema-
>
>
> Regards,
> Roman
>
> On Tue, Feb 8, 2022 at 5:43 PM HG wrote:
> >
> > Hi all,
> >
> > When I build this code:
> >
> > KafkaSource source = KafkaSource.builder()
> > .setProperties(kafkaProps)
> > .set
Hi all,
When I build this code:
KafkaSource source = KafkaSource.builder()
.setProperties(kafkaProps)
.setProperty("ssl.truststore.type",trustStoreType)
.setProperty("ssl.truststore.password",trustStorePassword)
.setProperty("ssl.truststore.location",trustStoreLoca
2aa10f925e9f63885cb7f6686303/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/statemachine/StateMachineExample.java#L104
>
> [2]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/csv/
> On 03/02/2022 16:
ializer will be the same for older Flink versions,
> the Kafka Source has appeared recently, to learn about the previous kafka
> source (FlinkKafkaConsumer) see: Kafka | Apache Flink
> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/#kafka-sourc
Super
Thanks
Op ma 7 feb. 2022 om 13:04 schreef Chesnay Schepler :
> I think you can safely ignore this warning. It shouldn't cause any harm,
> but I will file a ticket nonetheless.
>
> On 07/02/2022 12:52, HG wrote:
>
> I have nothing like that in the config (fli
leaked-classloader was set to
> false.
>
> On 07/02/2022 10:28, HG wrote:
>
> Hi,
>
> Well I have set :
>
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setMaxParallelism(5);
> env.setRuntimeMode(RuntimeExecutionMo
Hi,
I am developing my flink application.
For start I have built a class that reads events from Kafka and outputs
them datastream.print()
The job runs every time.
But starting with the 2nd time I see this in the standalone session log:
2022-02-04 15:16:30,801 WARN org.apache.kafka.common.utils.
Hello
Most examples available still use the FlinkKafkaConsumer unfortunately.
I need to consume events from Kafka.
The format is Long,Timestamp,String,String.
Do I need to create a custom deserializer?
What also confuses me is
KafkaSource** source = KafkaSource
How does it relate to the de
diani <
france...@ververica.com>:
> The latter link you posted refers to a very old flink release. You shold
> use the first link, which refers to latest release
>
> FG
>
> On Tue, Feb 1, 2022 at 8:20 AM HG wrote:
>
>> Hello all
>>
>> I am confused.
>
Hello all
I am confused.
What is the difference between KafkaSource as defined in :
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/
and FlinkKafkaConsumer010 as defined in
https://nightlies.apache.org/flink/flink-docs-release-
1.2/api/java/org/apache/fl
Because?
I am using IntelliJ.
What are the drawbacks of Java then?
On Fri, Jan 28, 2022, 19:05 sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:
> Yes Scala is the best.
>
> On Fri, Jan 28, 2022, 9:57 AM Nicolás Ferrario
> wrote:
>
>> Hi Seb.
>>
>> In my team we are migrating things
Thanks
On Fri, Jan 28, 2022, 07:47 Caizhi Weng wrote:
> Hi!
>
> This job will work as long as your SQL statement is valid. Did you meet
> some difficulties? Or what is your concern? A record of 100K is sort of
> large, but I've seen quite a lot of jobs with such record size
Hi,
I need to calculate elapsed times between steps of a transaction.
Each step is an event. All steps belonging to a single transaction have the
same transaction id. Every event has a handling time.
All information is part of a large JSON structure.
But I can have the incoming source supply trans
on it so that I can eventually get this into
> the
> Flink docs/quickstarts.
>
>
> Nico
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-24478
>
> On Wednesday, 19 January 2022 15:07:39 CET HG wrote:
> > Hi
> > For my first project I followed :
&
Hi
For my first project I followed :
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/project-configuration/
where there a build.gradle example
I created a directory, entered into it and did gradle init.
The I created the build.gradle as per the example.
I opened th
gards,
> Mariam
>
>
>
> On Friday, January 14, 2022, HG wrote:
>
>> Hi,
>>
>> Before starting programming myself I'd like to know whether there are
>> good examples with deserialization of JSON that I can borrow.
>> The structure of the JSON i
the Kafka Source has appeared recently, to learn about the previous kafka
> source (FlinkKafkaConsumer) see: Kafka | Apache Flink
> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/#kafka-sourcefunction>
> .
>
> Best Regards
&
Hi,
Before starting programming myself I'd like to know whether there are good
examples with deserialization of JSON that I can borrow.
The structure of the JSON is nested with multiple levels.
Any references?
'better well stolen than badly invented myself' we'd say in Dutch😁
Regards Hans
your application to solve your use case in the
>> way you want.
>> I suggest you start by looking at the documentation of Table API
>> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/overview/
>> and then, for your specific use case, check
>> https:/
by looking at the documentation of Table API
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/overview/
> and then, for your specific use case, check
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/queries/window-tvf/
> .
>
uot; elapsed time, and then inside
> the window you pick the beginning and end event and join them.
>
> Hope it helps,
> FG
>
> On Thu, Jan 6, 2022 at 3:25 PM HG wrote:
>
>> Hello all,
>>
>> My question is basically whether it is possible to group events by
Hello all,
My question is basically whether it is possible to group events by a key
(these will belong to a specific transaction) and then calculate the
elapsed times between them based on a timestamp that is present in the
event.
So a transaction my have x events all timestamped and with the
tran
fact, that even with
> this system property, some custom java HTTP clients, may simply decide to
> ignore this... :)
>
> does it explain your concerns?
>
> [1]
> https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html
>
> D.
>
> On Tue, Dec 21, 202
> nslookup to verify that). This is may be tricky to debug, unless you're
> familiar with networking. A good place to start might be checking the
> /etc/resolv.conf for details about the DNS server being used.
>
> Best,
> D.
>
> On Mon, Dec 20, 2021 at 3:39 PM HG w
Hello
I am trying to the flink-playground examples.
The docker-compose build fails on the mvn clean install command .
I am behind a proxy.
To diagnose this I started a container based on the already created image
docker run -it --name my-maven-project -v "$(pwd)":/usr/src/mymaven -w
/usr/src/my
0b87cece03b%40apache.org%3E
>
> pt., 17 gru 2021 o 13:04 HG napisał(a):
>
>> Hello all
>>
>> I am a flink newbie and trying to do the fraud detection example.
>> The maven command however fails for version 1.14.2 since it cannot find
>> flin
Hello all
I am a flink newbie and trying to do the fraud detection example.
The maven command however fails for version 1.14.2 since it cannot
find flink-walkthrough-datastream-java
for that version
mvn archetype:generate -DarchetypeGroupId=org.apache.flink
-DarchetypeArtifactId=flink-walkthr
59 matches
Mail list logo
