Hi John,
On Tue, Apr 27, 2021 at 9:47 AM John Smith wrote:
> Hi, I'm running flink as a systemd service with...
>
> [Service]
> Type=forking
> WorkingDirectory=/opt/flink
> User=flink
> Group=flink
> ExecStart=/opt/flink/bin/taskmanager.sh start
> ExecStop=/opt/flink/bin/taskmanager.sh stop
> Ti
Hi,
On Sat, May 15, 2021 at 5:07 PM wrote:
> First I was told that my application need only perform keyed aggregation
> of streaming IoT data on a sliding window. Flink seemed the obvious choice.
>
> Then I was told that the window size must be configurable, taking on one
> of 5 possible values,
Hi,
I am not familiar with hibench. Does the Flink UI show the configured
parallelism of 20 for the job, and there are indeed 20 partitions on the
Kafka topic you consume?
Which Flink version are you running? The repo
https://github.com/Intel-bigdata/HiBench mentions Flink 1.0.3, which is *very
*o
Hi,
Flink allows to enable latency tracking [1] and exposes several metrics
that might be what you are looking for [2, look for e.g. "numRecordsIn" or
"numBytesIn"]. You can query these metrics using the REST API [3] or by
registering a metrics reporter [4] that exposes them. As noted in the other
Hi Tao,
This is currently not possible using Table API, though this will likely
change in a future version. Currently, you would have to do that using the
Datastream API [1] and then switch to the Table API.
Best wishes,
Nico
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/
Hi Tamir,
I assume you want to use the Jdbc connector?
You can use three filters on your input stream to separate it into three
separate streams, then add a sink to each of those (see e.g. [1]). Then you
can have a different SQL statement for each of the three sinks. If you
specify the driver name
Hi Yuval,
I am not familiar with the Table API, but in the fragment you posted, the
generated job name is only used as default if configuration option
pipeline.name is not set. Can't you just set that to the name you want to
have?
Best wishes,
Nico
On Mon, Jun 7, 2021 at 10:09 AM Yuval Itzchakov
Hi Wanghui,
if I understand correctly, you are looking for the config option
security.ssl.algorithms [1]?
Best regards,
Nico
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/security/security-ssl/#cipher-suites
On Tue, Jul 6, 2021 at 3:46 AM Wanghui (HiCampus)
w
Hi Wanghui,
unfortunately, this is not supported to my knowledge. See also this similar
question on Stackoverflow:
https://stackoverflow.com/questions/60950594/flink-encryption-parameters-in-flink-conf-yaml
Best regards,
Nico
On Mon, Jul 5, 2021 at 3:45 PM Wanghui (HiCampus)
wrote:
> Hello, I
Hi Maciek,
is there a typo in the input data? Timestamp 2021-05-01 04:42:57 appears
twice, but timestamp 2021-05-01T15:28:34 (from the log lines) is not there
at all. I find it hard to correlate the logs with the input...
Best regards,
Nico
On Wed, Jul 7, 2021 at 11:16 AM Arvid Heise wrote:
>
Hi Mahima,
looks like you found the relevant parts of the code already: In
JarHandlerUtils.JarHandlerContext#fromRequest, the parallelism value is
extracted from the request body or query parameter (the latter is
deprecated, though). If none is found, it defaults to 1 and overwrites the
configured
that it works for internal, but not
external communication. Adding Nico (another one) to the loop here, who may
know more about whether there is a reason for this or whether this is a bug.
Best regards,
Nico
On Tue, Jul 6, 2021 at 5:28 PM Nicolaus Weidner <
nicolaus.weid...@data-artisans.com>
Hi Praneeth,
It does look like a failure constructing the serializer. Can you share the
serialization config you use for the Kafka producer? In particular, are you
using a custom serializer?
Do you use any custom classloading configuration?
Best regards,
Nico
On Wed, Sep 1, 2021 at 11:38 PM Pran
Hi Ragini,
On Fri, Sep 17, 2021 at 1:40 PM Ragini Manjaiah
wrote:
> Later I started encountering org.apache.kafka.common.errors.TimeoutException:
> Failed to update metadata after 6 ms.
>
This message can have several causes. There may be network issues, your
Kafka configuration might be br
Hi Jérémy,
objects are serialized when you store them in state. So when you retrieve
run_state from state, it is deserialized and you have a fresh instance.
Calling method_caller() then modifies this instance, but *not *the
serialized version stored in state.
In the second attempt you described, y
Hi Kamil,
On Tue, Oct 5, 2021 at 9:03 AM Kamil ty wrote:
> Hello,
>
> I'm trying to run a pyflink job in cluster mode (with yarn). My job
> contains source and sink definitions using Table API which are converted to
> a datastream and back. Unfortunately I'm getting an unusual exception at:
> *t
Hi Parag,
I am not so familiar with the setup you are using, but did you check out
[1]? Maybe the parameter
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]]
is what you are looking for?
Best regards,
Nico
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/re
Hi Tom,
On Mon, Oct 4, 2021 at 10:42 PM tom yang wrote:
> Hello,
>
>
>
> I have a recently ran into an issue with RichAsyncFunction and wanted to
> get some guidance from the community
>
>
>
> Please see snippet
>
>
>
> *class* AsyncFetchFromHttp *extends* RichAsyncFunction String, String>> {
>
Hi Marc,
I think you can just use keyed state in a
CheckpointedFunction. FunctionInitializationContext gives you access to
both keyed state and operator state (your stream needs to be keyed, of
course). So you could just update your local custom state on regular
invocations and update keyed state
ased on controlState ProcessedEvent
> result = customSystem.process(controlState.value() , event); // Save
> internal custom system state after processing: can be costly if high event
> throughput
> dataState.update(customSystem.getSnapshot(controlState.value().getUserId());
> // Outp
Hi Yuval,
Can you try
toDataStream[RowData](tableSchema.toPhysicalRowDataType.bridgedTo(classOf[RowData]))?
Best regards,
Nico
On Thu, Oct 28, 2021 at 10:15 PM Yuval Itzchakov wrote:
> Flink 1.14
> Scala 2.12.5
>
> Hi,
> I want to be able to convert a Table into a DataStream[RowData]. I need t
Hi Fuyao,
About your second question: You are right that taking and restoring from
savepoints will incur a performance loss. They cannot be incremental, and
cannot use native (low-level) data formats - for now. These issues are on
the list of things to improve for Flink 1.15, so if the changes mak
Hi,
I tried building Flink 1.13 with the Scala 2.12 profile and running some of
the examples with Java 11, without encountering the issue you describe
(with or without HA). Can you give more details on how exactly you built
Flink (ideally the full command), and how you ran the job?
Best,
Nico
On
Hi Fuyao,
I just wanted to say that the performance loss that you rightly suspected
when using savepoints (as opposed to checkpoints) may disappear with Flink
1.15. There should be no loss of functionality as far as checkpoints are
concerned.
I don't think the savepoint performance improvement goa
Hi,
to rescale, you should take a savepoint, stop the job, then restart from
the savepoint with your new desired parallelism. This way, no data will be
lost.
Best,
Nico
On Thu, Nov 25, 2021 at 10:53 AM 杨浩 wrote:
> Will Flink loss some old Keyed State when changing the parallelism, like 2
> ->
Hi Felipe,
just a quick question to make sure: did you do a full rebuild of your
project after changing the Spring boot version?
Best,
Nico
On Thu, Nov 25, 2021 at 8:01 PM Felipe Gutierrez <
felipe.o.gutier...@gmail.com> wrote:
> Hi community,
>
> I am using Flink 1.11 + Java 8 and I was updati
that only
Scala 2.11 or 2.12 (depending on your Flink version) is on the classpath,
else clashes can occur.
... that said, even with Flink 1.14+, while the RPC system should not
suffer this problem anymore, other components may.
Best,
Nico
On Fri, Nov 26, 2021 at 11:21 AM Nicolaus Weidner
3
>
>
>
>
> Thank you very much!
> Felipe
>
> *--*
> *-- Felipe Gutierrez*
> *-- skype: felipe.o.gutierrez*
>
>
> On Fri, Nov 26, 2021 at 12:08 PM Nicolaus Weidner <
> nicolaus.weid...@ververica.com> wrote:
>
>> Hi Felipe,
Hi Jai,
On Tue, Feb 22, 2022 at 9:19 PM Jai Patel
wrote:
> It seems like the errors are similar to those discussed here:
> - https://issues.apache.org/jira/browse/FLINK-14316
> - https://cdmana.com/2020/11/20201116104527255b.html
>
I couldn't find any other existing issue apart from the one you
Hi Marco,
I'm no expert on the Kafka producer, but I will try to help. [1] seems to
have a decent explanation of possible error causes for the error you
encountered.
Which leads me to two questions:
if (druidProducerTransactionMaxTimeoutMs > 0) {
>
> properties.setProperty("transaction.max
Hi Marco,
The documentation kind of suggestion this is the cause:
> https://nightlies.apache.org/flink/flink-docs-release-1.12/dev/connectors/kafka.html
>
> However, I think the documentation could benefit with a few examples and
> scenarios that can ill-considered configurations.
>
Matthias alre
Hi Jai,
Do writes to ValueStates/MapStates have a direct on churn of the Flink
> State or is the data buffered in between?
>
Writes to keyed state go directly to RocksDB. So there shouldn't be any
memory issues with buffers overflowing or similar. In general, more memory
should increase performan
Hi Sigalit,
first of all, have you read the docs page on metrics [1], and in particular
the Prometheus section on metrics reporters [2]?
Apart from that, there is also a (somewhat older) blog post about
integrating Flink with Prometheus, including a link to a repo with example
code [3].
Hope that
Hi Mahima,
have you checked whether the problematic dependency (seems to be avro) is
contained in Flink's /lib folder in your case? I would suggest to check on
JobManagers and TaskManagers, since you say it's in the business logic.
Have you tried the suggestions mentioned in the docs? Those would
34 matches
Mail list logo
