Hi Joshua,
could you try whether the job also fails when not using the gzip format?
This could help us narrow down the culprit. Moreover, you could try to run
your job and Flink with Java 11 now.
Cheers,
Till
On Tue, May 18, 2021 at 5:10 AM Joshua Fan wrote:
> Hi all,
>
> Most of the posts say
Arvid,
I found a jira related to my issue.
https://issues.apache.org/jira/browse/FLINK-18096
Added a comment and I think Seth's idea is way better than just renaming
the current name of the record from avro schema.
Thanks,
Youngwoo
On Mon, May 17, 2021 at 8:37 PM Youngwoo Kim (김영우)
wrote:
>
On Mon, May 17, 2021 at 01:22:16PM +0200, Arvid Heise wrote:
> Hi ChangZhuo,
>
> This looks indeed like a bug. I created FLINK-22686 [1] to track it. It
> looks unrelated to reactive mode to me and more related to unaligned
> checkpoints. So, you can try out reactive mode with aligned checkpoints.
Hi,
can you maybe share some details about the code you're running?
Regards
Ingo
On Tue, May 18, 2021 at 5:10 AM 杨建春/00250041 wrote:
> I'm using flink1.13.0, table Function, why report this error ? what
> reason ? Thanks!
>
>
>
> Traceback (most recent call last):
> File "D:/yjc/AIOPS/Flink
Hi,
1) The cause of the exception:
The dependencies added via pipeline.jars / pipeline.classpaths will be used to
construct user class loader.
For your job, the exception happens when HadoopUtils.getHadoopConfiguration is
called. The reason is that HadoopUtils is provided by Flink which is load
Hi all,
Most of the posts says that "Most of the times, the crashes in ZIP_GetEntry
occur when the jar file being accessed has been modified/overwritten while
the JVM instance was running. ", but do not know when and which jar file
was modified according to the job running in flink.
for your info
I'm using flink1.13.0, table Function, why report this error ? what reason ?
Thanks!
Traceback (most recent call last):
File "D:/yjc/AIOPS/Flink/UDTFcallstack.py", line 149, in
t_result.wait()
File "D:\Program Files
(x86)\python36\lib\site-packages\pyflink\table\table_result.py", line
ping?
On Tue, May 11, 2021 at 11:31 PM Jin Yi wrote:
> hello. thanks ahead of time for anyone who answers.
>
> 1. verifying my understanding: for a kafka source that's partitioned on
> the same piece of data that is later used in a keyBy, if we are relying on
> the kafka timestamp as the event
Hello,
Is new KafkaSource/KafkaSourceBuilder ready to be used ? If so, is KafkaSource
state compatible with legacy FlinkKafkaConsumer, for example if I replace
FlinkKafkaConsumer by KafkaSource, will offsets continue from what we had in
FlinkKafkaConsumer ?
Thanks,
Alexey
I think it should be possible to use Helm pre-upgrade hook to take savepoint
and stop currently running job and then Helm will upgrade image tags. The
problem is that if you hit timeout while taking savepoint, it is not clear how
to recover from this situation
Alexey
___
Hi Pedro,
There is currently no official Kubernetes Operator for Flink and, by
extension, there is no official Helm chart. It would be relatively easy to
create a chart for simply deploying standalone Flink resources via the
Kubernetes manifests described here[1], though it would leave out the
abi
Hi everyone!
We’re very excited to launch the Call for Presentations [1] for Flink
Forward Global 2021! If you have an inspiring Apache Flink use case,
real-world application or best practice, Flink Forward is the platform for
you to share your experiences.
We look forward to receiving your submi
On Mon, 17 May 2021, 17:38 Priyanka Manickam,
wrote:
> Hi Yang,
>
> I have checked the documents you have shared in the previous mail.
> But i am not sure what we have to give for the $(CONTAINER_SCRIPTS) in
> the line
>
>
> -Dkubernetes.pod-template-file=$(CONTAINER_SCRIPTS)/Kubernetes-pod-temp
Youngwoo,
I was trying to configure my flink/MINIO connector based on
"s3_setup_with_provider"
in the end-to-end test setup script; this requires that AWS_ACCES and AWS
_SECRET keys be set as ENV variables but I think that needs to be set
during the installation of MINIO, not flink.
I'm just not
Hello,
Forwarding this question from the dev mailing list in case this is a more
appropriate list.
Does flink have an official Helm Chart? I haven't been able to find any,
the closest most up-to-date one seems to be
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator.
Is this correct or
Hey Robert,
I’m not sure why you need to set env var. but That’s a flink configuration,
See
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/s3/#configure-path-style-access
Thanks,
Youngwoo
2021년 5월 17일 (월) 오후 10:47, Robert Cullen 님이 작성:
> Arvid, Is there
Hi,
Flink allows to enable latency tracking [1] and exposes several metrics
that might be what you are looking for [2, look for e.g. "numRecordsIn" or
"numBytesIn"]. You can query these metrics using the REST API [3] or by
registering a metrics reporter [4] that exposes them. As noted in the other
Arvid, Is there a way to set environment variables in the flink-conf.yaml?
Setting them on the CLI isn't working.
On Sat, May 15, 2021 at 3:56 PM Arvid Heise wrote:
> Hi Robert,
>
> we have an end-to-end-test with minio. You have to use
> s3.path.style.access; I think the actual key depends on t
Hi,
I am not familiar with hibench. Does the Flink UI show the configured
parallelism of 20 for the job, and there are indeed 20 partitions on the
Kafka topic you consume?
Which Flink version are you running? The repo
https://github.com/Intel-bigdata/HiBench mentions Flink 1.0.3, which is *very
*o
When using hibench to test Flink, the jobs submitted are built-in applications
of hibench, that is, the code logic of programs like wordcount cannot be
changed.
How can I get the throughput and processing delay of Flink?
In addition, in the /report/hibench.report file of hibench, we can't get t
Hello,
I was wondering whether anyone has tried and/or had any luck creating a
custom catalog with Iceberg + Flink via the Python API (
https://iceberg.apache.org/flink/#custom-catalog)?
When doing so, the docs mention that dependencies need to be specified via
*pipeline.jars* / *pipeline.classpa
Hi,
On Sat, May 15, 2021 at 5:07 PM wrote:
> First I was told that my application need only perform keyed aggregation
> of streaming IoT data on a sliding window. Flink seemed the obvious choice.
>
> Then I was told that the window size must be configurable, taking on one
> of 5 possible values,
Hi Suchithra,
this is very likely a version mixup: Can you make sure all jars in your
classpath are Flink 1.11.1?
On Mon, May 17, 2021 at 2:05 PM V N, Suchithra (Nokia - IN/Bangalore) <
suchithra@nokia.com> wrote:
> Hi,
>
>
>
> With flink 1.11.1 version, taskmanager initialization is failing
Hi,
With flink 1.11.1 version, taskmanager initialization is failing with below
error. Could you please help to debug the issue.
log":"[main] org.apache.flink.runtime.io.network.netty.NettyConfig NettyConfig
[server address: /0.0.0.0, server port: 4121, ssl enabled: false, memory
segment size
Hi Yun,
That’s very helpful and good to know that the problem/use-case has been thought
about. Since my need is probably shorter-term than later, I’ll likely need to
explore a workaround.
Do you know of an approach that might not require the use of check pointing and
restarting? I was looking
Hey Arvid,
I found that It's a constant from Flink.
https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSchemaConverter.java#L307
I believe, it would be good to substitute 'record' to 'Record'
What do you think?
Thanks,
Y
Hi ChangZhuo,
This looks indeed like a bug. I created FLINK-22686 [1] to track it. It
looks unrelated to reactive mode to me and more related to unaligned
checkpoints. So, you can try out reactive mode with aligned checkpoints.
If you can provide us with the topology, we can also fix it soonish:
Hi Youngwoo,
You can try to use aliases for it [1].
Even better would be to use a different name for the record. In general,
since Avro originally comes from the Java World, it's more common to use
camel case for record names.
[1] https://avro.apache.org/docs/current/spec.html#Aliases
On Mon, M
Hi Adi,
To me, this looks like a version conflict of some kind. Maybe you use
different Avro versions for your user program and on your Flink cluster.
Could you check that you don't have conflicting versions on your classpath?
It would also be helpful to have a minimal example that allows
reproduc
One small addition: The old mapping looks to use the
SubtaskStateMapper.RANGE whereas the new mapping looks to use the
SubtaskStateMapper.ROUND_ROBIN.
On Mon, May 17, 2021 at 11:56 AM Till Rohrmann wrote:
> Hi ChangZhuo Chen,
>
> This looks like a bug in Flink. Could you provide us with the logs
Hi ChangZhuo,
IIRC, even you have specified a savepoint when starting, the JobManager
could recover from the latest checkpoint when the JobManager failed.
Because when recovering, DefaultCompletedCheckpointStore will sort all the
checkpoints(including the savepoint) and pick the latest one.
So, c
Hi ChangZhuo Chen,
This looks like a bug in Flink. Could you provide us with the logs of the
run and more information about your job? In particular, how does your
topology look like?
My suspicion is the following: You have an operator with two inputs. One
input is keyed whereas the other input is
Hi Zerah,
I guess you could provide a Python implementation for
ConfluentRegistryAvroDeserializationSchema if needed. It’s just a wrapper for
the Java implementation and so it’s will be very easy to implement. You could
take a look at AvroRowDeserializationSchema [1] as an example.
Regards,
Di
Hi,
I have below use case
1. Read streaming data from Kafka topic using Flink Python API
2. Apply transformations on the data stream
3. Write back to different kafka topics based on the incoming data
Input data is coming from Confluent Avro Producer. By using the existing
pyflink.common.serializ
Hi,
I have a table backed by confluent avro format and the generated schema
from flink looks like following:
{
"type": "record",
"name": "record",
"fields": [
{
"name": "dt",
"type": [
"null",
{
"type": "int",
"logicalType": "date"
Could you share your pod-template.yaml or check whether the container name
is configured to "flink-main-container"?
Best,
Yang
ChangZhuo Chen (陳昌倬) 于2021年5月17日周一 下午5:20写道：
> On Mon, May 17, 2021 at 01:29:55PM +0530, Priyanka Manickam wrote:
> > Hi All,
> >
> > Do we required to add any image fo
On Mon, May 17, 2021 at 01:29:55PM +0530, Priyanka Manickam wrote:
> Hi All,
>
> Do we required to add any image for flink-main-container in
> pod-template.yaml file because it giving an error saying
> "spec.containers(0).image value required.
>
>
> Could anyone help with this please
Hi,
You n
Thanks for reading this email.
According to the introduction, the identity program in hibench reads data from
Kafka and then writes it back to Kafka.
When using the identity program in hibench to test the Flink, set the
parallelism to 20 in the flink.conf file in the conf directory of hiben
>
> Hi All,
>
> Do we required to add any image for flink-main-container in
> pod-template.yaml file because it giving an error saying
> "spec.containers(0).image value required.
>
>
> Could anyone help with this please
>
> Thanks,
> Priyanka Manickam
>
> .
>>
>>
>>
>>
>>
>>
Hi All,
Do we required to add any image for flink-main-container in
pod-template.yaml file because it giving an error saying
"spec.containers(0).image value required.
Could anyone help with this please
Thanks,
Priyanka Manickam
On Thu, 22 Apr 2021, 08:41 Milind Vaidya, wrote:
> Hi
>
> I see
Hi Milind,
A job can be stopped with a savepoint in the following way [1]: ./bin/flink
stop --savepointPath [:targetDirectory] :jobId
Best,
Matthias
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/state/savepoints/#stopping-a-job-with-savepoint
On Sun, May 16, 2021 at 1:12 AM
Yes, it does.
Internally, each re-scheduling is performed as stop-and-resume the job,
similar to a failover. Without checkpoints, the job will always restore
from the very beginning.
Thank you~
Xintong Song
On Mon, May 17, 2021 at 2:54 PM Alexey Trenikhun wrote:
> Hi Xintong,
> Does reactive
Hi,
I agree that both `connect` and `registerTableSource` are useful for
generating Table API pipelines. It is likely that both API methods will
get a replacement in the near future.
Let me explain the current status briefly:
connect(): The CREATE TABLE DDL evolved faster than connect(). The
43 matches
Mail list logo
