Show plan in UI not working.

[John Smith]

Just I'm case but in 1.14.x regardless of the job manager is leader or not.
Before submitting a job of you click on "Show Plan" it just shows a blank
window.

I'm assuming it's similar issue as the deserialozation ones.

Re: Flink Kinesis connector - EFO connection error with http proxy settings

[Gnanamoorthy, Saravanan]

Hi Danny,
I should be able to make the contribution to add proxy support. Please let me 
know the contribution process.

Thanks
-Saravan

From: Danny Cranmer 
Date: Wednesday, January 19, 2022 at 3:10 AM
To: Gnanamoorthy, Saravanan 
Cc: user@flink.apache.org 
Subject: Re: Flink Kinesis connector - EFO connection error with http proxy 
settings
NOTICE: This email is from an external sender - do not click on links or 
attachments unless you recognize the sender and know the content is safe.

Hello Saravanan,

Yes you are correct. EFO uses AWS SDK v2 and the builder does not set proxy 
configuration [1]. The polling (non EFO) mechanism is using AWS SDK v1 which 
has a more general configuration deserialiser, and hence proxy is configurable. 
I do not believe there is a workaround for this without modifying the connector.

If you are in a position to make a contribution to add support, we would 
appreciate this. Otherwise I can take this one. Please let me know your 
thoughts.

[1] 
https://github.com/apache/flink/blob/release-1.13/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AwsV2Util.java#L113

Thanks,
Danny Cranmer.

On Tue, Jan 18, 2022 at 12:52 AM Gnanamoorthy, Saravanan 
mailto:saravanan.gnanamoor...@fmr.com>> wrote:
Hello,
We are using Flink kinesis connector for processing the streaming data from 
kinesis. We are running the application behind the proxy. After the proxyhost 
and proxyport settings, the Connector works with default publisher 
type(Polling) but it doesn’t work when we enable the publisher type as Enhanced 
fanout (EFO). We tried with different connector version but it the behaviours 
is same. I am wondering if the proxy settings are ignored for EFO type. I am 
looking forward to your feedback/recommendations.

Flink version: 1.3.5
Java version: 11

Here is the error log:

2022-01-17 18:59:20,707 WARN  org.apache.flink.runtime.taskmanager.Task 
   [] - Source: Custom Source -> Sink: Print to Std. Out (1/1)#0 
(fbb512e099d031470403965ba1830e8c) switched from RUNNING to FAILED with failure 
cause: 
org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil$FlinkKinesisStreamConsumerRegistrarException:
 Error registering stream: a367945-consumer-stream-dit

at 
org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil.registerStreamConsumers(StreamConsumerRegistrarUtil.java:125)

at 
org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil.registerStreamConsumers(StreamConsumerRegistrarUtil.java:106)

at 
org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil.lazilyRegisterStreamConsumers(StreamConsumerRegistrarUtil.java:75)

at 
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.(KinesisDataFetcher.java:429)

at 
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.(KinesisDataFetcher.java:365)

at 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.createFetcher(FlinkKinesisConsumer.java:536)

at 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:308)

at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)

at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)

at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)

Suppressed: java.lang.NullPointerException

at 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.close(FlinkKinesisConsumer.java:421)

at 
org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:41)

at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:117)

at 
org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:864)

at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:843)

at 
org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:756)

at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cleanUpInvoke(SourceStreamTask.java:186)

at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:662)

at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)

at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.util.concurrent.ExecutionException: 
org.apache.flink.kinesis.shaded.software.amazo

Re: Apache Flink - Can AllWindowedStream be parallel ?

[Yun Tang]

Hi Singh,

All the output operator transformed by AllWindowedStream would be 
SingleOutputStreamOperator, which cannot be parallel.


[1] 
https://github.com/apache/flink/blob/master/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/AllWindowedStream.scala

Best
Yun Tang

From: M Singh 
Sent: Sunday, January 23, 2022 4:24
To: User-Flink 
Subject: Apache Flink - Can AllWindowedStream be parallel ?

Hi Folks:

The documentation for AllWindowedStream 
(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-rarr-allwindowedstream)
 has a note:


This is in many cases a non-parallel transformation. All records will be 
gathered in one task for the windowAll operator.


Does this mean that in some cases it might be parallel ?  If so, is there an 
example of such a scenario ?

Thanks

Re: Question about MapState size

[Yun Tang]

Hi Abdul,

What does "only count pertaining to the specific key of partition" mean? The 
counting size is for the map related to a specific selected key or the all the 
maps in the whole map state?

You can leverage RocksDB's native metrics to monitor the rocksDB usage, such as 
total-sst-files-size[1] to know the total sst files on disks of each rocksDB.

[1] 
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/config/#state-backend-rocksdb-metrics-total-sst-files-size


Best
Yun Tang

From: Abdul Rahman 
Sent: Saturday, January 22, 2022 14:51
To: user@flink.apache.org 
Subject: Question about MapState size

Hello,

I have a streaming application that has an operator based on the
KeyedCoProcessFunction. The operator has a MapState object.  I store
some data in this operator with a fixed ttl. I would like to monitor
the size/count of this state over time since its related to some
operational metrics we want to track. Seems like a simple thing to do;
but I havent come up with a way to do so

Given that iterating over the complete map is an expensive operation,
I only plan to do so periodically.  The first issue is that , the
stream is keyed, so any time i do a count of the mapstate, i dont get
the complete size of the state object, but only count pertaining to
the specific key of partition. Is there a way around this ?

Secondly, is there a way to monitor rocksdb usage over time. I can
find managed memory metrics. but this does not include disk space
rocksdb used. is there a way to get this from standard flink metrics;
either task manager or job manager ?

Query regarding Kafka Source and Kafka Sink in 1.14.3

[Mahima Agarwal]

Hi Team,

I am trying to set the following properties in Kafka Source API in flink
1.14.3 version.
-> client.id.prefix
-> partition.discovery.interval.ms

But I am getting the below mentioned warning in taskmanager logs:

1. WARN  org.apache.kafka.clients.consumer.ConsumerConfig [] -
The configuration 'client.id.prefix' was supplied but isn't a known config.
2. WARN  org.apache.kafka.clients.consumer.ConsumerConfig [] -
The configuration 'partition.discovery.interval.ms' was supplied but isn't
a known config.

What could be the reason for this warning?

Also, in flink version 1.13.2 we were able write timestamp to kafka
using setWriteTimestampToKafka(true) method of FlinkKafkaProducer class.
Similar to this how can we write timestamp to kafka using KafkaSink API in
flink 1.14.3?

Any leads would be appreciated.

Thanks and Regards
Mahima Agarwal