Re: read a finite number of messages from Kafka using Kafka connector without extending it?

[Konstantin Knauf]

Hi Yu,

I am not aware of a way to use the FlinkKafkaConsumer to generate a finite
data stream. You could, of course, use a FilterFunction or FlatMapFunction
to filter out events outside of the time interval right after the Kafka
Source. This way you would not need to modify it, but you have to stop the
job manually once no new data is processed.

Generally, I think, there is no way to only read messages from a certain
time interval from a Kafka topic (regardless of Flink). So, you would
always need to read more events and filter.

Cheers,

Konstantin

On Sat, Feb 16, 2019 at 1:10 AM Yu Yang  wrote:

> Hi,
>
> We are considering to use Flink SQL for ad hoc data analytics on real-time
> Kafka data, and want to limit the queries to process data in the past 5-10
> minutes. To achieve that, one possible approach is to extend the current
> Kafka connect to have it only read messages in a given period of time to
> generate a finite DataStream. I am wondering if there is an alternative to
> this approach. Any suggestions will be very much appreciated.
>
> Regards,
> -Yu
>
>
>

-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

Re: Reading messages from start - new job submission

[sohimankotia]

Hi,

Which version of flink you r using ?

Reset offset to earliest : 

https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration


Thanks




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: [ANNOUNCE] Apache Flink 1.7.2 released

[Robert Metzger]

Thank you Gordon!

Help spread the word here:
https://twitter.com/ApacheFlink/status/1097416946688102401



On Mon, Feb 18, 2019 at 7:41 AM Dian Fu  wrote:

> Great job. It's great to have a more stable 1.7 release available. Thanks
> @Gordon for making it happen.
>
> Regards,
> Dian
>
> 在 2019年2月18日，下午2:26，vino yang  写道：
>
> Great job! Thanks for being the release manager @Gordon.
>
> Best,
> Vino
>
> Hequn Cheng  于2019年2月18日周一 下午2:16写道：
>
>> Thanks a lot for the great release @Gordon.
>> Also thanks for the work by the whole community. :-)
>>
>> Best, Hequn
>>
>>
>> On Mon, Feb 18, 2019 at 2:12 PM jincheng sun 
>> wrote:
>>
>> > Thanks a lot for being our release manager Gordon > >,
>> > Great job!
>> >  And also a big thanks to the community for making this release
>> possible.
>> >
>> > Cheers,
>> > Jincheng
>> >
>> >
>> > Tzu-Li (Gordon) Tai  于2019年2月18日周一 上午10:29写道：
>> >
>> >> Hi,
>> >>
>> >> The Apache Flink community is very happy to announce the release of
>> >> Apache Flink 1.7.2, which is the second bugfix release for the Apache
>> >> Flink 1.7 series.
>> >>
>> >> Apache Flink® is an open-source stream processing framework for
>> >> distributed, high-performing, always-available, and accurate data
>> >> streaming applications.
>> >>
>> >> The release is available for download at:
>> >> https://flink.apache.org/downloads.html
>> >> 
>> >>
>> >> Please check out the release blog post for an overview of the
>> >> improvements for this bugfix release:
>> >> https://flink.apache.org/news/2019/02/15/release-1.7.2.html
>> >>
>> >> The full release notes are available in Jira:
>> >>
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344632
>> >>
>> >> We would like to thank all contributors of the Apache Flink community
>> >> who made this release possible!
>> >>
>> >> Regards,
>> >> Gordon
>> >>
>> >
>>
>
>

Re: StreamingFileSink causing AmazonS3Exception

[Kostas Kloudas]

Hi Padarn,

This is the jira issue:  https://issues.apache.org/jira/browse/FLINK-11187
and the fix, as you can see, was first included in version 1.7.2.

Cheers,
Kostas

On Mon, Feb 18, 2019 at 3:49 AM Padarn Wilson 
wrote:

> Hi Addison, Kostas, Steffan,
>
> I am also encountering this exact issue. I cannot find a JIRA ticket on
> this, is there some planned work on implementing a fix?
>
> @Addison - Did you manage to find a fix that you could apply without
> modifying the Flink codebase? If possible it would be better not patch the
> code base and compile a custom image.
>
> Thanks,
> Padarn
>
> On Tue, Dec 18, 2018 at 5:37 AM Addison Higham  wrote:
>
>> Oh this is timely!
>>
>> I hope I can save you some pain Kostas! (cc-ing to flink dev to get
>> feedback there for what I believe to be a confirmed bug)
>>
>>
>> I was just about to open up a flink issue for this after digging (really)
>> deep and figuring out the issue over the weekend.
>>
>> The problem arises due the flink hands input streams to the
>> S3AccessHelper. If you turn on debug logs for s3, you will eventually see
>> this stack trace:
>>
>> 2018-12-17 05:55:46,546 DEBUG
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient  -
>> FYI: failed to reset content inputstream before throwing up
>> java.io.IOException: Resetting to invalid mark
>>   at java.io.BufferedInputStream.reset(BufferedInputStream.java:448)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.internal.SdkBufferedInputStream.reset(SdkBufferedInputStream.java:106)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:112)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.event.ProgressInputStream.reset(ProgressInputStream.java:168)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:112)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.lastReset(AmazonHttpClient.java:1145)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1070)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3306)
>>   at
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3291)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:1576)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$uploadPart$8(WriteOperationHelper.java:474)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:123)
>>   at
>> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadPart(WriteOperationHelper.java:471)
>>   at
>> org.apache.flink.fs.s3hadoop.HadoopS3AccessHelper.uploadPart(HadoopS3AccessHelper.java:74)
>>   at
>> org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUploadImpl$UploadTask.run(RecoverableMultiPartUploadImpl.java:319)
>>   at
>> org.apache.flink.fs.s3.common.utils.BackPressuringExecutor$SemaphoreReleasingRunnable.run(BackPressuringExecutor.java:92)
>>   at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11

Re: Confusion in Heartbeat configurations

[zhijiang]

Hi sohimankotia,

In order not to strongly rely on the akka implementation, flink implements the 
heartbeat mechanism for health monitor for the components of TaskExecutor, 
JobMaster and ResourceManager from FLIP6. So you can see two sets of heartbeat 
setting, one is for akka internal implementation prefix with `akka` and the 
other is flink internal implementation.

Best,
Zhijiang
--
From:sohimankotia 
Send Time:2019年2月18日(星期一) 14:40
To:user 
Subject:Confusion in Heartbeat configurations

Hi, 

In
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html
link there are two heartbeat config are mentioned . 

akka.watch.heartbeat.interval
akka.watch.heartbeat.pause

Vs

heartbeat.interval
heartbeat.timeout


Can u guys pls explain what exactly is difference between them and which
component of job execution graph they impact . 

Thanks




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: [Meetup] Apache Flink+Beam+others in Seattle. Feb 21.

[Fabian Hueske]

Thank you Pablo!

Am Fr., 15. Feb. 2019 um 20:42 Uhr schrieb Pablo Estrada :

> Hello everyone,
> There is an upcoming meetup happening in the Google Seattle office, on
> February 21st, starting at 5:30pm:
> https://www.meetup.com/seattle-apache-flink/events/258723322/
>
> People will be chatting about Beam, Flink, Hive, and AthenaX
> . Anyone who is interested, please feel
> free to join : )
>
> FWIW, I am not organizing it, but didn't see it advertised on the lists.
> I'm bringing it here so people know.
> Best
> -P.
>

Submitting job to Flink on yarn timesout on flip-6 1.5.x

[Richard Deurwaarder]

Hello,

I am trying to upgrade our job from flink 1.4.2 to 1.7.1 but I keep running
into timeouts after submitting the job.

The flink job runs on our hadoop cluster and starts using Yarn.

Relevant config options seem to be:

jobmanager.rpc.port: 55501

recovery.jobmanager.port: 55502

yarn.application-master.port: 55503

blob.server.port: 55504


I've seen the following behavior:
  - Using the same flink-conf.yaml as we used in 1.4.2: 1.5.6 / 1.6.3 /
1.7.1 all versions timeout while 1.4.2 works.
  - Using 1.5.6 with "mode: legacy" (to switch off flip-6) works
  - Using 1.7.1 with "mode: legacy" gives timeout (I assume this option was
removed but the documentation is outdated?
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#legacy
)

When the timeout happens I get the following stacktrace:

INFO class java.time.Instant does not contain a getter for field seconds
2019-02-18T10:16:56.815+01:00
INFO class com.bol.fin_hdp.cm1.domain.Cm1Transportable does not contain a
getter for field globalId 2019-02-18T10:16:56.815+01:00
INFO Submitting job 5af931bcef395a78b5af2b97e92dcffe (detached: false).
2019-02-18T10:16:57.182+01:00
INFO 
2019-02-18T10:29:27.527+01:00
INFO The program finished with the following exception:
2019-02-18T10:29:27.564+01:00
INFO org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error. 2019-02-18T10:29:27.601+01:00
INFO at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
2019-02-18T10:29:27.638+01:00
INFO at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
2019-02-18T10:29:27.675+01:00
INFO at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
2019-02-18T10:29:27.711+01:00
INFO at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:798)
2019-02-18T10:29:27.747+01:00
INFO at
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:289)
2019-02-18T10:29:27.784+01:00
INFO at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:215)
2019-02-18T10:29:27.820+01:00
INFO at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1035)
2019-02-18T10:29:27.857+01:00
INFO at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:)
2019-02-18T10:29:27.893+01:00
INFO at java.security.AccessController.doPrivileged(Native Method)
2019-02-18T10:29:27.929+01:00
INFO at javax.security.auth.Subject.doAs(Subject.java:422)
2019-02-18T10:29:27.968+01:00
INFO at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
2019-02-18T10:29:28.004+01:00
INFO at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
2019-02-18T10:29:28.040+01:00
INFO at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:)
2019-02-18T10:29:28.075+01:00
INFO Caused by: java.lang.RuntimeException:
org.apache.flink.client.program.ProgramInvocationException: Could not
retrieve the execution result. 2019-02-18T10:29:28.110+01:00
INFO at
com.bol.fin_hdp.job.starter.IntervalJobStarter.startJob(IntervalJobStarter.java:43)
2019-02-18T10:29:28.146+01:00
INFO at
com.bol.fin_hdp.job.starter.IntervalJobStarter.startJobWithConfig(IntervalJobStarter.java:32)
2019-02-18T10:29:28.182+01:00
INFO at com.bol.fin_hdp.Main.main(Main.java:8) 2019-02-18T10:29:28.217+01:00
INFO at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-02-18T10:29:28.253+01:00
INFO at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-02-18T10:29:28.289+01:00
INFO at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-02-18T10:29:28.325+01:00
INFO at java.lang.reflect.Method.invoke(Method.java:498)
2019-02-18T10:29:28.363+01:00
INFO at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
2019-02-18T10:29:28.400+01:00
INFO ... 12 more 2019-02-18T10:29:28.436+01:00
INFO Caused by: org.apache.flink.client.program.ProgramInvocationException:
Could not retrieve the execution result. 2019-02-18T10:29:28.473+01:00
INFO at
org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:258)
2019-02-18T10:29:28.509+01:00
INFO at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
2019-02-18T10:29:28.544+01:00
INFO at
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
2019-02-18T10:29:28.581+01:00
INFO at com.bol.fin_hdp.cm1.job.Job.execute(Job.java:54)
2019-02-18T10:29:28.617+01:00
INFO at
com.bol.fin_hdp.job.starter.IntervalJobStarter.startJob(IntervalJobStarter.java:41)
2019-02-18T10:29:28.654+01:00
INFO ... 19 more 2019-02-18T10:29:28.693+01:00
INFO Caused by: org.apache.flink.runtime.client.JobSubmissionException:
Failed to submit JobGraph. 2019-02-18T10:29:28.730+01:00
IN

Re: [Table] Types of query result and tablesink do not match error

[Fabian Hueske]

Hi François,

I had a look at the code and the GenericTypeInfo checks equality by
comparing the classes the represent (Class == Class).
Class does not override the default implementation of equals, so this is an
instance equality check. The check can evaluate to false, if Map was loaded
by two different classloaders or if you use different implementations of
the interface.

Not sure if this explains the issue you are facing but it's a starting
point.

Best, Fabian

Am Fr., 8. Feb. 2019 um 18:08 Uhr schrieb françois lacombe <
francois.laco...@dcbrain.com>:

> Hi all,
>
> An error is currently raised when using table.insertInto("registeredSink")
> in Flink 1.7.0 when types of table and sink don't match.
>
> I've got the following :
> org.apache.flink.table.api.ValidationException: Field types of query
> result and registered TableSink null do not match.
> Query result schema: [dynamicFields: Map, staticFields: Map]
> TableSink schema:[dynamicFields: Map, staticFields: Map]
> at
> org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:876)
> at org.apache.flink.table.api.Table.insertInto(table.scala:918)
>
> Schemas are the same
> All fields got the GenericType type and I don't understand
> why they are so different.
>
> Have you any additional way to get extra debug information ?
> Any hint ?
>
> All the best
>
> François
>
>
>    
> 
> 
>
> [image: Arbre vert.jpg] Pensez à la planète, imprimer ce papier que si
> nécessaire
>

Re: [DISCUSS] Adding a mid-term roadmap to the Flink website

[Jark Wu]

Thanks Stephan for the proposal and a big +1 to this!

I also think it's a good idea to add a link of discussion/FLIP/JIRA to each
item as Zhijiang mentioned above.
This would be a great help for keeping track of progress and joining in the
discussion easily.

Best,
Jark

On Fri, 15 Feb 2019 at 11:34, jincheng sun  wrote:

> Hi Stephan,
>
> Thanks for the clarification! You are right, we have never initiated a
> discussion about supporting OVER Window on DataStream, we can discuss it in
> a separate thread. I agree with you add the item after move the discussion
> forward.
>
> +1 for putting the roadmap on the website.
> +1 for periodically update the roadmap, as mentioned by Fabian, we can
> update it at every feature version release.
>
> Thanks,
> Jincheng
>
> Stephan Ewen  于2019年2月14日周四 下午5:44写道：
>
>> Thanks Jincheng and Rong Rong!
>>
>> I am not deciding a roadmap and making a call on what features should be
>> developed or not. I was only collecting broader issues that are already
>> happening or have an active FLIP/design discussion plus committer support.
>>
>> Do we have that for the suggested issues as well? If yes , we can add
>> them (can you point me to the issue/mail-thread), if not, let's try and
>> move the discussion forward and add them to the roadmap overview then.
>>
>> Best,
>> Stephan
>>
>>
>> On Wed, Feb 13, 2019 at 6:47 PM Rong Rong  wrote:
>>
>>> Thanks Stephan for the great proposal.
>>>
>>> This would not only be beneficial for new users but also for
>>> contributors to keep track on all upcoming features.
>>>
>>> I think that better window operator support can also be separately group
>>> into its own category, as they affects both future DataStream API and batch
>>> stream unification.
>>> can we also include:
>>> - OVER aggregate for DataStream API separately as @jincheng suggested.
>>> - Improving sliding window operator [1]
>>>
>>> One more additional suggestion, can we also include a more extendable
>>> security module [2,3] @shuyi and I are currently working on?
>>> This will significantly improve the usability for Flink in corporate
>>> environments where proprietary or 3rd-party security integration is needed.
>>>
>>> Thanks,
>>> Rong
>>>
>>>
>>> [1]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html
>>> [2]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html
>>> [3]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html
>>>
>>>
>>>
>>>
>>> On Wed, Feb 13, 2019 at 3:39 AM jincheng sun 
>>> wrote:
>>>
 Very excited and thank you for launching such a great discussion,
 Stephan !

 Here only a little suggestion that in the Batch Streaming Unification
 section, do we need to add an item:

 - Same window operators on bounded/unbounded Table API and DataStream
 API
 (currently OVER window only exists in SQL/TableAPI, DataStream API does
 not yet support)

 Best,
 Jincheng

 Stephan Ewen  于2019年2月13日周三 下午7:21写道：

> Hi all!
>
> Recently several contributors, committers, and users asked about
> making it more visible in which way the project is currently going.
>
> Users and developers can track the direction by following the
> discussion threads and JIRA, but due to the mass of discussions and open
> issues, it is very hard to get a good overall picture.
> Especially for new users and contributors, is is very hard to get a
> quick overview of the project direction.
>
> To fix this, I suggest to add a brief roadmap summary to the homepage.
> It is a bit of a commitment to keep that roadmap up to date, but I think
> the benefit for users justifies that.
> The Apache Beam project has added such a roadmap [1]
> , which was received very well by
> the community, I would suggest to follow a similar structure here.
>
> If the community is in favor of this, I would volunteer to write a
> first version of such a roadmap. The points I would include are below.
>
> Best,
> Stephan
>
> [1] https://beam.apache.org/roadmap/
>
> 
>
> Disclaimer: Apache Flink is not governed or steered by any one single
> entity, but by its community and Project Management Committee (PMC). This
> is not a authoritative roadmap in the sense of a plan with a specific
> timeline. Instead, we share our vision for the future and major 
> initiatives
> that are receiving attention and give users and contributors an
> understanding what they can look forward to.
>
> *Future Role of Table API and DataStream API*
>   - Table API becomes first class citizen
>   - Table API becomes primary API for analytics use c

Re: Is group.id required in Kafka connector for offsets to be stored in checkpoint?

[Konstantin Knauf]

Hi David, Hi Sohi,

this should not be the case. If a savepoint/checkpoint is provided, Flink
should always take the offsets from the state regardless of the `group.id`
provided. Which Flink version and which FlinkKafkaConsumer version do you
use?

Best,

Konstantin

On Mon, Feb 18, 2019 at 5:50 AM sohimankotia  wrote:

> Hi David,
>
> We are also running streaming jobs over Kafka source .
>
> Yes : Consumer Group Id needs to be set for Kafka source explicitly t .
>
> We are also using checkpointing and save points for persisting state . Any
> time we change group id it starts from latest offset(default Kafka
> connector
> behavior) .
>
>
>
>
> Thanks
> Sohi
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>


-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

Re: Is group.id required in Kafka connector for offsets to be stored in checkpoint?

[sohimankotia]

Yes Konstantin Knauf-2 . You are right . 



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Confusion in Heartbeat configurations

[sohimankotia]

Thanks Zhijiang . 

Sorry to ask again . So both set of heartbeats are implementing same feature
. 

If Yes , which one has highest priority to detect failure .
If no , can you explain little more or point to some references to
understand difference .

Thanks
Sohi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Reading messages from start - new job submission

[avilevi]

Hi ,
I'm using 1.7.0

Cheers



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Reduce one event under multiple keys

[Fabian Hueske]

Hi Stephen,

Sorry for the late response.
If you don't need to match open and close events, your approach of using a
flatMap to fan-out for the hierarchical folder structure and a window
operator (or two for open and close) for counting and aggregating should be
a good design.

Best, Fabian

Am Mo., 11. Feb. 2019 um 11:29 Uhr schrieb Stephen Connolly <
stephen.alan.conno...@gmail.com>:

>
>
> On Mon, 11 Feb 2019 at 09:42, Fabian Hueske  wrote:
>
>> Hi Stephen,
>>
>> A window is created with the first record that is assigned to it.
>> If the windows are based on time and a key, than no window will be
>> created (and not space be occupied) if there is not a first record for a
>> key and time interval.
>>
>> Anyway, if tracking the number of open files & average opening time is
>> your use case, you might want to implement the logic with a ProcessFunction
>> instead of a window.
>> The reason is that it is that time windows don't share state, i.e., the
>> information about an opened but not yet closed file would not be "carried
>> over" to the next window.
>> However, if you use a ProcessFunction, you are responsible for cleaning
>> up the state.
>>
>
> Ahh but I am cheating by ensuring the events are rich enough that I do not
> need to match them.
>
> I get the "open" (they are not really "open" events - I have mapped to an
> analogy... it might be more like a build job start events... or not... I'm
> not at liberty to say ;-) ) events because I need to count the number of
> "open"s per time period.
>
> I get the "close" events and they include the duration plus other
> information that can then be transformed into the required metrics... yes I
> could derive the "open" from the "close" by subtracting the duration but:
>
> 1. they would cross window boundaries quite often, leading to repeated
> fetch-update-write operations on the backing data store
> 2. they wouldn't be as "live" and one of the things we need to know is how
> many "open"s there are in the previous window... given some durations can
> be many days, waiting for the "close" event to create the "open" metric
> would not be a good plan.
>
> Basically, I am pushing some of the calculations to the edge where there
> is state that makes those calculations cheap and then the rich events are
> *hopefully* easy to aggregate with just simple aggregation functions that
> only need to maintain the running total... at least that's what the PoC I
> am experimenting with Flink should show
>
>
>>
>> Hope this helps,
>> Fabian
>>
>> Am So., 10. Feb. 2019 um 20:36 Uhr schrieb Stephen Connolly <
>> stephen.alan.conno...@gmail.com>:
>>
>>>
>>>
>>> On Sun, 10 Feb 2019 at 09:09, Chesnay Schepler 
>>> wrote:
>>>
 This sounds reasonable to me.

 I'm a bit confused by this question: "*Additionally, I am (naïevely)
 hoping that if a window has no events for a particular key, the
 memory/storage costs are zero for that key.*"

 Are you asking whether a key that was received in window X (as part of
 an event) is still present in window x+1? If so, then the answer is no; a
 key will only be present in a given window if an event was received that
 fits into that window.

>>>
>>> To confirm:
>>>
>>> So let's say I'l tracking the average time a file is opened in folders.
>>>
>>> In window N we get the events:
>>>
>>> {"source":"ca:fe:ba:be","action":"open","path":"/foo/bar/README.txt"}
>>>
>>> {"source":"ca:fe:ba:be","action":"close","path":"/foo/bar/README.txt","duration":"67"}
>>> {"source":"ca:fe:ba:be","action":"open","path":"/foo/bar/User guide.txt"}
>>> {"source":"ca:fe:ba:be","action":"open","path":"/foo/bar/Admin
>>> guide.txt"}
>>>
>>> So there will be aggregates stored for
>>> ("ca:fe:ba:be","/"), ("ca:fe:ba:be","/foo"), ("ca:fe:ba:be","/foo/bar"),
>>> ("ca:fe:ba:be","/foo/bar/README.txt"), etc
>>>
>>> In window N+1 we do not get any events at all.
>>>
>>> So the memory used by my aggregation functions from window N will be
>>> freed and the storage will be effectively zero (modulo any follow on
>>> processing that might be on a longer window)
>>>
>>> This seems to be what you are saying... in which case my naïeve hope was
>>> not so naïve! w00t!
>>>
>>>

 On 08.02.2019 13:21, Stephen Connolly wrote:

 Ok, I'll try and map my problem into something that should be familiar
 to most people.

 Consider collection of PCs, each of which has a unique ID, e.g.
 ca:fe:ba:be, de:ad:be:ef, etc.

 Each PC has a tree of local files. Some of the file paths are
 coincidentally the same names, but there is no file sharing between PCs.

 I need to produce metrics about how often files are opened and how long
 they are open for.

 I need for every X minute tumbling window not just the cumulative
 averages for each PC, but the averages for each file as well as the
 cumulative averegaes for each folder and their sub-folders.

 I have a stream of events like
>>>

Re: Limit in batch flink sql job

[Fabian Hueske]

Thanks for pointing this out!
This is indeed a bug in the documentation.

I'll fix that.

Thank you,
Fabian

Am Mi., 13. Feb. 2019 um 02:04 Uhr schrieb yinhua.dai <
yinhua.2...@outlook.com>:

> OK, thanks.
> It might be better to update the document which has the following example
> that confused me.
>
> SELECT *
> FROM Orders
> LIMIT 3
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Re: KafkaTopicPartition internal class treated as generic type serialization

[Fabian Hueske]

Hi Eric,

I did a quick search in our Jira to check if this is a known issue but
didn't find anything.
Maybe Gordon (in CC) knows a bit more about this problem.

Best, Fabian

Am Fr., 15. Feb. 2019 um 11:08 Uhr schrieb Eric Troies :

> Hi, I'm having the exact same issue with flink 1.4.0 using scala 2.11 .
>
> Do you have any suggestion on how to fix this ?
>
> I don't see how to register a custom serializer for a class I did not write.
>
> Thanks !
>
>
>
> > I disabled generic type serialization via
>
> > env.getConfig.disableGenericTypes()
>
> > and got the following exception when running my job on a standalone cluster.
>
> > Caused by: java.lang.UnsupportedOperationException: Generic types have been
> > disabled in the ExecutionConfig and type
> > org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition is
> > treated as a generic type.
>
> > Is this expected?  I figure I may just have to register a custom serializer
> > for this, but (wrongly?) expected the class to have its own serializer,
> > although there's not much overhead for this class.
>
> > I do understand the documentation behind using disableGenericTypes() only in
> > application development, but would this possibly hide other classes since it
> > could possibly eagerly fail on this class?
>
>

Re: Test FileUtilsTest.testDeleteDirectory failed when building Flink

[Fabian Hueske]

Hi Paul,

Which components (Flink, JDK, Docker base image, ...) are you upgrading and
which versions do you come from?
I think it would be good to check how (and with which options) the JVM in
the container is started.

Best, Fabian


Am Fr., 15. Feb. 2019 um 09:50 Uhr schrieb Paul Lam :

> Hi all,
>
> Recently we migrate Flink build to a new docker image, after which the
> build job always fails with test errors on local file system permissions.
>
> For example: FileUtilsTest.testDeleteDirectory:129 this should fail with
> an exception.
>
> I notice the following statements in the javadoc of
> `java.io.File.setWritable`:
>
> > On some platforms it may be possible to start the Java virtual machine
> with special privileges that allow it to modify files that disallow write
> operations.
>
> I think it’s what the test is designed for and where the problem lies.
>
> Could anyone help me with this? Thanks a lot!
>
> WRT the environment:
>
> - Flink version: 1.7.1
> - JDK: open jdk 1.8.0_111
> - OS version: debian 8
>
> Best,
> Paul Lam
>
>

Dataset sampling

[Flavio Pompermaier]

Hi to all,
is there any plan to support different sampling techniques?
This would be very helpful when interactive table API will be available..

Best,
Flavio

Re: [DISCUSS] Adding a mid-term roadmap to the Flink website

[Shaoxuan Wang]

Hi Stephan,

Thanks for summarizing the work&discussions into a roadmap. It really helps
users to understand where Flink will forward to. The entire outline looks
good to me. If appropriate, I would recommend to add another two attracting
categories in the roadmap.

*Flink ML Enhancement*
  - Refactor ML pipeline on TableAPI
  - Python support for TableAPI
  - Support streaming training & inference.
  - Seamless integration of DL engines (Tensorflow, PyTorch etc)
  - ML platform with a group of AI tooling
Some of these work have already been discussed in the dev mail list.
Related JIRA (FLINK-11095) and discussion:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Embracing-Table-API-in-Flink-ML-td25368.html
;
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Python-and-Non-JVM-Language-Support-in-Flink-td25905.html


*Flink-Runtime-Web Improvement*
  - Much of this comes via Blink
  - Refactor the entire module to use latest Angular (7.x)
  - Add resource information at three levels including Cluster, TaskManager
and Job
  - Add operator level topology and and data flow tracing
  - Add new metrics to track the back pressure, filter and data skew
  - Add log association to Job, Vertex and SubTasks
Related JIRA (FLINK-10705) and discussion:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html


What do you think?

Regards,
Shaoxuan



On Wed, Feb 13, 2019 at 7:21 PM Stephan Ewen  wrote:

> Hi all!
>
> Recently several contributors, committers, and users asked about making it
> more visible in which way the project is currently going.
>
> Users and developers can track the direction by following the discussion
> threads and JIRA, but due to the mass of discussions and open issues, it is
> very hard to get a good overall picture.
> Especially for new users and contributors, is is very hard to get a quick
> overview of the project direction.
>
> To fix this, I suggest to add a brief roadmap summary to the homepage. It
> is a bit of a commitment to keep that roadmap up to date, but I think the
> benefit for users justifies that.
> The Apache Beam project has added such a roadmap [1]
> , which was received very well by the
> community, I would suggest to follow a similar structure here.
>
> If the community is in favor of this, I would volunteer to write a first
> version of such a roadmap. The points I would include are below.
>
> Best,
> Stephan
>
> [1] https://beam.apache.org/roadmap/
>
> 
>
> Disclaimer: Apache Flink is not governed or steered by any one single
> entity, but by its community and Project Management Committee (PMC). This
> is not a authoritative roadmap in the sense of a plan with a specific
> timeline. Instead, we share our vision for the future and major initiatives
> that are receiving attention and give users and contributors an
> understanding what they can look forward to.
>
> *Future Role of Table API and DataStream API*
>   - Table API becomes first class citizen
>   - Table API becomes primary API for analytics use cases
>   * Declarative, automatic optimizations
>   * No manual control over state and timers
>   - DataStream API becomes primary API for applications and data pipeline
> use cases
>   * Physical, user controls data types, no magic or optimizer
>   * Explicit control over state and time
>
> *Batch Streaming Unification*
>   - Table API unification (environments) (FLIP-32)
>   - New unified source interface (FLIP-27)
>   - Runtime operator unification & code reuse between DataStream / Table
>   - Extending Table API to make it convenient API for all analytical use
> cases (easier mix in of UDFs)
>   - Same join operators on bounded/unbounded Table API and DataStream API
>
> *Faster Batch (Bounded Streams)*
>   - Much of this comes via Blink contribution/merging
>   - Fine-grained Fault Tolerance on bounded data (Table API)
>   - Batch Scheduling on bounded data (Table API)
>   - External Shuffle Services Support on bounded streams
>   - Caching of intermediate results on bounded data (Table API)
>   - Extending DataStream API to explicitly model bounded streams (API
> breaking)
>   - Add fine fault tolerance, scheduling, caching also to DataStream API
>
> *Streaming State Evolution*
>   - Let all built-in serializers support stable evolution
>   - First class support for other evolvable formats (Protobuf, Thrift)
>   - Savepoint input/output format to modify / adjust savepoints
>
> *Simpler Event Time Handling*
>   - Event Time Alignment in Sources
>   - Simpler out-of-the box support in sources
>
> *Checkpointing*
>   - Consistency of Side Effects: suspend / end with savepoint (FLIP-34)
>   - Failed checkpoints explicitly aborted on TaskManagers (not only on
> coordinator)
>
> *Automatic scaling (adjusting parallelism)*
>   - Reactiv

Re: Flink 1.6 Yarn Session behavior

[Jins George]

Thank you Gary. That was helpful.

Thanks,

Jins George

On 2/17/19 10:03 AM, Gary Yao wrote:
Hi Jins George,

Every TM brings additional overhead, e.g., more heartbeat messages. However, a
cluster with 28 TMs would not be considered big as there are users that are
running Flink applications on thousands of cores [1][2].

Best,
Gary

[1] 
https://flink.apache.org/flink-architecture.html#run-applications-at-any-scale
[2] 
https://de.slideshare.net/FlinkForward/flink-forward-sf-2017-stephan-ewen-experiences-running-flink-at-very-large-scale

On Thu, Feb 14, 2019 at 6:59 PM Jins George 
mailto:jins.geo...@aeris.net>> wrote:

Thanks Gary. Understood the behavior.

I am leaning towards running 7 TM on each machine(8 core), I have 4 nodes, that 
will end up 28 taskmanagers and 1 job manager. I was wondering if this can 
bring additional burden on jobmanager? Is it recommended?

Thanks,

Jins George

On 2/14/19 8:49 AM, Gary Yao wrote:
Hi Jins George,

This has been asked before [1]. The bottom line is that you currently cannot
pre-allocate TMs and distribute your tasks evenly. You might be able to
achieve a better distribution across hosts by configuring fewer slots in your
TMs.

Best,
Gary

[1] 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-5-job-distribution-over-cluster-nodes-td21588.html


On Wed, Feb 13, 2019 at 6:20 AM Tzu-Li (Gordon) Tai 
mailto:tzuli...@apache.org>> wrote:
Hi,

I'm forwarding this question to Gary (CC'ed), who most likely would have an 
answer for your question here.

Cheers,
Gordon

On Wed, Feb 13, 2019 at 8:33 AM Jins George 
mailto:jins.geo...@aeris.net>> wrote:

Hello community,

I am trying to  upgrade a  Flink Yarn session cluster running BEAM pipelines  
from version 1.2.0 to 1.6.3.

Here is my session start command: yarn-session.sh -d -n 4  -jm 1024 -tm 3072 -s 
7

Because of the dynamic resource allocation,  no taskmanager gets created 
initially. Now once I submit a job with parallelism 5, I see that 1 
task-manager gets created and all 5 parallel instances are scheduled on the 
same taskmanager( because I have 7 slots).  This can create hot spot as only 
one physical node ( out of 4 in my case) is utilized for processing.

I noticed the legacy mode, which would provision all task managers at cluster 
creation, but since legacy mode is expected to go away soon, I didn't want to 
try that route.

Is there a way I can configure the multiple jobs or parallel instances of same 
job spread across all the available Yarn nodes and continue using the 'new' 
mode ?

Thanks,

Jins George

Re: Dataset sampling

[Fabian Hueske]

Hi Flavio,

I'm not aware of any particular plan to add sampling operators to the Table
API or SQL.
However, I agree. It would be a good feature.

Best, Fabian

Am Mo., 18. Feb. 2019 um 15:44 Uhr schrieb Flavio Pompermaier <
pomperma...@okkam.it>:

> Hi to all,
> is there any plan to support different sampling techniques?
> This would be very helpful when interactive table API will be available..
>
> Best,
> Flavio
>
>

Flink Streaming Job with OutputFormat stops without error message

[Marke Builder]

Hi,

I'm using a flink streaming job which read from kafka and write to hbase
with the OutputFormat. Like:
https://github.com/apache/flink/blob/master/flink-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteStreamExample.java

But after a certain time, the job ends without any error message.

Does anyone have an idea?

Thanks!
Marke

subscribe

[Artur Mrozowski]

art...@gmail.com

Re: ProducerFencedException when running 2 jobs with FlinkKafkaProducer

[Rohan Thimmappa]

Hi Tzu-Li,

Any updated on this. This is consistently reproducible.

Same jar - Separate source topic to Separate  destination topic.

This sort of blocker for flink upgrada. i tried with 1.7.2 but no luck.

org.apache.flink.streaming.connectors.kafka.FlinkKafka011Exception:
Failed to send data to Kafka: Producer attempted an operation with an
old epoch. Either there is a newer producer with the same
transactionalId, or the producer's transaction has been expired by the
broker.
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.checkErroneous(FlinkKafkaProducer011.java:994)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.invoke(FlinkKafkaProducer011.java:615)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.invoke(FlinkKafkaProducer011.java:94)
at 
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.invoke(TwoPhaseCommitSinkFunction.java:230)
at 
org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
at 
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
at com.comcast.ips.transformation.EMMFlatMap.flatMap(EMMFlatMap.java:98)
at com.comcast.ips.transformation.EMMFlatMap.flatMap(EMMFlatMap.java:33)
at 
org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
at 
org.apache.flink.streaming.runtime.operators.TimestampsAndPeriodicWatermarksOperator.processElement(TimestampsAndPeriodicWatermarksOperator.java:67)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)





Rohan

On Wed, Feb 13, 2019 at 12:33 AM Tzu-Li (Gordon) Tai 
wrote:

> Hi,
>
> I think this is unexpected. The generated transactional ids should not be
> clashing.
> Looking at the FlinkKafkaProducer code, it seems like the generation is
> only a function of the subtask id of the FlinkKafkaProducer, which could be
> the same across 2 different Kafka sources.
>
> I'm not completely certain about this. Piotr (in CC) might have more
> insights for this.
>
> Cheers,
> Gordon
>
> On Wed, Feb 13, 2019 at 9:15 AM Slotterback, Chris <
> chris_slotterb...@comcast.com> wrote:
>
>> Hey all,
>>
>>
>>
>> I am running into an issue where if I run 2 flink jobs (same jar,
>> different configuration), that produce to different kafka topics on the
>> same broker, using the 1.7 FlinkKafkaProducer set with EXACTLY_ONCE
>> semantics, both jobs go into a checkpoint exception loop every 15 seconds
>> or so:
>>
>>
>>
>> Caused by: org.apache.kafka.common.errors.ProducerFencedException:
>> Producer attempted an operation with an old epoch. Either there is a newer
>> producer with the same transactionalId, or the producer's transaction has
>> been expired by the broker.
>>
>>
>>
>> As soon as one of the jobs is cancelled, things go back to normal for the
>> other job. I tried manually setting the TRANSACTIONAL_ID_CONFIG config in
>> the producer to be unique for each of the jobs. My producer transaction
>> timeout is set to 5 minutes, and flink checkpointing is set to 1 minute. Is
>> there some way to prevent these jobs from tripping over each other in
>> execution while retaining exactly once processing?
>>
>

-

Re: ProducerFencedException when running 2 jobs with FlinkKafkaProducer

[Tzu-Li (Gordon) Tai]

Hi,

I just saw a JIRA opened for this:
https://issues.apache.org/jira/browse/FLINK-11654.

The JIRA ticket's description matches what I had in mind and can confirm
the bug assessment. Unfortunately, I currently do not have the capacity to
provide a fix and test for this.
For the meantime, I've made this a blocker for releasing 1.8.0. It would be
great if someone can try out the proposed fix mentioned in the JIRA, see if
it fixes the issue in your cases, and provide a PR for the patch.

Thanks,
Gordon

On Tue, Feb 19, 2019 at 9:46 AM Rohan Thimmappa 
wrote:

> Hi Tzu-Li,
>
> Any updated on this. This is consistently reproducible.
>
> Same jar - Separate source topic to Separate  destination topic.
>
> This sort of blocker for flink upgrada. i tried with 1.7.2 but no luck.
>
> org.apache.flink.streaming.connectors.kafka.FlinkKafka011Exception: Failed to 
> send data to Kafka: Producer attempted an operation with an old epoch. Either 
> there is a newer producer with the same transactionalId, or the producer's 
> transaction has been expired by the broker.
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.checkErroneous(FlinkKafkaProducer011.java:994)
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.invoke(FlinkKafkaProducer011.java:615)
>   at 
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.invoke(FlinkKafkaProducer011.java:94)
>   at 
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.invoke(TwoPhaseCommitSinkFunction.java:230)
>   at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
>   at 
> org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>   at com.comcast.ips.transformation.EMMFlatMap.flatMap(EMMFlatMap.java:98)
>   at com.comcast.ips.transformation.EMMFlatMap.flatMap(EMMFlatMap.java:33)
>   at 
> org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
>   at 
> org.apache.flink.streaming.runtime.operators.TimestampsAndPeriodicWatermarksOperator.processElement(TimestampsAndPeriodicWatermarksOperator.java:67)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
>   at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
>
>
>
>
>
> Rohan
>
> On Wed, Feb 13, 2019 at 12:33 AM Tzu-Li (Gordon) Tai 
> wrote:
>
>> Hi,
>>
>> I think this is unexpected. The generated transactional ids should not be
>> clashing.
>> Looking at the FlinkKafkaProducer code, it seems like the generation is
>> only a function of the subtask id of the FlinkKafkaProducer, which could be
>> the same across 2 different Kafka sources.
>>
>> I'm not completely certain about this. Piotr (in CC) might have more
>> insights for this.
>>
>> Cheers,
>> Gordon
>>
>> On Wed, Feb 13, 2019 at 9:15 AM Slotterback, Chris <
>> chris_slotterb...@comcast.com> wrote:
>>
>>> Hey all,
>>>
>>>
>>>
>>> I am running into an issue where if I run 2 flink jobs (same jar,
>>> different configuration), that produce to different kafka topics on the
>>> same broker, using the 1.7 FlinkKafkaProducer set with EXACTLY_ONCE
>>> semantics, both jobs go into a checkpoint exception loop every 15 seconds
>>> or so:
>>>
>>>
>>>
>>> Caused by: org.apache.kafka.c

The submitting is hanging when register a hdfs file as registerCacheFile in 1.7 based on RestClusterClient

[Joshua Fan]

Hi, all

As the title says, the submitting is always hanging there when the cache
file is not reachable, actually because the RestClient uses a java.io.File
to get the cache file.

I use RestClusterClient to submit job in Flink 1.7.

Below is instructions shown in
https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/#distributed-cache
:

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

// register a file from HDFS
env.registerCachedFile("hdfs:///path/to/your/file", "hdfsFile")

// register a local executable file (script, executable, ...)
env.registerCachedFile("file:///path/to/exec/file", "localExecFile", true)

Unfortunately, both the two examples can not be submitted, because
either hdfs:///path/to/your/file
or file:///path/to/exec/file is not reachable by the java.io.File, the http
post will not finish and the submitting is hanging.
When use env.registerCachedFile("/path/to/exec/file", "localExecFile",
true), the path is a regular local Path , the job can be submitted and the
cache file is available.

Is there some problems in the code or should I fire a jira?

Yours
Joshua

Re: subscribe

[Fabian Hueske]

Hi Artur,

In order to subscribe to Flink's user mailing list you need to send a mail
to user-subscr...@flink.apache.org

Best, Fabian

Am Mo., 18. Feb. 2019 um 20:34 Uhr schrieb Artur Mrozowski :

> art...@gmail.com
>