Re: [VOTE] Release 1.11.1, release candidate #1
+1 (non-binding) - Built from source - Verified hive connector tests for all hive versions - Played some simple cases with hive connector and everything seems fine On Sat, Jul 18, 2020 at 12:24 AM Rui Li wrote: > OK, I agree FLINK-18588 can wait for the next release. > > On Fri, Jul 17, 2020 at 11:56 PM Leonard Xu wrote: > >> +1 (non-binding) >> >> - checked/verified signatures and hashes >> - built from source code with scala 2.11 succeeded >> - checked that there are no missing artifacts >> - started a cluster, the Web UI was accessible, submitted a wordcount job >> and ran succeeded, no suspicious log output >> - test using SQL Client to submit job and the query result is expected >> - go through all issues which fix version property is 1.11.1, all issues >> are closed except FLINK-15794, >> and FLINK-15794 has fixed in master and 1.11.1 just wait for being >> fixed in 1.10.2. >> - the web PR looks good >> >> For FLINK-18588, I also agree with Timo to put it to 1.11.2 because >> it's a `Major` bug rather than `Blocker`. >> >> Best, >> Leonard > > > > -- > Best regards! > Rui Li > -- Best regards! Rui Li
Re: Flink Sinks
Prasanna, The Flink project does not have an SQS connector, and a quick google search hasn't found one. Nor does Flink have an HTTP sink, but with a bit of googling you can find that various folks have implemented this themselves. As for implementing SQS as a custom sink, if you need exactly once guarantees I believe you will need to limit yourself to using SQS FIFO queues, and you'll need to attach a MessageDeduplicationId to each record. SQS offers to do deduplication for you, but only supports deduplication within a 5 minute window -- which I would find too short to rely on during recovery or redeployment. See https://stackoverflow.com/a/28111986/2000823 for more on exactly once with SQS. Regards, David On Fri, Jul 17, 2020 at 9:55 PM Prasanna kumar < prasannakumarram...@gmail.com> wrote: > Hi , > > I did not find out of box flink sink connector for http and SQS > mechanism. > > Has anyone implemented it? > Wanted to know if we are writing a custom sink function , whether it > would affect semantic exactly one guarantees ? > > > Thanks , > Prasanna >
Re: thrift support
Hi Chen, Are building something like hive thrift server ? Chen Qin 于2020年7月18日周六 上午8:50写道： > Hi there, > > Here in Pinterest, we utilize thrift end to end in our tech stack. As we > have been building Flink as a service platform, the team spent time working > on supporting Flink jobs with thrift format and successfully launched a > good number of important jobs in Production in H1. > > In H2, we are looking at supporting Flink SQL with native Thrift support. > We have some prototypes already running in development settings and plan to > move forward on this approach. > > In the long run, we thought out of box thrift format support would benefit > other folks as well. So the question is if there is already some effort > around this space we can sync with? > > Chen > Pinterest Data > -- Best Regards Jeff Zhang
Re: thrift support
Jeff, Are you referring something like this SPIP? https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 Not at this moment, we are working on desr/ser work at the moment. Would be good to starts discussion and learn if folks working on related areas and align. Chen On Sat, Jul 18, 2020 at 6:41 AM Jeff Zhang wrote: > Hi Chen, > > Are building something like hive thrift server ? > > Chen Qin 于2020年7月18日周六 上午8:50写道： > > > Hi there, > > > > Here in Pinterest, we utilize thrift end to end in our tech stack. As we > > have been building Flink as a service platform, the team spent time > working > > on supporting Flink jobs with thrift format and successfully launched a > > good number of important jobs in Production in H1. > > > > In H2, we are looking at supporting Flink SQL with native Thrift support. > > We have some prototypes already running in development settings and plan > to > > move forward on this approach. > > > > In the long run, we thought out of box thrift format support would > benefit > > other folks as well. So the question is if there is already some effort > > around this space we can sync with? > > > > Chen > > Pinterest Data > > > > > -- > Best Regards > > Jeff Zhang >
[jira] [Created] (FLINK-18635) Typo in 'concepts/timely stream processing' part of the website
Xuhui Mao created FLINK-18635: - Summary: Typo in 'concepts/timely stream processing' part of the website Key: FLINK-18635 URL: https://issues.apache.org/jira/browse/FLINK-18635 Project: Flink Issue Type: Improvement Components: Documentation Affects Versions: 1.11.0 Reporter: Xuhui Mao [https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html] [Introduction Part] "Timely *steam* processing is an extension of [stateful stream processing|https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/stateful-stream-processing.html] in which time plays some role in the computation." I think this sentence has a typo, should change the 'steam' to 'stream'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: thrift support
Hi Chen, Right, this is what I mean. Could you provide more details about the desr/ser work ? Giving a concrete example or usage scenario would be helpful. Chen Qin 于2020年7月18日周六 下午11:09写道： > Jeff, > > Are you referring something like this SPIP? > > https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 > Not at this moment, we are working on desr/ser work at the moment. Would be > good to starts discussion and learn if folks working on related areas and > align. > > Chen > > On Sat, Jul 18, 2020 at 6:41 AM Jeff Zhang wrote: > > > Hi Chen, > > > > Are building something like hive thrift server ? > > > > Chen Qin 于2020年7月18日周六 上午8:50写道： > > > > > Hi there, > > > > > > Here in Pinterest, we utilize thrift end to end in our tech stack. As > we > > > have been building Flink as a service platform, the team spent time > > working > > > on supporting Flink jobs with thrift format and successfully launched a > > > good number of important jobs in Production in H1. > > > > > > In H2, we are looking at supporting Flink SQL with native Thrift > support. > > > We have some prototypes already running in development settings and > plan > > to > > > move forward on this approach. > > > > > > In the long run, we thought out of box thrift format support would > > benefit > > > other folks as well. So the question is if there is already some effort > > > around this space we can sync with? > > > > > > Chen > > > Pinterest Data > > > > > > > > > -- > > Best Regards > > > > Jeff Zhang > > > -- Best Regards Jeff Zhang
