Re: [PROPOSAL] HiveIO - updated link to document

Thank you for your helpful comments. We will look into the HadoopIO code that you have suggested. We are following incremental development and we will consider more generic approach to develop for other runners. Going ahead we will ask questions when we need help. Madhu Borkar On Tue, Apr 25, 20

Re: An Update on Jenkins

Thanks for the update ! Regards JB On Apr 26, 2017, 05:51, at 05:51, Jason Kuster wrote: >Hey folks, > >There have been a couple of different issues over the last couple of >days >related to some necessary updates Infra has been working on. We've >tracked >down the last couple of issues, and th

Re: An Update on Jenkins

Thanks for the update, Jason! On Wed, Apr 26, 2017 at 6:51 AM Jason Kuster wrote: > Hey folks, > > There have been a couple of different issues over the last couple of days > related to some necessary updates Infra has been working on. We've tracked > down the last couple of issues, and the late

An Update on Jenkins

Hey folks, There have been a couple of different issues over the last couple of days related to some necessary updates Infra has been working on. We've tracked down the last couple of issues, and the latest one seems to be that we're being hit by the rate limiter as a result of everything starting

Re: Can application specify how watermarks should be generated?

Thank you, Thomas. That clears my confusions. :) Shen On Tue, Apr 25, 2017 at 7:30 PM, Thomas Groh wrote: > getCurrentTimestamp returns the timestamp of the current element. Both > Bounded and Unbounded Readers have this method. > > For a bounded source, this is safe - the source watermark can

Re: Can application specify how watermarks should be generated?

getCurrentTimestamp returns the timestamp of the current element. Both Bounded and Unbounded Readers have this method. For a bounded source, this is safe - the source watermark can be held to negative infinity while elements remain in the source and advance to infinity after all elements are read,

Re: [PROPOSAL] HiveIO - updated link to document

Hello, I created the HiveIO JIRA and followed the initial discussions about the best approach for HiveIO so I want first to suggest you to read the previous thread(s) on the mailing list. https://www.mail-archive.com/dev@beam.incubator.apache.org/msg02313.html The main idea I concluded from that

[PROPOSAL] HiveIO - updated link to document

Please, use following link for the HiveIO proposal. https://docs.google.com/document/d/1JOzihFiXkQjtv6rur8-vCixSK- nHhIoIij9MwJZ_Dp0/edit?usp=sharing Madhu Borkar

Re: Community hackathon

No problem, Sean. Invite sent. On Tue, Apr 25, 2017 at 6:14 PM Sean Story wrote: > I'd also love to be added to the slack channel > > > Thanks, > > Sean Story > > > > On Apr 25, 2017, at 12:54 AM, Davor Bonaci wrote: > > > > Thanks everyone for the enthusiasm! > > > > Let's go with this Wednesd

Re: Community hackathon

I'd also love to be added to the slack channel Thanks, Sean Story > On Apr 25, 2017, at 12:54 AM, Davor Bonaci wrote: > > Thanks everyone for the enthusiasm! > > Let's go with this Wednesday, 4/26, starting at 10 AM Pacific time, and > running for the following 24 hours. I'll try to seed th

Re: [DISCUSSION] Encouraging more contributions

I think it is important to clarify that the developer documentation discussed in this thread is of two kinds: 6.1. Documents with proposals and new designs, those covered by the Beam Improvement Proposal (BEAM-566), and that we need to put with a single file index (I remember there was a google di

Re: [DISCUSSION] Encouraging more contributions

These are the ones I've come across so far, are there others? * Dynamic DoFn https://s.apache.org/a-new-dofn ** Splittable DoFn (Obsoletes Source API) http://s.apache.org/splittable-do-fn ** State and Timers for DoFn: https://s.apache.org/beam-state * Lateness https://s.apache.org/beam-latenes

Re: [DISCUSSION] Encouraging more contributions

You're right Kenn, discussing the "table of contents" is a good start. WDYT about having this table of contents audience and code package oriented? something like 1. SDK writers 1.1 Core components 1.1.1 Transforms 1.1.2 Metrics 1.2 IO components

Re: [DISCUSSION] Encouraging more contributions

Hi Stephen, Yes, it is a good answer, the doc "Authoring I/O Transforms - Overview" (https://beam.apache.org/documentation/io/authoring-overview/) is exactly what I had in mind. IMHO, writing this kind of docs (beam contributor oriented) for all the key concepts, and stored at one only (sear

Re: Towards a spec for robust streaming SQL, Part 1

Hi Tyler, thanks for pushing this effort and including the Flink list. I haven't managed to read the doc yet, but just wanted to thank you for the write-up and let you know that I'm very interested in this discussion. We are very close to the feature freeze of Flink 1.3 and I'm quite busy getting