How to add/alter/drop a Hive partition from a Beam app

Hi Beam community, I am wondering how we can use some Beam APIs or Beam SQL to perform some Hive DDL operations such as add/alter/drop a partition. I guess I might need to use HCatalogIO, however I am not sure about what syntax to use. Please advise. Thanks!

Re: [DISCUSS] Drop support for Flink 1.8 and 1.9

> Do we now support 1.8 through 1.12? Yes and that's clearly too much given that the Flink community only support the two latest release. It also hits us because we run tests for all those versions on precommit. On Fri, Mar 12, 2021 at 7:27 PM Robert Bradshaw wrote: > > Do we now support 1.8 thr

Re: [DISCUSS] Drop support for Flink 1.8 and 1.9

Do we now support 1.8 through 1.12? Unless there are specific objections, makes sense to me. On Fri, Mar 12, 2021 at 8:29 AM Alexey Romanenko wrote: > +1 too but are there any potential objections for this? > > On 12 Mar 2021, at 11:21, David Morávek wrote: > > +1 > > D. > > On Thu, Mar 11, 20

Re: Setting rowGroupSize in ParquetIO

Tbh mate, I reckon it would be quicker if you progress your PR. Cheers, David From: Bashir Sadjad Sent: 12 March 2021 16:29 To: user@beam.apache.org Subject: Re: Setting rowGroupSize in ParquetIO Thanks David. Yes, I looked at passing it through the HadoopConfi

Re: Setting rowGroupSize in ParquetIO

Thanks David. Yes, I looked at passing it through the HadoopConfiguration but it seems row-group size is not there or at least ParquetWriter.Builder seems to set that directly from its rowGroupSize property. I filed BEAM-11969 for this so if you ca

Re: [DISCUSS] Drop support for Flink 1.8 and 1.9

+1 too but are there any potential objections for this? > On 12 Mar 2021, at 11:21, David Morávek wrote: > > +1 > > D. > > On Thu, Mar 11, 2021 at 8:33 PM Ismaël Mejía > wrote: > +user > > > Should we add a warning or something to 2.29.0? > > Sounds like a good ide

Re: Setting rowGroupSize in ParquetIO

Hi Bashir, I think it is just a case of somebody bothering to plumbing it in explicitly, e.g. /** Specifies row group size. By default, DEFAULT_BLOCK_SIZE. */ public Sink withRowGroupSize(int rowGroupSize) { return toBuilder().setRowGroupSize(rowGroupSize).build(); } and this.writer =

Re: [DISCUSS] Drop support for Flink 1.8 and 1.9

+1 D. On Thu, Mar 11, 2021 at 8:33 PM Ismaël Mejía wrote: > +user > > > Should we add a warning or something to 2.29.0? > > Sounds like a good idea. > > > > > On Thu, Mar 11, 2021 at 7:24 PM Kenneth Knowles wrote: > > > > Should we add a warning or something to 2.29.0? > > > > On Thu, Mar 11,