That sounds good On Thu, 1 Jun 2023 at 18:09, Steve Loughran <ste...@cloudera.com.invalid> wrote:
> hadoop-api-shim ? > > On Thu, 1 Jun 2023 at 04:07, Ayush Saxena <ayush...@gmail.com> wrote: > > > +1, for the new repo. > > > > The name sounds fine, but good if we have scope of having “hadoop-” > > prefix, we have that for almost all of the subprojects/modules > > > > Can hadoop-shims or hadoop-shims-api or something on similar lines work? > > > > -Ayush > > > > > On 01-Jun-2023, at 1:18 AM, Steve Loughran <ste...@cloudera.com.invalid > > > > wrote: > > > > > > I want to create a new repository to put a shim library to allow > > previous > > > > > releases to access the more recent hadoop filesystem APIs -currently > the > > > open source implementations of parquet, ORC can't use vectored io, in > > > particular, even though we can in Cloudera. Providing a shim opens them > > up > > > to all *and* gets the APIs more broadly stressed/tested. > > > This needs to be in its own repository, not just for rapid initial > > release, > > > but because it is designed to be built as old a version of hadoop we > can > > > reasonably support, which IMO means hadoop 3.1.0+. I know parquet still > > > wants to build against 2.8.x, but to claim support for hadoop 2 means > > > "build and test on java7", which is unrealistic in 2023. > > > > > > Initial WiP implementation, which works with 3.1.0 and tests against > > others > > > https://github.com/steveloughran/fs-api-shim > > > the complexity is about testing this -I have contract tests which then > > need > > > to be executed on every supported hadoop release, which will need a > > > separate module for each one. > > > > > > I can create the repo easily enough, just would like approval. And is > the > > > name OK? > > > > > > steve > > >
