Initial Shim library up https://github.com/apache/hadoop-api-shim
much of this relates to testing; we need a module to test a common set of tests on every supported hadoop build, with validation that expected features are present, and that the same operations work when both reflection and fall back implementations are used. This needs to include integration tests against the stores...the good news is that being downstream of things we can include google gcs tests without creating a loop in the DAG. If anyone wants to help with that test design/impl, it'd be wonderful steve On Mon, 5 Jun 2023 at 12:11, Steve Loughran <ste...@cloudera.com> wrote: > > > On Thu, 1 Jun 2023 at 18:46, Owen O'Malley <owen.omal...@gmail.com> wrote: > >> For the case in question, ORC can extend its shim module to do exactly >> this. Let me take a look at the async ORC patch to see what needs to >> happen. >> > > > >> BTW, Steve I'll be at Buzzwords for your talk. >> > > ooh, cool. it's always good to have people in the audience to pick on and > I don't like doing it to strangers or people who, if they bear a grudge, > can be a problem in future. Fokka from the parquet team will be at the conf > tool. > > Hope you will also be at my smart home one, where I will even show how > straightforward it is for anyone (*) to add their EV charge info to home > assistant, so coming closer to my goal of having a light bulb change colour > when it is charging. > > >> >> It may make sense to have such a library, but ORC already has that >> mechanism. >> > > The vector iO shim a reasonable amount of work if you want to provide a > fallback for older versions. if you just want to have an optional feature > which is only available on some platforms, easier. > > > I'll be around over the w/e and the day after if you bring your laptop and > want to do some code > > (*) anyone who understands what OAuth tokens are, how to create one from a > specific web site, knows how to set up home assistant with the > community store (github a/c, registering a device with github, etc). > > >> >
