I do have a WiP library to hide that hoop-jumping behind a normal API, with
a goal of 3.2+ support only. It does actually compile against hadoop 2, but
it isn't tested there
https://github.com/steveloughran/fs-api-shim
1. my goal is to make it a hadoop library like hadoop-third-party, with its
ow
On Mon, Mar 27, 2023 at 20:29 Wei-Chiu Chuang wrote:
> For complex applications such as
> HBase it is almost impossible to achieve true FS agnosticity without proper
> contract tests, as now I am starting to realize.
>
This is absolutely true. HBase jumps through all sorts of painful
reflective
I think moving up interfaces to FileSystem or some abstract FileSystem
class has a few benefits:
1. Application can potentially be made FS-agnostic, with
hasPathCapabilities() check.
At least, make the code to compile.
2. We will be able to add a contract test to ensure behavior is expected.
The
side issue, as i think about what bulk delete call would also keep hbase
happy
https://issues.apache.org/jira/browse/HADOOP-18679
should we think about new API calls only raising RuntimeExceptions?
The more work I do on futures the more the way we always raise IOEs
complicates life. java has outg
On Thu, 23 Mar 2023 at 10:07, Ayush Saxena wrote:
>
> Second idea mentioned in the original mail is also similar to mentioned in
> the comment in the above ticket and is still quite acceptable, name can be
> negotiated though, Add an interface to pull the relevant methods up in that
> without tou
They both need it for a similar use case: "to support Ozone", not anything
core that we handle as part of "Apache Hadoop" and I suppose both are
working fine with HDFS, because of adding dependency with HDFS? and now
they don't want to add Ozone for whatever reasons and folks chasing this
integrati
Well reflections are good or not will drag this somewhere else. I will
respect what Tsz-Wo said and put this in my rule book for future :)
If I get into Why we don’t have “all” the API in FileSystem itself will
drag it to another area, What and where to use Abstraction and stuff like
that, Which n
I am not sure what classifies as a Hack and what not, I thought reflections
are part of Java.
Whatever solution but pulling in just the HDFS specific stuff to FileSystem
just for Ozone, because Hbase guys didn’t agree and we have people in
Hadoop who we can convince, I am -1 to such an approach an
Hbase doesn’t want to add Ozone as a dependency sounds to me like a ‘Hbase
having resistance against the people proposing or against Ozone’
Anyway doesn’t ViewDistributedFileSystem not solve this Ozone problem, I
remember Uma chasing that to solve these problems only?
Pulling up the core HDFS A
Thank you. Makes sense to me. Yes, as part of this effort we are going to
need contract tests.
On Fri, Mar 17, 2023 at 3:52 AM Steve Loughran
wrote:
>1. I think a new interface would be good as FileContext could do the
>same thing
>2. using PathCapabilities probes should still be man
1. I think a new interface would be good as FileContext could do the
same thing
2. using PathCapabilities probes should still be mandatory as for
FileContext it would depend on the back end
3. Whoever does this gets to specify what the API does and write the
contract tests. Saying
Hi,
Stephen and I are working on a project to make HBase to run on Ozone.
HBase, born out of the Hadoop project, depends on a number of HDFS specific
APIs, including recoverLease() and isInSafeMode(). The HBase community [1]
strongly voiced that they don't want the project to have direct dependen
12 matches
Mail list logo
