Hi Steve,
Thanks again for your in depth replies, we found your comments quite
useful. A few responses inline:
On 8/9/13 10:31 AM, Steve Loughran wrote:
On 8 August 2013 21:51, Matevz Tadel <mta...@ucsd.edu> wrote:
We already do fallback to xrootd on open failures from our application
framework ... the job gets redirected to xrootd proxy which downloads the
whole file and serves data as the job asks for it. The changes described by
Jeff are an optimization for cases when a single block becomes unavailable.
Like other said, there's work on heterogenous storage. Maybe you could make
sure that there is some handling there for block unavailablity events -then
you can hook in that to handle it.
We discussed this a bit more and agree, the best way forward is to
carefully watch the progress of this development, and we will look into
how it will fit our needs.
Thanks also for the comments below, Jeff will appreciate them more as he
actually went through all the HDFS code and did all the changes in the
patch, I did the JNI + Xrootd part.
One more question ... if we'd converge on an acceptable solution (I find
it a bit hard at the moment), how long would it take for the changes to be
released and what release cycle would it end up in?
all changes go into trunk, if it was go to into the 2.x line then it would
be 2.3 at the earliest; 2.1 is going through its beta right now. If you
work against 2.1 and report bugs now, that would help the beta and make it
easier for you to have a private fork of 2.1.x with the extensions
In the long term this sounds like a reasonable approach. For the short
term, we will continue to work with OSG to include patches needed to
support our use case, as their upcoming release will be based the 2.0
branch we have already been experimenting with. We will take into
careful consideration the requirements you outlined that would make our
patches reasonable candidates to be considered in the official Hadoop trunk.
Jeff Dost
UCSD Physics
