Hi Sanjay,
Actually I was not aware of that work… This seems to be a better way of
achieving some of the same things we do externally to the DN process. I will
look into reimplementing some parts on top of this; seems it should just
require some very small extensions to DataNodeCluster. Thank y
Erik
Great stuff.
BTW did you build on top of the “simulated data nodes” in HDFS which has a way
to storing only the length of data (but not real data)? That work allowed
supplementing with a matching editsLog for the NN. Your approach of using a
real image has the advantage of being able to
Hi Anu,
Thanks for the interest!
1. Unfortunately I am very doubtful that LinkedIn security would let us
release our traces. If you collect your audit logs that's the only thing
necessary to build traces.
2. Our current approach is to use 'truncate' to create sparse files which
have the expected
Hi Erik,
Looking forward to the release of this tool. Thank you very much for the
contribution.
Had a couple of questions about how the tool works.
1. Would you be able to provide the traces along with this tool? In other
words, would I be able to use this out of the box, or do I have to build
forking off of the 2.7.4 release thread to answer this question about
Dynamometer
Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
specifically the NameNode. We have been using it for some time now and have
recently been making some enhancements to ease of use and reproducibilit
