Re: profiling hdfs write path

On 8 December 2012 04:39, Radim Kolar wrote: > > if you want to keep code in safe state you need: > 1. good unit test > 2. high unit test coverage > 3. clean code > 4. documented code > 5. good javadoc + good functional tests, which explores the deployment state of the world, especially differe

Re: profiling hdfs write path

I'm not saying that the Hadoop process is perfect, far from it, but from where I sit (like you I'm a contributor but not yet a committer) it seems to be working OK so far for both you and I. It does not work for me OK. Its way too slow. i got just 2k LOC in committed and still floating around p

Re: profiling hdfs write path

On Wed, Dec 5, 2012 at 6:02 PM, Radim Kolar wrote: > YARN-223 > YARN-211 > YARN-210 > MAPREDUCE-4839

Re: profiling hdfs write path

YARN-223 YARN-211 YARN-210 MAPREDUCE-4839 MAPREDUCE-4827

Re: profiling hdfs write path

On Tue, Dec 4, 2012 at 6:00 PM, Radim Kolar wrote: > Its not about writing patches, its about to get them committed. I have > experience that getting something committed takes months even on simple > patch. I have about 10 patches floating around none of them was committed in > last 4 weeks. Coul

Re: profiling hdfs write path

On 5 December 2012 02:00, Radim Kolar wrote: > > Agree. Want to write some? >> > Its not about writing patches, its about to get them committed. I have > experience that getting something committed takes months even on simple > patch. I have about 10 patches floating around none of them was com

Re: profiling hdfs write path

Agree. Want to write some? Its not about writing patches, its about to get them committed. I have experience that getting something committed takes months even on simple patch. I have about 10 patches floating around none of them was committed in last 4 weeks. They are really simple stuff. I

Re: profiling hdfs write path

Thank you Todd! I have been seeing similar attitude in many jiras, that I have tried hard to ignore and was wondering how to respond to this email. I could not have said it better. On Tue, Dec 4, 2012 at 9:27 AM, Todd Lipcon wrote: > f you're just going to insult us, please stay away. We don't

Re: profiling hdfs write path

On Tue, Dec 4, 2012 at 9:39 AM, Radim Kolar wrote: > >> If you're just going to insult us, please stay away. We don't need >> your help unless you're going to be constructive. > > Good units tests will catch code modifications like: > Agree. Want to write some? Would love to see patches like t

Re: profiling hdfs write path

If you're just going to insult us, please stay away. We don't need your help unless you're going to be constructive. Good units tests will catch code modifications like: from: long getLastByteOffsetBlock() { return lastByteOffsetInBlock; } to from: long getLastByteOffsetBlo

Re: profiling hdfs write path

On Tue, Dec 4, 2012 at 9:07 AM, Radim Kolar wrote: > >> It is definitely buggy, it might not actually be faster, and it >> probably isn't well commented. But feel free to have a go at it. > > thank you for your code, i got it merged with trunk. HDFS is crap code, > private methods not documented a

Re: profiling hdfs write path

It is definitely buggy, it might not actually be faster, and it probably isn't well commented. But feel free to have a go at it. thank you for your code, i got it merged with trunk. HDFS is crap code, private methods not documented at all, and unit tests are joke. I did some random code chang

Re: profiling hdfs write path

Hi Radim, My work-in-progress branch is online here: https://github.com/toddlipcon/hadoop-common/commits/trunk-write-pipeline-fast It is definitely buggy, it might not actually be faster, and it probably isn't well commented. But feel free to have a go at it. -Todd On Thu, Nov 29, 2012 at 7:17

Re: profiling hdfs write path

> Hoping to find time to get back to finishing the patches in the next few months. Todd, just attach these pathes to jira, they do not even needs to apply cleanly to trunk. I will get them finished within day. I do not have months which i can spare on waiting for work be done by you. If you

Re: profiling hdfs write path

Currently it's CPU-intensive for several reasons: 1) It doesn't yet use the native CRC code 2) It makes several unnecessary copies and byte buffer allocations, both in the client and in the DataNode There are open JIRAs for these, and I have a preliminary patch which helped a lot, but it hasn't

Re: profiling hdfs write path

Hi Radim, Currently it's CPU-intensive for several reasons: 1) It doesn't yet use the native CRC code 2) It makes several unnecessary copies and byte buffer allocations, both in the client and in the DataNode There are open JIRAs for these, and I have a preliminary patch which helped a lot, but i

profiling hdfs write path

anybody tried to profile why HDFS write path is so much CPU intensive?