Questions on HDFS-8880

Folks, I’ve been looking over HDFS-8880 and it’s various follow-on JIRAs. The intentions are good, but the implementation is mostly/effectively a duplicate of the FileSink that’s already part of the Hadoop metrics subsystem. (which therefore means it works with all daemons, out of the

Re: Questions on HDFS-8880

sed guide on how to > configure Metrics2 would probably be a welcome addition. > > HBase has a blog at https://blogs.apache.org/hbase/ this could also be good > content for a blog post. > > Best, > Andrew > > On Tue, Oct 6, 2015 at 11:12 AM, Allen Wittenauer wrote: > &

Re: Questions on HDFS-8880

27;. > > - Not all NN metrics are logged. These last two are likely bugs. > > > > > > > On 10/6/15, 11:46 AM, "Allen Wittenauer" wrote: > >> Current state is the custom stuff in HDFS-8880 is mostly undocumented except >> for some tuning

Re: Questions on HDFS-8880

is configured like any other logger > by adding '-Dnamenode.metrics.logger=DEBUG,NNMETRICSRFA' or equivalent to > HADOOP_NAMENODE_OPTS. Your concerns sound specific to 9114 and not inherent > to 8880. > > > On 10/6/15, 1:28 PM, "Allen Wittenauer" wrote:

Re: [DISCUSS] About the details of JDK-8 support

If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after mo

Re: [DISCUSS] About the details of JDK-8 support

FYI: I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watchi

Re: [DISCUSS] About the details of JDK-8 support

On Oct 17, 2015, at 8:55 AM, Steve Loughran wrote: >> > > thx for doing this, I'll keep an eye on the builds Thanks for helping out too. You never know how software actually runs in the wild. I’m already hitting bugs that didn’t show up in testing. *sigh* > one question: which jdk8? Som

Re: [DISCUSS] Looking to a 2.8.0 release

> On Nov 11, 2015, at 12:13 PM, Vinod Vavilapalli > wrote: > >— HDFS-6200 Create a separate jar for hdfs-client: Compatible improvement > - no dimension of alpha/betaness here. IMO: this feels like a massive break in backwards compatibility. Anyone who is looking for specific met

Re: [DISCUSS] Looking to a 2.8.0 release

> On Nov 11, 2015, at 1:11 PM, Vinod Vavilapalli > wrote: > > I’ll let others comment on specific features. > > Regarding the 3.x vs 2.x point, as I noted before on other threads, given all > the incompatibilities in trunk it will be ways off before users can run their > production workloads

Jenkins on H6 for HDFS is sick

FYI, I’ve filed INFRA-10785.

Re: [DISCUSS] Looking to a 2.8.0 release

> On Nov 25, 2015, at 11:23 AM, Vinod Kumar Vavilapalli > wrote: > > There are 40 odd incompatible changes in 3.x: > https://issues.apache.org/jira/issues/?jql=project%20in%20%28HADOOP%2C%20YARN%2C%20HDFS%2C%20MAPREDUCE%29%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%203.0.0%20AND

Re: [VOTE] Merge yarn-native-services branch into trunk

> On Aug 31, 2017, at 8:33 PM, Jian He wrote: > I would like to call a vote for merging yarn-native-services to trunk. 1) Did I miss it or is there no actual end-user documentation on how to use this? I see hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-serv

Re: [VOTE] Merge yarn-native-services branch into trunk

> On Sep 5, 2017, at 2:53 PM, Jian He wrote: > >> Based on the documentation, this doesn’t appear to be a fully function DNS >> server as an admin would expect (e.g., BIND, Knot, whatever). Where’s >> forwarding? How do I setup notify? Are secondaries even supported? etc, etc. > > It seems l

Re: [VOTE] Merge yarn-native-services branch into trunk

> On Sep 5, 2017, at 3:12 PM, Gour Saha wrote: > > 2) Lots of markdown problems in the NativeServicesDiscovery.md document. > This includes things like Œyarnsite.xml¹ (missing a dash.) > > The md patch uploaded to YARN-5244 had some special chars. I fixed those > in YARN-7161. It’s a

Re: [VOTE] Merge yarn-native-services branch into trunk

> On Sep 5, 2017, at 6:23 PM, Jian He wrote: > >> If it doesn’t have all the bells and whistles, then it shouldn’t be on >> port 53 by default. > Sure, I’ll change the default port to not use 53 and document it. >> *how* is it getting launched on a privileged port? It sounds like the

Re: [VOTE] Merge yarn-native-services branch into trunk

> On Sep 8, 2017, at 9:25 AM, Jian He wrote: > > Hi Allen, > The documentations are committed. Please check QuickStart.md and others in > the same folder. > YarnCommands.md doc is updated to include new commands. > DNS default port is also documented. > Would you like to give a look and see if

Re: qbt is failiing///RE: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

> On Sep 19, 2017, at 6:35 AM, Brahma Reddy Battula > wrote: > > qbt is failing from two days with following errors, any idea on this..? Nothing to be too concerned about. This is what it looks like when a build server gets bounced or crashed. INFRA team knows our jobs take

Re: 2017-10-06 Hadoop 3 release status update

> On Oct 6, 2017, at 1:31 PM, Andrew Wang wrote: > > - Still waiting on Allen to review YARN native services feature. Fake news. I’m still -1 on it, at least prior to a patch that posted late yesterday. I’ll probably have a chance to play with it early next week. Key pro

YARN native services Re: 2017-10-06 Hadoop 3 release status update

> On Oct 6, 2017, at 5:51 PM, Eric Yang wrote: > yarn application -deploy –f spec.json > yarn application -stop > yarn application -restart > yarn application -remove > > and > > yarn application –list will display both application list from RM as well as > docker services?

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

To whoever set this up: There was a job config problem where the Jenkins branch parameter wasn’t passed to Yetus. Therefore both of these reports have been against trunk. I’ve fixed this job (as well as the other jobs) to honor that parameter. I’ve kicked off a new run with these changes.

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

I’m really confused why this causes the Yahoo! QA boxes to go catatonic (!?!) during the run. As in, never come back online, probably in a kernel panic. It’s pretty consistently in hadoop-hdfs, so something is going wrong there… is branch-2 hdfs behaving badly? Someone needs to run the hadoop

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

. > > Thanks, > Subru > > On Mon, Oct 23, 2017 at 11:26 AM, Vrushali C wrote: > Hi Allen, > > I have filed https://issues.apache.org/jira/browse/YARN-7380 for the > timeline service findbugs warnings. > > thanks > Vrushali > > > On Mon, Oct 23, 2017

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer > wrote: > > > > With no other information or access to go on, my current hunch is that one of > the HDFS unit tests is ballooning in memory size. The easiest way to kill a > Linux machine is to eat all of the RAM,

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

t tell which daemons/components of HDFS consumes unexpected high >> memory. Don't sounds like a solid bug report to me. >> >> >> >> Thanks,? >> >> >> Junping >> >> >> >> From: Sean Bus

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 24, 2017, at 4:10 PM, Andrew Wang wrote: > > FWIW we've been running branch-3.0 unit tests successfully internally, though > we have separate jobs for Common, HDFS, YARN, and MR. The failures here are > probably a property of running everything in the same JVM, which I've found > pro

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

> On Nov 3, 2017, at 12:08 PM, Stack wrote: > > On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko > wrote: > >> It is an interesting question whether Ozone should be a part of Hadoop. > > I don't see a direct answer to this question. Is there one? Pardon me if > I've not seen it but I'm in

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

The original release script and instructions broke the build up into three or so steps. When I rewrote it, I kept that same model. It’s probably time to re-think that. In particular, it should probably be one big step that even does the maven deploy. There’s really no harm in doing th

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

> On Nov 21, 2017, at 2:16 PM, Vinod Kumar Vavilapalli > wrote: > >>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even >>> work. Not just deprecated in favor of timelineserver as was advertised. >> >> This works for me in trunk and the bash code doesn’t appear to

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

> On Nov 30, 2017, at 1:07 AM, Rohith Sharma K S > wrote: > > > >. If ATSv1 isn’t replaced by ATSv2, then why is it marked deprecated? > Ideally it should not be. Can you point out where it is marked as deprecated? > If it is in historyserver daemon start, that change made very long back when

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

It’s significantly more concerning that 3.0.0-beta1 doesn’t show up here: http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/release/index.html It looks like they are missing from the source tag too. I wonder what else is missing. > On Dec 18, 2017, at 11:15 AM, Andrew Wa

Fwd: Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

For my part of the HDFS bug bash, I’ve gotten the ASF Windows build working again. Starting tomorrow, results will be sent to the *-dev lists. A few notes: * It only runs the unit tests. There’s not much point in running the other Yetus plugins since those are covered by the Linux one

Re: [NOTIFICATION] Hadoop trunk rebased

Did the patch that fixes the mountain of maven warnings get missed? > On Apr 26, 2018, at 11:52 PM, Akira Ajisaka wrote: > > + common-dev and mapreduce-dev > > On 2018/04/27 6:23, Owen O'Malley wrote: >> As we discussed in hdfs-dev@hadoop, I did a force push to Hadoop's trunk to >> replace the

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

FYI: I’m going to disable the branch-2 nightly jobs. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On May 15, 2018, at 10:16 AM, Chris Douglas wrote: > > They've been failing for a long time. It can't install bats, and > that's fatal? -C The bats error is new and causes the build to fail enough that it produces the email output. For the past few months, it hasn’t been producing

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

> On Jun 7, 2018, at 3:46 AM, Lokesh Jain wrote: > > Hi Yongjun > > I followed Nanda’s steps and I see the same issues as reported by Nanda. This situation is looking like an excellent opportunity for PMC members to mentor people on how the build works since it’s apparent that three days la

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

> On Jun 7, 2018, at 11:47 AM, Steve Loughran wrote: > > Actually, Yongjun has been really good at helping me get set up for a 2.7.7 > release, including "things you need to do to get GPG working in the docker > image” *shrugs* I use a different release script after some changes broke

Re: [DISCUSS] Alpha Release of Ozone

Given that there are some Ozone components spread out past the core maven modules, is the plan to release a Hadoop Trunk + Ozone tar ball or is more work going to go into segregating the Ozone components prior to release? Has anyone verified that a Hadoop release doesn't have _any_ of the extr

Re: [DISCUSS] Alpha Release of Ozone

> On Aug 8, 2018, at 12:56 PM, Anu Engineer wrote: > >> Has anyone verified that a Hadoop release doesn't have _any_ of the extra >> ozone bits that are sprinkled outside the maven modules? > As far as I know that is the state, we have had multiple Hadoop releases > after ozone has been merg

Re: Checkstyle shows false positive report

> On Aug 15, 2018, at 4:49 AM, Kitti Nánási > wrote: > > Hi All, > > We noticed that the checkstyle run by the pre commit job started to show > false positive reports, so I created HADOOP-15665 > . > > Until that is fixed, keep in mind to r

Re: Hadoop encryption module as Apache Chimera incubator project

Why is this discussion taking place on hdfs-dev when other parts of Hadoop use encryption code? That alone makes me uncomfortable moving this code outside of the Hadoop umbrella. Also, since Yetus was invoked, it’s worthwhile pointing out that we had _many_ projects, inside

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

> On Feb 9, 2016, at 6:27 PM, Junping Du wrote: > > Thanks Yongjun for identifying and proposing this change to 2.6.4. I think > this is the right thing to do and check for following releases. For 2.6.4, it > seems unnecessary to create another release candidate for this issue as we > only ki

Re: 2.7.3 release plan

This is probably a good time to remind/point folks to HADOOP-12893. Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components. I fairly confident this means releases are off the table until someone aud

Re: Looking to a Hadoop 3 release

> On Apr 22, 2016, at 5:38 PM, Vinod Kumar Vavilapalli > wrote: > > On an unrelated note, offline I was pitching to a bunch of contributors > another idea to deal with rotting trunk post 3.x: *Make 3.x releases off of > trunk directly*. > > What this gains us is that > - Trunk is always near

Re: Looking to a Hadoop 3 release

> On Apr 22, 2016, at 6:10 PM, Vinod Kumar Vavilapalli > wrote: > > Nope. > > I’m proposing making a new 3.x release (as has been discussed in this thread) > off today’s trunk (instead of creating a fresh branch-3) and create a new > trunk-incompt where incompatible changes that we don’t wan

Re: Recent PreCommit Jenkins failures

> On Apr 27, 2016, at 3:53 PM, Arun Suresh wrote: > > which I guess is due to the haskell site throwing a 502 and prior to that I > saw a couple of failures with > hackage.haskell.org has been down all day for some reason. I’ve got a patch sitting in HADOOP-13067 which will switch us over t

[VOTE] Merge feature branch HADOOP-12930

Hey gang! I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch was developed exclusively by me as per the discussion two months ago as a way to make what would be a rather large patch hopefully ea

Re: [DISCUSS] Merge Jersey upgrade(HADOOP-9613) to trunk

+1 agreed. > On May 9, 2016, at 2:13 PM, Akira AJISAKA wrote: > > +1 for marking HADOOP-9613 as incompatible change and upgrading the version > in trunk. > > Thanks, > Akira > > On 5/9/16 12:12, Tsuyoshi Ozawa wrote: >> Hi developers, >> >> We’ve worked on upgrading jersey(HADOOP-9613) for

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

> On May 10, 2016, at 8:32 AM, Akira AJISAKA wrote: > > Hi developers, > > Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in trunk. +1 yes please. > One concern I think is, when the minimum version is set to JDK8, we need to > configure Jenkins to disable multi JDK te

Re: [VOTE] Merge feature branch HADOOP-12930

h >> with and without bats installed), tried making an example custom >> command (valid and malformed). everything looks great. >> >> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer wrote: >>> >>>Hey gang! >>> >>>I’d

Re: [VOTE] Merge feature branch HADOOP-12930

> On May 14, 2016, at 9:04 PM, Masatake Iwasaki > wrote: > > + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} > ${HADOOP_SUBCMD_ARGS[$*]}” Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec. Thanks!

Re: [VOTE] Merge feature branch HADOOP-12930

g overwhelmed since it would trigger full unit tests on pretty much the entire code base…. > While testing this, I discovered a bug in the distro build for Windows. > Could someone please code review my patch on HADOOP-13149? Done! > > --Chris Nauroth > > >

Re: [VOTE] Merge feature branch HADOOP-12930

> On May 15, 2016, at 7:27 AM, Allen Wittenauer wrote: > > >> On May 14, 2016, at 9:04 PM, Masatake Iwasaki >> wrote: >> >> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} >> ${HADOOP_SUBCMD_ARGS[$*]}” > >

Re: Looking to a Hadoop 3 release

Is there going to be a general upgrade of dependencies? I'm thinking of jetty & jackson in particular. On Mar 5, 2015, at 5:24 PM, Andrew Wang wrote: > I've taken the liberty of adding a Hadoop 3 section to the Roadmap wiki > page. In addition to the two things I've been pushing, I also looke

Re: Looking to a Hadoop 3 release

witch). > > On Thu, Mar 5, 2015 at 9:21 PM, Allen Wittenauer wrote: > >> >> Is there going to be a general upgrade of dependencies? I'm thinking of >> jetty & jackson in particular. >> >> On Mar 5, 2015, at 5:24 PM, Andrew Wang wrote: >> &g

Re: Hadoop 3.x: what about shipping trunk as a 2.x release in 2015?

Between this and the other thread, I’m seeing: * companies that were forced to make internal forks because their patches were ignored are now considered the deciders for whether we move forward * 5 years since the last branch off of trunk is considered ‘soon’ * M

Re: Hadoop 3.x: what about shipping trunk as a 2.x release in 2015?

On Mar 10, 2015, at 12:40 PM, Karthik Kambatla wrote: > > Are we okay with breaking other forms of compatibility for Hadoop-3, like > behavior, dependencies, JDK, classpath, environment? I think so. Are we > okay with breaking these forms of compatibility in future Hadoop-2.x? > Likely not. Doe

Save the Date: Apache Hadoop Bug Bash / May 8th!

Hi folks, There are ~6,000 Hadoop JIRA issues that have gone unaddressed, including ~900 with patches waiting to be reviewed. Among other things, this lack of attention to our backlog is making the Hadoop project very unfriendly to contributors--which is ultimately very unhealthy for

PSA: Various JIRA commit-time issues

It would be great if everyone could make sure that when resolving a JIRA Issue that they: A) Make sure that their final message is actually in the comment box and not in the release notes box. For 2.7, there are currently 5 out of the 24 release notes that are +1 type messages

Re: How to enable debug log in HDFS

This obviously doesn't help you with 2.4, but in trunk/3.x this is significantly easier to do: hdfs --loglevel DEBUG namenode (interactive/non-daemon mode, output to console, usually) hdfs --loglevel DEBUG --daemon start namenode (daemon mode, output to log file, usually) On Mar 25, 2015,

Re: Uncertainty of pre-commit builds

Very likely. One of the things I noticed during HADOOP-11746 is that there is a HUGE, catastrophic race if Jenkins doesn’t setup the environment correctly or leaks variables between runs. shellcheck prints out so many messages on the current code I’m surprised it doesn’t crash. On Ma

Re: Uncertainty of pre-commit builds

nch for working on > improving our test-patch scripts, Allen. > > On Tue, Mar 31, 2015 at 9:37 AM, Allen Wittenauer wrote: > >> >>Very likely. One of the things I noticed during HADOOP-11746 is >> that there is a HUGE, catastrophic race if Jenkins doesn’t s

IMPORTANT: automatic changelog creation

Hello everyone! (to: and reply-to: set to common-dev, cc: the rest of ‘em, to concentrate the discussion) HADOOP-11731 has just been committed to *trunk*. This change does two things: a) Removes dev-support/relnotes.py b) Adds dev-support/releasedocmaker.py releasedoc

Re: IMPORTANT: automatic changelog creation

On Apr 2, 2015, at 9:51 AM, Karthik Kambatla wrote: >> >> a) remove CHANGES.TXT from trunk >> > > Removing this from trunk makes it particularly hard to cherry-pick changes > from trunk to branch-2. I would gate this on the removal of CHANGES.txt on > branch-2 as well, at least until we have s

Re: IMPORTANT: automatic changelog creation

On Apr 2, 2015, at 11:36 AM, Mai Haohui wrote: > Hi Allen, > > Thanks for driving this. Just some quick questions: > >>> Removing changes.txt, relnotes.py, etc from branch-2 would be an >>> incompatible change. Pushing aside the questions of that document’s >>> quality (hint: lots of

Re: IMPORTANT: automatic changelog creation

On Apr 2, 2015, at 12:40 PM, Vinod Kumar Vavilapalli wrote: > > We'd then doing two commits for every patch. Let's simply not remove > CHANGES.txt from trunk, keep the existing dev workflow, but doc the release > process to remove CHANGES.txt in trunk at the time of a release going out of

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

Someone should look into HDFS-8132, which appears to have been filed against RC0. On Apr 11, 2015, at 1:44 AM, Vinod Kumar Vavilapalli wrote: > Hi all, > > I've created a release candidate RC0 for Apache Hadoop 2.7.0. > > The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.

Re: committing HADOOP-11746 test-patch improvements

16, 2015, at 7:38 PM, Chris Nauroth wrote: > I'd like to thank Allen Wittenauer for his work on HADOOP-11746 to rewrite > test-patch.sh. There is a lot of nice new functionality in there. My > favorite part is that some patches will execute much faster, so I expect > this will

Re: committing HADOOP-11746 test-patch improvements

FYI, MAPREDUCE-6324 is the first JIRA to be tested with the new code in place if someone hasn’t seen the new output. On Apr 21, 2015, at 9:06 PM, Allen Wittenauer wrote: > > > Just a heads up that I’ll be committing this to trunk and branch-2 here > in a bit. I’ll watch

Re: committing HADOOP-11746 test-patch improvements

good news! The new mvn site test does appear to be capable of catching issues! YARN-3410 was committed just prior to the new test code and has markdown-to-html syntax issues. HADOOP-11627 (for example) shows the results. :D On Apr 21, 2015, at 10:48 PM, Allen Wittenauer wrote: > &g

Re: committing HADOOP-11746 test-patch improvements

Err, first jira mentioned should be HADOOP-11861. On Apr 22, 2015, at 8:10 AM, Allen Wittenauer wrote: > > Some status: > > * So far, HADOOP-11627 was filed which is luckily an extremely easy bug to > fix. > > * There have been a few runs which seems to indicat

IMPORTANT: testing patches for branches

Hey gang, Just so everyone is aware, if you are working on a patch for either a feature branch or a major branch, if you name the patch with the branch name following the spec in HowToContribute (and a few other ways… test-patch tries to figure it out!), test-patch.sh *should*

Re: IMPORTANT: testing patches for branches

Vavilapalli wrote: > Does this mean HADOOP-7435 is no longer needed / closeable as dup? > > Thanks > +Vinod > > On Apr 22, 2015, at 12:34 PM, Allen Wittenauer wrote: > >> >> Hey gang, >> >> Just so everyone is aware, if you are working on

Re: IMPORTANT: testing patches for branches

Oh, this is also in the release notes, but one can use a git reference # as well. :) (with kudos to OOM for the idea.) On Apr 22, 2015, at 8:57 PM, Allen Wittenauer wrote: > > More than likely. It probably needs more testing (esp under Jenkins). > > It should be noted that

Re: IMPORTANT: testing patches for branches

On Apr 22, 2015, at 11:34 PM, Zheng, Kai wrote: > Hi Allen, > > This sounds great. > >>> Naming a patch foo-HDFS-7285.00.patch should get tested on the HDFS-7285 >>> branch. > Does it happen locally in developer's machine when running test-patch.sh, or > also mean something in Hadoop Jen

Re: fsck output compatibility question with regard to HDFS-7281

On Apr 24, 2015, at 5:53 AM, Yongjun Zhang wrote: > > Basically we are adding two additional lines to the report (as highlighted > above). > > Theoretically if a tool parses existing fsck report and expects the > 'Corrupt blocks" entry to be right after the "Average block replication" > entry,

HDFS audit log

I think we need to have a discussion about the HDFS audit log. The purpose of the HDFS audit log* is for operations and security people to keep track of actual, bits-on-disk changes to HDFS and related metadata changes. It is not meant as a catch-all for any and all HDFS

Bug Bash Details/Registration: May 8th, 2015

(Reply-to set to common-dev@) With over 900 patches not yet reviewed and approved for Apache Hadoop, it's time to make some strong progress on the bug list! A number of Apache Hadoop committers and Hadoop-related tech companies are hosting an Apache Hadoop Community event on Fri

Re: fsck output compatibility question with regard to HDFS-7281

On Apr 28, 2015, at 10:59 AM, Andrew Wang wrote: > > This is also not something typically upheld by unix-y commands. BSD vs. GNU > already leads to incompatible flags and output. Most of these commands > haven't been changed in 20 years, but that doesn't constitute a compat > guarantee.

Re: fsck output compatibility question with regard to HDFS-7281

f > making RPC calls to avoid duplicated code. > > ~Haohui > > On Tue, Apr 28, 2015 at 11:54 AM, Andrew Wang > wrote: >> On Tue, Apr 28, 2015 at 11:25 AM, Allen Wittenauer >> wrote: >> >>> >>> On Apr 28, 2015, at 10:59 AM, Andrew Wang

REMINDER! Register for the Bug Bash on May 8th!

Here’s the link in case you didn’t see it: https://www.eventbrite.com/e/apache-hadoop-global-bug-bash-tickets-16507188445 Whether you are working virtually or in-person, it’s important for everyone participating to register so that we can keep track of global p

Update: checkstyle, shellcheck, and whitespace tests

Hey gang. With the commit of HADOOP-11866, the output for checkstyle and whitespace has been greatly enhanced to actually be useful now. Instead of getting weird output, you'll see the file, the line number and (in the case of checkstyle) error. Note that line numbers are aft

we need a fix: precommit failures correlate to hdfs patches

So, as some may have noticed, I slammed the Jenkins servers over the weekend to get some recent patch test runs in JIRA for the bug bash this week. I've had a suspicion for a while now that either the long run times of the hadoop-hdfs module unit tests (typically 2+ hours) or t

Re: we need a fix: precommit failures correlate to hdfs patches

gt;> guarantees about files out of the main workspace. >> >> They all write to ${WORKSPACE}/../patchProcess, which will probably >> collide >> if multiple runs happen on the same machine. They also all blindly move >> that directory at the end of the run. >>

Jenkins precommit-*-build

TL;DR: Heads up: I’m going to hack on these scripts to fix the race conditions. Presently, we have a pretty wicked race condition in the precommit scripts, namely writing to a directory that is outside of Jenkins' workspace directory. I’ve committed changes this morning

Re: Jenkins precommit-*-build

HDFS, MAPREDUCE, and YARN have been migrated. Let me know of any issues and I’ll try to get to them as I can. This should be the end of the Jenkins race conditions for our pre commits! *crosses fingers*

REMINDER! REGISTRATIONS CLOSING 5/6!

Hey gang! A couple of things in regards to the Bug Bash on Friday: * We’ll be closing registrations to the Bug Bash on May 6th at 3PM Pacific time. So make sure you do it son: https://www.eventbrite.com/e/apache-hadoop-global-bug-bash-tickets-165071884

Re: REMINDER! REGISTRATIONS CLOSING 5/6!

On May 5, 2015, at 8:10 PM, Allen Wittenauer wrote: > * We’ll be closing registrations to the Bug Bash on May 6th at > 3PM Pacific time. So make sure you do it son: > https://www.eventbrite.com/e/apache-hadoop-global-bug-bash-tickets-16507188445 That

[DISCUSS] branch-1

May we declare this branch dead and just close bugs (but not necessarily concepts, ideas, etc) with won’t fix? I don’t think anyone has any intention of working on the 1.3 release, especially given that 1.2.1 was Aug 2013 …. I guess we need a PMC member to declare a vo

Re: How to add branch-2.7 to Jenkins white list

On May 11, 2015, at 10:23 AM, Wangda Tan wrote: > Hi Hadoop-dev, > > Now we are working on branch-2.7 for 2.7.1 release but I found branch-2.7 > may not be added to Jenkins white list. For example: YARN-3434 cannot > trigger Jenkins build. > > Could anybody help to add it? It’s not a white l

Re: How to add branch-2.7 to Jenkins white list

On May 11, 2015, at 10:34 AM, Allen Wittenauer wrote: > > It’s not a white list; it’s done heuristically. If you read through the > release notes of HADOOP-11746, you’ll see that it only supports major > branches. It does not (currently?) support minor or micro branches. (Th

Bug Bash Post-mortem

Hey gang. I promised a post-mortem, but I got distracted with making everyone's lives miserable with new test-patch code. Luckily my coworker Barbara (who some of you met!) wrote some wrap up info about it on our corporate blog, including some interesting stats…. htt

Re: DISCUSS: is the order in FS.listStatus() required to be sorted?

The POSIX spec for readdir (http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir.html) doesn’t spell out a sort order, so it should be assumed that the ordering isn’t guaranteed. Chris Siebenmann has written a few relative blog posts on the topic that might be of i

Re: set up jenkins test for branch-2

On Jun 14, 2015, at 1:00 PM, Yongjun Zhang wrote: > From time to time we saw changes that work fine in trunk but not branch-2, > and we don't catch the issue in a timely manner. The difference between > trunk and branch-2 is sufficient to justify periodic jenkins test and even > pre-commit test

Re: IMPORTANT: automatic changelog creation

I think it defeats the purpose ofu On Jul 8, 2015, at 12:13 AM, Tsuyoshi Ozawa wrote: > +1, thanks Allen and Andrew for taking lots effort! > >> Is there any possibility that, we can restrict someone from editing the >> issue in jira once its marked as "closed" after release? > > Vinay's comm

Re: Request for installing ISA-L library in build machines

On Jul 28, 2015, at 3:15 AM, Vinayakumar B wrote: > Have you tried package available at > https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version > ? > > I think this will work for Ubuntu also. Don’t forget to add it to the Dockerfile, otherwise those that use it won’t be

Re: Hadoop 2.6.1 Release process thread

I hope someone tests this against JDK6, otherwise this is an incompatible change. On Aug 12, 2015, at 2:21 PM, Chris Nauroth wrote: > I've just applied the 2.6.1-candidate label to HADOOP-10786. Since this > is somewhat late in the process, I thought I'd better follow up over email > too. >

Re: Even after HDFS-2856 JSVC References are require..?

FWIW, I still use and prefer jsvc, esp with the sudo trick in place. On Sep 9, 2015, at 9:35 AM, Chris Nauroth wrote: > AFAIK, the majority of existing deployments still use jsvc to run a > secured DataNode. It would be a backwards-incompatible change to remove > support for this deployment mo

Re: [VOTE] Release Apache Hadoop 2.6.1 RC0

On Sep 9, 2015, at 6:00 PM, Vinod Kumar Vavilapalli wrote: > - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these > fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off > 2.6.1. Isn’t there a risk that there are things in branch-2.6 that aren’t

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

There’s a bunch of stuff that needs to happen at the Jenkins level: * Kill off the JDK7 trunk builds for HADOOP, HDFS, MAPRED, YARN * Remove JDK7 from pre-commit for HADOOP, HDFS, MAPRED, YARN One thing that needs to happen in the Apache Yetus project: * Wait until YETUS-369 has been written and

[RESULT] Re: [VOTE] Merge feature branch HADOOP-12930

Vote passes. 1 = +1 non-binding 4 = +1 binding I’ll squash and commit here in a sec. Thanks everyone! - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache

  1   2   3   4   5   6   >