[jira] [Resolved] (HADOOP-15051) FSDataOutputStream returned by LocalFileSystem#createNonRecursive doesn't have hflush capability

2017-11-19 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-15051.
-
Resolution: Won't Fix

> FSDataOutputStream returned by LocalFileSystem#createNonRecursive doesn't 
> have hflush capability
> 
>
> Key: HADOOP-15051
> URL: https://issues.apache.org/jira/browse/HADOOP-15051
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Ted Yu
>
> See HBASE-19289 for background information.
> Here is related hbase code (fs is instance of LocalFileSystem):
> {code}
> this.output = fs.createNonRecursive(path, overwritable, bufferSize, 
> replication, blockSize,
>   null);
> // TODO Be sure to add a check for hsync if this branch includes 
> HBASE-19024
> if (!(CommonFSUtils.hasCapability(output, "hflush"))) {
>   throw new StreamLacksCapabilityException("hflush");
> {code}
> StreamCapabilities is used to poll "hflush" capability.
> [~busbey] suggested fixing this in hadoop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

2017-11-19 Thread Steve Loughran 


On 14 Nov 2017, at 00:10, Arun Suresh 
mailto:asur...@apache.org>> wrote:

Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2.

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
*

The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
756ebc8394e473ac25feac05fa493f6d612e6c50.

The maven artifacts are available via 
repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
*

We are carrying over the votes from the previous RC given that the delta is
the license fix.

Given the above - we are also going to stick with the original deadline for
the vote : ending on Friday 17th November 2017 2pm PT time.

Thanks,
-Arun/Subru

I hadn't finished my testing yet; I'd been assuming that it was 5 days from thw 
new RC. I believe every RC should (must?) still have that 5 day test period. I 
have done all my core tests, but haven't done the final tests of the S3Guard 
CLI tests. I did have to actually put in an afternoon making my spark cloud 
integration tests work against branch-2, so it wasn't like I could just D/L and 
test in half an hour


As it was, I had filed one minor bug, but didnt' consider that an issue as it 
was only with the new  FileSystem.create(path)" Builder. I was going to go vote 
+! unless those CLI tests were completly broken.


What I do want is  release notes to highlight which things we think 
unstable/experimental features to use with caution

  1.  Filesystem.create(Path)
  2.  S3Guard
  3.  AlyiunOSS (it hasn't been out long enough to be trusted)

We ourselves know what's still stabilising, other should too


Download:

D/L tar file, & .asc, check signature

pg2 --verify hadoop-2.9.0.tar.gz.asc
gpg: assuming signed data in 'hadoop-2.9.0.tar.gz'
gpg: Signature made Mon 13 Nov 23:45:49 2017 GMT
gpg:using RSA key 0x7ECDEEEA64ECB6E6
gpg: Good signature from "Arun Suresh 
mailto:asur...@apache.org>>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:  There is no indication that the signature belongs to the owner.
Primary key fingerprint: 412B BFB1 27CB 48DA 6BA2  E3EE 7ECD EEEA 64EC B6E6


This is a valid signature; Arun is listed in KEYS. He's not trusted though: at 
the time of checking his public key hadn't been signed by anyone I trust
 though it shows we need more cross-trust between them. (Since then Andrew Wang 
has authenticated him, so he is now transitively trusted).


Downstream build

Clean build of Spark through the staged artifacts (must remember to rm the 
local copies when the release ships). This verifies artifacts and source code 
compatibility.

mvn clean install -Pyarn,hadoop-2.7,hadoop-cloud,snapshots-and-staging  
-Dhadoop.version=2.9.0

Everything compiled fine. I did not do a full test run; I didn't have the time 
and without knowing whether everything worked locally on the 2.8.1 release, 
could have been a distraction on a failure.


Downstream tests:

Ran all my spark cloud integration tests: s3 (S3 ireland + S3Guard) , azure 
(Azure ireland(, swift (RAX US);

https://github.com/hortonworks-spark/cloud-integration/tree/branch-2

there were 3 failures of SparkSQL & ORC in test setup, but I think that's 
related to the latest spark code & my injected-into-org.apache.spark-namespace 
tests; they're a bit brittle to changes in spark-master.

All the work used the "legacy" FileOutputCommitter; with s3guard turned on. 
This delivers the consistency needed for Job and task commit to be correct, 
though without the performance needed for it to be usable in production. That's 
dependent on HADOOP-13786, which I don't propose to backport. (it loves its 
Java 8 & targets 3.1)


hadoop cloud module object store tests

Checked out the relevant git commit & did a local run of hadoop-aws, 
hadoop-azure and hadoop-openstack


hadoop-amazon
* s3a happy, including s3guard
* All the s3n & s3:// tests skipped/failed because I didn't have the test 
bindings for those set up: untested.

Azure wasb: happy; one transient failure which

ADLs: Not set up test this
OSS: not set up to test this

Swift:

Found one test failure due to swift not supporting 
FileSystem.createNonRecursive(); this means the new builder based 
FileSystem.

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-11-19 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 208] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 
   
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.getWeight()
 does not release lock on all exception paths At FSAppAttempt.java:on all 
exception paths At FSAppAttempt.java:[line 1310] 

Unreaped Processes :

   hadoop-hdfs:13 
   hadoop-mapreduce-client-jobclient:1 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 
   hadoop.hdfs.TestFileConcurrentReader 
   hadoop.hdfs.TestDFSRename 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 
   hadoop.hdfs.TestRenameWhileOpen 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 
   hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy 
   hadoop.hdfs.TestEncryptedTransfer 
   hadoop.fs.TestUnbuffer 
   hadoop.hdfs.TestErasureCodingPolicies 
   hadoop.hdfs.TestDFSStripedOutputStream 
   hadoop.hdfs.TestDecommission 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 

Timed out junit tests :

   org.apache.hadoop.mapred.pipes.TestPipeApplication 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   UnreapedProcessesLog:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-reaper.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-reaper.txt
  [4.0K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.1M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/598/a