[jira] [Resolved] (HADOOP-14736) S3AInputStream to implement an efficient skip() call through seeking
[
https://issues.apache.org/jira/browse/HADOOP-14736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14736.
-
Resolution: Duplicate
I should fix HADOOP-14606 instead of filing the same bug whenever I look at the
code
> S3AInputStream to implement an efficient skip() call through seeking
>
>
> Key: HADOOP-14736
> URL: https://issues.apache.org/jira/browse/HADOOP-14736
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Priority: Major
>
> {{S3AInputStream}} implements skip() naively through the byte class: Reading
> and discarding all data. Efficient on classic "sequential" reads, provided
> the forward skip is <1MB. For larger skip values or on random IO, seek()
> should be used.
> After some range checks/handling past-EOF skips to seek (EOF-1), a seek()
> should handle the skip file.
> *there are no FS contract tests for skip semantics*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15220) Über-jira: S3a phase V: Hadoop 3.2 features
Steve Loughran created HADOOP-15220: --- Summary: Über-jira: S3a phase V: Hadoop 3.2 features Key: HADOOP-15220 URL: https://issues.apache.org/jira/browse/HADOOP-15220 Project: Hadoop Common Issue Type: Bug Reporter: Steve Loughran Assignee: Steve Loughran Über-jira for S3A work for Hadoop 3.2.x The items from HADOOP-14831 which didn't get into Hadoop-3.1, and anything else -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13371.
-
Resolution: Won't Fix
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13648) s3a home directory to be "/"
[
https://issues.apache.org/jira/browse/HADOOP-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13648.
-
Resolution: Won't Fix
too late to fix this, I suspect
> s3a home directory to be "/"
>
>
> Key: HADOOP-13648
> URL: https://issues.apache.org/jira/browse/HADOOP-13648
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> The home directory of an s3a instances is {{/user/" +
> System.getProperty("user.name"))}}. As HADOOP-12774 notes, it gets the user
> wrong: if it were to be correct it should use the shortname of the current
> principal.
> I don't think the username is valid here at all. s3a buckets are not
> filesystems with users and permissions; all this per-user home dir appears to
> do is cause confusion, and end up putting the output of an {{hadoop fs -rm}}
> operation into a directory under it.
> If we made it "/" then it'd be the same for all users, and "/.Trash" would be
> where deleted files get copied to
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13892) use s3 tags/headers to record permissions on objects, so preserving them through distcp round trips
[ https://issues.apache.org/jira/browse/HADOOP-13892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13892. - Resolution: Won't Fix > use s3 tags/headers to record permissions on objects, so preserving them > through distcp round trips > --- > > Key: HADOOP-13892 > URL: https://issues.apache.org/jira/browse/HADOOP-13892 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Major > > S3 now supports object tags, attributes which can be updated during the life > of an object. > S3A could use that to preserve the permissions/ACLs of objects when copied > from elsewhere, in particular from HDFS. This would ensure that data backed > up from HDFS preserves all the permission information needed when doing a > recovery from S3 to HDFS. > Azure WASB does exactly this already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13974) S3Guard CLI to support list/purge of pending multipart commits
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13974. - Resolution: Fixed > S3Guard CLI to support list/purge of pending multipart commits > -- > > Key: HADOOP-13974 > URL: https://issues.apache.org/jira/browse/HADOOP-13974 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Aaron Fabbri >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch, > HADOOP-13974.003.patch, HADOOP-13974.004.patch, HADOOP-13974.005.patch, > HADOOP-13974.006.patch, HADOOP-13974.007.patch > > > The S3A CLI will need to be able to list and delete pending multipart > commits. > We can do the cleanup already via fs.s3a properties. The CLI will let scripts > stat for outstanding data (have a different exit code) and permit batch jobs > to explicitly trigger cleanups. > This will become critical with the multipart committer, as there's a > significantly higher likelihood of commits remaining outstanding. > We may also want to be able to enumerate/cancel all pending commits in the FS > tree -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14975) S3AInputStream/OutputStream statistics aren't getting into StorageStatistics
[ https://issues.apache.org/jira/browse/HADOOP-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14975. - Resolution: Duplicate Duplicate of HADOOP-15161; that is fixed so closing this one > S3AInputStream/OutputStream statistics aren't getting into StorageStatistics > > > Key: HADOOP-14975 > URL: https://issues.apache.org/jira/browse/HADOOP-14975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > when the input and output stream stats are merged into the > S3AInstrumentation, the FS statistics aren't updated to match, so FS > statistics don't track things like aggregate throttle count, TCP aborts, > bytes discarded etc. They are metrics, but not sotrage stats > They should be, which requires S3AInstrumentation to take the StorageStats in > its constructor and then update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
How to get Jenkins job create/edit permission?
Hi guys, I was trying to run JACC build for 3.1.0 release. When I run JACC to compare branch-2.9 or branch-3.0 with branch-3.1, I found the Jenkins build got killed after 1 hour. For example: https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/1828/consoleFull I think there're some settings of the JACC job, however, I'm not sure how can I tweak the Jenkins job. Could you share how to get permission to tweak Jenkins jobs? (Apache ID: wangda) Thanks, Wangda
[jira] [Created] (HADOOP-15221) Swift driver should not fail if JSONUtils reports UnknowPropertyException
Chen He created HADOOP-15221: Summary: Swift driver should not fail if JSONUtils reports UnknowPropertyException Key: HADOOP-15221 URL: https://issues.apache.org/jira/browse/HADOOP-15221 Project: Hadoop Common Issue Type: Improvement Components: fs/swift Reporter: Chen He Assignee: Chen He org.apache.hadoop.fs.swift.exceptions.SwiftJsonMarshallingException: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field We know system is keep involving and new field will be added. However, for compatibility point of view, extra field added to json should be logged but may not lead to failure from the robustness point of view. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reopened HADOOP-14961:
-
> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console
> {noformat}
> Downloading Oracle Java 8...
> [0m[91m--2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving download.oracle.com (download.oracle.com)...
> [0m[91m23.59.190.131, 23.59.190.130
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily
> Location:
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
> [following]
> --2017-10-18 18:28:11--
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)...
> [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e
> Connecting to edelivery.oracle.com
> (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected.
> [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved
> Temporarily
> Location:
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
> [following]
> --2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m404 Not Found
> [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found.
> [0m[91mdownload failed
> Oracle JDK 8 is NOT installed.
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial"
> for a long term fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge resolved HADOOP-14961.
-
Resolution: Duplicate
Fixed by HADOOP-14816.
> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console
> {noformat}
> Downloading Oracle Java 8...
> [0m[91m--2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving download.oracle.com (download.oracle.com)...
> [0m[91m23.59.190.131, 23.59.190.130
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily
> Location:
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
> [following]
> --2017-10-18 18:28:11--
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)...
> [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e
> Connecting to edelivery.oracle.com
> (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected.
> [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved
> Temporarily
> Location:
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
> [following]
> --2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m404 Not Found
> [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found.
> [0m[91mdownload failed
> Oracle JDK 8 is NOT installed.
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial"
> for a long term fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
Eric Yang created HADOOP-15222: -- Summary: Refine proxy user authorization to support multiple ACL list Key: HADOOP-15222 URL: https://issues.apache.org/jira/browse/HADOOP-15222 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Eric Yang This Jira is responding to follow up work for HADOOP-14077. The original goal of HADOOP-14077 is to have ability to support multiple ACL lists. When checking for proxy user authorization in AuthenticationFilter to ensure there is a way to authorize normal users and admin users using separate proxy users ACL lists. This was suggested in HADOOP-14060 to configure AuthenticationFilterWithProxyUser this way: AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser This enables the second AuthenticationFilterWithProxyUser validates both credentials claim by proxy user, and end user. However, there is a side effect that unauthorized users are not properly rejected with 403 FORBIDDEN message if there is no other web filter configured to handle the required authorization work. This JIRA is intend to discuss the work of HADOOP-14077 by either combine StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a AuthorizationFilterWithProxyUser as a final filter to evict unauthorized user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Request edit permission for wiki.apache.org/hadoop
Thanks -Xing
