[jira] [Updated] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

[ https://issues.apache.org/jira/browse/SPARK-40128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated SPARK-40128: --- Docs Text: Added support for keeping vectorized reads enabled for Parquet files using the DELTA_LENG

[jira] [Updated] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

[ https://issues.apache.org/jira/browse/SPARK-40128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated SPARK-40128: --- Description: Even though https://issues.apache.org/jira/browse/SPARK-36879 added implementations fo

[jira] [Updated] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

[ https://issues.apache.org/jira/browse/SPARK-40128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated SPARK-40128: --- Attachment: delta_length_byte_array.parquet > Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone

[jira] [Created] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

Dennis Huo created SPARK-40128: -- Summary: Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader Key: SPARK-40128 URL: https://issues.apache.org/jira/browse/SPARK-40128

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Release Note: This feature adds a new `COMPOSITE_CRC` FileChecksum type which uses CRC composition

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848276#comment-16848276 ] Dennis Huo commented on HDFS-13056: --- Sorry, I somehow missed the note from [~xiaochen]

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Release Note: This feature adds a new `COMPOSITE_CRC` FileChecksum type which uses CRC composition

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Release Note: This feature adds a new `COMPOSITE_CRC` FileChecksum type which uses CRC composition

[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch

[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722014#comment-16722014 ] Dennis Huo commented on HDFS-14147: --- The branch-2.8 confusion might be because I had st

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424935#comment-16424935 ] Dennis Huo commented on HDFS-13056: --- Good observation, [~xiaochen], that seems to save a

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.014.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416732#comment-16416732 ] Dennis Huo commented on HDFS-13056: --- Thanks for taking another look [~xiaochen]! Applied

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.013.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.012.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.011.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412452#comment-16412452 ] Dennis Huo commented on HDFS-13056: --- Merged trunk to produce [^HDFS-13056.011.patch] >

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412449#comment-16412449 ] Dennis Huo commented on HDFS-13056: --- Thanks for taking a look [~ste...@apache.org]! Appl

[jira] [Created] (HDFS-13345) Add support for new COMPOSITE_CRC in WebHDFS

Dennis Huo created HDFS-13345: - Summary: Add support for new COMPOSITE_CRC in WebHDFS Key: HDFS-13345 URL: https://issues.apache.org/jira/browse/HDFS-13345 Project: Hadoop HDFS Issue Type: Sub-ta

[jira] [Created] (HDFS-13345) Add support for new COMPOSITE_CRC in WebHDFS

Dennis Huo created HDFS-13345: - Summary: Add support for new COMPOSITE_CRC in WebHDFS Key: HDFS-13345 URL: https://issues.apache.org/jira/browse/HDFS-13345 Project: Hadoop HDFS Issue Type: Sub-ta

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.010.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.009.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410650#comment-16410650 ] Dennis Huo commented on HDFS-13056: --- Thanks for the review [~xiaochen]! Responses:   -

[jira] [Commented] (HDFS-13256) Make FileChecksumHelper propagate per-datanode exceptions as part of top-level exception when all datanode attempts fail

[ https://issues.apache.org/jira/browse/HDFS-13256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393794#comment-16393794 ] Dennis Huo commented on HDFS-13256: --- Uploaded [^HDFS-13256-poc.patch] which is diffbased

[jira] [Updated] (HDFS-13256) Make FileChecksumHelper propagate per-datanode exceptions as part of top-level exception when all datanode attempts fail

[ https://issues.apache.org/jira/browse/HDFS-13256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13256: -- Attachment: HDFS-13256-poc.patch > Make FileChecksumHelper propagate per-datanode exceptions as part of

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393595#comment-16393595 ] Dennis Huo commented on HDFS-13056: --- [~ajayydv], unfortunately, the FileChecksumHelper a

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393023#comment-16393023 ] Dennis Huo commented on HDFS-13056: --- Thanks for helping validate the mixed-chunksize sup

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392176#comment-16392176 ] Dennis Huo commented on HDFS-13056: --- Variable-sized blocks within a file are indeed supp

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390617#comment-16390617 ] Dennis Huo commented on HDFS-13056: --- Some observations - the FileSystem interface doesn'

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.008.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.005.patch > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390272#comment-16390272 ] Dennis Huo commented on HDFS-13056: --- Good catch, thanks for the report, [~ajayydv]. I ac

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387080#comment-16387080 ] Dennis Huo commented on HDFS-13056: --- Uploaded [^HDFS-13056-branch-2.8.004.patch] and [^H

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.007.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.004.patch > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385621#comment-16385621 ] Dennis Huo commented on HDFS-13191: --- Thanks for the input! I personally don't feel too s

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384469#comment-16384469 ] Dennis Huo commented on HDFS-13056: --- While testing some integration with non-HDFS filesy

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.006.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.003.patch > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383178#comment-16383178 ] Dennis Huo commented on HDFS-13056: --- Added [^HDFS-13056.005.patch] to fix TestHdfsConfig

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.005.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383162#comment-16383162 ] Dennis Huo commented on HDFS-13056: --- Also added a backport to branch-2.8 which I was abl

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.002.patch > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382938#comment-16382938 ] Dennis Huo commented on HDFS-13056: --- [~ajayydv] Thanks for helping test and reviewing! A

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.004.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13191: -- Status: Patch Available (was: Open) > Internal buffer-sizing details are inadvertently baked into FileC

[jira] [Commented] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375253#comment-16375253 ] Dennis Huo commented on HDFS-13191: --- [^HDFS-13191.001.patch] illustrates a basic fix wit

[jira] [Updated] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13191: -- Attachment: HDFS-13191.001.patch > Internal buffer-sizing details are inadvertently baked into FileCheck

[jira] [Created] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

Dennis Huo created HDFS-13191: - Summary: Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum Key: HDFS-13191 URL: https://issues.apache.org/jira/browse/HDFS-13191

[jira] [Created] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

Dennis Huo created HDFS-13191: - Summary: Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum Key: HDFS-13191 URL: https://issues.apache.org/jira/browse/HDFS-13191

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373729#comment-16373729 ] Dennis Huo commented on HDFS-13056: --- [~szetszwo] and [~drankye] may also be interested i

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.003.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372423#comment-16372423 ] Dennis Huo commented on HDFS-13056: --- Fixed all the checkstyle issues; the failed unittes

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.003.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.002.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: hdfs-file-composite-crc32-v3.pdf > Expose file-level composite CRCs in HDFS which are compar

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347851#comment-16347851 ] Dennis Huo commented on HDFS-13056: --- Uploaded initial end-to-end working draft against t

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.001.patch > Expose file-level composite CRCs in HDFS which are comparable across

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347407#comment-16347407 ] Dennis Huo commented on HDFS-13056: --- I ended up going with modifying the existing protoc

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: hdfs-file-composite-crc32-v2.pdf > Expose file-level composite CRCs in HDFS which are compar

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341933#comment-16341933 ] Dennis Huo commented on HDFS-13056: --- Thanks for sharing! I was also weighing whether to

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338488#comment-16338488 ] Dennis Huo commented on HDFS-13056: --- Cleaned up the POC code against branch-2.8 to sketc

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.001.patch > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336852#comment-16336852 ] Dennis Huo commented on HDFS-13056: --- Attached a quick and dirty poc patch which just har

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.poc1.patch > Expose file-level composite CRCs in HDFS which are compar

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: hdfs-file-composite-crc32-v1.pdf > Expose file-level composite CRCs in HDFS which are compar

[jira] [Created] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

Dennis Huo created HDFS-13056: - Summary: Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts Key: HDFS-13056 URL: https://issues.apache.org/jira/browse/HDFS-13056

[jira] [Created] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

Dennis Huo created HDFS-13056: - Summary: Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts Key: HDFS-13056 URL: https://issues.apache.org/jira/browse/HDFS-13056

[jira] [Commented] (MAPREDUCE-6931) Remove TestDFSIO "Total Throughput" calculation

[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142510#comment-16142510 ] Dennis Huo commented on MAPREDUCE-6931: --- Done. > Remove TestDFSIO "Total Throug

[jira] [Created] (AIRFLOW-1535) Add support for Dataproc serviceAccountScopes in DataprocClusterCreateOperator

Dennis Huo created AIRFLOW-1535: --- Summary: Add support for Dataproc serviceAccountScopes in DataprocClusterCreateOperator Key: AIRFLOW-1535 URL: https://issues.apache.org/jira/browse/AIRFLOW-1535 Projec

[jira] [Commented] (MAPREDUCE-6931) Remove TestDFSIO "Total Throughput" calculation

[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125997#comment-16125997 ] Dennis Huo commented on MAPREDUCE-6931: --- Right, that line is part of the refacto

[jira] [Commented] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation

[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124768#comment-16124768 ] Dennis Huo commented on MAPREDUCE-6931: --- Hmm, looking at github I only see the r

[jira] [Updated] (MAPREDUCE-6931) Remove TestDFSIO "Total Throughput" calculation

[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated MAPREDUCE-6931: -- Summary: Remove TestDFSIO "Total Throughput" calculation (was: Fix TestDFSIO "Total Through

[jira] [Commented] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation

[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122505#comment-16122505 ] Dennis Huo commented on MAPREDUCE-6931: --- Fair enough, makes sense. I went ahead

[jira] [Commented] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation

[ https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122308#comment-16122308 ] Dennis Huo commented on MAPREDUCE-6931: --- Thanks for the explanation! I have no s

[jira] [Created] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation

Dennis Huo created MAPREDUCE-6931: - Summary: Fix TestDFSIO "Total Throughput" calculation Key: MAPREDUCE-6931 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation

Dennis Huo created MAPREDUCE-6931: - Summary: Fix TestDFSIO "Total Throughput" calculation Key: MAPREDUCE-6931 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931 Project: Hadoop Map/Reduce

[jira] [Resolved] (HDFS-12253) CLONE - Pretty-format the output for DFSIO

[ https://issues.apache.org/jira/browse/HDFS-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo resolved HDFS-12253. --- Resolution: Fixed > CLONE - Pretty-format the output for DFSIO > -

[jira] [Resolved] (HDFS-12253) CLONE - Pretty-format the output for DFSIO

[ https://issues.apache.org/jira/browse/HDFS-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo resolved HDFS-12253. --- Resolution: Fixed > CLONE - Pretty-format the output for DFSIO > -

[jira] [Created] (HDFS-12253) CLONE - Pretty-format the output for DFSIO

Dennis Huo created HDFS-12253: - Summary: CLONE - Pretty-format the output for DFSIO Key: HDFS-12253 URL: https://issues.apache.org/jira/browse/HDFS-12253 Project: Hadoop HDFS Issue Type: Test

[jira] [Created] (HDFS-12253) CLONE - Pretty-format the output for DFSIO

Dennis Huo created HDFS-12253: - Summary: CLONE - Pretty-format the output for DFSIO Key: HDFS-12253 URL: https://issues.apache.org/jira/browse/HDFS-12253 Project: Hadoop HDFS Issue Type: Test

[jira] [Created] (MAPREDUCE-6759) JobSubmitter/JobResourceUploader should parallelize upload of -libjars, -files, -archives

Dennis Huo created MAPREDUCE-6759: - Summary: JobSubmitter/JobResourceUploader should parallelize upload of -libjars, -files, -archives Key: MAPREDUCE-6759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6759

[jira] [Created] (MAPREDUCE-6759) JobSubmitter/JobResourceUploader should parallelize upload of -libjars, -files, -archives

Dennis Huo created MAPREDUCE-6759: - Summary: JobSubmitter/JobResourceUploader should parallelize upload of -libjars, -files, -archives Key: MAPREDUCE-6759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6759

[jira] [Created] (MAPREDUCE-6758) TestDFSIO should parallelize its creation of control files on setup

Dennis Huo created MAPREDUCE-6758: - Summary: TestDFSIO should parallelize its creation of control files on setup Key: MAPREDUCE-6758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6758 Project:

  1   2   >