Re: [VOTE] Add 'inherited' and 'namespaces' Fields to GetApplicablePolicies API Response

2025-04-09 Thread Dennis Huo
+1, thanks! On Mon, Apr 7, 2025 at 3:12 PM Yufei Gu wrote: > +1. Thanks Jonas for the change! > Yufei > > > On Sat, Apr 5, 2025 at 6:45 PM Honah J. wrote: > > > Hi folks, > > > > Thanks for the feedback and discussion so far. Based on our PR discussion > > about naming, we’ve simplified “namesp

[DISCUSS] Secrets Management for Catalog Federation

2025-04-04 Thread Dennis Huo
Hello! I wanted to continue discussion that had started during review of the initial Catalog Federation API spec ( https://github.com/apache/polaris/pull/1026) regarding how we want to handle secrets. I drafted this document describing how secrets management fits in to Catalog Federation, while e

Re: [VOTE] REST API changes for Catalog Federation - addition of ConnnectionConfigInto to ExternalCatalog

2025-03-26 Thread Dennis Huo
t Singh > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > Best, > > > > Prashant > > > > > > > > On Tue, Mar 25, 2025 at 7:01 AM Dmitri Bourlatchkov < > di...@apache.org> > > >

Re: Clarification on Unique Entity Identifier in Polaris

2025-03-26 Thread Dennis Huo
Correct, the entityId must be unique across all types. On Wed, Mar 26, 2025 at 12:30 PM Eric Maynard wrote: > I believe that entity ID by itself is meant to be a unique identifier. > > On Wed, Mar 26, 2025 at 12:27 PM Honah J. wrote: > > > Hi folks, > > > > I have a question about what constitu

[VOTE] REST API changes for Catalog Federation - addition of ConnnectionConfigInto to ExternalCatalog

2025-03-24 Thread Dennis Huo
Hello all, We've had some productive discussion in various places on the mailing list, in the github PR, and in the live community sync now about the initial minimal updates to the Polaris REST API (under the "management API") for adding a ConnectionConfigInfo field to ExternalCatalogs to represen

Re: [Discuss] Change EntityTable properties and internal properties from TEXT to JSONB

2025-03-21 Thread Dennis Huo
Re: "non-transactional", I guess it's more descriptive to refer to this new implementation as "AtomicOperationJdbcPersistence" if it's confusing to say "non-transactional"; the key is simply that we don't expose any "runInTransaction" to the AtomicOperationMetaStoreManager layer. Regarding Robert'

Re: Catalog Federation API Spec updates - adding ConnectionConfigInfo

2025-03-20 Thread Dennis Huo
Manager) and secret storage. > > However, that will likely lead to significant and non-backward-compatible > changes in Admin API areas that deal with federated catalogs. I am > personally fine with this approach. Still, I think it would be good to hear > what other people think

Catalog Federation API Spec updates - adding ConnectionConfigInfo

2025-03-19 Thread Dennis Huo
Hi all, For anyone who may not have seen the PR on github, just wanted to ensure folks have a chance to give input on the mailing list here about the latest API spec proposal for supporting Catalog Federation: https://github.com/apache/polaris/pull/1026 This adds a ConnectionConfigInfo field to

Re: Donate Nessie Iceberg Catalog migrator

2025-02-11 Thread Dennis Huo
This is great to hear, thanks for helping arrange this! +1 to having this tool in Polaris and +1 inviting one of the original main contributors of the code. I don't feel too strongly about separate repo vs additional module, since I could see pros/cons of either approach. Would be a bit cleaner t

Re: [DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-05 Thread Dennis Huo
it would be great to discuss. > We can give time to anyone interested to read the proposals, and > discuss next week ? > > Maybe we can do it on Thursday, Feb 13, 9am PST (same slot as Polaris > Community Meeting) ? > > Thanks > Regards > JB > > On Tue, Feb 4, 2025 at

Re: [DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-04 Thread Dennis Huo
ion of > handling concurrent reads / writes made by different instances of the > Polaris Server. Would you be able to elaborate on that? > > Thanks, > Dmitri. > > On Tue, Feb 4, 2025 at 2:03 PM Dennis Huo wrote: > > > Hello all, > > > > We've had s

Re: [DISCUSS] Clarification of Persistence SPI

2025-02-04 Thread Dennis Huo
For posterity, https://docs.google.com/document/d/1U9rprj8w8-Q0SnQvRMvoVlbX996z-89eOkVwWTQaZG0/edit?tab=t.0#bookmark=id.cmp67w7aja0o explains a bit about why the Postgres implementation hits this. To recap from the doc comment: 1. The current Postgres impl naively follows the same design pattern a

[DISCUSS] Polaris Persistence architecture, refactor, support for new backends (including MongoDB, DynamoDB)

2025-02-04 Thread Dennis Huo
Hello all, We've had some discussions and github issues ( https://github.com/apache/polaris/issues/775, https://github.com/apache/polaris/issues/766, etc) scattered between community syncs, slack threads, etc., related to how to adapt the Polaris persistence layer to new DB backends, so I'm hoping

Re: Next steps for design discussion of Apache Polaris Catalog Federation

2025-01-15 Thread Dennis Huo
ink: https://meet.google.com/ryj-mueq-csf Thanks JB for setting it up! On Mon, Jan 13, 2025 at 11:15 AM Russell Spitzer wrote: > +1 Works for me (meeting on thursday) > > On Sun, Jan 12, 2025 at 8:56 PM Dennis Huo wrote: > > > Ah so sorry, I meant to include the proposed date of Thur

Re: Welcome Dennis Huo as new committer

2025-01-13 Thread Dennis Huo
t; > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Jan 13, 2025 at 2:38 AM Jean-Baptiste Onofré < > j...@nanthrax.net> > > > > wrote: > > > > > > > > > Hi folks, > > > >

Re: Next steps for design discussion of Apache Polaris Catalog Federation

2025-01-12 Thread Dennis Huo
between catalogs. Wdyt? > > Thanks, > > Alex > > > On Fri, Jan 10, 2025 at 3:36 AM Dennis Huo wrote: > > > > Hello! > > > > I wanted to kick off next steps to drive forward the design discussion > for > > Polaris Catalog Federation: > > >

Next steps for design discussion of Apache Polaris Catalog Federation

2025-01-09 Thread Dennis Huo
Hello! I wanted to kick off next steps to drive forward the design discussion for Polaris Catalog Federation: https://docs.google.com/document/d/1Q6eEytxb0btpOPcL8RtkULskOlYUCo_3FLvFRnHkzBY/edit?tab=t.0 In some of the prior community syncs I think folks expressed a desire to have some dedicated d

Re: Iceberg Catalog Federation

2024-12-11 Thread Dennis Huo
I just filed https://github.com/apache/polaris/issues/540 to track this feature request, and wrote up this document to try to better solidify our terminology and concepts on related federation features, while also sketching out a proposal for a basic MVP of catalog federation that could be easily e

Re: [DISCUSS] Defining a concept of "externally owned" tables in the REST spec

2024-10-09 Thread Dennis Huo
Please feel free to chime in here if I missed anything or got anything wrong from today's discussion. On Fri, Sep 20, 2024 at 9:05 PM Dennis Huo wrote: > Thanks for the input, Christian! > > I agree a comprehensive solution would likely require some notion of > pull-based ap

Re: [DISCUSS] Defining a concept of "externally owned" tables in the REST spec

2024-09-20 Thread Dennis Huo
ur catalog we are just using CloudEvents wrapped around > `TableUpdates` enriched with a few extensions. > > > > For both pull and push based approaches, your building block 1) is needed > anyway – so that’s surely a common ground. > > > > I would be interested to hear

[DISCUSS] Defining a concept of "externally owned" tables in the REST spec

2024-09-18 Thread Dennis Huo
Hi all, I wanted to follow up on some discussions that came up in one of the Iceberg Catalog community syncs awhile back relating to the concept of tables that can be registered in an Iceberg REST Catalog but which have their "source of truth" in some external Catalog. The original context was th

Re: Support Securable Objects in Iceberg REST Catalog

2024-07-02 Thread Dennis Huo
he request will loadTable(table_name, >> context={privilege=DELETE}). Would that be something feasible to solve the >> concern? >> >> > mapping INSERT/DELETE/UPDATE all to TABLE_WRITE_DATA since at least for >> now, from the Catalog's perspective, any deletes requi

RE: Re: Support Securable Objects in Iceberg REST Catalog

2024-06-28 Thread Dennis Huo
to handle directly in the REST request/response. Would love to hear anyone's thoughts on these areas. Cheers, Dennis Huo On 2024/06/08 19:12:10 Walaa Eldin Moustafa wrote: > Thanks Jack and team for working on this proposal. I went over it and it is > very well written. I particularl

[jira] [Updated] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

2022-08-17 Thread Dennis Huo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated SPARK-40128: --- Docs Text: Added support for keeping vectorized reads enabled for Parquet files using the

[jira] [Updated] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

2022-08-17 Thread Dennis Huo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated SPARK-40128: --- Description: Even though https://issues.apache.org/jira/browse/SPARK-36879 added implementations

[jira] [Updated] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

2022-08-17 Thread Dennis Huo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated SPARK-40128: --- Attachment: delta_length_byte_array.parquet > Add DELTA_LENGTH_BYTE_ARRAY as a recogni

[jira] [Created] (SPARK-40128) Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

2022-08-17 Thread Dennis Huo (Jira)
Dennis Huo created SPARK-40128: -- Summary: Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader Key: SPARK-40128 URL: https://issues.apache.org/jira/browse/SPARK-40128

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2019-05-25 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Release Note: This feature adds a new `COMPOSITE_CRC` FileChecksum type which uses CRC composition

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2019-05-25 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848276#comment-16848276 ] Dennis Huo commented on HDFS-13056: --- Sorry, I somehow missed the note from [~xiao

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2019-05-25 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Release Note: This feature adds a new `COMPOSITE_CRC` FileChecksum type which uses CRC composition

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2019-05-25 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Release Note: This feature adds a new `COMPOSITE_CRC` FileChecksum type which uses CRC composition

[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch

2018-12-14 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722014#comment-16722014 ] Dennis Huo commented on HDFS-14147: --- The branch-2.8 confusion might be because I

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-04-03 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424935#comment-16424935 ] Dennis Huo commented on HDFS-13056: --- Good observation, [~xiaochen], that seems to

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-04-03 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.014.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-27 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416732#comment-16416732 ] Dennis Huo commented on HDFS-13056: --- Thanks for taking another look [~xiao

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-27 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.013.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-24 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.012.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.011.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412452#comment-16412452 ] Dennis Huo commented on HDFS-13056: --- Merged trunk to produce [^HDFS-13056.011.p

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412449#comment-16412449 ] Dennis Huo commented on HDFS-13056: --- Thanks for taking a look [~ste...@apache

[jira] [Created] (HDFS-13345) Add support for new COMPOSITE_CRC in WebHDFS

2018-03-23 Thread Dennis Huo (JIRA)
Dennis Huo created HDFS-13345: - Summary: Add support for new COMPOSITE_CRC in WebHDFS Key: HDFS-13345 URL: https://issues.apache.org/jira/browse/HDFS-13345 Project: Hadoop HDFS Issue Type: Sub

[jira] [Created] (HDFS-13345) Add support for new COMPOSITE_CRC in WebHDFS

2018-03-23 Thread Dennis Huo (JIRA)
Dennis Huo created HDFS-13345: - Summary: Add support for new COMPOSITE_CRC in WebHDFS Key: HDFS-13345 URL: https://issues.apache.org/jira/browse/HDFS-13345 Project: Hadoop HDFS Issue Type: Sub

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.010.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.009.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410650#comment-16410650 ] Dennis Huo commented on HDFS-13056: --- Thanks for the review [~xiaochen]! Respo

[jira] [Commented] (HDFS-13256) Make FileChecksumHelper propagate per-datanode exceptions as part of top-level exception when all datanode attempts fail

2018-03-09 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393794#comment-16393794 ] Dennis Huo commented on HDFS-13256: --- Uploaded [^HDFS-13256-poc.patch] whic

[jira] [Updated] (HDFS-13256) Make FileChecksumHelper propagate per-datanode exceptions as part of top-level exception when all datanode attempts fail

2018-03-09 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13256: -- Attachment: HDFS-13256-poc.patch > Make FileChecksumHelper propagate per-datanode exceptions as part

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-09 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393595#comment-16393595 ] Dennis Huo commented on HDFS-13056: --- [~ajayydv], unfortunately, the FileChecksumHe

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-09 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393023#comment-16393023 ] Dennis Huo commented on HDFS-13056: --- Thanks for helping validate the mixed-chunk

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-08 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392176#comment-16392176 ] Dennis Huo commented on HDFS-13056: --- Variable-sized blocks within a file are in

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-07 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-07 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390617#comment-16390617 ] Dennis Huo commented on HDFS-13056: --- Some observations - the FileSystem inter

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-07 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-07 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.008.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-07 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.005.patch > Expose file-level composite CRCs in HDFS which

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-07 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390272#comment-16390272 ] Dennis Huo commented on HDFS-13056: --- Good catch, thanks for the report, [~ajayydv

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-05 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387080#comment-16387080 ] Dennis Huo commented on HDFS-13056: --- Uploaded [^HDFS-13056-branch-2.8.004.patch]

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-05 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.007.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-05 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.004.patch > Expose file-level composite CRCs in HDFS which

[jira] [Commented] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

2018-03-04 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385621#comment-16385621 ] Dennis Huo commented on HDFS-13191: --- Thanks for the input! I personally don't

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-02 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-02 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-02 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384469#comment-16384469 ] Dennis Huo commented on HDFS-13056: --- While testing some integration with non-

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-02 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.006.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-02 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.003.patch > Expose file-level composite CRCs in HDFS which

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383178#comment-16383178 ] Dennis Huo commented on HDFS-13056: --- Added [^HDFS-13056.005.patch] to

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.005.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383162#comment-16383162 ] Dennis Huo commented on HDFS-13056: --- Also added a backport to branch-2.8 which I

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.002.patch > Expose file-level composite CRCs in HDFS which

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382938#comment-16382938 ] Dennis Huo commented on HDFS-13056: --- [~ajayydv] Thanks for helping test and revie

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-03-01 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.004.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

2018-02-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13191: -- Status: Patch Available (was: Open) > Internal buffer-sizing details are inadvertently baked i

[jira] [Commented] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

2018-02-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375253#comment-16375253 ] Dennis Huo commented on HDFS-13191: --- [^HDFS-13191.001.patch] illustrates a basic

[jira] [Updated] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

2018-02-23 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13191: -- Attachment: HDFS-13191.001.patch > Internal buffer-sizing details are inadvertently baked i

[jira] [Created] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

2018-02-23 Thread Dennis Huo (JIRA)
Dennis Huo created HDFS-13191: - Summary: Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum Key: HDFS-13191 URL: https://issues.apache.org/jira/browse/HDFS-13191

[jira] [Created] (HDFS-13191) Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum

2018-02-23 Thread Dennis Huo (JIRA)
Dennis Huo created HDFS-13191: - Summary: Internal buffer-sizing details are inadvertently baked into FileChecksum and BlockGroupChecksum Key: HDFS-13191 URL: https://issues.apache.org/jira/browse/HDFS-13191

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373729#comment-16373729 ] Dennis Huo commented on HDFS-13056: --- [~szetszwo] and [~drankye] may also be intere

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-22 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.003.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-21 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-21 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-21 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372423#comment-16372423 ] Dennis Huo commented on HDFS-13056: --- Fixed all the checkstyle issues; the fa

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-21 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.003.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-21 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-21 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-20 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are compara

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-02-20 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.002.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-01-31 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: hdfs-file-composite-crc32-v3.pdf > Expose file-level composite CRCs in HDFS which

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-01-31 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347851#comment-16347851 ] Dennis Huo commented on HDFS-13056: --- Uploaded initial end-to-end working draft aga

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-01-31 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.001.patch > Expose file-level composite CRCs in HDFS which are comparable acr

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-01-31 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347407#comment-16347407 ] Dennis Huo commented on HDFS-13056: --- I ended up going with modifying the exis

[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-01-31 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: hdfs-file-composite-crc32-v2.pdf > Expose file-level composite CRCs in HDFS which

[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts

2018-01-26 Thread Dennis Huo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341933#comment-16341933 ] Dennis Huo commented on HDFS-13056: --- Thanks for sharing! I was also weighing whethe

  1   2   >