Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Jack Ye
Nice, just saw that. We are adding the definitions as a part of ttps:// github.com/apache/iceberg/pull/9695, we can help review the PRs listed here and then update the OpenAPI spec accordingly. -Jack On Wed, Feb 14, 2024 at 4:12 PM Steven Wu wrote: > @Ryan, the JSON serialization is also used

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Steven Wu
@Ryan, the JSON serialization is also used by Flink for checkpoint state. so it is not purely a REST API thing. @Jack, Ryan also had the same suggestion in the PR comment. I have updated the naming On Wed, Feb 14, 2024 at 4:08 PM Jack Ye wrote: > > It would fail if the FileScanTask is some othe

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Jack Ye
> It would fail if the FileScanTask is some other implementation (like StaticDataTask). Actually we faced exactly the same issue, and we have an internal patch to fix the parser for that. +1 for the proposal. For the type names, can we come up with a different name from " base-file-task"? "base" i

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Ryan Blue
Thanks, Steven! Looks like the right direction to add other task types with their own serialization. I hadn't realized that these were in the table spec and not just the REST spec. What do you think about keeping JSON serialization that isn't part of table metadata in the REST spec? I'm actually p

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Steven Wu
The first linked reference is the PR for spec update. [3] https://github.com/apache/iceberg/pull/9728 On Wed, Feb 14, 2024 at 3:36 PM Steven Wu wrote: > We just ran out of time and didn't get a chance to discuss this in the > community sync meeting today. Hence, I am raising the discussion here

[Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Steven Wu
We just ran out of time and didn't get a chance to discuss this in the community sync meeting today. Hence, I am raising the discussion here. We added JSON parsers for content file and file scan task a year ago [1]. Recently, I just realized the implementation only handles BaseFileScanTask. It wou

Deprecate DynamodbCatalog

2024-02-14 Thread Jack Ye
Hi everyone, As discussed in community sync, I am raising a thread about potentially deprecating DynamodbCatalog. I added it back in the day because of some customer collaboration effort. Later another implementation was also added in pybut at this point the specific customer has forked the imple

[VOTE] Release Apache PyIceberg 0.6.0rc6

2024-02-14 Thread Sung Yun (BLOOMBERG/ 120 PARK)
Hi Everyone, We are moving onto the next RC with some important fixes. This RC includes: * Bug Fix in passing configuration through environment variables #423 * Arm wheels #416 * Correction to the NOTICE and LICENSE #413 Again, here's a summary of the high level features included in this release