Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-16 Thread Steven Wu
I have updated the PR with Java implementation for StaticDataTask. As I updated the PR description, we can add support for other task types incrementally to keep the PRs relatively small. [3] https://github.com/apache/iceberg/pull/9728 On Wed, Feb 14, 2024 at 4:24 PM Jack Ye wrote: > Nice, just

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Jack Ye
Nice, just saw that. We are adding the definitions as a part of ttps:// github.com/apache/iceberg/pull/9695, we can help review the PRs listed here and then update the OpenAPI spec accordingly. -Jack On Wed, Feb 14, 2024 at 4:12 PM Steven Wu wrote: > @Ryan, the JSON serialization is also used

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Steven Wu
@Ryan, the JSON serialization is also used by Flink for checkpoint state. so it is not purely a REST API thing. @Jack, Ryan also had the same suggestion in the PR comment. I have updated the naming On Wed, Feb 14, 2024 at 4:08 PM Jack Ye wrote: > > It would fail if the FileScanTask is some othe

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Jack Ye
> It would fail if the FileScanTask is some other implementation (like StaticDataTask). Actually we faced exactly the same issue, and we have an internal patch to fix the parser for that. +1 for the proposal. For the type names, can we come up with a different name from " base-file-task"? "base" i

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Ryan Blue
Thanks, Steven! Looks like the right direction to add other task types with their own serialization. I hadn't realized that these were in the table spec and not just the REST spec. What do you think about keeping JSON serialization that isn't part of table metadata in the REST spec? I'm actually p

Re: [Discuss] add a new task-type to file scan task JSON serialization

2024-02-14 Thread Steven Wu
The first linked reference is the PR for spec update. [3] https://github.com/apache/iceberg/pull/9728 On Wed, Feb 14, 2024 at 3:36 PM Steven Wu wrote: > We just ran out of time and didn't get a chance to discuss this in the > community sync meeting today. Hence, I am raising the discussion here