Hello Polaris Community, I would like to share my proposal for a new service, the Polaris Delegation Service, and to share the design document for discussion and feedback. The Delegation Service is intended to optionally be deployed alongside Polaris to handle the execution of certain long-running tasks.
1. Motivation The Polaris Catalog is optimized for low-latency metadata operations. However, certain tasks such as purging data files for dropped tables are resource-intensive and can impact its core performance. The motivation for this new service is to decouple these I/O-heavy background tasks from the main catalog, ensuring it remains highly responsive while allowing the task execution workload to be managed and scaled independently. 2. Proposal We propose an optional, independent Delegation Service responsible for executing these offloaded operations. The MVP will focus on synchronously handling the data file deletion process for DROP TABLE WITH PURGE commands. 3. Relationship to the "Asynchronous & Reliable Tasks" Proposal This proposal is designed to be highly synergistic with the existing "Asynchronous & Reliable Tasks" proposal. The Asynchronous Task proposal describes a general internal framework for reliably scheduling and managing the lifecycle of any task within Polaris. On the other hand, this proposal defines a specific, external worker service optimized for executing a particular class of I/O-heavy tasks. The Delegation Service does not alter the core Polaris task schema. This allows it to seamlessly act as a specialized "backend" worker that can execute tasks scheduled and managed by the more advanced Asynchronous Task Framework, which would serve as the reliable "frontend." This relationship is explored further in section 10.2 of the document. Please find the detailed design document here for review: - https://docs.google.com/document/d/1AhR-cZ6WW6M-z8v53txOfcWvkDXvS-0xcMe3zjLMLj8/edit?usp=sharing Best Regards, William
