Thanks you Nicholas for driving this. Best, Giannis
On Fri, 29 May 2026 at 9:15 PM, Nicholas <[email protected]> wrote: > Summary of Apache Fluss Community Call 29th May: > > > > > Meeting recording: > > > > > https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo > > > > > Meeting Notes: > > > > > Community sync focused on incubation graduation adjustments, volunteer > co-hosting, deletion vectors and union read (Iceberg/Paimon) progress and > conference plans, FlusRust merge and roadmap, Spark optimizer, Append > Column proposal, and Kafka compatibility scope. > > > > > Issues & Tickets: > > > > > 1. Incubation Graduation Guidelines and PMC List Update > > > > > Context: > > - New ASF incubation graduation guidelines updated ~6 months ago require > active contributions from all PMC nominees. > > Status: > > - Mentors and community were unaware; received challenges from incubator > regarding PMC list compliance. > > Concerns, Challenges, and Blockers > > - Need to validate PMC nominees' activity; may delay graduation. > > - Coordination overhead to revise PMC list on private list. > > Ideas and Explanations > > - Discuss and refresh PMC roster based on recent contribution activity. > > - Proceed with other graduation requirements in parallel. > > Decisions and Next Steps > > - Action: Start private list discussion to refine PMC list and add > active members (Owner: Jark; Mentor Chair looped). > > - Continue graduation process post-list update. > > Current Objective: > > - Align PMC list with new guidelines to unblock graduation and 1.0 > timing. > > > > > 2. Community Call Co-host Roster > > > > > Context: > > - Co-hosting burden previously concentrated; Mehul drafted how-to guide. > > Status: > > - Multiple volunteers offered to rotate hosting. > > Concerns, Challenges, and Blockers > > - Single point of failure when primary host unavailable. > > Ideas and Explanations > > - Maintain a public roster; pick alternates as needed. > > Decisions and Next Steps > > - Action: Start a public mailing-list thread to collect volunteers and > schedule months (Owner: 蒋晓峰/Niklas; Mehul to initiate if needed). > > Current Objective: > > - Establish a rotating co-host roster for resilience. > > > > > 3. Deletion Vectors, Union Read: Iceberg/Paimon Progress and Demos > > > > > Context: > > - Design posted last month; focus on deletion vectors and union read. > POC efforts on Paimon; interest in Iceberg path. > > Status: > > - Paimon POC nearly ready; complexity higher than expected. > > - Plan to demo at Flink Forward Shenzhen. > > - Community sharing blogs/examples (Iceberg vs Paimon feeds). > > Concerns, Challenges, and Blockers > > - Complexity of Paimon compaction/erase semantics. > > - Time/ bandwidth to cover both Paimon and Iceberg before events. > > Ideas and Explanations > > - Target Iceberg union read alongside Flus to leverage market interest > (e.g., Redpanda feature parity) and openness vs paywalled offerings. > > - Explore DuckDB extension for analytics consumption beyond Flink SQL. > > Decisions and Next Steps > > - Action: Coordinate Yuxia + Mehul + Muhammet to push Flus+Iceberg union > read for Flus Forward; ensure demo readiness (Owner: Yuxia). > > - Action: Start Flink Dev discussion after refining design this month > (Owner: Yuxia). > > - George to share dual-ingest blog/resources in community chat. > > Current Objective: > > - Deliver working union read (Iceberg first) and Paimon POC/demo; refine > design and socialize on mailing lists. > > > > > 4. Iceberg Table Maintenance and Gaps > > > > > Context: > > - Giannis exploring improvements (orphan file deletion, maintenance > service). > > Status: > > - Open question on whether maintenance should live as a service vs > user-managed jobs. > > Concerns, Challenges, and Blockers > > - Unclear ownership model for maintenance orchestration. > > - Need prioritized list of small, impactful tasks. > > Ideas and Explanations > > - Compare approaches (Gravitino, Apache Amoro). > > - Start thread to enumerate needs; Giannis to tackle smaller items. > > Decisions and Next Steps > > - Action: Open design/prioritization thread for Iceberg maintenance gaps > (Owner: Giannis). > > Current Objective: > > - Define and prioritize Iceberg maintenance enhancements aligned with > Flus usage. > > > > > 5. Catalogs, Access Control, and Cross-Engine Constraints (Snowflake) > > > > > Context: > > - Discussion on catalog limitations (e.g., Snowflake external catalog > disallows deletes) impacting interoperability. > > Status: > > - Raised as architectural risk; not yet an active proposal. > > Concerns, Challenges, and Blockers > > - Catalog feature disparity blocks deletes/equality deletes. > > - Fragmented auth across Flus, storage, and table formats. > > Ideas and Explanations > > - Explore unified auth token and delegated credentials across layers. > > - Monitor catalog capabilities as central to analyst workflows. > > Decisions and Next Steps > > - Spin out mailing list/side meetings for deeper exploration (Owners: > interested parties). > > Current Objective: > > - Document constraints and potential integration patterns; avoid scope > creep in core agenda. > > > > > Fluss Rust Merge and Roadmap > > > > > Context: > > - Consensus to merge FlusRust into main Flus repo and align releases. > > Status: > > - PR ready; CI passing; unified protobufs. Progress on complex types, > limit/prefix scans, metrics, schema evolution for log tables. Python/C++ > lagging. > > Concerns, Challenges, and Blockers > > - Bindings (Python/C++) lack parity. > > - Coordination on gateway work vs core gaps. > > Ideas and Explanations > > - Limit 1.0 Rust scope to core roadmap items; bindings follow later. > > - Stream interface to ease DataFusion integration. > > Decisions and Next Steps > > - Action: Review and merge FlusRust PR (Owners: reviewers/core). > > - Action: Create issues for stream interface and DataFusion provider > alignment (Owner: Anton). > > - Action: Coordinate with Leonard/Hongshun/Junbo on Flus Gateway + Rust > client roadmap; public Slack channel created (Owner: Leonard; Anton to plan > issues). > > - Note: 1.0 release delayed pending graduation; FlusRust considered > milestone for 1.0. > > Current Objective: > > - Complete merge; close core Rust gaps; plan gateway integration path; > align release cadence with Flus 1.0. > > > > > Spark Union Read and Optimizer > > > > > Context: > > - 1.0 roadmap item covering union read and scan/pushdown optimizations. > > Status: > > - Union read implemented (except streaming); scan optimizations in > progress (limits, partition/column pruning; pushdown/join interfaces under > umbrella issues). > > Concerns, Challenges, and Blockers > > - Streaming union read lower priority; polish needed across features. > > Ideas and Explanations > > - Deliver a cohesive set: scan optimizations + union read + pushdown > interfaces for 1.0. > > Decisions and Next Steps > > - Ongoing: Yang/Jark/Anton to continue collaboration and reviews. > > Current Objective: > > - Ship Spark optimizer features and union read as key 1.0 deliverables. > > > > > Append Column Proposal (Local Enrichment) > > > > > Context: > > - Keith's FIP proposes append-only column files on immutable log tables > to support enrichment (e.g., embeddings) without duplicating data. > > Status: > > - Discussion thread open; early feedback likens approach to LANs append > columns and aligns with future feature engineering needs; Iceberg exploring > similar. > > Concerns, Challenges, and Blockers > > - Need concrete real-world use cases to validate scope and design. > > - Cross-feature implications (replication, durability, tiering). > > Ideas and Explanations > > - Use cases: inline vector embeddings on log tables; LANs tiering > without duplicating tables. > > - Collect scenarios in FIP comments; consider recurring focused meetings > if interest is high. > > Decisions and Next Steps > > - Action: Community to review doc and add concrete use cases; Keith to > facilitate follow-ups (Owner: All; Keith). > > - Allow time due to many concurrent proposals. > > Current Objective: > > - Refine design with use-case evidence and subsystem considerations; > build consensus for a robust implementation. > > > > > Kafka Compatibility vs Migration Tool > > > > > Context: > > - Two approaches debated via FIPs: full Kafka API compatibility vs > migration-focused tooling. > > Status: > > - Ben authored broad-scope designs; community leaning to narrow initial > scope; agreement to move to dev list for broader input. > > Concerns, Challenges, and Blockers > > - Large scope risks slow progress. > > - Messaging: Flus is not "another Kafka"; avoid confusion. > > Ideas and Explanations > > - Stage implementation: start with producer API to ingest via Kafka > clients/log proxies; expand to consumer/admin later. > > - Clarify terminology (compatibility vs migration) in proposals. > > Decisions and Next Steps > > - Action: Define bare-minimum initial scope (producer-focused) and > proceed with dev-list proposal (Owners: Yang with Ben; Giannis to encourage > scoping). > > - Action: Full design sketched for major APIs; phased delivery plan > documented (Owner: Jark/Ben). > > Current Objective: > > - Reach community agreement on initial scope and terminology; begin > incremental implementation toward broader compatibility. > > > > > Other & Incidental Topics: > > > > > 1. Volunteer Energy and Load Sharing > > > > > Community appreciated increased co-host volunteers; aim to avoid > single-person burden; jokes aside, roster will distribute responsibilities. > > > > > 2. Event Targets and Messaging > > > > > Plan demos/talks at Flink Forward Shenzhen, Flus Forward, Lakehouse Days, > and Community Over Code; Iceberg union read and DuckDB integration > highlighted as high-impact narratives. > > > > > 3. Public Channels for Coordination > > > > > Created public Slack channel "Flus Rust Gateway" to coordinate Rust client > and gateway work; open invite for contributors. > > Action Items > > - Start mailing-list thread re: next month's community call co-host - > WATCH (5 secs): > https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=350.9999 > > - Share Iceberg vs Paimon blog links in community chat - WATCH (5 secs): > https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=679.9999 > > - Start mailing-list thread re: Iceberg table maintenance > gaps/priorities - WATCH (5 secs): > https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=967.9999 > > - Start mailing-list thread re: deletion vectors/Iceberg union read; > then refine design doc - WATCH (5 secs): > https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=3127.9999 > > - Create GitHub issues for FlusRust stream interface + Gateway; then > coordinate w/ Leonard, Hongshun, Junbo - WATCH (5 secs): > https://fathom.video/share/gCdmW2HDsdQp6ypsSsZuW5KK_n8G6fCo?timestamp=3149.9999 > > > > > Best Regards, > > Nicholas Jiang
