GitHub user lipengming created a discussion: Integrating with lake table stores (e.g., Apache Paimon): a table type or index format instead of a second copy of graph data?
### Background GraphAr is positioned as "a standardized system-independent format for storing graph data" (README), with its own metadata and chunked data files. Meanwhile, Apache Paimon already supports third-party table types (`ICEBERG_TABLE`, `LANCE_TABLE` in `org.apache.paimon.TableType`) and has mechanisms for maintaining secondary index structures over its tables. ### The concern: a second physical copy of the data In a typical lakehouse, the source of truth lives once in the lake — e.g., a Paimon vertex table and an edge table. To use GraphAr today, the data has to be re-materialized into GraphAr chunks (fixed-size vertex chunks, adjacency layout with offset tables), because lake-managed Parquet files cannot be directly pointed to by GraphAr metadata. That means **two physical copies of the same graph data** (plus two metadata trees), which seems to conflict with the data-lake principle of "one copy of data, many views/materializations". ### Possible integration shapes Has the community considered making GraphAr work *with* lake table stores rather than alongside them? Two concrete shapes: 1. **A lake table type** — a `GRAPHAR_TABLE`-style table type (similar to Paimon's existing Iceberg/Lance table types): the graph is stored once in GraphAr layout, while the lake engine handles registration, schema evolution, compaction, and snapshot-based time travel for graph versions. 2. **A graph index format** — GraphAr's CSR-like adjacency layout (offset + adjacency chunks) is essentially a secondary index structure. It could be defined as a standard *index* format that engines like Paimon maintain over their primary tables, instead of GraphAr maintaining a parallel copy of both property data and adjacency. For context on demand: I've been prototyping graph indexes directly in Paimon — adjacency index tables, node-map reverse index, and a k-hop traversal SDK over Paimon tables (`paimon-graph` module on the `feature/graph-index` branch of my fork, [lipengming/paimon](https://github.com/lipengming/paimon)). The experiment confirms there is demand for graph retrieval on lake tables, but the index layout is Paimon-internal. A system-independent layout spec — which is very close to what GraphAr already specifies — would let this interoperate across engines instead of being Paimon-specific. ### Questions 1. Is there a principled reason (beyond incubation priorities) for staying fully independent of lake table formats — e.g., interchange-first positioning, or avoiding coupling with any single engine's snapshot/compaction semantics? 2. Would a "GraphAr as table type / lake-managed graph" path be in scope for the spec, or is the intended model always "an extra GraphAr copy as interchange/export format"? 3. If the second copy is accepted by design, is there guidance on keeping it consistent with the lake (incremental sync, CDC, version alignment)? Happy to work on a PoC if there's interest, and glad to move this discussion to the dev mailing list ([email protected]) if that's a better venue. --- ### 中文摘要 **问题背景**:GraphAr 定位为"标准化的、系统无关的图数据存储格式",有自己的元数据和分块数据文件。但在典型湖仓架构中,图数据的 source of truth 已经在湖里(如 Paimon 的点表/边表)。今天要用 GraphAr,必须把数据重新物化成 GraphAr 分块(固定大小的顶点分块、带 offset 的邻接布局),湖里管理的 Parquet 文件无法直接被 GraphAr 元数据引用——这意味着**同一份图数据会存两份物理副本**(外加两套元数据树),似乎与数据湖"一份数据、多种视图"的初衷相悖。 **两种可能的整合形态**: 1. **作为湖的表类型**:类似 Paimon 已有的 Iceberg/Lance 表类型,引入 `GRAPHAR_TABLE`——图数据只存一份(GraphAr 布局),注册、schema 演进、compaction、快照时间旅行等由湖引擎负责。 2. **作为图索引格式**:GraphAr 的类 CSR 邻接布局(offset + 邻接分块)本质上是一种二级索引结构,可以定义为标准化的索引格式,由 Paimon 等引擎在主表之上维护,而不是由 GraphAr 平行维护一份属性数据 + 邻接数据的副本。 **个人实践佐证需求**:我在自己的 fork(`lipengming/paimon` 的 `feature/graph-index` 分支)上做了 `paimon-graph` 原型——Paimon 表之上的邻接索引表、node-map 反查索引和 k-hop 遍历 SDK。实验证明基于湖表的图检索确有需求,但索引布局是 Paimon 私有的;一个系统无关的布局规范(与 GraphAr 已有规范非常接近)能让这类能力跨引擎互通。 **想请教的问题**: 1. 除孵化期优先级外,是否有原则性的理由让 GraphAr 完全独立于湖表格式(比如交换格式优先的定位,或避免与单一引擎的快照/compaction 语义耦合)? 2. "GraphAr 作为表类型 / 由湖管理的图" 这条路线是否在规范的目标范围内?还是设计意图始终是"额外一份 GraphAr 副本作为交换/导出格式"? 3. 如果第二份副本是设计上接受的,是否有关于与湖保持一致的指导(增量同步、CDC、版本对齐)? 如有兴趣我乐意做 PoC,也欢迎转到 dev 邮件列表讨论。 GitHub link: https://github.com/apache/incubator-graphar/discussions/988 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
