Copilot commented on code in PR #142: URL: https://github.com/apache/gravitino-site/pull/142#discussion_r4140774477
########## versioned_docs/version-1.3.1/open-api/credentials.yaml: ########## @@ -0,0 +1,119 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +--- + +paths: + + /metalakes/{metalake}/objects/{metadataObjectType}/{metadataObjectFullName}/credentials: + parameters: + - $ref: "./openapi.yaml#/components/parameters/metalake" + - $ref: "./openapi.yaml#/components/parameters/metadataObjectType" + - $ref: "./openapi.yaml#/components/parameters/metadataObjectFullName" + get: + tags: + - credentials + summary: Get credentials + operationId: getCredentials + responses: + "200": + description: Returns the list of credential objects associated with specified metadata object. + content: + application/vnd.gravitino.v1+json: + schema: + $ref: "#/components/responses/CredentialResponse" Review Comment: A media type's `schema` must reference a Schema Object, but this points into `components.responses`; that entry is itself schema-shaped and lacks the required Response Object fields. Move `CredentialResponse` under `components.schemas` and update this reference, otherwise the published OpenAPI document is invalid. ########## versioned_docs/version-1.3.1/trino-connector/udf-support.md: ########## @@ -0,0 +1,79 @@ +--- +title: "Trino Connector UDF Support" +slug: "/trino-connector/udf-support" +keyword: "gravitino connector trino udf function" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +The Gravitino Trino connector supports user-defined functions (UDFs) registered in Apache Gravitino. +Functions with `RuntimeType.TRINO` and SQL language implementations are automatically exposed as +[Trino language functions](https://trino.io/docs/current/routines/function.html), making them available for use in Trino queries. + +## Mechanism + +When Gravitino catalogs contain registered functions, the Trino connector: + +1. Lists functions from the Gravitino server for each schema. +2. Filters to include only functions with `RuntimeType.TRINO` and `Language.SQL`. +3. Maps each function implementation to a Trino `LanguageFunction` with a signature token derived from the function name and parameter types. + +Only functions with language `SQL` and runtime `TRINO` are visible to and callable from Trino. Functions registered for other languages or runtimes (for example a Python or Java implementation with runtime `SPARK`) are managed in Gravitino but are **not** exposed through this connector: they do not appear in `SHOW FUNCTIONS`, and invoking one fails with a Trino `Function "<catalog>.<schema>.<name>" not registered` error. The function still exists in Gravitino; the connector simply filters it out. The Gravitino UI function detail view shows, per implementation, whether it is exposed through the Trino connector. + +### SQL body format + +The `sql` field of a `SQL`/`TRINO` implementation is the function body. The connector assembles a complete [Trino SQL routine](https://trino.io/docs/current/routines/function.html) specification (`FUNCTION <name>(<params>) RETURNS <type> [NOT] DETERMINISTIC SECURITY INVOKER ...`) from the function name, parameters, return type and deterministic flag before handing it to Trino. The body may be: + +- A bare expression, e.g. `x + 1`. The connector wraps it as `RETURN x + 1`. +- A control statement, e.g. `RETURN x + 1` or `BEGIN ... END`. + +The form is decided by the first token of the body, ignoring leading SQL comments. Since `return`, `begin` and `function` are also valid identifiers, a parameter with one of these names shadows the keyword: the body is then always treated as an expression, e.g. `return + 1` for a parameter named `return`. A body that is itself a complete `FUNCTION ...` specification is not supported and the function is skipped with a warning. + +Function, parameter and row field names are quoted in the generated specification. Trino resolves routine and parameter names case-insensitively regardless of quoting, so the body can reference parameters as plain identifiers. + +## Prerequisites + +- The Gravitino catalog must support function operations (i.e., implement `FunctionCatalog`). +- Functions must be registered in Gravitino via the Gravitino client or REST API before they can be queried from Trino. + +## Register a UDF + +Use the Gravitino Java client to register a function: + +```java +FunctionCatalog functionCatalog = catalog.asFunctionCatalog(); +functionCatalog.registerFunction( + NameIdentifier.of("my_schema", "add_one"), + "Adds one to input", + FunctionType.SCALAR, + true, + FunctionDefinitions.of( + FunctionDefinitions.of( + FunctionParams.of(FunctionParams.of("x", Types.IntegerType.get())), + Types.IntegerType.get(), + FunctionImpls.of( + FunctionImpls.ofSql(FunctionImpl.RuntimeType.TRINO, "RETURN x + 1"))))); Review Comment: This Java example nests factory calls with incompatible argument types: `FunctionParams.of` is passed a `FunctionParam`, and the inner definition receives scalar values where the API uses arrays. The snippet will not compile; construct the parameter and implementation arrays as shown in the main function-management guide. ########## versioned_sidebars/version-1.3.1-sidebars.json: ########## @@ -0,0 +1,1664 @@ +{ + "docs": [ + { + "id": "index", + "label": "What's Apache Gravitino?", + "type": "doc" + }, + { + "id": "overview", + "label": "Overview", + "type": "doc" + }, + { + "label": "Getting started", + "type": "category", + "items": [ + { + "id": "getting-started/index", + "label": "Overview", + "type": "doc" + }, + { + "id": "getting-started/hive", + "label": "Installing Apache Hive", + "type": "doc" + }, + { + "id": "getting-started/aws-remote-access", + "label": "Remote Access on AWS", + "type": "doc" + }, + { + "id": "getting-started/playground", + "label": "Installing Apache Gravitino Playground", + "type": "doc" + } + ] + }, + { + "label": "Install", + "type": "category", + "items": [ + { + "id": "how-to-install", + "label": "How to install Gravitino", + "type": "doc" + }, + { + "id": "chart", + "label": "Install Gravitino on Kubernetes", + "type": "doc" + }, + { + "id": "iceberg-rest-catalog-chart", + "label": "Install Iceberg Rest catalog server on Kubernetes", + "type": "doc" + } + ] + }, + { + "id": "iceberg-rest-service", + "label": "Iceberg REST catalog service", + "type": "doc" + }, + { + "label": "Lance REST service", + "type": "category", + "link": { + "type": "doc", + "id": "lance-rest-service" + }, + "items": [ + { + "id": "lance-rest-integration", + "type": "doc", + "label": "Lance integration" + }, + { + "id": "lance-rest-server-chart", + "type": "doc", + "label": "Install Lance REST Server on Kubernetes" + } + ] + }, + { + "id": "gravitino-mcp-server", + "label": "Gravitino MCP server", + "type": "doc" + }, + { + "id": "webui", + "label": "Web UI", + "type": "doc" + }, + { + "id": "webui-v2", + "label": "Web UI v2", + "type": "doc" + }, + { + "label": "Catalogs", + "type": "category", + "items": [ + { + "label": "Relational", + "type": "category", + "items": [ + { + "id": "jdbc-clickhouse-catalog", + "label": "ClickHouse catalog", + "type": "doc" + }, + { + "id": "jdbc-doris-catalog", + "label": "Doris catalog", + "type": "doc" + }, + { + "label": "Generic Lakehouse Catalog", + "type": "category", + "link": { + "type": "doc", + "id": "lakehouse-generic-catalog" + }, + "items": [ + { + "id": "lakehouse-generic-lance-table", + "label": "Lance table support", + "type": "doc" + }, + { + "id": "lakehouse-generic-delta-table", + "label": "Delta table support", + "type": "doc" + } + ] + }, + { + "id": "aws-glue-catalog", + "label": "Glue catalog", + "type": "doc" + }, + { + "label": "Hive catalog", + "type": "category", + "link": { + "type": "doc", + "id": "apache-hive-catalog" + }, + "items": [ + { + "id": "hive-catalog-with-cloud-storage", + "label": "Hive catalog with cloud storage", + "type": "doc" + } + ] + }, + { + "id": "jdbc-hologres-catalog", + "label": "Hologres catalog", + "type": "doc" + }, + { + "id": "lakehouse-hudi-catalog", + "label": "Hudi catalog", + "type": "doc" + }, + { + "id": "lakehouse-iceberg-catalog", + "label": "Iceberg catalog", + "type": "doc" + }, + { + "id": "jdbc-mysql-catalog", + "label": "MySQL catalog", + "type": "doc" + }, + { + "id": "jdbc-oceanbase-catalog", + "label": "OceanBase catalog", + "type": "doc" + }, + { + "id": "lakehouse-paimon-catalog", + "label": "Paimon catalog", + "type": "doc" + }, + { + "id": "jdbc-postgresql-catalog", + "label": "PostgreSQL catalog", + "type": "doc" + }, + { + "id": "jdbc-starrocks-catalog", + "label": "StarRocks catalog", + "type": "doc" + } + ] + }, + { + "label": "Fileset", + "type": "category", + "items": [ + { + "id": "fileset-catalog-index", + "label": "Fileset catalog index", + "type": "doc" + }, + { + "id": "fileset-catalog", + "label": "Fileset catalog", + "type": "doc" + }, + { + "id": "fileset-catalog-with-s3", + "label": "Fileset catalog with s3", + "type": "doc" + }, + { + "id": "fileset-catalog-with-gcs", + "label": "Fileset catalog with gcs", + "type": "doc" + }, + { + "id": "fileset-catalog-with-oss", + "label": "Fileset catalog with oss", + "type": "doc" + }, + { + "id": "fileset-catalog-with-adls", + "label": "Fileset catalog with adls", + "type": "doc" + } + ] + }, + { + "label": "Messaging", + "type": "category", + "items": [ + { + "id": "kafka-catalog", + "label": "Kafka catalog", + "type": "doc" + } + ] + }, + { + "label": "Model", + "type": "category", + "items": [ + { + "id": "model-catalog", + "label": "Model catalog", + "type": "doc" + } + ] + } + ] + }, + { + "label": "Table Maintenance Service", + "type": "category", + "items": [ + { + "id": "table-maintenance-service/optimizer", + "label": "Optimizer", + "type": "doc" + }, + { + "id": "table-maintenance-service/optimizer-cli-reference", + "label": "Optimizer CLI reference", + "type": "doc" + }, + { + "id": "table-maintenance-service/optimizer-configuration", + "label": "Optimizer configuration", + "type": "doc" + }, + { + "id": "table-maintenance-service/optimizer-extension-guide", + "label": "Optimizer extension guide", + "type": "doc" + }, + { + "id": "table-maintenance-service/optimizer-quick-start", + "label": "Optimizer quick start", + "type": "doc" + }, + { + "id": "table-maintenance-service/optimizer-troubleshooting", + "label": "Optimizer troubleshooting", + "type": "doc" + }, + { + "id": "iceberg-compaction-policy", + "label": "Iceberg compaction policy", + "type": "doc" + } + ] + }, + { + "label": "Engine Connectivity", + "type": "category", + "items": [ + { + "label": "Gravitino Engine Connectors", + "type": "category", + "items": [ + { + "label": "Trino", + "type": "category", + "items": [ + { + "id": "trino-connector/trino-connector", + "label": "Overview", + "type": "doc" + }, + { + "id": "trino-connector/requirements", + "label": "Requirements", + "type": "doc" + }, + { + "id": "trino-connector/installation", + "label": "Installation", + "type": "doc" + }, + { + "id": "trino-connector/configuration", + "label": "Configuration", + "type": "doc" + }, + { + "id": "trino-connector/authentication", + "label": "Authentication", + "type": "doc" + }, + { + "label": "Supported catalogs", + "type": "category", + "items": [ + { + "id": "trino-connector/supported-catalog", + "label": "Introduction", + "type": "doc" + }, + { + "id": "trino-connector/catalog-glue", + "label": "Glue", + "type": "doc" + }, + { + "id": "trino-connector/catalog-hive", + "label": "Hive", + "type": "doc" + }, + { + "id": "trino-connector/catalog-iceberg", + "label": "Iceberg", + "type": "doc" + }, + { + "id": "trino-connector/catalog-mysql", + "label": "MySQL", + "type": "doc" + }, + { + "id": "trino-connector/catalog-postgresql", + "label": "PostgreSQL", + "type": "doc" + } + ] + }, + { + "id": "trino-connector/sql-support", + "label": "Supported SQL", + "type": "doc" + }, + { + "id": "trino-connector/udf-support", + "label": "UDF Support", + "type": "doc" + }, + { + "id": "trino-connector/trino-cascading-query", + "label": "Trino cascading query", + "type": "doc" + }, + { + "id": "trino-connector/development", + "label": "Development", + "type": "doc" + } + ] + }, + { + "label": "Spark", + "type": "category", + "items": [ + { + "id": "spark-connector/spark-connector", + "label": "Overview", + "type": "doc" + }, + { + "id": "spark-connector/spark-catalog-glue", + "label": "Glue", + "type": "doc" + }, + { + "id": "spark-connector/spark-catalog-hive", + "label": "Hive", + "type": "doc" + }, + { + "id": "spark-connector/spark-catalog-iceberg", + "label": "Iceberg", + "type": "doc" + }, + { + "id": "spark-connector/spark-catalog-jdbc", + "label": "JDBC", + "type": "doc" + }, + { + "id": "spark-connector/spark-catalog-paimon", + "label": "Paimon", + "type": "doc" + }, + { + "id": "spark-connector/spark-connector-udf", + "label": "User-defined functions", + "type": "doc" + }, + { + "id": "spark-connector/spark-authentication-with-gravitino", + "label": "Authentication", + "type": "doc" + } + ] + }, + { + "label": "Flink", + "type": "category", + "items": [ + { + "id": "flink-connector/flink-connector", + "label": "Overview", + "type": "doc" + }, + { + "id": "flink-connector/flink-catalog-hive", + "label": "Hive", + "type": "doc" + }, + { + "id": "flink-connector/flink-catalog-iceberg", + "label": "Iceberg", + "type": "doc" + }, + { + "id": "flink-connector/flink-catalog-paimon", + "label": "Paimon", + "type": "doc" + }, + { + "id": "flink-connector/flink-catalog-jdbc", + "label": "JDBC", + "type": "doc" + }, + { + "id": "flink-connector/flink-authentication-with-gravitino", + "label": "Authentication", + "type": "doc" + } + ] + }, + { + "id": "daft-connector/daft-connector", + "label": "Daft", + "type": "doc" + } + ] + }, + { + "label": "Iceberg REST", + "type": "category", + "items": [ + { + "id": "iceberg-rest-engine/trino", + "label": "Trino", + "type": "doc" + }, + { + "id": "iceberg-rest-engine/spark", + "label": "Spark", + "type": "doc" + }, + { + "id": "iceberg-rest-engine/flink", + "label": "Flink", + "type": "doc" + }, + { + "id": "iceberg-rest-engine/doris", + "label": "Doris", + "type": "doc" + }, + { + "id": "iceberg-rest-engine/starrocks", + "label": "StarRocks", + "type": "doc" + }, + { + "id": "iceberg-rest-engine/pyiceberg", + "label": "PyIceberg", + "type": "doc" + } + ] + } + ] + }, + { + "label": "Server administration", + "type": "category", + "items": [ + { + "id": "gravitino-server-config", + "label": "Configurations", + "type": "doc" + }, + { + "id": "metrics", + "label": "Metrics", + "type": "doc" + }, + { + "id": "how-to-use-relational-backend-storage", + "label": "How to use relational backend storage", + "type": "doc" + } + ] + }, + { + "label": "Lineage", + "type": "category", + "items": [ + { + "id": "lineage/lineage", + "label": "Overview", + "type": "doc" + }, + { + "id": "lineage/gravitino-spark-lineage", + "label": "Gravitino Spark Lineage support", + "type": "doc" + }, + { + "id": "lineage/gravitino-server-lineage", + "label": "Gravitino Server Lineage support", + "type": "doc" + } + ] + }, + { + "label": "Security", + "type": "category", + "items": [ + { + "id": "security/security", Review Comment: This sidebar still points to removed pages (`security/security`, `security/how-to-use-built-in-idp`, and `table-maintenance-service/optimizer-quick-start`). The 1.3.1 snapshot contains none of those IDs, so Docusaurus cannot resolve the versioned sidebar. Update the current sidebar for the renamed/removed pages and regenerate this file. ########## docs/filesets.md: ########## @@ -0,0 +1,107 @@ +--- +title: "Filesets" +slug: "/filesets" +keyword: "fileset, files, storage location, GVFS, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +A fileset is a named pointer to files. Where a table describes rows and columns, a fileset describes +a location, so unstructured and semi-structured data gets the same treatment as everything else in +the catalog: a name, a place in the hierarchy, tags, policies, and an owner. + +The point is indirection. Code that reads a fileset by name does not carry a bucket path, so moving +data between clusters or storage systems changes the fileset rather than every job that reads it. + +That indirection is delivered by the Gravitino Virtual File System, or GVFS. GVFS is a filesystem +implementation that resolves a fileset name to its storage location and then reads and writes +through to the underlying system, whether that is HDFS, S3, GCS, ADLS, or OSS. Paths take the form +`gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}`, so a Spark job, a pandas script, or a +Hadoop shell command reads a catalog name rather than a bucket URL. + +GVFS comes in two implementations. The Java one implements the Hadoop Compatible File System +interface, so anything that already speaks HDFS paths works, and it requires Hadoop 3.3.1 or later. +The Python one is built on fsspec, so it works with pandas, PyArrow, and anything else in that +ecosystem. There is also a FUSE implementation for mounting a fileset as a local directory. + +Credentials are the other half of it. A caller reaching storage through GVFS can be given +short-lived credentials vended by Gravitino rather than holding long-lived cloud keys of its own, +which is what makes the indirection a governance boundary rather than only a convenience. See +[Credential vending](./security/credential-vending.md). + +Filesets live in a schema inside a fileset catalog. Gravitino manages that catalog itself rather +than federating an external one, so no provider is needed when creating it. + +## Quick Start + +**1. Create a fileset catalog and schema.** See +[Catalogs and Schemas](./catalogs-and-schemas.md). A fileset catalog usually carries a base +`location`, and schemas can narrow it further. + +**2. Create the fileset.** Give it a name, a type, and a storage location. Creating a managed +fileset creates the directory; pointing at an existing path makes it external. + +**3. Read it by name.** Engines reach the files through GVFS or the client rather than the raw path. +See [How to use GVFS](./how-to-use-gvfs.md). + +## The Fileset Model + +### Managed and External + +A managed fileset belongs to Gravitino. Creating it creates the directory, and deleting it deletes +the data. + +An external fileset points at a location that already exists and stays under someone else's control. +Deleting it removes the Gravitino record and leaves the files alone. + +The distinction only matters at deletion time, and it is the single most important thing to get +right when creating one. + +### Storage Locations + +A fileset has at least one storage location and can have several, each with a name. The default is +chosen by the `default-location-name` property, and a location supplied without a name is recorded +as `unknown`. + +Several locations on one fileset is how the same logical dataset is described across clusters or +regions, with readers selecting the one they should use rather than each carrying its own path. + +Locations can also be inherited: a catalog or schema with a `location` property supplies a base, and +a fileset created beneath it can take its placement from that rather than spelling out a full path. + +A location can be a template rather than a fixed path. Placeholders such as `{{catalog}}`, +`{{schema}}`, `{{user}}`, and `{{project}}` are filled in when the fileset is created, with values +supplied by `placeholder-` properties. One catalog-level template then generates a consistent +directory layout for every fileset created beneath it, rather than each one being spelled out by +hand. + +### Properties + +Properties are free-form, with `default-location-name` and the `location-{name}` keys reserved for +placement. Everything else is yours, and travels with the fileset. + +## Working With Filesets in the UI + +Opening a fileset catalog lists its schemas and the filesets inside them. A fileset shows its type, +its storage locations, and its properties. + +Filesets display the tags they carry, including inherited ones, but cannot be tagged from the UI +today. Attaching a tag to a fileset goes through the API. + +## Permissions + +| Privilege | Grantable on | What it allows | +|------------------|---------------------------------------|-----------------------------| +| `CREATE_FILESET` | Metalake, catalog, schema, or fileset | Creating filesets | Review Comment: The detailed access-control matrix limits `CREATE_FILESET` to Metalake, Catalog, and Schema (`docs/security/access-control.md:182`); Fileset is not a valid grant target for creating a new object. Correct this row and the versioned copy. ########## docs/open-api/datatype.yaml: ########## @@ -201,20 +202,20 @@ components: ] } - ExternalType: - type: object - required: - - type - - catalogString - properties: - type: - type: string - enum: - - "external" - catalogString: - type: string - description: The string representation of this type in the catalog - example: { - "type": "external", - "externalType": "user-defined" - } \ No newline at end of file + ExternalType: + type: object + required: + - type + - catalogString + properties: + type: + type: string + enum: + - "external" + catalogString: + type: string + description: The string representation of this type in the catalog + example: { + "type": "external", + "externalType": "user-defined" Review Comment: The example does not satisfy its own schema: `catalogString` is required, but the example supplies an undefined `externalType` property. This produces misleading generated API documentation and can fail strict example validation. ########## docs/open-api/policies.yaml: ########## @@ -249,7 +249,7 @@ paths: tags: - policy summary: Associate policies with metadata object - description: Associate and disassociate policies with metadata object, please be aware that supported metadata objects are CATALOG, SCHEMA, TABLE, FILESET, TOPIC, COLUMN + description: Associate and disassociate policies with metadata object, please be aware that supported metadata objects are CATALOG, SCHEMA, TABLE, FILESET, TOPIC, MODEL, VIEW, FUNCTION Review Comment: The operation description omits `COLUMN`, although the request schema still accepts `COLUMN` at line 502. Add it so the documented supported object types match the API schema, including in the versioned specification. ########## docs/table-maintenance-service/optimizer-extension-guide.md: ########## @@ -124,7 +125,7 @@ If you also extend Gravitino server job execution, see [Manage jobs in Gravitino ## Related -- [Table Maintenance Service (Optimizer)](./optimizer.md) -- [Optimizer Configuration](./optimizer-configuration.md) -- [Optimizer CLI Reference](./optimizer-cli-reference.md) -- [Optimizer Troubleshooting](./optimizer-troubleshooting.md) +- [Table Maintenance Service](./optimizer.md) +- [Configuration](./optimizer-configuration.md) +- [CLI Reference](./optimizer-configuration.md) Review Comment: The CLI Reference link targets the configuration page instead of `optimizer-cli-reference.md`. The same bad target was copied into the 1.3.1 snapshot, so both pages need correction. ########## docs/topics.md: ########## @@ -0,0 +1,90 @@ +--- +title: "Topics" +slug: "/topics" +keyword: "topic, messaging, Kafka, streaming metadata, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +A topic is a stream of messages, and in Gravitino it is a metadata object like any other. Registering +a Kafka cluster as a messaging catalog puts its topics in the same hierarchy as tables and filesets, +so a stream can be found, classified, and governed alongside the data it feeds. + +Gravitino holds the reference rather than the messages. Listing topics asks the cluster at request +time, so a topic created directly in Kafka appears the next time Gravitino is asked, and deleting a +topic through Gravitino deletes it in the cluster. + +The value is coverage rather than new capability. Without it, streams sit outside the catalog and +outside whatever classification and access rules apply to everything else. + +## Quick Start + +**1. Connect a messaging catalog.** A messaging catalog takes the `kafka` provider and the cluster's +bootstrap servers. See [Catalogs and Schemas](./catalogs-and-schemas.md) and +[Apache Kafka catalog](./kafka-catalog.md). + +**2. Browse or create topics.** A connected cluster surfaces the topics already there. Creating one +through Gravitino creates it in the cluster. + +**3. Classify what matters.** Topics carry tags and policies the same way tables do. + +## The Topic Model + +### Messaging Systems + +Kafka is the only messaging system with a catalog today, connected with the `kafka` provider and the +cluster's `bootstrap.servers`. Anything speaking the Kafka protocol works through the same +connector. Other messaging systems have no catalog. + +### The Default Schema + +A messaging catalog presents one schema, named `default`, holding every topic in the cluster. Kafka +has no namespace of its own to map onto, so creating or dropping a schema in a messaging catalog is +rejected rather than silently ignored. + +The consequence for a large cluster is that topics are not grouped in the catalog the way tables are +grouped by database. Tags are the way to organize them. + +### Names and Properties + +A topic name is unique within its schema, and matches the topic name in the cluster. + +Two properties are settable at creation. `partition-count` sets the number of partitions and can be +changed afterward. `replication-factor` sets the replication and is immutable once the topic exists. +Leaving either unset takes the broker's own default, from `num.partition` and +`default.replication.factor` respectively. Review Comment: Kafka's broker default is named `num.partitions` (plural), not `num.partition`. As written, users looking up or configuring the referenced broker setting will use a nonexistent property; correct this in all current and versioned occurrences. ########## docs/functions.md: ########## @@ -0,0 +1,93 @@ +--- +title: "Functions" +slug: "/functions" +keyword: "function, UDF, user-defined function, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +A function is a piece of logic registered in the catalog so engines can find it and people can +govern it. Registering one puts it in the same hierarchy as tables, with a name, an owner, tags, and +permissions. + +Gravitino stores the definition rather than executing it. Whether a function runs, and how, is a +question for the engine that reads it. + +## Quick Start + +**1. Open a catalog.** Functions live in a schema, alongside the tables they usually operate on. See +[Catalogs and Schemas](./catalogs-and-schemas.md). + +**2. Register the function.** Registration is done through the API rather than the UI. A function +needs a name, its type, and at least one definition: a parameter list, a return type, and an +implementation in SQL, Python, or Java. + +**3. Use it from an engine.** An engine reading through Gravitino resolves the function by name. + +## The Function Model + +### Function Types + +| Type | Returns | +|-------------|----------------------------------------------------| +| `SCALAR` | One value per row | +| `AGGREGATE` | One value per group | +| `TABLE` | A set of rows, for table-valued operations | + +The type is declared when the function is registered and tells an engine how to call it. + +### Determinism + +A function is marked deterministic or not. A deterministic function returns the same result for the +same arguments every time, which lets an engine cache, reorder, or eliminate calls to it. Marking a +function deterministic when it is not, such as one that reads the clock or a random source, invites +an engine to optimize in ways that change the answer. + +Gravitino records the flag and does not verify it. + +### Definitions and Implementations + +A function has two levels beneath it. + +A definition is an overload: one parameter list and one return type. A function with several +definitions is the same name accepting different arguments, the way `round(x)` and `round(x, digits)` +are one function in most engines. + +An implementation is how a definition is expressed for a given engine. Each declares a language and +a runtime, so one definition can carry a SQL version for Trino and a Python version for Spark, and +each engine uses the one it can execute. Review Comment: This says Spark uses a Python implementation, but the new Spark UDF guide explicitly states that only Java implementations with `RuntimeType.SPARK` are supported and Python cannot be invoked. Change this example to a Java implementation in both the current and versioned pages. This issue also appears in the following locations of the same file: - line 82 - line 86 ########## versioned_docs/version-1.3.1/lance-rest-server-chart.md: ########## @@ -0,0 +1,127 @@ +--- +title: "Install Lance REST Server on Kubernetes" +slug: "/lance-rest-server-chart" +keyword: + - Lance REST Server Helm Chart +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +This Helm chart deploys Apache Gravitino Lance REST Server on Kubernetes with customizable configurations. + +## Prerequisites + +- Kubernetes 1.29+ +- Helm 3+ + +## Installation + +### Install from OCI Registry (Recommended for Released Versions) + +Pull the chart from Docker Hub OCI registry: + +```console +helm pull oci://registry-1.docker.io/apache/gravitino-lance-rest-server-helm --version <VERSION> +``` + +Or install directly: + +```console +helm upgrade --install gravitino-lance oci://registry-1.docker.io/apache/gravitino-lance-rest-server-helm --version <VERSION> -n gravitino --create-namespace +``` + +### Install from Local Repository (for Development or Unreleased Versions) + +Clone the repository and navigate to the chart directory: + +```console +git clone https://github.com/apache/gravitino.git +cd gravitino/dev/charts +``` + +Update chart dependencies: + +```console +helm dependency update gravitino-lance-rest-server +``` + +Install the chart: + +```console +helm upgrade --install gravitino-lance ./gravitino-lance-rest-server -n gravitino --create-namespace +``` + +## View Chart Values + +Customize values.yaml parameters to override chart default settings. Additionally, Gravitino Lance REST Server configurations in [gravitino-lance-rest-server.conf](../dev/charts/gravitino-lance-rest-server/resources/gravitino-lance-rest-server.conf) can be modified through Helm values.yaml. Review Comment: This relative target points outside the documentation tree to a `dev/charts` path that does not exist in this site repository, so the published link is broken. Link to the corresponding file in the Apache Gravitino repository instead. ########## docs/topics.md: ########## @@ -0,0 +1,90 @@ +--- +title: "Topics" +slug: "/topics" +keyword: "topic, messaging, Kafka, streaming metadata, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +A topic is a stream of messages, and in Gravitino it is a metadata object like any other. Registering +a Kafka cluster as a messaging catalog puts its topics in the same hierarchy as tables and filesets, +so a stream can be found, classified, and governed alongside the data it feeds. + +Gravitino holds the reference rather than the messages. Listing topics asks the cluster at request +time, so a topic created directly in Kafka appears the next time Gravitino is asked, and deleting a +topic through Gravitino deletes it in the cluster. + +The value is coverage rather than new capability. Without it, streams sit outside the catalog and +outside whatever classification and access rules apply to everything else. + +## Quick Start + +**1. Connect a messaging catalog.** A messaging catalog takes the `kafka` provider and the cluster's +bootstrap servers. See [Catalogs and Schemas](./catalogs-and-schemas.md) and +[Apache Kafka catalog](./kafka-catalog.md). + +**2. Browse or create topics.** A connected cluster surfaces the topics already there. Creating one +through Gravitino creates it in the cluster. + +**3. Classify what matters.** Topics carry tags and policies the same way tables do. + +## The Topic Model + +### Messaging Systems + +Kafka is the only messaging system with a catalog today, connected with the `kafka` provider and the +cluster's `bootstrap.servers`. Anything speaking the Kafka protocol works through the same +connector. Other messaging systems have no catalog. + +### The Default Schema + +A messaging catalog presents one schema, named `default`, holding every topic in the cluster. Kafka +has no namespace of its own to map onto, so creating or dropping a schema in a messaging catalog is +rejected rather than silently ignored. + +The consequence for a large cluster is that topics are not grouped in the catalog the way tables are +grouped by database. Tags are the way to organize them. + +### Names and Properties + +A topic name is unique within its schema, and matches the topic name in the cluster. + +Two properties are settable at creation. `partition-count` sets the number of partitions and can be +changed afterward. `replication-factor` sets the replication and is immutable once the topic exists. +Leaving either unset takes the broker's own default, from `num.partition` and +`default.replication.factor` respectively. + +### What Gravitino Stores and What It Does Not + +Gravitino stores the topic's place in the hierarchy and anything attached to it, including tags, +policies, and ownership. Message content, offsets, consumer groups, and lag stay entirely in the +cluster. + +Message schemas are also outside the catalog. Gravitino does not integrate with a schema registry, +so the structure of the messages in a topic is not described here and cannot be classified per field +the way table columns can. Tags and policies attach to the topic as a whole. + +## Working With Topics in the UI + +Opening a messaging catalog lists its topics, and selecting one shows its properties. + +Topics display the tags and policies they carry, including inherited ones, but cannot be tagged from +the UI today. Attaching a tag to a topic goes through the API. + +## Permissions + +| Privilege | Grantable on | What it allows | +|-----------------|-------------------------------------|-----------------------| +| `CREATE_TOPIC` | Metalake, catalog, schema, or topic | Creating topics | Review Comment: The detailed access-control matrix limits `CREATE_TOPIC` to Metalake, Catalog, and Schema (`docs/security/access-control.md:179`); Topic is not a valid grant target for a create privilege. Correct this row and the versioned copy. ########## versioned_docs/version-1.3.1/iceberg-rest-catalog-chart.md: ########## @@ -0,0 +1,108 @@ +--- +title: "Install Iceberg REST Catalog Server on Kubernetes" +slug: "/iceberg-rest-catalog-chart" +keyword: + - Iceberg REST Helm Chart +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +This Helm chart deploys Apache Gravitino Iceberg REST Catalog Server on Kubernetes with customizable configurations. + +## Prerequisites + +- Kubernetes 1.29+ +- Helm 3+ + +## Installation + +### Install from OCI Registry (Recommended for Released Versions) + +Pull the chart from Docker Hub OCI registry: + +```console +helm pull oci://registry-1.docker.io/apache/gravitino-iceberg-rest-server-helm --version <VERSION> +``` + +Or install directly: + +```console +helm upgrade --install gravitino-iceberg oci://registry-1.docker.io/apache/gravitino-iceberg-rest-server-helm --version <VERSION> -n gravitino --create-namespace +``` + +### Install from Local Repository (for Development or Unreleased Versions) + +Clone the repository and navigate to the chart directory: + +```console +git clone https://github.com/apache/gravitino.git +cd gravitino/dev/charts +``` + +Update chart dependencies: + +```console +helm dependency update gravitino-iceberg-rest-server +``` + +Install the chart: + +```console +helm upgrade --install gravitino-iceberg ./gravitino-iceberg-rest-server -n gravitino --create-namespace +``` + +## View Chart Values + +Customize values.yaml parameters to override chart default settings. Additionally, Gravitino Iceberg REST Catalog Server configurations in [gravitino-iceberg-rest-server.conf](../dev/charts/gravitino-iceberg-rest-server/resources/gravitino-iceberg-rest-server.conf) can be modified through Helm values.yaml. Review Comment: This relative target points outside the documentation tree to a `dev/charts` path that does not exist in this site repository, so the published link is broken. Link to the corresponding file in the Apache Gravitino repository instead. ########## versioned_docs/version-1.3.1/open-api/idp/openapi.yaml: ########## @@ -0,0 +1,73 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +--- +openapi: 3.0.3 +info: + title: Gravitino Local Users and Groups REST API + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0.html + version: 1.3.0-SNAPSHOT Review Comment: The released 1.3.1 IDP specification still identifies itself as `1.3.0-SNAPSHOT`, unlike the main specification and generated 1.3.1 API docs. Set this to the release version so clients and published metadata do not report the wrong API version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
