chovy-3012 opened a new issue, #12723: URL: https://github.com/apache/seatunnel/issues/12723
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description The existing `connector-bigquery` module only supports writing data **to** BigQuery (sink). There is no way to read data **from** BigQuery. This feature adds a BigQuery Source so users can ingest data out of BigQuery into other systems. The source should support: - **Project:** `project_id` — GCP project ID (reuses the sink's existing option). - **Dataset:** `dataset_id` — BigQuery dataset ID (reuses the sink's existing option). Serves as `defaultDataset` so unqualified table names in `query` resolve automatically. When paired with `table_id`, enables direct table reads via `listTableData()` without starting a query job. - **Table:** `table_id` — BigQuery table ID (reuses the sink's existing option). Paired with `dataset_id` as an alternative to `query` for full-table reads via `listTableData()`. - **Auth:** `service_account_key_path` / `service_account_key_json` — reuses the sink's existing auth options. Service Account only. - **Read query:** `query` — SQL query to execute via `BigQuery.query(QueryJobConfiguration)`. Either `query` or the `dataset_id` + `table_id` pair is required. - **Pagination:** `max_results` — BigQuery API page size (default 100). The SDK's `TableResult` implements `Page<FieldValueList>` and handles pagination internally via `getNextPage()` / `getNextPageToken()`. - **Fetch size:** `fetch_size` — SeaTunnel internal batch size (not a BigQuery API parameter; controls SeaTunnel's internal buffer, follows the JDBC source convention). - **Location:** `location` — BigQuery job location, set on `JobId.setLocation()` (required for non-US/EU jobs). - **Query cache:** `use_query_cache` — whether to use BigQuery query result caching (default true), mapped to `QueryJobConfiguration.useQueryCache()`. - **Dry run:** `dry_run` — when true, executes a dry run that returns estimated bytes scanned and cost without actually running the query, mapped to `QueryJobConfiguration.dryRun()`. - **Maximum bytes billed:** `maximum_bytes_billed` — cost cap; the query fails if estimated bytes exceed this limit, preventing accidental full-table scans, mapped to `QueryJobConfiguration.getMaximumBytesBilled()`. - **Emulator:** `emulator_host` — reuses the sink's existing option for testing without real GCP credentials. - **Universe domain:** `universe_domain` — reuses the sink's existing option for sovereign cloud support. - **Type conversion:** reuses the existing `BigQueryTypeConverter` for BigQuery-to-SeaTunnel type mapping. - **Dependencies:** no new third-party dependencies — the `google-cloud-bigquery` SDK and shading config already exist in the module. Deliverables: `BigQuerySource`, `BigQuerySourceFactory`, source reader; `plugin-mapping.properties` entry (`seatunnel.source.BigQuery = connector-bigquery`); EN/ZH documentation; unit tests; E2E test with a BigQuery emulator. ### Usage Scenario Data engineers use BigQuery for analytics and reporting. They need to export query results from BigQuery into data lakes or warehouses (Iceberg, ClickHouse, MySQL) for downstream processing, without writing custom scripts or relying on manual CSV exports. ### Related issues #10753 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
