Hi Robin! IIUC, the difference is:
- Pipeline connectors can be used as a fully contained yaml configured CDC pipeline job <https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/core-concept/data-pipeline/> - Flink CDC sources are Flink Table connectors that can connect directly to source database tables and binlogs. They allow you to use Flink SQL / Table API to query external source databases. They are used internally by pipelines. E.g. The mysql-cdc connector is used by a source type: mysql pipeline connector. > is the point that Flink CDC provides CDC connectors, and they are documented here <https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/connectors/flink-sources/overview/> when they could as logically be documented here <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/> under the main Flink docs? Flink CDC connectors are Flink Table connectors, but specifically for doing CDC. Compare that to e.g. the Flink JDBC table connector <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/>, which allows you to query a MySQL table with Flink, but won't read changes in a streaming fashion. (IIUC, that is why the JDBC docs have a "Scan Source: Bounded" heading) I'm not an expert though, so please someone correct me if I am wrong! On Mon, Dec 2, 2024 at 12:52 PM Robin Moffatt via user < user@flink.apache.org> wrote: > I'm struggling to grok the difference between pipeline connectors > <https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/connectors/pipeline-connectors/overview/> > and Flink sources > <https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/connectors/flink-sources/overview/> > in > Flink CDC. > > I understand pipeline connectors, and have been through the quickstart and > they make sense. > > But how are Flink sources any different from what I'd build in Flink SQL > itself directly? How do they fit into Flink CDC? Or is the point that Flink > CDC provides CDC connectors, and they are documented here > <https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/connectors/flink-sources/overview/> > when they could as logically be documented here > <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/> > under > the main Flink docs? > > Thanks in advance, > Robin >
