I don't know what your dependency issue is (post it here if you want help!), but I generally recommend using mvn dependency:tree to debug version clashes (and then pin or exclude versions)
On Tue, Feb 2, 2021 at 9:23 PM Sebastián Magrí <sebasma...@gmail.com> wrote: > The root of the previous error seemed to be the flink version the > connector was compiled for. I've tried compiling my own postgresql-cdc > connector, but still have some issues with dependencies. > > On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <sebasma...@gmail.com> > wrote: > >> Applied that parameter and that seems to get me some progress here. >> >> I still get the shade overlapping classes warning, but I get the >> PostgreSQLTableFactory in the merged table.factories.Factory service file. >> >> However, now on runtime the application fails to find the debezium source >> function class coming down to this error: >> >> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: >> Cannot load user class: >> com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction >> ClassLoader info: URL ClassLoader: >> Class not resolvable through given classloader. >> >> The class is indeed in jar, though. >> >> Any thougths? >> >> On Thu, 28 Jan 2021 at 09:57, Jark Wu <imj...@gmail.com> wrote: >> >>> Hi Sebastián, >>> >>> Could you try to add combine.children="append" attribute to the >>> transformers configuration? >>> You can also see the full shade plugin configuration here [1]. >>> >>> Best, >>> Jark >>> >>> [1]: >>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources >>> >>> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <sebasma...@gmail.com> >>> wrote: >>> >>>> Hi Jark! >>>> >>>> Please find the full pom file attached. >>>> >>>> Best Regards, >>>> >>>> On Thu, 28 Jan 2021 at 03:21, Jark Wu <imj...@gmail.com> wrote: >>>> >>>>> Hi Sebastián, >>>>> >>>>> I think Dawid is right. >>>>> >>>>> Could you share the pom file? I also tried to >>>>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and >>>>> the Factory file contains >>>>> >>>>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory >>>>> >>>>> >>>>> Best, >>>>> Jark >>>>> >>>>> >>>>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <sebasma...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks a lot for looking into it Dawid, >>>>>> >>>>>> In the >>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory >>>>>> file I only see >>>>>> >>>>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory >>>>>> >>>>>> Even after applying the ServicesResourceTransformer. >>>>>> >>>>>> >>>>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz < >>>>>> dwysakow...@apache.org> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Unfortunately I am not familiar with the packaging of >>>>>>> flink-connector-postgres-cdc. Maybe @Jark could help here? >>>>>>> >>>>>>> However, I think the problem that you cannot find the connector is >>>>>>> caused because of lack of entry in the resulting Manifest file. If there >>>>>>> are overlapping classes maven does not exclude whole dependencies, but >>>>>>> rather picks the overlapping class from one of the two. Could you check >>>>>>> if >>>>>>> you see entries for all tables in >>>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory. >>>>>>> >>>>>>> If not, you could try applying the ServicesResourceTransformer[1] >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Dawid >>>>>>> >>>>>>> [1] >>>>>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer >>>>>>> On 26/01/2021 12:29, Sebastián Magrí wrote: >>>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> I've reported an issue with the postgresql-cdc connector apparently >>>>>>> caused by the maven shade plugin excluding either the JDBC connector or >>>>>>> the >>>>>>> cdc connector due to overlapping classes. The issue for reference is >>>>>>> here: >>>>>>> >>>>>>> https://github.com/ververica/flink-cdc-connectors/issues/90 >>>>>>> >>>>>>> In the meantime, however, I've been trying to figure out if I can >>>>>>> set up an exclusion rule to fix this in my pom.xml file, without >>>>>>> success. >>>>>>> >>>>>>> The `org.postgresql:postgresql` dependency is being added manually >>>>>>> by me to have a sink on a postgresql table and injected by the cdc >>>>>>> connector seemingly via its debezium connector dependency. >>>>>>> >>>>>>> Any guidance or hints I could follow would be really appreciated. >>>>>>> >>>>>>> -- >>>>>>> Sebastián Ramírez Magrí >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Sebastián Ramírez Magrí >>>>>> >>>>> >>>> >>>> -- >>>> Sebastián Ramírez Magrí >>>> >>> >> >> -- >> Sebastián Ramírez Magrí >> > > > -- > Sebastián Ramírez Magrí >
