Re: When does kafka connect commit messages to kafka broker ?

[Yu Watanabe]

I found the reason. I am not sure if this is Azure problem  or Jdbc
problem though...

First of all my apology that I had not elaborated my environment.

I use,

DataSource: Azure PostgresSQL Server (Read Replica)
Kafka Connect 2.3.1 (strimzi 0.15.0)
Kafka Broker 2.3.1  (strimzi 0.15.0)

In this condition,  I had below connect property.

=============================================================================================================
{
  "name": "custom_query",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url" :
"jdbc:postgresql://XXXX:5432/YYY?user=BBBB&password=CCCC&ssl=true&readOnly=true&defaultRowFetchSize=5000",
    "mode" : "incrementing",
    "incrementing.column.name" : "sid",
    "validate.non.null": false,
    "table.whitelist" : "DDD",
    "topic.prefix" : "db_",
    "tasks.max" : "1"
  }
}
=============================================================================================================

Above property executes below query.

=============================================================================================================
SELECT * FROM "public"."DDD" WHERE "public"."DDD"."sid" > ? ORDER BY
"public"."DDD"."sid" ASC
=============================================================================================================

However, while testing kafka connect, I found that when "*"  is used,
jdbc does not return any records.
So, I changed the query condition by specifying specify column.

=============================================================================================================
{
  "name": "custom_query",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url" :
"jdbc:postgresql://XXXX:5432/YYY?user=BBBB&password=CCCC&ssl=true&readOnly=true&defaultRowFetchSize=5000",
     "incrementing.column.name" : "sid",
     "validate.non.null": false,
     "query": "SELECT sid, m FROM DDD",
     "topic.prefix" : "custom_query",
     "tasks.max" : "1"
  }
}
=============================================================================================================

Now kafka connect started producing message to broker.

Thanks,
Yu Watanabe

On Wed, Jan 15, 2020 at 12:29 PM Yu Watanabe <yu.w.ten...@gmail.com> wrote:
>
> Hello.
>
> I would like to ask question regarding to kafka connect.
>
> I am trying to import over 1 million+ records from postgres to kafka
> broker using kafka connect using jdbc driver .
> However, it seems that topic for storing messages is not created immediately.
>
> I have built my kafka-connect and broker using strimzi (0.15.0 , kafka 2.3.1 
> ).
>
> When does kafka connect create topic to store messages in broker ?
>
> Thanks,
> Yu Watnaabe
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter:   twitter.com/yuwtennis



-- 
Yu Watanabe

linkedin: www.linkedin.com/in/yuwatanabe1/
twitter:   twitter.com/yuwtennis