Hi Tom will do. for now I have 4 specific file types I need to ingest.
1. reading apache web server log files, http.log's. 2. reading in our custom log files 3. reading in log4j log files 4. mysql connection as a source 5. cassandra connection, as a sink I can not use NFS mounting the source file system to the Connect cluster, we don't allow NFS. I'm hoping to pull #1-#3 in as each line a the value field of a JSON message, then maybe use stream process, or kSQL to unpack into a 2nd message which can then be consumed, analysed etc. bit amazed there is not a predefined connector for http logs files though G On Wed, Jan 15, 2020 at 12:32 PM Tom Bentley <tbent...@redhat.com> wrote: > Hi George, > > Since you mentioned CDC specifically you might want to check out Debezium ( > https://debezium.io/) which operates as a connector of the sort Robin > referred to and does CDC for MySQL and others. > > Cheers, > > Tom > > On Wed, Jan 15, 2020 at 10:18 AM Robin Moffatt <ro...@confluent.io> wrote: > > > The integration part of Apache Kafka that you're talking about is > > called Kafka Connect. Kafka Connect runs as its own process, known as > > a Kafka Connect Worker, either on its own or as part of a cluster. Kafka > > Connect will usually be deployed on a separate instance from the Kafka > > brokers. > > > > Kafka Connect connectors will usually connect to the external system over > > the network if that makes sense (e.g. a database) but not always (e.g. if > > its acting as a syslog endpoint, or maybe processing local files). > > > > You can learn more about Kafka Connect and its deployment model here: > > https://rmoff.dev/crunch19-zero-to-hero-kafka-connect > > > > > > -- > > > > Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff > > > > > > On Wed, 15 Jan 2020 at 03:43, George <george...@gmail.com> wrote: > > > > > Hi all. > > > > > > Please advise, a real noob here still, unpacking how the stack still > > > works... > > > > > > if I have a mySQL server, or a web server, or a 2 node JBOSS cluster. > > > > > > If I want to use the mysql connector to connect to the MySQL DB to pull > > > data using CDC... then I need to install the Kafka stack on the DB > > server, > > > I understand that this will be a stand alone install, assume with no > > > zookeeper involved. > > > > > > Similarly for the apache web server and the 2 JBOSS servers > > > > > > G > > > > > > -- > > > You have the obligation to inform one honestly of the risk, and as a > > person > > > you are committed to educate yourself to the total risk in any > activity! > > > > > > Once informed & totally aware of the risk, > > > every fool has the right to kill or injure themselves as they see fit! > > > > > > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit!
