> And while a lot of the connector documentation is on the Confluent website, you can still use them with FOSS Kafka so long as you're in line with the Confluent Community Licence
*Drive by unhelpful comment:* While this is true legally, the fact that (most?) actual connector implementations are CCL and not FOSS, means that organizations that use purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect effectively unusable. Okay carry on! :) - Andrew Otto On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson <lclar...@redhat.com> wrote: > Hi Andrew, > > So if you've downloaded Apache Kafka, you can run a standalone connect > instance using the bin/connect-standalone.sh script mentioned. And while a > lot of the connector documentation is on the Confluent website, you can > still use them with FOSS Kafka so long as you're in line with the Confluent > Community Licence (basically, IIRC, you can use them for free, but not to > run a SAAS or similar that competes with Confluent, but IANAL). > > I agree that there's not much useful documentation for your use case. I > will look into writing a tutorial for your use case, would you be happy to > give me feedback on it as I go? > > The most important configuration initially is the plugin.path, where your > standalone KC process will look for those JARs. You can see an example > properties file for standalone Connect under the config/ dir in the Kafka > you downloaded. Note that it has the plugin path commented out initially. > > So, Kafka ships with a connector that exposes a file source and file sink, > which is good for testing out KC and getting used to it. You can either > build it from source, or download it from here: > https://mvnrepository.com/artifact/org.apache.kafka/connect-file - choose > the version that matches the version of Kafka you've downloaded, and then > you can download the JAR under files up the top. This documentation from > Confluent is useful: > > https://docs.confluent.io/platform/current/connect/filestream_connector.html > > Note that if you don't provide a file property, (this isn't documented > either(!)) it will use standard input for the file source, and standard > output for the file sink. You can see example configs for this connector > reading from a file or console under that same config/ directory, and ditto > for writing. > > These connectors might also be useful for playing with KC, and are all free > and downloadable: > https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <- > generates a stream of test data > https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <- > disregard, I saw you mentioned not having Twitter > https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't > used this, but could be interesting? > > > I hope this helps you get started, and please let me know if I can help > with anything else :) > > Cheers, > > Liam Clarke > > > > On Wed, 30 Mar 2022 at 11:54, andrew davidson <a...@santacruzanalytics.com > > > wrote: > > > I found the quick start https://kafka.apache.org/quickstart example very > > helpful. It made it really easy to understand how download, start up, > > create topic, push some data through the Kafka. I did not find > > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful. > > > > I am looking for something very simple to learning how to configure and > > use connectors using Apache Kafka distribution, not Confluent. I can run > on > > my mac or Linux server. Being a newbie I want to keep things super > simple. > > I do not want to have to debug firewalls, ACL, … > > > > I do not have a data base, access to twitter, … > > > > I thought maybe something some sort source/sink using the local file > > system? > > > > Any suggestions? > > > > Kind regards > > > > Andy > > > > p.s. I have read a lot of documentation most of it is very high level. > Can > > anyone recommend a “hand on” tutorial? > > > > > > >
