Hi Andrew Otto (Andrew 2? :D ), Well noted. I should have worded it as "You don't need a Confluent subscription to use them" :)
Cheers, Liam On Wed, 30 Mar 2022 at 13:49, Andrew Otto <o...@wikimedia.org> wrote: > > And while a lot of the connector documentation is on the Confluent > website, you can still use them with FOSS Kafka so long as you're in line > with the Confluent Community Licence > > *Drive by unhelpful comment:* > While this is true legally, the fact that (most?) actual connector > implementations are CCL and not FOSS, means that organizations that use > purely FOSS software (like the Wikimedia Foundation) makes Kafka Connect > effectively unusable. > > Okay carry on! :) > > - Andrew Otto > > > > > > On Tue, Mar 29, 2022 at 8:27 PM Liam Clarke-Hutchinson < > lclar...@redhat.com> > wrote: > > > Hi Andrew, > > > > So if you've downloaded Apache Kafka, you can run a standalone connect > > instance using the bin/connect-standalone.sh script mentioned. And while > a > > lot of the connector documentation is on the Confluent website, you can > > still use them with FOSS Kafka so long as you're in line with the > Confluent > > Community Licence (basically, IIRC, you can use them for free, but not to > > run a SAAS or similar that competes with Confluent, but IANAL). > > > > I agree that there's not much useful documentation for your use case. I > > will look into writing a tutorial for your use case, would you be happy > to > > give me feedback on it as I go? > > > > The most important configuration initially is the plugin.path, where your > > standalone KC process will look for those JARs. You can see an example > > properties file for standalone Connect under the config/ dir in the Kafka > > you downloaded. Note that it has the plugin path commented out initially. > > > > So, Kafka ships with a connector that exposes a file source and file > sink, > > which is good for testing out KC and getting used to it. You can either > > build it from source, or download it from here: > > https://mvnrepository.com/artifact/org.apache.kafka/connect-file - > choose > > the version that matches the version of Kafka you've downloaded, and then > > you can download the JAR under files up the top. This documentation from > > Confluent is useful: > > > > > https://docs.confluent.io/platform/current/connect/filestream_connector.html > > > > Note that if you don't provide a file property, (this isn't documented > > either(!)) it will use standard input for the file source, and standard > > output for the file sink. You can see example configs for this connector > > reading from a file or console under that same config/ directory, and > ditto > > for writing. > > > > These connectors might also be useful for playing with KC, and are all > free > > and downloadable: > > https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <- > > generates a stream of test data > > https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <- > > disregard, I saw you mentioned not having Twitter > > https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I haven't > > used this, but could be interesting? > > > > > > I hope this helps you get started, and please let me know if I can help > > with anything else :) > > > > Cheers, > > > > Liam Clarke > > > > > > > > On Wed, 30 Mar 2022 at 11:54, andrew davidson < > a...@santacruzanalytics.com > > > > > wrote: > > > > > I found the quick start https://kafka.apache.org/quickstart example > very > > > helpful. It made it really easy to understand how download, start up, > > > create topic, push some data through the Kafka. I did not find > > > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful. > > > > > > I am looking for something very simple to learning how to configure > and > > > use connectors using Apache Kafka distribution, not Confluent. I can > run > > on > > > my mac or Linux server. Being a newbie I want to keep things super > > simple. > > > I do not want to have to debug firewalls, ACL, … > > > > > > I do not have a data base, access to twitter, … > > > > > > I thought maybe something some sort source/sink using the local file > > > system? > > > > > > Any suggestions? > > > > > > Kind regards > > > > > > Andy > > > > > > p.s. I have read a lot of documentation most of it is very high level. > > Can > > > anyone recommend a “hand on” tutorial? > > > > > > > > > > > >
