Thanks Andy, and glad I could help somewhat :) I'll start writing something up over the next week, and will share the link here for feedback :)
Just a caveat, the Filestream Connector could be considered a security risk in a production environment given its direct access to the filesystem. Not that you're looking to use this in prod, but just wanted to be sure I mentioned it. On Thu, 31 Mar 2022 at 09:24, andrew davidson <a...@santacruzanalytics.com> wrote: > Thanks Liam. > > What is 'FOSS Kafka'? google did not find any useful definitions > > A tutorial would be great! I would be happy to provide feed back > > > > https://docs.confluent.io/platform/current/connect/filestream_connector.html > looks really helpful. > > Your list of connectors is very helpful. I wonder if your email would be a > good thing to add to the kafka documentation quick start guide or FAQ? > > Kind regards > > Andy > > On 3/29/22, 5:27 PM, "Liam Clarke-Hutchinson" <lclar...@redhat.com> > wrote: > > Hi Andrew, > > So if you've downloaded Apache Kafka, you can run a standalone connect > instance using the bin/connect-standalone.sh script mentioned. And > while a > lot of the connector documentation is on the Confluent website, you can > still use them with FOSS Kafka so long as you're in line with the > Confluent > Community Licence (basically, IIRC, you can use them for free, but not > to > run a SAAS or similar that competes with Confluent, but IANAL). > > I agree that there's not much useful documentation for your use case. I > will look into writing a tutorial for your use case, would you be > happy to > give me feedback on it as I go? > > The most important configuration initially is the plugin.path, where > your > standalone KC process will look for those JARs. You can see an example > properties file for standalone Connect under the config/ dir in the > Kafka > you downloaded. Note that it has the plugin path commented out > initially. > > So, Kafka ships with a connector that exposes a file source and file > sink, > which is good for testing out KC and getting used to it. You can either > build it from source, or download it from here: > https://mvnrepository.com/artifact/org.apache.kafka/connect-file - > choose > the version that matches the version of Kafka you've downloaded, and > then > you can download the JAR under files up the top. This documentation > from > Confluent is useful: > > https://docs.confluent.io/platform/current/connect/filestream_connector.html > > Note that if you don't provide a file property, (this isn't documented > either(!)) it will use standard input for the file source, and standard > output for the file sink. You can see example configs for this > connector > reading from a file or console under that same config/ directory, and > ditto > for writing. > > These connectors might also be useful for playing with KC, and are all > free > and downloadable: > https://www.confluent.io/hub/confluentinc/kafka-connect-datagen <- > generates a stream of test data > https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter <- > disregard, I saw you mentioned not having Twitter > https://www.confluent.io/hub/C0urante/kafka-connect-reddit <- I > haven't > used this, but could be interesting? > > > I hope this helps you get started, and please let me know if I can help > with anything else :) > > Cheers, > > Liam Clarke > > > > On Wed, 30 Mar 2022 at 11:54, andrew davidson < > a...@santacruzanalytics.com> > wrote: > > > I found the quick start https://kafka.apache.org/quickstart example > very > > helpful. It made it really easy to understand how download, start up, > > create topic, push some data through the Kafka. I did not find > > https://kafka.apache.org/quickstart#quickstart_kafkaconnect useful. > > > > I am looking for something very simple to learning how to configure > and > > use connectors using Apache Kafka distribution, not Confluent. I can > run on > > my mac or Linux server. Being a newbie I want to keep things super > simple. > > I do not want to have to debug firewalls, ACL, … > > > > I do not have a data base, access to twitter, … > > > > I thought maybe something some sort source/sink using the local file > > system? > > > > Any suggestions? > > > > Kind regards > > > > Andy > > > > p.s. I have read a lot of documentation most of it is very high > level. Can > > anyone recommend a “hand on” tutorial? > > > > > > >
