You will need to find or create something to accomplish this. Topics in Kafka primarily act as queues. If you search the web for more information about "kafka connector http" you will find a few projects that do this. You could also take a look at the Confluent S3 Connector as an example of how to accomplish what you want to do. If you are interested in a quick and dirty solution you could put a script using curl in cron to pull the web content and write it to a topic using the kafka-rest-proxy.
On Mon, Jan 15, 2018 at 1:52 PM James Smyth <smyth.james...@gmail.com> wrote: > Hi Kafka people, > > I am very new to kafka, so perhaps my question is naive. I spent some time > searching around at resources of kafka but only became more confused. > > What are common ways to pull info from a web site and send to kafka to > become a log entry? > There is a web site that I want to pull a piece of data from once/month > and have that data written to Kafka log. Consumers will be listening for > that message to do processing on it. > I am not sure about common ways to do this. > > I am thinking I could have some scheduler (e.g. cron) wake up once per > month and trigger the pull of the data from the web site and then send it > to a kafka stream. > Does kafka have ability to trigger events once/month or is using cron a > better idea? > What is the scheduler triggering a stand-alone batch job or the running of > a some service like a kafka producer? Should I worry about a service > running all the time when it is likely to only do a few seconds of work > each month? > > Many thanks, > > James. > >
