thanks! I will write my reply into the mail group once I have fiddled with your suggestions.
James. > On Jan 15, 2018, at 2:20 PM, Jacob Sheck <shec0...@gmail.com> wrote: > > You will need to find or create something to accomplish this. Topics in > Kafka primarily act as queues. If you search the web for more information > about "kafka connector http" you will find a few projects that do this. > You could also take a look at the Confluent S3 Connector as an example of > how to accomplish what you want to do. If you are interested in a quick > and dirty solution you could put a script using curl in cron to pull the > web content and write it to a topic using the kafka-rest-proxy. > > On Mon, Jan 15, 2018 at 1:52 PM James Smyth <smyth.james...@gmail.com> > wrote: > >> Hi Kafka people, >> >> I am very new to kafka, so perhaps my question is naive. I spent some time >> searching around at resources of kafka but only became more confused. >> >> What are common ways to pull info from a web site and send to kafka to >> become a log entry? >> There is a web site that I want to pull a piece of data from once/month >> and have that data written to Kafka log. Consumers will be listening for >> that message to do processing on it. >> I am not sure about common ways to do this. >> >> I am thinking I could have some scheduler (e.g. cron) wake up once per >> month and trigger the pull of the data from the web site and then send it >> to a kafka stream. >> Does kafka have ability to trigger events once/month or is using cron a >> better idea? >> What is the scheduler triggering a stand-alone batch job or the running of >> a some service like a kafka producer? Should I worry about a service >> running all the time when it is likely to only do a few seconds of work >> each month? >> >> Many thanks, >> >> James. >> >>
