Hi there, I would like to automate some of my tasks using Apache Kafka. Previously i used to do the same using Apache Airflow and which worked fine. But i want to explore the same using Kafka whether this works better than Airflow or not.
1) Kafka runs on Server A 2) Kafka searches for a file named test.xml on Server B, here kafka search for every 10 or 20 mins whether this file created or not. 3) Once kafka sense the file created, then the job starts as follows a)Create a jira ticket and update all the executions on jira for each events b) Trigger a rsync command c) Then unarchive the files using tar command d) Some script to execute using the unarchive files e) Then archive the files and rsync to different location f) Send email once all task finished Please advise if this is something kafka intelligent to begin with? Or if you have any other open source products which can do this actions , please let me know. By the way i prefer to setup these on docker-compose based installation. Thanks Jay
