Hello, I just started learning Kafka and have the environment setup on my hortonworks sandbox at home vmware.
test.csv is what I want the producer to send out: more test1.csv ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic kafka-topic2 1, abc 2, def ... 8, vwx 9, zzz What I received are all the content of test.csv, however, not in their original order; kafka-console-consumer.sh --zookeeper 192.168.112.129:2181 --topic kafka-topic2 2, def 1, abc ... 9, zzz 8, vwx I read from google that partition could be the feasible solution, however, my questions are: 1. for small files like this one, shall I really do the partitioning? how small a partition would be acceptable to ensure the sequence? 2. for big files, each partition could still contain multiple lines, how to ensure all the lines in each partition won't get messed up on consumer side? I also want to know what is the best practice to process large volume of data through kafka? There should be better way other than console command. Thank you very much. *------------------------------------------------* *Sincerely yours,* *Raymond*
