Re: Question on ETL while replau

2015-02-02 Thread Jun Rao
You can't change existing messages. You can republish messages with new fields and manually set the consumer offsets. Thanks, Jun On Thu, Jan 29, 2015 at 1:12 PM, Joshua Schumacher wrote: > What's the best way to add two 'fields' to my kafka messages once they are > stored? Can I just do a re

Question on ETL while replau

2015-01-29 Thread Joshua Schumacher
What's the best way to add two 'fields' to my kafka messages once they are stored? Can I just do a replay on all of them and add the field? How would I throw out the old kafka messages that don't have the field then? I am using Druid to process the data, but not sure of how to propagate data cha