Re: migrating log data to new locations

2014-10-08 Thread Javier Alba
Thanks guys for your support. @Jay, this approach seems the easiest right now, I will try it. @Jun, following your suggestion I’ve submitted a Jira: https://issues.apache.org/jira/browse/KAFKA-1689 thanks, Javier On Wednesday 8 October 2014 at 00:45, Jay Kreps wrote: > I think the mo

Re: migrating log data to new locations

2014-10-07 Thread Jay Kreps
I think the more automated/lazy way right now would be to shutdown one broker, rm -rf all its data, add the data directories in config, and restart to let the broker restore off the replicas. This may actually be okay though it is a little slower. -Jay On Tue, Oct 7, 2014 at 3:25 PM, Jun Rao wro

Re: migrating log data to new locations

2014-10-07 Thread Jun Rao
You can stop the broker and copy some of the log directories to the new volumes. You have to be a bit careful when you do that. There are two metadata files recovery-point-offset-checkpoint and replication-offset-checkpoint that you have to manually split and copy over. Ideally, we should be able

Re: migrating log data to new locations

2014-10-07 Thread Steve Morin
Neha, I log volume or can it be volumes plural? -Steve On Tue, Oct 7, 2014 at 6:41 AM, Neha Narkhede wrote: > Is it possible to perform this migration without losing the data currently > stored in the kafka cluster? > > Though I haven't tested this, the way this is designed should allow you to

Re: migrating log data to new locations

2014-10-07 Thread Neha Narkhede
Is it possible to perform this migration without losing the data currently stored in the kafka cluster? Though I haven't tested this, the way this is designed should allow you to shut down a broker, move some partition directories over to the new log volume and restart the broker. You will have to