Re: corrupted edits log after power failure

2011-09-26 Thread Steve Loughran
On 22/09/11 20:15, Brian Bockelman wrote: Hi Gabi, I'd be a bit scared of that backup strategy; what happens if the TCP connection gets cut suddenly during curl? What happens if there's a TCP corruption? Such things have happened before. Curl might work for long-haul backups, but I'd use H

Re: corrupted edits log after power failure

2011-09-22 Thread Brian Bockelman
Hi Gabi, I'd be a bit scared of that backup strategy; what happens if the TCP connection gets cut suddenly during curl? What happens if there's a TCP corruption? Such things have happened before. Personally, we have the SNN merge the edits every 15 minutes. If it hasn't happened in 30 minut

Re: corrupted edits log after power failure

2011-09-22 Thread Kihwal Lee
Does the backup process include syncing? On-drive write cache can also trick you. For absolutely critical data, it is a good idea to use a controller with battery-backed write cache or a service/product that guarantees durability. Kihwal On 9/22/11 3:48 AM, "Gabi Kazav" wrote: Hi, I had Powe

corrupted edits log after power failure

2011-09-22 Thread Gabi Kazav
Hi, I had Power Failure. I have backup of files: edits, fsimage. I am backing it up with: curl -s http://nameNode:50070/getimage?getimage=1 > fsimage curl -s http://nameNode:50070/getimage?getedits=1 > edits When I am trying to start the HDFS with the recovered files, I got error about the edi