That looks like it was corrupted and then repaired.
Since running on a laptop, suspended disks, write caches
and a lot of other things could come to mind for the corruption?
On Jun 15, 2014, at 10:23 AM, Octavian Covalschi <octavian.covals...@gmail.com>
wrote:
> Hello,
>
> I'm evaluating Apollo for our current project and so far it worked well for
> our use case, however today I've encountered a problem, corrupted LevelDB...
>
> What could possibly corrupt it? I have Apollo 1.7 installed on my laptop
> and everything was fine till today when I discovered the problem (see some
> log messages below). Nothing unusual happened, so cannot link this with any
> incident...
>
> So I was wondering:
>
> What's the proper fix for this? This time i deleted all files in data dir
> and restarted apollo, but I'm not sure it's the way to go in a production
> environment...
>
> What can be done to avoid disasters? A msg queue is a critical component of
> our architecture so, if that stops most of the functionalities are stopped
> too...
>
> Is there an easy way to replicate persisted messages it?
>
> Is there a more verbose logging that I could use to monitor/find the
> problem in future?
>
> Should I use different storage engine?
>
> Is there a best practice/pattern to recover from this kind of situations?
> Like re-publishing messages, but it increases complexity of our app.
>
> Thank you in advance.
>
>
> [Log messages]
>
> 2014-06-15 11:57:16,866 | INFO | OS : Linux 3.14.5-200.fc20.x86_64
> ("Fedora release 20 (Heisenbug)") |
> 2014-06-15 11:57:16,870 | INFO | JVM : Java HotSpot(TM) 64-Bit Server
> VM 1.7.0_51 (Oracle Corporation) |
> 2014-06-15 11:57:16,870 | INFO | Apollo : 1.7 (at: /opt/apollo/home) |
> 2014-06-15 11:57:16,871 | INFO | OS is restricting the open file limit to:
> 100000 |
> 2014-06-15 11:57:17,077 | INFO | Starting store: leveldb store at
> /opt/apollo/brokers/local/data |
> 2014-06-15 11:57:17,139 | INFO | Accepting connections at: tcp://
> 0.0.0.0:60013 |
> 2014-06-15 11:57:17,144 | INFO | Opening the log file took: 27.97 ms |
> 2014-06-15 11:57:17,196 | WARN | DB operation failed. (entering recovery
> mode): org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption:
> CURRENT file does not end with newline | 146a03f386f
> 2014-06-15 11:57:18,081 | INFO | virtual host startup is waiting on store
> startup |
> 2014-06-15 11:57:18,199 | INFO | DB recovered from failure. |
> 2014-06-15 11:57:18,200 | ERROR | Store startup failure:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption:
> CURRENT file does not end with newline | 146a03f3870
> 2014-06-15 11:57:18,201 | INFO | virtual host startup is no longer
> waiting. It waited a total of 1 seconds. |
> 2014-06-15 11:57:18,305 | INFO | Administration interface available at:
> http://127.0.0.1:60080/
