@Clebert After investigation, looking problems in our remote storage system.
But I was afraid to have corrupted data, since googling I found error could be related to that. Now looking everything is back. Thanks <http://www.williamhill.com/> <http://www.whenthefunstops.co.uk/> Jose Manuel Vega Monroy Java Developer / Software Developer Engineer in Test Direct: +0035 0 2008038 (Ext. 8038) Email: jose.mon...@williamhill.com William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA On 22/07/2020, 23:50, "Clebert Suconic" <clebert.suco...@gmail.com> wrote: the error is self explanatory.. isn't? The broker will be stopped after a critical IO error. I'm glad it did as this is the intention.. you probably need to evaluate how you add and consume messages.. make sure you consume your messages. .you can't just write until you run out of space.. or add more space to your device. On Wed, Jul 22, 2020 at 4:58 PM Jose Manuel Vega Monroy < jose.mon...@williamhill.com> wrote: > Hi there, > > > > Facing this on *brokers, and crashing*: > > > > 2020-07-22 20:39:45,857 [main] INFO (Server.java:379) - Started @19462ms > > 2020-07-22 20:39:48,713 [KahaDB Page Writer] INFO (PageFile.java:1110) - > Unexpected io error on pagefile write of 1 pages. > > java.io.IOException: No space left on device > > at sun.nio.ch.FileDispatcherImpl.force0(Native Method) > > at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) > > at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) > > at > org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:397) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1095) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.pollWrites(PageFile.java:1025) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.access$000(PageFile.java:66) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile$2.run(PageFile.java:1226) > > 2020-07-22 20:39:48,719 [KahaDB Page Writer] INFO (PageFile.java:1028) - > An exception was raised while performing poll writes > > java.io.IOException: No space left on device > > at sun.nio.ch.FileDispatcherImpl.force0(Native Method) > > at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) > > at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) > > at > org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:397) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1095) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.pollWrites(PageFile.java:1025) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.access$000(PageFile.java:66) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile$2.run(PageFile.java:1226) > > 2020-07-22 20:39:53,720 [ActiveMQ Journal Checkpoint Worker] ERROR > (MessageDatabase.java:441) - Checkpoint failed > > java.lang.IllegalStateException: PageFile is not loaded > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.assertLoaded(PageFile.java:819) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.tx(PageFile.java:304) > > at > org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1698) > > at > org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:1070) > > at > org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:433) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-07-22 20:39:53,723 [ActiveMQ Journal Checkpoint Worker] INFO (DefaultIOExceptionHandler.java:177) > - Stopping BrokerService[localhost] due to exception, java.io.IOException: > PageFile is not loaded > > java.io.IOException: PageFile is not loaded > > at > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40) > > at > org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:442) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.IllegalStateException: PageFile is not loaded > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.assertLoaded(PageFile.java:819) > > at > org.apache.activemq.store.kahadb.disk.page.PageFile.tx(PageFile.java:304) > > at > org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1698) > > at > org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:1070) > > at > org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:433) > > ... 7 more > > 2020-07-22 20:39:53,725 [IOExceptionHandler: stopping > BrokerService[localhost]] INFO (BrokerService.java:836) - Apache > ActiveMQ 5.15.0 (localhost, > ID:prdxpeb21b2b001.prod.williamhill.plc-34078-1595446783723-0:1) is > shutting down > > 2020-07-22 20:39:53,726 [ActiveMQ Transport: tcp:///10.121.68.129:58814@61321] > WARN (TransportConnection.java:237) - Transport Connection to: tcp:// > 10.121.68.129:58814 failed: java.io.IOException: Unexpected error > occurred: org.apache.activemq.broker.BrokerStoppedException: Broker > BrokerService[localhost] is being stopped > > 2020-07-22 20:39:53,736 [ActiveMQ Transport Server Thread Handler: tcp:// > 0.0.0.0:61321?maximumConnections=1000&wireFormat.maxFrameSize=104857600] > INFO (TcpTransportServer.java:496) - socketQueue interrupted - stopping > > 2020-07-22 20:39:53,739 [ActiveMQ Transport Server Thread Handler: tcp:// > 0.0.0.0:61321?maximumConnections=1000&wireFormat.maxFrameSize=104857600] > INFO (TransportConnector.java:243) - Could not accept connection during > shutdown : {} > > java.lang.InterruptedException > > > > Any idea about the problem? Using *remote NFS mount partition.* > > > > Thanks > > > > [image: > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_WHlogo.gif-3Fwidth-3D180&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=N7KmekZaFn0h5F4ebyFKa7jmRkLYB4wUngChP43XmyY&s=D0ZCY54nt54ADquI1YDvXM77BBoZry9TAkLy3SiywLg&e= ] > <http://www.williamhill.com/> > > [image: > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_senet.gif-3Fwidth-3D180&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=N7KmekZaFn0h5F4ebyFKa7jmRkLYB4wUngChP43XmyY&s=AbN_tPjBETgsat_zGSIEDIJzn5FR8fIkp1AqNmhyWdg&e= ] > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=N7KmekZaFn0h5F4ebyFKa7jmRkLYB4wUngChP43XmyY&s=awqtCrRZ6VNCKSCbXum-FxwqVMWnr71mWuOucch5QsE&e= > > > *Jose Manuel Vega Monroy * > *Java Developer / Software Developer Engineer in Test* > > Direct: +*0035 0 2008038 (Ext. 8038)* > Email: jose.mon...@williamhill.com > > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > > > Confidentiality: The contents of this e-mail and any attachments > transmitted with it are intended to be confidential to the intended > recipient; and may be privileged or otherwise protected from disclosure. If > you are not an intended recipient of this e-mail, do not duplicate or > redistribute it by any means. Please delete it and any attachments and > notify the sender that you have received it in error. This e-mail is sent > by a William Hill PLC group company. The William Hill group companies > include, among others, William Hill PLC (registered number 4212563), > William Hill Organization Limited (registered number 278208), William Hill > US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr > Green Limited (registered number C43260). Each of William Hill PLC and > William Hill Organization Limited is registered in England and Wales and > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle > County DE 19801 Delaware, United States of America. WHG (International) > Limited is registered in Gibraltar and has its registered office at 6/1 > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has > its registered office at Tagliaferro Business Centre, Level 7, 14 High > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, > the contents of this e-mail are subject to contract; and are not an > official statement, and do not necessarily represent the views, of William > Hill PLC, its subsidiaries or affiliated companies. Please note that > neither William Hill PLC, nor its subsidiaries and affiliated companies can > accept any responsibility for any viruses contained within this e-mail and > it is your responsibility to scan any emails and their attachments. William > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail > traffic data and also the content of e-mails for effective operation of the > e-mail system, or for security, purposes. > -- Clebert Suconic Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
