I can reproduce this with the mx4j lib loaded by setting to max hint window to 'empty': max_hint_window_in_ms:
I guess you could call this a bug, but given that it has 2 defaults, you have to explicitly set this to empty in the configuration to cause any exceptions. On Thu, Nov 28, 2013 at 10:12 AM, Nate McCall <n...@thelastpickle.com>wrote: > What happens when you don't start the JMX service? That field has a > default in both cassandra.yaml and in Config.java: > > https://github.com/apache/cassandra/blob/cassandra-1.2/src/java/org/apache/cassandra/config/Config.java#L43 > > This may be a bug that could be fixed with simply adding a null check for > such in DatabaseDescriptor#loadYaml() > > > On Thu, Nov 28, 2013 at 3:00 AM, Xavier Fustero <xav...@rightscale.com>wrote: > >> Mikhail, >> >> tested it setting it to default value 3600000 and it stopped crashing! >> That's awesome! I love this mailing list ;-) >> >> Thanks a lot, >> Xavi >> >> >> On Thu, Nov 28, 2013 at 2:59 AM, Mikhail Stepura < >> mikhail.step...@outlook.com> wrote: >> >>> What’s the value of “max_hint_window_in_ms” in your cassandra.yaml? >>> >>> -M >>> >>> >>> "Xavier Fustero" <xav...@rightscale.com> wrote in message >>> news:cah7zuusuh7s_9vvjaxg75fo5xd2rpwy6rgpw_ubxmwbwd4_...@mail.gmail.com. >>> .. >>> Hi all, >>> >>> I am installing a cassandra 1.2 on Ubuntu. I followed the Debian/Ubuntu >>> guidelines but even following the procedure to get rid of openJDK it was >>> always there. I read the packages a build using openJDK so I am not sure >>> who really get rid of it. >>> >>> Anyway, I finally decided to install it from apache-cassandra tar file. >>> However, after my instance is booted with all configuration, starting it on >>> foreground shows me the error below. I got the same before using deb >>> package but I thought it was related to the openJDK but it seems it is not >>> as I am using >>> >>> # java -version >>> java version "1.6.0_31" >>> Java(TM) SE Runtime Environment (build 1.6.0_31-b04) >>> Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) >>> >>> I also updated the mx4j-tools.jar in case it was related. The system.log >>> doesn't seem to show any error thoug. I need these MBeans working as I am >>> monitoring Cassandra with some collectd plugins accessing to them. >>> >>> Please let me know if anyone understand what is wrong here. I pasted >>> below all the logs I am talking about. >>> >>> Thanks a lot, >>> Xavi >>> >>> >>> root@xaviring1:~# /usr/share/cassandra/bin/cassandra -f >>> xss = -XX:HeapDumpPath=/mnt/cassandra -XX:+CMSIncrementalPacing >>> -XX:+CMSIncrementalMode >>> -javaagent:/usr/share/cassandra/bin/../lib/jamm-0.2.5.jar >>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn100M >>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k >>> com.rightscale.rmiregistry.port:8081 >>> Getting the platform's MBean Server >>> Connection URL: >>> service:jmx:rmi://localhost:8081/jndi/rmi://localhost:8081/jmxrmi >>> Creating RMI connector server >>> HttpAdaptor version 3.0.2 started on port 8082 >>> Exception during http request >>> javax.management.RuntimeMBeanException: java.lang.NullPointerException >>> at >>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856) >>> at >>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869) >>> at >>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670) >>> at >>> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) >>> at >>> mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119) >>> at >>> mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56) >>> at >>> mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980) >>> Caused by: java.lang.NullPointerException >>> at >>> org.apache.cassandra.config.DatabaseDescriptor.getMaxHintWindow(DatabaseDescriptor.java:1157) >>> at >>> org.apache.cassandra.service.StorageProxy.getMaxHintWindow(StorageProxy.java:1506) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) >>> at >>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) >>> at >>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) >>> at >>> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) >>> at >>> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) >>> at >>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) >>> ... 4 more >>> >>> >>> Here is my system.log: >>> >>> Nov 27 16:41:05 xaviring1 cassandra: Logging initialized >>> Nov 27 16:41:05 xaviring1 cassandra: JVM vendor/version: Java >>> HotSpot(TM) 64-Bit Server VM/1.6.0_31 >>> Nov 27 16:41:05 xaviring1 cassandra: Your JVM is out of date. Please >>> upgrade to the newest Oracle Java 6. >>> Nov 27 16:41:05 xaviring1 cassandra: Heap size: 1063256064/1063256064 >>> Nov 27 16:41:05 xaviring1 cassandra: Classpath: >>> /usr/share/cassandra/bin/../conf:/usr/share/cassandra/bin/../build/classes/main:/usr/share/cassandra/bin/../build/classes/thrift:/usr/share/cassandra/bin/../lib/CustomAgent.jar:/usr/share/cassandra/bin/../lib/antlr-3.2.jar:/usr/share/cassandra/bin/../lib/apache-cassandra-1.2.12.jar:/usr/share/cassandra/bin/../lib/apache-cassandra-clientutil-1.2.12.jar:/usr/share/cassandra/bin/../lib/apache-cassandra-thrift-1.2.12.jar:/usr/share/cassandra/bin/../lib/apache-cassandra.jar:/usr/share/cassandra/bin/../lib... >>> Nov 27 16:41:06 xaviring1 cassandra: JNA mlockall successful >>> Nov 27 16:41:06 xaviring1 cassandra: Loading settings from >>> file:/etc/cassandra/cassandra.yaml >>> Nov 27 16:41:07 xaviring1 cassandra: Data files directories: >>> [/mnt/cassandra/data] >>> Nov 27 16:41:07 xaviring1 cassandra: Commit log directory: >>> /mnt/cassandra/commitlog >>> Nov 27 16:41:07 xaviring1 cassandra: DiskAccessMode 'auto' determined to >>> be mmap, indexAccessMode is mmap >>> Nov 27 16:41:07 xaviring1 cassandra: disk_failure_policy is stop >>> Nov 27 16:41:07 xaviring1 cassandra: Global memtable threshold is >>> enabled at 338MB >>> Nov 27 16:41:07 xaviring1 cassandra: Not using multi-threaded >>> compaction >>> Nov 27 16:41:07 xaviring1 cassandra: Please rename encryption_options as >>> server_encryption_options in the yaml >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing key cache with >>> capacity of 100 MBs. >>> Nov 27 16:41:08 xaviring1 cassandra: Scheduling key cache save to each >>> 14400 seconds (going to save all keys). >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing row cache with >>> capacity of 100 MBs and provider >>> org.apache.cassandra.cache.SerializingCacheProvider >>> Nov 27 16:41:08 xaviring1 cassandra: Scheduling row cache save to each 0 >>> seconds (going to save all keys). >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.batchlog >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.peer_events >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.hints >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.Schema >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing >>> system.schema_keyspaces >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-1 >>> (220 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-1 >>> (220 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-2 >>> (220 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-2 >>> (220 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.range_xfers >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing >>> system.HintsColumnFamily >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing >>> system.schema_columnfamilies >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-2 >>> (4593 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-2 >>> (4593 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-1 >>> (4593 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-1 >>> (4593 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.NodeIdInfo >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.schema_columns >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columns/system-schema_columns-ic-1 (4164 >>> bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columns/system-schema_columns-ic-1 (4164 >>> bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columns/system-schema_columns-ic-2 (4177 >>> bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/schema_columns/system-schema_columns-ic-2 (4177 >>> bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.IndexInfo >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.Migrations >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.peers >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.local >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/local/system-local-ic-11 (5746 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/local/system-local-ic-11 (5746 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/local/system-local-ic-12 (120 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/local/system-local-ic-12 (120 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/local/system-local-ic-13 (5421 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Opening >>> /mnt/cassandra/data/system/local/system-local-ic-13 (5421 bytes) >>> Nov 27 16:41:08 xaviring1 cassandra: Initializing system.LocationInfo >>> Nov 27 16:41:09 xaviring1 cassandra: Initializing >>> system_traces.sessions >>> Nov 27 16:41:09 xaviring1 cassandra: Initializing system_traces.events >>> Nov 27 16:41:09 xaviring1 cassandra: completed pre-loading (3 keys) key >>> cache. >>> Nov 27 16:41:09 xaviring1 cassandra: Replaying >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023050.log, >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023051.log, >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023052.log, >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023053.log >>> Nov 27 16:41:09 xaviring1 cassandra: Replaying >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023050.log >>> Nov 27 16:41:09 xaviring1 cassandra: Finished reading >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023050.log >>> Nov 27 16:41:09 xaviring1 cassandra: Replaying >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023051.log >>> Nov 27 16:41:09 xaviring1 cassandra: Finished reading >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023051.log >>> Nov 27 16:41:09 xaviring1 cassandra: Replaying >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023052.log >>> Nov 27 16:41:09 xaviring1 cassandra: Finished reading >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023052.log >>> Nov 27 16:41:09 xaviring1 cassandra: Replaying >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023053.log >>> Nov 27 16:41:09 xaviring1 cassandra: Finished reading >>> /mnt/cassandra/commitlog/CommitLog-2-1385569023053.log >>> Nov 27 16:41:09 xaviring1 cassandra: Enqueuing flush of >>> Memtable-local@420824846(52/520 serialized/live bytes, 2 ops) >>> Nov 27 16:41:09 xaviring1 cassandra: Writing Memtable-local@420824846(52/520 >>> serialized/live bytes, 2 ops) >>> Nov 27 16:41:09 xaviring1 cassandra: Enqueuing flush of >>> Memtable-schema_keyspaces@1503990546(251/2510 serialized/live bytes, 8 >>> ops) >>> Nov 27 16:41:09 xaviring1 cassandra: Enqueuing flush of >>> Memtable-schema_columns@1686581432(23741/237410 serialized/live bytes, >>> 386 ops) >>> Nov 27 16:41:09 xaviring1 cassandra: Enqueuing flush of >>> Memtable-schema_columnfamilies@1737470871(22187/221870 serialized/live >>> bytes, 369 ops) >>> Nov 27 16:41:10 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/local/system-local-ic-14-Data.db (84 bytes) for >>> commitlog position ReplayPosition(segmentId=1385570469088, position=142) >>> Nov 27 16:41:10 xaviring1 cassandra: Writing >>> Memtable-schema_keyspaces@1503990546(251/2510 serialized/live bytes, 8 >>> ops) >>> Nov 27 16:41:10 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-3-Data.db >>> (218 bytes) for commitlog position ReplayPosition(segmentId=1385570469088, >>> position=142) >>> Nov 27 16:41:10 xaviring1 cassandra: Writing >>> Memtable-schema_columns@1686581432(23741/237410 serialized/live bytes, >>> 386 ops) >>> Nov 27 16:41:10 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/schema_columns/system-schema_columns-ic-3-Data.db >>> (4164 bytes) for commitlog position ReplayPosition(segmentId=1385570469088, >>> position=142) >>> Nov 27 16:41:10 xaviring1 cassandra: Writing >>> Memtable-schema_columnfamilies@1737470871(22187/221870 serialized/live >>> bytes, 369 ops) >>> Nov 27 16:41:10 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-3-Data.db >>> (4593 bytes) for commitlog position ReplayPosition(segmentId=1385570469088, >>> position=142) >>> Nov 27 16:41:10 xaviring1 cassandra: Log replay complete, 12 replayed >>> mutations >>> Nov 27 16:41:10 xaviring1 cassandra: Cassandra version: 1.2.12 >>> Nov 27 16:41:10 xaviring1 cassandra: Thrift API version: 19.36.1 >>> Nov 27 16:41:10 xaviring1 cassandra: CQL supported versions: 2.0.0,3.0.5 >>> (default: 3.0.5) >>> Nov 27 16:41:10 xaviring1 cassandra: Loading persisted ring state >>> Nov 27 16:41:10 xaviring1 cassandra: Starting up server gossip >>> Nov 27 16:41:11 xaviring1 cassandra: Enqueuing flush of >>> Memtable-local@413461956(249/2490 serialized/live bytes, 9 ops) >>> Nov 27 16:41:11 xaviring1 cassandra: Writing >>> Memtable-local@413461956(249/2490 >>> serialized/live bytes, 9 ops) >>> Nov 27 16:41:11 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/local/system-local-ic-15-Data.db (245 bytes) for >>> commitlog position ReplayPosition(segmentId=1385570469088, position=52887) >>> Nov 27 16:41:11 xaviring1 cassandra: Compacting >>> [SSTableReader(path='/mnt/cassandra/data/system/local/system-local-ic-12-Data.db'), >>> SSTableReader(path='/mnt/cassandra/data/system/local/system-local-ic-13-Data.db'), >>> SSTableReader(path='/mnt/cassandra/data/system/local/system-local-ic-14-Data.db'), >>> SSTableReader(path='/mnt/cassandra/data/system/local/system-local-ic-15-Data.db'), >>> SSTableReader(path='/mnt/cassandra/data/system/local/system-local-ic-11-Data.db')] >>> >>> Nov 27 16:41:11 xaviring1 cassandra: Compacted 5 sstables to >>> [/mnt/cassandra/data/system/local/system-local-ic-16,]. 11,616 bytes to >>> 5,748 (~49% of original) in 529ms = 0.010362MB/s. 5 total rows, 1 unique. >>> Row merge counts were {1:0, 2:0, 3:0, 4:0, 5:1, } >>> Nov 27 16:41:12 xaviring1 cassandra: Filtering out >>> TLS_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket >>> Nov 27 16:41:12 xaviring1 cassandra: Starting Encrypted Messaging >>> Service on SSL port 7000 >>> Nov 27 16:41:12 xaviring1 cassandra: Using saved token >>> [-1025783784642367561, -123766659789318196, -1295312444089505247, >>> -1362940946402062950, -1505674377441592984, -1510151953022964931, >>> -1595956917215185478, -1618024116320884346, -1661847706648338640, >>> -1718952508753916570, -1723563561873974918, -1771501414901960857, >>> -1998768013558002607, -2032719025450193591, -2062350291372813732, >>> -216371939563228238, -2214918226708625680, -2219986229292502225, >>> -2224304248500816743, -2229405184252594928, -2245632632301636047, >>> -2262650862561623702, -2303010585556132278, -2544761475287241917, >>> -2549894955562210213, -2550092719946622364, -2554549736787432603, >>> -2634414036433968806, -2672468728734129175, -2719614996320648573, -27... >>> Nov 27 16:41:12 xaviring1 cassandra: Enqueuing flush of >>> Memtable-local@209168200(84/840 serialized/live bytes, 4 ops) >>> Nov 27 16:41:12 xaviring1 cassandra: Writing Memtable-local@209168200(84/840 >>> serialized/live bytes, 4 ops) >>> Nov 27 16:41:12 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/local/system-local-ic-17-Data.db (120 bytes) for >>> commitlog position ReplayPosition(segmentId=1385570469088, position=53162) >>> Nov 27 16:41:12 xaviring1 cassandra: Enqueuing flush of >>> Memtable-local@60652275(10098/100980 serialized/live bytes, 257 ops) >>> Nov 27 16:41:12 xaviring1 cassandra: Writing >>> Memtable-local@60652275(10098/100980 >>> serialized/live bytes, 257 ops) >>> Nov 27 16:41:12 xaviring1 cassandra: Completed flushing >>> /mnt/cassandra/data/system/local/system-local-ic-18-Data.db (5421 bytes) >>> for commitlog position ReplayPosition(segmentId=1385570469088, >>> position=65169) >>> Nov 27 16:41:12 xaviring1 cassandra: Node /10.114.18.206 state jump to >>> normal >>> Nov 27 16:41:12 xaviring1 cassandra: Startup completed! Now serving >>> reads. >>> Nov 27 16:41:12 xaviring1 cassandra: mx4j successfuly loaded >>> Nov 27 16:41:13 xaviring1 cassandra: Starting listening for CQL clients >>> on /0.0.0.0:9042... >>> Nov 27 16:41:13 xaviring1 cassandra: Binding thrift service to / >>> 0.0.0.0:9160 >>> Nov 27 16:41:13 xaviring1 cassandra: Using TFramedTransport with a max >>> frame size of 15728640 bytes. >>> Nov 27 16:41:13 xaviring1 cassandra: Using synchronous/threadpool thrift >>> server on 0.0.0.0 : 9160 >>> Nov 27 16:41:13 xaviring1 cassandra: Listening for thrift clients... >>> >>> >> >> > > > -- > ----------------- > Nate McCall > Austin, TX > @zznate > > Co-Founder & Sr. Technical Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > -- ----------------- Nate McCall Austin, TX @zznate Co-Founder & Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
