Hello flume users! I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did that with 1.5 but now I seem to fail :(
I got this error in the log (full exception below): java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer information does not match signer information of other classes in the same package Basically, what I did was to install Flume 1.6, install Solr 5.3, compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's classpath. Here's the full command line: exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application -n agent -f conf/flume-conf.properties Here's my Java version, if it matters: java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) >From what I can understand out off the Internet (thanks, Internet!) this may be caused by multiple jars having different versions of the same thing. But I can only find it in Morphlines (via grep -RHls in all the classpaths). Maybe I'm bumping into https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to verify this by compiling Kite 0.12.0 but it failed with: [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data: Too many files with unapproved license: 2 See RAT report.... So then I compiled only morphlines and that worked, only to bump into the same error. As promised, below is the full exception. For completeness, I'm also including the main config and the morphline config: 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 counterGroup:{ name:null counters:{} } } - Exception follows. java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer information does not match signer information of other classes in the same package at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) at java.lang.ClassLoader.defineClass(ClassLoader.java:794) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224) at org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176) at org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91) at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43) at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101) at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) > cat conf/flume-conf.properties agent.sources = spoolDir agent.channels = spillableMem agent.sinks = solr agent.sources.spoolDir.type = spooldir agent.sources.spoolDir.spoolDir = /opt/logs agent.sources.spoolDir.batchSize = 1000 agent.sources.spoolDir.channels = spillableMem agent.channels.spillableMem.type = SPILLABLEMEMORY agent.channels.spillableMem.memoryCapacity = 100000 agent.channels.spillableMem.overflowCapacity = 1000000 agent.channels.spillableMem.checkpointDir = /var/run/flume agent.channels.spillableMem.dataDirs = /var/run/flume-data agent.sinks.solr.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink agent.sinks.solr.channel = spillableMem agent.sinks.solr.morphlineFile = conf/morphline.conf agent.sinks.solr.morphlineId = 1 agent.sinks.solr.batchSize = 10000 > cat conf/morphline.conf SOLR_LOCATOR : { collection : gettingstarted solrUrl : "http://192.168.1.3:8983/solr/"; batchSize : 5000 } morphlines : [ { id : 1 commands : [ { readLine { charset : UTF-8 } } { grok { dictionaryFiles : [conf/grok-patterns] expressions : { message : """%{COMBINEDAPACHELOG}""" } } } # { generateUUID { # field : id # } # } { loadSolr { solrLocator : ${SOLR_LOCATOR} } } ] } ] I also have a grok-patterns file there, but I don't think it's relevant. Any ideas are much appreciated! Thanks and best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/
