Hi, We haven't documented it yet as it's not in any release. I've been meaning to do that! :)
I just created https://issues.apache.org/jira/browse/FLUME-1845 for this. Brock On Tue, Jan 15, 2013 at 9:12 AM, Jeff Bachtel <[email protected]> wrote: > No, I hadn't separated the jarfiles out in that manner (I didn't realize the > order they appeared in the CLASSPATH mattered). I'll certainly go change > that now. > > Is there documentation on the plugins.d directory anywhere? I discovered it > in the code by accident, and it met my needs while using this custom sink. > > Jeff > > > On Mon, Jan 14, 2013 at 9:23 PM, Brock Noland <[email protected]> wrote: >> >> Great to hear the plugin directory is being used. Just to confirm, do >> you have the following: >> >> /usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/ >> >> correct? >> >> Generally the plugin.d was created so this: >> >> /usr/local/flume/plugins.d/plugin/lib/ >> >> would have the plugin jar and this: >> >> /usr/local/flume/plugins.d/plugin/libext/ >> >> would have any jars the plugin depended on. >> >> Brock >> >> On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <[email protected]> >> wrote: >> > The reason I don't recommend using it is that I didn't know it existed >> > :P >> > >> > That seems like a great place to put it. I've done quite a bit of >> > customization, so for me it's easier to just point Flume to the ./target >> > directory of my java projects. But the plugins.d seems like a great >> > place >> > for third-party libraries. >> > >> > Glad I could help! >> > >> > - Connor >> > >> > >> > On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <[email protected]> >> > wrote: >> >> >> >> Awesome, thanks; it was that typo. >> >> >> >> The plugins.d directory seems to be working, is there a particular >> >> reason >> >> you don't recommend using? I'm brand-spanking new to Flume, so I'm >> >> curious. >> >> >> >> Thanks again, >> >> >> >> Jeff >> >> >> >> >> >> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson >> >> <[email protected]> >> >> wrote: >> >>> >> >>> Leave the Cassandra sink jar with its other jars in a folder some >> >>> where. >> >>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH >> >>> line. >> >>> Uncomment it if needed. Append the path to your cassandra sink and its >> >>> associated jars, and see if that works. >> >>> >> >>> There doesn't appear to be a version incompatability; simply, the >> >>> Cassandra jar is not found in the classpath. And the reason for that >> >>> is it >> >>> looks like you have a typo in your flume.conf. >> >>> >> >>> com.tbtoddb.flume.sinks.cassandra.CassandraSink >> >>> >> >>> should probably be >> >>> >> >>> com.btoddb.flume.sinks.cassandra.CassandraSink >> >>> >> >>> - Connor >> >>> >> >>> >> >>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <[email protected]> >> >>> wrote: >> >>>> >> >>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into >> >>>> /usr/local/flume . I manually created >> >>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred >> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local >> >>>> pull/build >> >>>> of btoddb's sources on github), so I have a lib/ directory that's >> >>>> completely >> >>>> populated with both the sink plugin and any possible jarfiles it >> >>>> might rely >> >>>> on. >> >>>> >> >>>> I've likewise tried just copying >> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it to >> >>>> get >> >>>> picked up. >> >>>> >> >>>> When I have a CassandraSink define in flume.conf as such: >> >>>> webserver.sinks.cassandraSink.type = >> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink >> >>>> webserver.sinks.cassandraSink.channel = memoryChannel >> >>>> webserver.sinks.cassandraSink.hosts = localhost >> >>>> webserver.sinks.cassandraSink.cluster-name = MyCluster >> >>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs >> >>>> webserver.sinks.cassandraSink.records-coldfam = Requests >> >>>> >> >>>> and try to start Flume with: >> >>>> >> >>>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f >> >>>> /usr/local/flume/conf/flume.conf -n webserver >> >>>> -Dflume.root.logger=DEBUG,console >> >>>> >> >>>> >> >>>> I get the following output (and obviously, no success): >> >>>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR - >> >>>> >> >>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)] >> >>>> Failed to load configuration data. Exception follows. >> >>>> org.apache.flume.FlumeException: Unable to load sink type: >> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class: >> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink >> >>>> at >> >>>> >> >>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69) >> >>>> at >> >>>> >> >>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41) >> >>>> at >> >>>> >> >>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415) >> >>>> at >> >>>> >> >>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103) >> >>>> at >> >>>> >> >>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) >> >>>> at >> >>>> >> >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >>>> at >> >>>> >> >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) >> >>>> at >> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >> >>>> at >> >>>> >> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> >>>> at >> >>>> >> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> >>>> at >> >>>> >> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> >>>> at >> >>>> >> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> >>>> at java.lang.Thread.run(Thread.java:722) >> >>>> Caused by: java.lang.ClassNotFoundException: >> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink >> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> >>>> at java.security.AccessController.doPrivileged(Native Method) >> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423) >> >>>> at >> >>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356) >> >>>> at java.lang.Class.forName0(Native Method) >> >>>> at java.lang.Class.forName(Class.java:186) >> >>>> at >> >>>> >> >>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67) >> >>>> ... 12 more >> >>>> >> >>>> >> >>>> does anyone have any idea why this plugin isn't being picked up by >> >>>> Flume? Is it an incompatibility with the plugin expecting to run >> >>>> under Flume >> >>>> 1.3.1, perhaps? I'm at a loss on debugging this. >> >>>> >> >>>> Thanks, >> >>>> Jeff >> >>> >> >>> >> >> >> > >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - >> http://incubator.apache.org/mrunit/ > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
