Thank you. it turns out that the elasticsearch sink in flume is written against very old versions of ES. There was a patch someone provided for 5.x version. I tried that. It didn't function.
We're right now debating on scrapping flume entirely and using logstash instead ________________________________ From: أنس الليثي <dev.fano...@gmail.com> Sent: Sunday, April 9, 2017 2:01:26 PM To: user@flume.apache.org Subject: Re: Elasticsearch sink fails with method not found NoSuchMethodError usually appears when there is a mismatch in the libraries versions. Check the Elasticsearch version you are using and check the Elasticsearch and Solr jar versions in the lib directory of Flume. On 9 April 2017 at 23:57, jeff saremi <jeffsar...@hotmail.com<mailto:jeffsar...@hotmail.com>> wrote: How old is the ElasticSearch sink code? I'm using flume 1.7.0 with ES 5.3.0 I have copied the following jars to flume\lib: elasticsearch-5.3.0 lucene-core-6.4.1 and here's my config: agent.sinks.es.type = elasticsearch agent.sinks.es.hostNames = localhost:9300 agent.sinks.es.indexName = logs agent.sinks.es.indexType = logs agent.sinks.es.clusterName = logcluster agent.sinks.es.batchSize = 500 agent.sinks.es.ttl = 30d agent.sinks.es.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer agent.sinks.es.channel = memoryChannel and this is the error I get: 09 Apr 2017 13:43:14,481 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@65b52c32 counterGroup:{ name:null counters:{} } } - Exception follows.java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:141) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77) at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48) at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:358) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:45) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 09 Apr 2017 13:43:14,496 INFO [lifecycleSupervisor-1-2] (org.apache.flume.sink.elasticsearch.ElasticSearchSink.stop:377) - ElasticSearch sink {} stopping -- Anas Rabei Senior Software Developer Mubasher.info anas.ra...@mubasher.info<mailto:anas.ra...@mubasher.info>
