Please go ahead and file a jira. If you are willing to submit a patch,
you can post it on the jira.
Viral Bajaria wrote:
I have a similar use case that cropped up yesterday. I saw the archive
and found that there was a recommendation to build it as Sharninder
suggested.
For now, I went down the route of writing a python script which
downloads from S3 and puts the files in a directory which is
configured to be picked up via a spooldir.
I would prefer to get a direct S3 source, and maybe we could
collaborate on it and open-source it. Let me know if you prefer that
and we can work directly on it by creating a JIRA.
Thanks,
Viral
On Thu, Jul 31, 2014 at 10:26 AM, Hari Shreedharan
<hshreedha...@cloudera.com <mailto:hshreedha...@cloudera.com>> wrote:
In both cases, Sharninder is right :)
Sharninder wrote:
As far as I know, there is no (open source) implementation of an S3
source, so yes, you'll have to implement your own. You'll have to
implement a Pollable source and the dev documentation has an outline
that you can use. You can also look at the existing Execsource and
work your way up.
As far as I know, there is no way to configure flume without
using the
configuration file.
On Thu, Jul 31, 2014 at 7:57 PM, Paweł <pro...@gmail.com
<mailto:pro...@gmail.com>
<mailto:pro...@gmail.com <mailto:pro...@gmail.com>>> wrote:
Hi,
I'm wondering if Flume is able to read directly from S3.
I'll describe my case. I have log files stored in AWS S3. I have
to fetch periodically new S3 objects and read log lines from it.
Than use log lines (events) are processed in standard flume's way
(as with other sources).
*1) Is there any way to fetch S3 objects or I have to write
my own
Source?*
There is also second case. I want to have flume configuration
dynamic. Flume sources can change in time. New AWS key and S3
bucket can be added or deleted.
*2) Is there any other way to configure Flume than by static
configuration file?*
--
Paweł Róg
