What happens when you change the colNames parameter to col1val,col2val? The line should be:
agent.sinks.hbaseSink.serializer.colNames=col1val,col2val On Thu, Jul 31, 2014 at 1:45 AM, Tinte garcia, Miguel Angel < miguel.ti...@atos.net> wrote: > Hi Jonathan, > > My current configuration is the following: > > agent.sinks.hbaseSink.type=hbase > > agent.sinks.hbaseSink.channel=memoryChannel > > agent.sinks.hbaseSink.table=hbase_table > > # filling first column > > agent.sinks.hbaseSink.columnFamily=column1 > > agent.sinks.hbaseSink.batchSize = 5000 > > # splitting input parameters > > > agent.sinks.hbaseSink.serializer=org.apache.flume.sink.hbase.RegexHbaseEventSerializer > > agent.sinks.hbaseSink.serializer.regex=^[^,]+,(.+),(.+)$ > > agent.sinks.hbaseSink.serializer.colNames=col1val > > # filling second column > > #agent.sinks.hbaseSink.columnFamily=column2 > > # splitting input parameters > > > #agent.sinks.hbaseSink.serializer=org.apache.flume.sink.hbase.RegexHbaseEventSerializer > > #agent.sinks.hbaseSink.serializer.regex=^[^,]+,(.+),(.+)$ > > #agent.sinks.hbaseSink.serializer.colNames=col2val > > > > As you can see above, I have only been able to put one colName into a > columnFamily, the second declaration commented didn’t work. The flume event > is generated to store in the HBase table as a POST method with the next > information: > > > http://localhost:8080/flumeEvent/rest/data/inject?colval11=1&colval2=005&colval3=test > > > With the following content: “This is a test for different columns” > > > > Thanks again > > > > > > *From:* Jonathan Natkins [mailto:na...@streamsets.com] > *Sent:* Thursday, July 31, 2014 12:10 AM > > *To:* user@flume.apache.org > *Subject:* Re: Flume to Hbase columns with regexp > > > > Hi Miguel, > > > > What does your configuration look like after you made the initial changes? > As far as I can tell, the HbaseSink only has the ability to load into one > column family, so declaring two of them probably won't help. > > > > The fact that you're getting different values in the same column leads me > to believe that your column values are split across multiple events. Is > that accurate? What did the event that produced two values in one column > look like? > > > > Thanks, > > Natty > > > > On Wed, Jul 30, 2014 at 2:10 AM, Tinte garcia, Miguel Angel < > miguel.ti...@atos.net> wrote: > > Hi Jonathan, > > Thanks for your comments below. This is what I have been able to do so far > (result copied from hbasexplorer): > > *rowkey - timestamp* > > *column1* > > *column2* > > *column3* > > 1406706418563-47PT7nzRvW-0 > > Show 1 Timestamp > > col1val: firstPart > col2val: This is the first part of the result > > > > > > > > Therefore, I have been able to split the different colNval tokens (which > is great) but I am still unable to store these split tokens into the > different hbase table columns. I have tried it declaring two columnFamily > and one value in the subsequent colNames parameter, but it didn’t work. > > Is it possible inserting these values into different columns? > > > > Thanks again > > > > > > *From:* Jonathan Natkins [mailto:na...@streamsets.com] > *Sent:* Tuesday, July 29, 2014 1:15 AM > *To:* user@flume.apache.org > *Subject:* Re: Flume to Hbase columns with regexp > > > > Alright, a couple things: > > > > 1) It looks like my intuition was correct. Changing your config to be > colNames from columns seems to get things working. > > > > 2) Based on the description of what you're trying to do, it looks like > your regex might be slightly off. For example, if I had a row: > > > > familyName,col1val,col2val > > > > Your regex will result in column1 containing 'familyName', and column2 > containing 'col1val,col2val', which I don't think is what you're trying to > do. Probably you want to use this regex, or something like it: > > > > ^[^,]+,(.+),(.+)$ > > > > This regex will result in column1 containing 'col1val', column2 containing > 'col2val', and the first value (which appears to be the family name) being > thrown away. Is this what you were trying to do? > > > > As an aside, the mechanics of the RegexHbaseEventSerializer are to take > the matching groups and map those to the list of column names defined by > the colNames config parameter. If you want to toss any data away, just make > sure it's not within a set of parentheses. > > > > Let me know if you have any more questions, or if you have trouble getting > this to work. > > > > Thanks! > > Natty > > > > On Mon, Jul 28, 2014 at 3:48 PM, Jonathan Natkins <na...@streamsets.com> > wrote: > > I haven't tested this myself, but a quick look at the code suggests that > your column name specification may be configured incorrectly. It looks like > it should be: > > > > agent.sinks.hbaseSink.serializer.colNames = column1,column2 > > > > I'm trying this out myself, though, so if I find something definitive, > I'll let you know. > > > > On Mon, Jul 28, 2014 at 4:19 AM, Tinte garcia, Miguel Angel < > miguel.ti...@atos.net> wrote: > > Hi, > > I am sending a Flume event to insert some information into a concrete > HBase table. My flume conf.properties looks like this: > > agent.sinks.hbaseSink.table=table_name > > agent.sinks.hbaseSink.columnFamily=idColumn > > > agent.sinks.hbaseSink.serializer=org.apache.flume.sink.hbase.RegexHbaseEventSerializer > > agent.sinks.hbaseSink.serializer.regex=^([^,]+),(.+)$ > > agent.sinks.hbaseSink.serializer.columns = column1,column2 > > > > Basically, what I am trying to do is splitting the input values into three > different columns: idColumn,column1,column2 > > With this configuration, no error is returned but no input is recorded > into the table. Any idea about what am I doing wrong? > > > > Thanks in advance > > > > > > > > > > > > > > >
