Hi,
performance seems to be good with elastic search
2 minutes and change for 5 lakh entries
However, i have a problem.
I can enter the messages into elasticsearch only when rsyslogd is running
in foreground (rsyslogd -n running in command line)
not sure why this is so.
Secondly, my requirement would be to query based on part of message.
i.e my message may look like
<date><time>ip=x.x.x.x name=abcd loglevel=3 <actual log message>
Is it possible to query using curl alll messages that have ip address as
y.y.y.y ?
Is 4000 (500000 entries / 125 odd seconds) the max I can get per second in
my system or can I get some more tuning parameters.
My rsyslog.conf has the following lines
------------------------------------------------------------
$ActionQueueDequeueBatchSize 1000
template (name="apsimTemplate" type="list" option.json="on") {
constant(value="{")
constant(value="\"@message\":\"")
property(name="msg")
constant(value="\"}")
}
*.* action(type="omelasticsearch" template="apsimTemplate"
server="localhost" serverport="9200")
My elasticsearch.yml has folliwing lines
------------------------------------------------------------
cluster:
name: APSIM
network:
host: localhost
root@localhost rsyslog]# date; ./a.out ; date
Mon Jun 17 23:33:29 IST 2013
openlog: Success
Mon Jun 17 23:35:32 IST 2013
[root@localhost rsyslog]#
[root@localhost rsyslog]# curl 'http://localhost:9200/_search?pretty=true'
-d '
{
"from" : 0, "size" : 1000000,
"query" : {
"matchAll" : {}
}
}' > e
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 1688k 100 1688k 0 86 19.1M 998 --:--:-- --:--:-- --:--:--
19.6M
[root@localhost rsyslog]# cat e | grep "this is a test" | wc -l
500000
[root@localhost rsyslog]#
void main()
{
int i = 0;
char * string = "{\n \"name\":\"joys of programming\" \n }";
setlogmask (LOG_UPTO (LOG_NOTICE));
openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
perror("openlog");
for(i = 0; i < 500000; i++) {
syslog (LOG_NOTICE, "this is a test %d", i);
}
closelog ();
}
On Tue, Jun 18, 2013 at 5:38 PM, Mahesh V <[email protected]>wrote:
> sorry.
>
> got the answer from
> http://www.elasticsearch.org/guide/reference/api/search/from-size/
>
>
>
>
> On Tue, Jun 18, 2013 at 5:33 PM, Mahesh V
> <[email protected]>wrote:
>
>> Hi
>>
>> I changed the elasticsearch.yml with following lines
>>
>> cluster:
>>
>> name: APSIM
>>
>> network:
>> host: localhost
>>
>>
>> and i ran the rsyslogd in foreground with debug enabled.
>>
>> and with this program
>>
>>
>> void main()
>> {
>> int i = 0;
>> char * string = "{\n \"name\":\"joys of programming\" \n }";
>>
>> setlogmask (LOG_UPTO (LOG_NOTICE));
>>
>> openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
>> perror("openlog");
>>
>> for(i = 0; i < 50; i++) {
>> syslog (LOG_NOTICE, "this is a test %d", i);
>>
>> //usleep(200);
>> }
>> closelog ();
>> }
>>
>>
>> I got only some lines registered in the log. I should have had 50 of such
>> lines but could get only so much.
>> Any ideas on what could be wrong?
>>
>> [root@localhost rsyslog]# curl '
>> http://localhost:9200/_search?pretty=1&q=*:*'
>> {
>> "took" : 2,
>> "timed_out" : false,
>> "_shards" : {
>> "total" : 5,
>> "successful" : 5,
>> "failed" : 0
>> },
>> "hits" : {
>> "total" : 108,
>> "max_score" : 1.0,
>> "hits" : [ {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "ZM-HY3IGRLaW7X_BlhN-Vw",
>> "_score" : 1.0, "_source" : {"@message":" [origin
>> software=\"rsyslogd\" swVersion=\"7.4.0\" x-pid=\"16988\" x-info=\"
>> http://www.rsyslog.com\";] start"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "J-EqhZMBSw64Lts2SKD-yg",
>> "_score" : 1.0, "_source" : {"@message":"type=1400
>> audit(1371489908.934:42327): avc: denied { name_connect } for pid=16877
>> comm=72733A6D61696E20513A526567 dest=9200
>> scontext=unconfined_u:system_r:syslogd_t:s0
>> tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "jK6sFd-CTO-0llfj_zp_zw",
>> "_score" : 1.0, "_source" : {"@message":"type=1400
>> audit(1371489908.934:42328): avc: denied { name_connect } for pid=16877
>> comm=72733A6D61696E20513A526567 dest=9200
>> scontext=unconfined_u:system_r:syslogd_t:s0
>> tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "QyMvKocsTXWYNdusOAuUQA",
>> "_score" : 1.0, "_source" : {"@message":"type=1400
>> audit(1371489955.439:42329): avc: denied { write } for pid=17022
>> comm=\"sshd\" name=\"log\" dev=devtmpfs ino=51770750
>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
>> tcontext=unconfined_u:object_r:device_t:s0 tclass=sock_file"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "8UvsmtlpTu22him_rKN6bw",
>> "_score" : 1.0, "_source" : {"@message":" this is a test 2"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "yA7VpDWmQIWOJLty_F9DEw",
>> "_score" : 1.0, "_source" : {"@message":" this is a test 11"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "eJGdab4NRTGeuWJfIAFa3A",
>> "_score" : 1.0, "_source" : {"@message":" this is a test 15"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "1vOyvn67SwmSPsZqU29rEQ",
>> "_score" : 1.0, "_source" : {"@message":" this is a test 27"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "EKZAVd20TE6rPpoiHTooHQ",
>> "_score" : 1.0, "_source" : {"@message":" this is a test 38"}
>> }, {
>>
>> "_index" : "system",
>> "_type" : "events",
>> "_id" : "cp460Z1QRsa4etFxrJWilA",
>> "_score" : 1.0, "_source" : {"@message":" this is a test 41"}
>> } ]
>> }
>> }[root@localhost rsyslog]#
>>
>>
>>
>> On Tue, Jun 18, 2013 at 4:46 PM, Mahesh V <[email protected]
>> > wrote:
>>
>>> Hello Radu,
>>>
>>> None of the above worked.
>>> elasticsearch.yml is blank for me. (all lines are commented).
>>>
>>> logging.yml had the following lines
>>>
>>> rootLogger: INFO, file
>>> logger:
>>> # log action execution errors for easier debugging
>>> action: DEBUG
>>> # reduce the logging for aws, too much is logged under the default INFO
>>> com.amazonaws: WARN
>>>
>>>
>>> # gateway
>>> #gateway: DEBUG
>>> #index.gateway: DEBUG
>>>
>>> # peer shard recovery
>>> #index.shard.recovery: DEBUG
>>>
>>> # discovery
>>> #discovery: TRACE
>>>
>>> appender:
>>> console:
>>> type: console
>>> layout:
>>> type: consolePattern
>>> conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n"
>>>
>>> syslog:
>>> type: syslog
>>> syslogrHost: localhost:514
>>> facility: LOCAL6
>>> layout:
>>> type: pattern
>>> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>>>
>>> rootLogger: INFO, file
>>> logger:
>>> # log action execution errors for easier debugging
>>> action: DEBUG
>>> # reduce the logging for aws, too much is logged under the default INFO
>>> com.amazonaws: WARN
>>>
>>>
>>> # gateway
>>> #gateway: DEBUG
>>> #index.gateway: DEBUG
>>>
>>> # peer shard recovery
>>> #index.shard.recovery: DEBUG
>>>
>>> # discovery
>>> #discovery: TRACE
>>>
>>> appender:
>>> console:
>>> type: console
>>> layout:
>>> type: consolePattern
>>> conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n"
>>>
>>> syslog:
>>> type: syslog
>>> syslogrHost: localhost:514
>>> facility: LOCAL6
>>> layout:
>>> type: pattern
>>> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>>>
>>>
>>>
>>> The entries are in rsyslog.conf are
>>>
>>>
>>> template (name="apsimTemplate" type="list" option.json="on") {
>>> constant(value="{")
>>> constant(value="\"@message\":\"")
>>> property(name="msg")
>>> constant(value="\"}")
>>> }
>>>
>>> *.* action(type="omelasticsearch" template="apsimTemplate"
>>> server="localhost" serverport="9200")
>>>
>>>
>>>
>>>
>>> Thanks
>>> Mahesh
>>>
>>>
>>> On Tue, Jun 18, 2013 at 4:10 PM, Radu Gheorghe
>>> <[email protected]>wrote:
>>>
>>>> Interesting. The log is there, so it's just about getting it. The
>>>> correct
>>>> command to search for everything would be:
>>>>
>>>> curl 'http://localhost:9200/_search?pretty=1&q=*:*'
>>>>
>>>> Note the quotes around the URI - they're needed because the & sign will
>>>> fork the thing to background and ignore the rest. But it should still
>>>> work
>>>> only with the first part:
>>>>
>>>> curl http://localhost:9200/_search?pretty
>>>>
>>>> You can also try Elasticsearch
>>>> Head<https://github.com/mobz/elasticsearch-head>to see what's
>>>> happening. If logs don't show with curl, try refreshing first:
>>>> curl localhost:9200/_refresh
>>>>
>>>> If you still got issues, please post your elasticsearch.yml and tell
>>>> what's
>>>> your ES version.
>>>>
>>>> 2013/6/18 Mahesh V <[email protected]>
>>>>
>>>> > Hi
>>>> > I see the following when running rsyslogd -nd (debug mode)
>>>> >
>>>> > 359.892976482:7f9f631cf700: omelasticsearch: result doAction: 0
>>>> (bulkmode
>>>> > 0)
>>>> > 1359.892983879:7f9f631cf700: Action 0x7f9f6b748300 transitioned to
>>>> state:
>>>> > rdy
>>>> > 1359.892990396:7f9f631cf700: action 0x7f9f6b748300 call returned 0
>>>> > 1359.892995969:7f9f631cf700: omelasticsearch: beginTransaction
>>>> > 1359.893001589:7f9f631cf700: Action 0x7f9f6b748300 transitioned to
>>>> state:
>>>> > itx
>>>> > 1359.893007132:7f9f631cf700: entering actionCalldoAction(), state: itx
>>>> > 1359.894200154:7f9f631cf700: omelasticsearch: es reply:
>>>> >
>>>> >
>>>> '{"ok":true,"_index":"system","_type":"events","_id":"r2M4dw28QYGr-FuacMQ-SA","_version":1}'
>>>> >
>>>> > which means the insert has happened.
>>>> >
>>>> > but when I run
>>>> > curl http://localhost:9200/_search?pretty=1&q=*.*
>>>> >
>>>> > I dont see the message.
>>>> >
>>>> > /var/log/elasticsearch/elasticsearch.log has nothing as well.
>>>> >
>>>> > Where does the message go?
>>>> >
>>>> > thanks
>>>> > Mahesh
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Tue, Jun 18, 2013 at 2:21 PM, Radu Gheorghe <
>>>> [email protected]
>>>> > >wrote:
>>>> >
>>>> > > Hi Mahesh,
>>>> > >
>>>> > > Your template is suppose to output a valid JSON. Your message can be
>>>> > > whatever you want (JSON or not). Here's a reference:
>>>> > >
>>>> > >
>>>> >
>>>> http://blog.sematext.com/2013/05/28/structured-logging-with-rsyslog-and-elasticsearch/
>>>> > >
>>>> > > A valid template would be this one:
>>>> > > template (name="apsimTemplate" type="list" option.json="on") {
>>>> > > constant(value="{")
>>>> > > constant(value="\"@message\":\"")
>>>> > > property(name="msg")
>>>> > > constant(value="\"}")
>>>> > > }
>>>> > >
>>>> > >
>>>> > > Which differs only a bit from the one you pasted, which had an extra
>>>> > quote
>>>> > > and comma before @message:
>>>> > > constant(value="\",\"@message\":\"") instead of
>>>> > > constant(value="\"@message\":\"")
>>>> > >
>>>> > > Best regards,
>>>> > > Radu
>>>> > >
>>>> > > 2013/6/18 Mahesh V <[email protected]>
>>>> > >
>>>> > > > Hello,
>>>> > > >
>>>> > > > i would like to know how to format the rsyslog message in the
>>>> client
>>>> > > > programs to get it parsed by json to be entered
>>>> > > > in elasticsearch.
>>>> > > >
>>>> > > > My rsyslog.conf has the following entries
>>>> > > >
>>>> > > > template (name="apsimTemplate" type="list" option.json="on") {
>>>> > > > constant(value="{") constant(value="\",\"@message\":\"")
>>>> > > > property(name="msg") constant(value="\"}") }
>>>> > > >
>>>> > > > *.* action(type="omelasticsearch" template="apsimTemplate" )
>>>> > > >
>>>> > > >
>>>> > > > and I use syslog with the formatted message as shown here : syslog
>>>> > > > (LOG_NOTICE, "{ \"@message\":\"A tree falls in a forest %d\" }",
>>>> i);
>>>> > > >
>>>> > > > But the entries do not show up in elasticsearch.log
>>>> > > >
>>>> > > > what am I missing?
>>>> > > >
>>>> > > > thanks
>>>> > > > Mahesh
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > On Mon, Jun 17, 2013 at 7:16 PM, Mahesh V <
>>>> > > [email protected]
>>>> > > > >wrote:
>>>> > > >
>>>> > > > > Hello Folks,
>>>> > > > >
>>>> > > > > if I put the below two lines in rsyslog.conf and run a simple
>>>> > > executable
>>>> > > > > as shown
>>>> > > > > below I do not get any entries in elasticsearch logs.
>>>> > > > >
>>>> > > > > void main()
>>>> > > > > {
>>>> > > > > int i = 0;
>>>> > > > > setlogmask (LOG_UPTO (LOG_NOTICE));
>>>> > > > > openlog ("exampleprog", LOG_PID | LOG_NDELAY,
>>>> LOG_DAEMON);
>>>> > > > > perror("openlog");
>>>> > > > > for(i = 0; i < 500000; i++) {
>>>> > > > > //syslog (LOG_NOTICE, "A tree falls in a forest
>>>> %d",
>>>> > > i);
>>>> > > > > syslog (LOG_NOTICE, "{ \"@message\":\"A tree
>>>> falls
>>>> > in a
>>>> > > > > forest %d\" }", i);
>>>> > > > > //usleep(200);
>>>> > > > > }
>>>> > > > > closelog ();
>>>> > > > > }
>>>> > > > > template (name="apsimTemplate" type="list" option.json="on") {
>>>> > > > > constant(value="{") constant(value="\",\"@message\":\"")
>>>> > > > > property(name="msg") constant(value="\"}") }
>>>> > > > >
>>>> > > > > *.* action(type="omelasticsearch" template="apsimTemplate" ) #
>>>> > > > > searchIndex="srchidx")
>>>> > > > >
>>>> > > > >
>>>> > > > > whereas if I put the below line in /etc/rsyslog.conf, the lines
>>>> > appear.
>>>> > > > > jasonlint validates the line in syslog function.
>>>> > > > >
>>>> > > > > *.* /var/log/elasticsearch/elasticsearch.log
>>>> > > > > What could be the issue?
>>>> > > > > thanks
>>>> > > > > Mahesh
>>>> > > > >
>>>> > > > _______________________________________________
>>>> > > > rsyslog mailing list
>>>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> > > > http://www.rsyslog.com/professional-services/
>>>> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> > myriad
>>>> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you
>>>> > > > DON'T LIKE THAT.
>>>> > > >
>>>> > > _______________________________________________
>>>> > > rsyslog mailing list
>>>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> > > http://www.rsyslog.com/professional-services/
>>>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad
>>>> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>> you
>>>> > > DON'T LIKE THAT.
>>>> > >
>>>> > _______________________________________________
>>>> > rsyslog mailing list
>>>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> > http://www.rsyslog.com/professional-services/
>>>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad
>>>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> > DON'T LIKE THAT.
>>>> >
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>> you DON'T LIKE THAT.
>>>>
>>>
>>>
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
