Hi
I changed the elasticsearch.yml with following lines
cluster:
name: APSIM
network:
host: localhost
and i ran the rsyslogd in foreground with debug enabled.
and with this program
void main()
{
int i = 0;
char * string = "{\n \"name\":\"joys of programming\" \n }";
setlogmask (LOG_UPTO (LOG_NOTICE));
openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
perror("openlog");
for(i = 0; i < 50; i++) {
syslog (LOG_NOTICE, "this is a test %d", i);
//usleep(200);
}
closelog ();
}
I got only some lines registered in the log. I should have had 50 of such
lines but could get only so much.
Any ideas on what could be wrong?
[root@localhost rsyslog]# curl '
http://localhost:9200/_search?pretty=1&q=*:*'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 108,
"max_score" : 1.0,
"hits" : [ {
"_index" : "system",
"_type" : "events",
"_id" : "ZM-HY3IGRLaW7X_BlhN-Vw",
"_score" : 1.0, "_source" : {"@message":" [origin
software=\"rsyslogd\" swVersion=\"7.4.0\" x-pid=\"16988\" x-info=\"
http://www.rsyslog.com\";] start"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "J-EqhZMBSw64Lts2SKD-yg",
"_score" : 1.0, "_source" : {"@message":"type=1400
audit(1371489908.934:42327): avc: denied { name_connect } for pid=16877
comm=72733A6D61696E20513A526567 dest=9200
scontext=unconfined_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "jK6sFd-CTO-0llfj_zp_zw",
"_score" : 1.0, "_source" : {"@message":"type=1400
audit(1371489908.934:42328): avc: denied { name_connect } for pid=16877
comm=72733A6D61696E20513A526567 dest=9200
scontext=unconfined_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "QyMvKocsTXWYNdusOAuUQA",
"_score" : 1.0, "_source" : {"@message":"type=1400
audit(1371489955.439:42329): avc: denied { write } for pid=17022
comm=\"sshd\" name=\"log\" dev=devtmpfs ino=51770750
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:device_t:s0 tclass=sock_file"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "8UvsmtlpTu22him_rKN6bw",
"_score" : 1.0, "_source" : {"@message":" this is a test 2"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "yA7VpDWmQIWOJLty_F9DEw",
"_score" : 1.0, "_source" : {"@message":" this is a test 11"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "eJGdab4NRTGeuWJfIAFa3A",
"_score" : 1.0, "_source" : {"@message":" this is a test 15"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "1vOyvn67SwmSPsZqU29rEQ",
"_score" : 1.0, "_source" : {"@message":" this is a test 27"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "EKZAVd20TE6rPpoiHTooHQ",
"_score" : 1.0, "_source" : {"@message":" this is a test 38"}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "cp460Z1QRsa4etFxrJWilA",
"_score" : 1.0, "_source" : {"@message":" this is a test 41"}
} ]
}
}[root@localhost rsyslog]#
On Tue, Jun 18, 2013 at 4:46 PM, Mahesh V <[email protected]>wrote:
> Hello Radu,
>
> None of the above worked.
> elasticsearch.yml is blank for me. (all lines are commented).
>
> logging.yml had the following lines
>
> rootLogger: INFO, file
> logger:
> # log action execution errors for easier debugging
> action: DEBUG
> # reduce the logging for aws, too much is logged under the default INFO
> com.amazonaws: WARN
>
>
> # gateway
> #gateway: DEBUG
> #index.gateway: DEBUG
>
> # peer shard recovery
> #index.shard.recovery: DEBUG
>
> # discovery
> #discovery: TRACE
>
> appender:
> console:
> type: console
> layout:
> type: consolePattern
> conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n"
>
> syslog:
> type: syslog
> syslogrHost: localhost:514
> facility: LOCAL6
> layout:
> type: pattern
> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>
> rootLogger: INFO, file
> logger:
> # log action execution errors for easier debugging
> action: DEBUG
> # reduce the logging for aws, too much is logged under the default INFO
> com.amazonaws: WARN
>
>
> # gateway
> #gateway: DEBUG
> #index.gateway: DEBUG
>
> # peer shard recovery
> #index.shard.recovery: DEBUG
>
> # discovery
> #discovery: TRACE
>
> appender:
> console:
> type: console
> layout:
> type: consolePattern
> conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n"
>
> syslog:
> type: syslog
> syslogrHost: localhost:514
> facility: LOCAL6
> layout:
> type: pattern
> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>
>
>
> The entries are in rsyslog.conf are
>
>
> template (name="apsimTemplate" type="list" option.json="on") {
> constant(value="{")
> constant(value="\"@message\":\"")
> property(name="msg")
> constant(value="\"}")
> }
>
> *.* action(type="omelasticsearch" template="apsimTemplate"
> server="localhost" serverport="9200")
>
>
>
>
> Thanks
> Mahesh
>
>
> On Tue, Jun 18, 2013 at 4:10 PM, Radu Gheorghe <[email protected]>wrote:
>
>> Interesting. The log is there, so it's just about getting it. The correct
>> command to search for everything would be:
>>
>> curl 'http://localhost:9200/_search?pretty=1&q=*:*'
>>
>> Note the quotes around the URI - they're needed because the & sign will
>> fork the thing to background and ignore the rest. But it should still work
>> only with the first part:
>>
>> curl http://localhost:9200/_search?pretty
>>
>> You can also try Elasticsearch
>> Head<https://github.com/mobz/elasticsearch-head>to see what's
>> happening. If logs don't show with curl, try refreshing first:
>> curl localhost:9200/_refresh
>>
>> If you still got issues, please post your elasticsearch.yml and tell
>> what's
>> your ES version.
>>
>> 2013/6/18 Mahesh V <[email protected]>
>>
>> > Hi
>> > I see the following when running rsyslogd -nd (debug mode)
>> >
>> > 359.892976482:7f9f631cf700: omelasticsearch: result doAction: 0
>> (bulkmode
>> > 0)
>> > 1359.892983879:7f9f631cf700: Action 0x7f9f6b748300 transitioned to
>> state:
>> > rdy
>> > 1359.892990396:7f9f631cf700: action 0x7f9f6b748300 call returned 0
>> > 1359.892995969:7f9f631cf700: omelasticsearch: beginTransaction
>> > 1359.893001589:7f9f631cf700: Action 0x7f9f6b748300 transitioned to
>> state:
>> > itx
>> > 1359.893007132:7f9f631cf700: entering actionCalldoAction(), state: itx
>> > 1359.894200154:7f9f631cf700: omelasticsearch: es reply:
>> >
>> >
>> '{"ok":true,"_index":"system","_type":"events","_id":"r2M4dw28QYGr-FuacMQ-SA","_version":1}'
>> >
>> > which means the insert has happened.
>> >
>> > but when I run
>> > curl http://localhost:9200/_search?pretty=1&q=*.*
>> >
>> > I dont see the message.
>> >
>> > /var/log/elasticsearch/elasticsearch.log has nothing as well.
>> >
>> > Where does the message go?
>> >
>> > thanks
>> > Mahesh
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jun 18, 2013 at 2:21 PM, Radu Gheorghe <[email protected]
>> > >wrote:
>> >
>> > > Hi Mahesh,
>> > >
>> > > Your template is suppose to output a valid JSON. Your message can be
>> > > whatever you want (JSON or not). Here's a reference:
>> > >
>> > >
>> >
>> http://blog.sematext.com/2013/05/28/structured-logging-with-rsyslog-and-elasticsearch/
>> > >
>> > > A valid template would be this one:
>> > > template (name="apsimTemplate" type="list" option.json="on") {
>> > > constant(value="{")
>> > > constant(value="\"@message\":\"")
>> > > property(name="msg")
>> > > constant(value="\"}")
>> > > }
>> > >
>> > >
>> > > Which differs only a bit from the one you pasted, which had an extra
>> > quote
>> > > and comma before @message:
>> > > constant(value="\",\"@message\":\"") instead of
>> > > constant(value="\"@message\":\"")
>> > >
>> > > Best regards,
>> > > Radu
>> > >
>> > > 2013/6/18 Mahesh V <[email protected]>
>> > >
>> > > > Hello,
>> > > >
>> > > > i would like to know how to format the rsyslog message in the client
>> > > > programs to get it parsed by json to be entered
>> > > > in elasticsearch.
>> > > >
>> > > > My rsyslog.conf has the following entries
>> > > >
>> > > > template (name="apsimTemplate" type="list" option.json="on") {
>> > > > constant(value="{") constant(value="\",\"@message\":\"")
>> > > > property(name="msg") constant(value="\"}") }
>> > > >
>> > > > *.* action(type="omelasticsearch" template="apsimTemplate" )
>> > > >
>> > > >
>> > > > and I use syslog with the formatted message as shown here : syslog
>> > > > (LOG_NOTICE, "{ \"@message\":\"A tree falls in a forest %d\" }", i);
>> > > >
>> > > > But the entries do not show up in elasticsearch.log
>> > > >
>> > > > what am I missing?
>> > > >
>> > > > thanks
>> > > > Mahesh
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jun 17, 2013 at 7:16 PM, Mahesh V <
>> > > [email protected]
>> > > > >wrote:
>> > > >
>> > > > > Hello Folks,
>> > > > >
>> > > > > if I put the below two lines in rsyslog.conf and run a simple
>> > > executable
>> > > > > as shown
>> > > > > below I do not get any entries in elasticsearch logs.
>> > > > >
>> > > > > void main()
>> > > > > {
>> > > > > int i = 0;
>> > > > > setlogmask (LOG_UPTO (LOG_NOTICE));
>> > > > > openlog ("exampleprog", LOG_PID | LOG_NDELAY,
>> LOG_DAEMON);
>> > > > > perror("openlog");
>> > > > > for(i = 0; i < 500000; i++) {
>> > > > > //syslog (LOG_NOTICE, "A tree falls in a forest
>> %d",
>> > > i);
>> > > > > syslog (LOG_NOTICE, "{ \"@message\":\"A tree falls
>> > in a
>> > > > > forest %d\" }", i);
>> > > > > //usleep(200);
>> > > > > }
>> > > > > closelog ();
>> > > > > }
>> > > > > template (name="apsimTemplate" type="list" option.json="on") {
>> > > > > constant(value="{") constant(value="\",\"@message\":\"")
>> > > > > property(name="msg") constant(value="\"}") }
>> > > > >
>> > > > > *.* action(type="omelasticsearch" template="apsimTemplate" ) #
>> > > > > searchIndex="srchidx")
>> > > > >
>> > > > >
>> > > > > whereas if I put the below line in /etc/rsyslog.conf, the lines
>> > appear.
>> > > > > jasonlint validates the line in syslog function.
>> > > > >
>> > > > > *.* /var/log/elasticsearch/elasticsearch.log
>> > > > > What could be the issue?
>> > > > > thanks
>> > > > > Mahesh
>> > > > >
>> > > > _______________________________________________
>> > > > rsyslog mailing list
>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > http://www.rsyslog.com/professional-services/
>> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> > myriad
>> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>> > > > DON'T LIKE THAT.
>> > > >
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > > DON'T LIKE THAT.
>> > >
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > DON'T LIKE THAT.
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
