sorry. got the answer from http://www.elasticsearch.org/guide/reference/api/search/from-size/
On Tue, Jun 18, 2013 at 5:33 PM, Mahesh V <[email protected]>wrote: > Hi > > I changed the elasticsearch.yml with following lines > > cluster: > name: APSIM > > network: > host: localhost > > > and i ran the rsyslogd in foreground with debug enabled. > > and with this program > > > void main() > { > int i = 0; > char * string = "{\n \"name\":\"joys of programming\" \n }"; > > setlogmask (LOG_UPTO (LOG_NOTICE)); > > openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON); > perror("openlog"); > > for(i = 0; i < 50; i++) { > syslog (LOG_NOTICE, "this is a test %d", i); > > //usleep(200); > } > closelog (); > } > > > I got only some lines registered in the log. I should have had 50 of such > lines but could get only so much. > Any ideas on what could be wrong? > > [root@localhost rsyslog]# curl ' > http://localhost:9200/_search?pretty=1&q=*:*' > { > "took" : 2, > "timed_out" : false, > "_shards" : { > "total" : 5, > "successful" : 5, > "failed" : 0 > }, > "hits" : { > "total" : 108, > "max_score" : 1.0, > "hits" : [ { > > "_index" : "system", > "_type" : "events", > "_id" : "ZM-HY3IGRLaW7X_BlhN-Vw", > "_score" : 1.0, "_source" : {"@message":" [origin > software=\"rsyslogd\" swVersion=\"7.4.0\" x-pid=\"16988\" x-info=\" > http://www.rsyslog.com\";] start"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "J-EqhZMBSw64Lts2SKD-yg", > "_score" : 1.0, "_source" : {"@message":"type=1400 > audit(1371489908.934:42327): avc: denied { name_connect } for pid=16877 > comm=72733A6D61696E20513A526567 dest=9200 > scontext=unconfined_u:system_r:syslogd_t:s0 > tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "jK6sFd-CTO-0llfj_zp_zw", > "_score" : 1.0, "_source" : {"@message":"type=1400 > audit(1371489908.934:42328): avc: denied { name_connect } for pid=16877 > comm=72733A6D61696E20513A526567 dest=9200 > scontext=unconfined_u:system_r:syslogd_t:s0 > tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "QyMvKocsTXWYNdusOAuUQA", > "_score" : 1.0, "_source" : {"@message":"type=1400 > audit(1371489955.439:42329): avc: denied { write } for pid=17022 > comm=\"sshd\" name=\"log\" dev=devtmpfs ino=51770750 > scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 > tcontext=unconfined_u:object_r:device_t:s0 tclass=sock_file"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "8UvsmtlpTu22him_rKN6bw", > "_score" : 1.0, "_source" : {"@message":" this is a test 2"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "yA7VpDWmQIWOJLty_F9DEw", > "_score" : 1.0, "_source" : {"@message":" this is a test 11"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "eJGdab4NRTGeuWJfIAFa3A", > "_score" : 1.0, "_source" : {"@message":" this is a test 15"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "1vOyvn67SwmSPsZqU29rEQ", > "_score" : 1.0, "_source" : {"@message":" this is a test 27"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "EKZAVd20TE6rPpoiHTooHQ", > "_score" : 1.0, "_source" : {"@message":" this is a test 38"} > }, { > > "_index" : "system", > "_type" : "events", > "_id" : "cp460Z1QRsa4etFxrJWilA", > "_score" : 1.0, "_source" : {"@message":" this is a test 41"} > } ] > } > }[root@localhost rsyslog]# > > > > On Tue, Jun 18, 2013 at 4:46 PM, Mahesh V > <[email protected]>wrote: > >> Hello Radu, >> >> None of the above worked. >> elasticsearch.yml is blank for me. (all lines are commented). >> >> logging.yml had the following lines >> >> rootLogger: INFO, file >> logger: >> # log action execution errors for easier debugging >> action: DEBUG >> # reduce the logging for aws, too much is logged under the default INFO >> com.amazonaws: WARN >> >> >> # gateway >> #gateway: DEBUG >> #index.gateway: DEBUG >> >> # peer shard recovery >> #index.shard.recovery: DEBUG >> >> # discovery >> #discovery: TRACE >> >> appender: >> console: >> type: console >> layout: >> type: consolePattern >> conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n" >> >> syslog: >> type: syslog >> syslogrHost: localhost:514 >> facility: LOCAL6 >> layout: >> type: pattern >> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" >> >> rootLogger: INFO, file >> logger: >> # log action execution errors for easier debugging >> action: DEBUG >> # reduce the logging for aws, too much is logged under the default INFO >> com.amazonaws: WARN >> >> >> # gateway >> #gateway: DEBUG >> #index.gateway: DEBUG >> >> # peer shard recovery >> #index.shard.recovery: DEBUG >> >> # discovery >> #discovery: TRACE >> >> appender: >> console: >> type: console >> layout: >> type: consolePattern >> conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n" >> >> syslog: >> type: syslog >> syslogrHost: localhost:514 >> facility: LOCAL6 >> layout: >> type: pattern >> conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" >> >> >> >> The entries are in rsyslog.conf are >> >> >> template (name="apsimTemplate" type="list" option.json="on") { >> constant(value="{") >> constant(value="\"@message\":\"") >> property(name="msg") >> constant(value="\"}") >> } >> >> *.* action(type="omelasticsearch" template="apsimTemplate" >> server="localhost" serverport="9200") >> >> >> >> >> Thanks >> Mahesh >> >> >> On Tue, Jun 18, 2013 at 4:10 PM, Radu Gheorghe >> <[email protected]>wrote: >> >>> Interesting. The log is there, so it's just about getting it. The correct >>> command to search for everything would be: >>> >>> curl 'http://localhost:9200/_search?pretty=1&q=*:*' >>> >>> Note the quotes around the URI - they're needed because the & sign will >>> fork the thing to background and ignore the rest. But it should still >>> work >>> only with the first part: >>> >>> curl http://localhost:9200/_search?pretty >>> >>> You can also try Elasticsearch >>> Head<https://github.com/mobz/elasticsearch-head>to see what's >>> happening. If logs don't show with curl, try refreshing first: >>> curl localhost:9200/_refresh >>> >>> If you still got issues, please post your elasticsearch.yml and tell >>> what's >>> your ES version. >>> >>> 2013/6/18 Mahesh V <[email protected]> >>> >>> > Hi >>> > I see the following when running rsyslogd -nd (debug mode) >>> > >>> > 359.892976482:7f9f631cf700: omelasticsearch: result doAction: 0 >>> (bulkmode >>> > 0) >>> > 1359.892983879:7f9f631cf700: Action 0x7f9f6b748300 transitioned to >>> state: >>> > rdy >>> > 1359.892990396:7f9f631cf700: action 0x7f9f6b748300 call returned 0 >>> > 1359.892995969:7f9f631cf700: omelasticsearch: beginTransaction >>> > 1359.893001589:7f9f631cf700: Action 0x7f9f6b748300 transitioned to >>> state: >>> > itx >>> > 1359.893007132:7f9f631cf700: entering actionCalldoAction(), state: itx >>> > 1359.894200154:7f9f631cf700: omelasticsearch: es reply: >>> > >>> > >>> '{"ok":true,"_index":"system","_type":"events","_id":"r2M4dw28QYGr-FuacMQ-SA","_version":1}' >>> > >>> > which means the insert has happened. >>> > >>> > but when I run >>> > curl http://localhost:9200/_search?pretty=1&q=*.* >>> > >>> > I dont see the message. >>> > >>> > /var/log/elasticsearch/elasticsearch.log has nothing as well. >>> > >>> > Where does the message go? >>> > >>> > thanks >>> > Mahesh >>> > >>> > >>> > >>> > >>> > >>> > On Tue, Jun 18, 2013 at 2:21 PM, Radu Gheorghe < >>> [email protected] >>> > >wrote: >>> > >>> > > Hi Mahesh, >>> > > >>> > > Your template is suppose to output a valid JSON. Your message can be >>> > > whatever you want (JSON or not). Here's a reference: >>> > > >>> > > >>> > >>> http://blog.sematext.com/2013/05/28/structured-logging-with-rsyslog-and-elasticsearch/ >>> > > >>> > > A valid template would be this one: >>> > > template (name="apsimTemplate" type="list" option.json="on") { >>> > > constant(value="{") >>> > > constant(value="\"@message\":\"") >>> > > property(name="msg") >>> > > constant(value="\"}") >>> > > } >>> > > >>> > > >>> > > Which differs only a bit from the one you pasted, which had an extra >>> > quote >>> > > and comma before @message: >>> > > constant(value="\",\"@message\":\"") instead of >>> > > constant(value="\"@message\":\"") >>> > > >>> > > Best regards, >>> > > Radu >>> > > >>> > > 2013/6/18 Mahesh V <[email protected]> >>> > > >>> > > > Hello, >>> > > > >>> > > > i would like to know how to format the rsyslog message in the >>> client >>> > > > programs to get it parsed by json to be entered >>> > > > in elasticsearch. >>> > > > >>> > > > My rsyslog.conf has the following entries >>> > > > >>> > > > template (name="apsimTemplate" type="list" option.json="on") { >>> > > > constant(value="{") constant(value="\",\"@message\":\"") >>> > > > property(name="msg") constant(value="\"}") } >>> > > > >>> > > > *.* action(type="omelasticsearch" template="apsimTemplate" ) >>> > > > >>> > > > >>> > > > and I use syslog with the formatted message as shown here : syslog >>> > > > (LOG_NOTICE, "{ \"@message\":\"A tree falls in a forest %d\" }", >>> i); >>> > > > >>> > > > But the entries do not show up in elasticsearch.log >>> > > > >>> > > > what am I missing? >>> > > > >>> > > > thanks >>> > > > Mahesh >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Mon, Jun 17, 2013 at 7:16 PM, Mahesh V < >>> > > [email protected] >>> > > > >wrote: >>> > > > >>> > > > > Hello Folks, >>> > > > > >>> > > > > if I put the below two lines in rsyslog.conf and run a simple >>> > > executable >>> > > > > as shown >>> > > > > below I do not get any entries in elasticsearch logs. >>> > > > > >>> > > > > void main() >>> > > > > { >>> > > > > int i = 0; >>> > > > > setlogmask (LOG_UPTO (LOG_NOTICE)); >>> > > > > openlog ("exampleprog", LOG_PID | LOG_NDELAY, >>> LOG_DAEMON); >>> > > > > perror("openlog"); >>> > > > > for(i = 0; i < 500000; i++) { >>> > > > > //syslog (LOG_NOTICE, "A tree falls in a forest >>> %d", >>> > > i); >>> > > > > syslog (LOG_NOTICE, "{ \"@message\":\"A tree >>> falls >>> > in a >>> > > > > forest %d\" }", i); >>> > > > > //usleep(200); >>> > > > > } >>> > > > > closelog (); >>> > > > > } >>> > > > > template (name="apsimTemplate" type="list" option.json="on") { >>> > > > > constant(value="{") constant(value="\",\"@message\":\"") >>> > > > > property(name="msg") constant(value="\"}") } >>> > > > > >>> > > > > *.* action(type="omelasticsearch" template="apsimTemplate" ) # >>> > > > > searchIndex="srchidx") >>> > > > > >>> > > > > >>> > > > > whereas if I put the below line in /etc/rsyslog.conf, the lines >>> > appear. >>> > > > > jasonlint validates the line in syslog function. >>> > > > > >>> > > > > *.* /var/log/elasticsearch/elasticsearch.log >>> > > > > What could be the issue? >>> > > > > thanks >>> > > > > Mahesh >>> > > > > >>> > > > _______________________________________________ >>> > > > rsyslog mailing list >>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > http://www.rsyslog.com/professional-services/ >>> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> > myriad >>> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> > > > DON'T LIKE THAT. >>> > > > >>> > > _______________________________________________ >>> > > rsyslog mailing list >>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > http://www.rsyslog.com/professional-services/ >>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> > > DON'T LIKE THAT. >>> > > >>> > _______________________________________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > http://www.rsyslog.com/professional-services/ >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> > DON'T LIKE THAT. >>> > >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
