Multiline regex does not need to match entire message. We have Elastic support, so I turned to them for Filebeat help. The only thing I was missing was checking the negate checkbox (still not entirely sure what it does). Now it works perfectly fine with all of our messages.
--ab On Fri, Feb 17, 2017 at 5:44 AM, Jan Doberstein <[email protected]> wrote: > Hej Andy, > > maybe you should separate the multiple messages you have by type into > different log files to be able to have one pattern for every logfile. > > I didn’t dig into NXLog that deep but again - someone in the NXLog > community might help with that. > > /jd > > From: Andrew Badera <[email protected]> <[email protected]> > Reply: [email protected] <[email protected]> > <[email protected]> > Date: 17. Februar 2017 at 11:58:37 > To: [email protected] <[email protected]> > <[email protected]> > Subject: Re: [graylog2] Re: Multiline message problems > > Hi Jan, > > Thanks for the reply. > > Before I share our million different log messages, can we discuss on the > basis that a single regex won't capture our messages? We have multiline > exceptions, multiline SQL statements, multiline various other types of > messages. If NXLog multiline handling is stronger, is there anything I may > have missed in terms of NXLog setup? Are there other alternatives (other > than decorating our messages) I haven't considered, or obviously missed? > > Thanks- > --ab > > > On Fri, Feb 17, 2017 at 2:49 AM, Jan Doberstein <[email protected]> wrote: > >> Hej Andy, >> >> if you want help with the multiline detection of filebeat, we would need >> to have some information about your logfile. examples welcome. >> >> with your question about nxlog the limit for one message is reached - you >> would need to configure this limit. But for this the NXLog Community might >> be the best place to ask. >> >> regards >> Jan >> >> On Thursday, February 16, 2017 at 11:16:55 PM UTC+1, Andy Badera wrote: >>> >>> Hello all- >>> >>> Windows app server into Graylog 2.1.0. >>> >>> Like many, we have multiline log messages. There is presently no clearly >>> defined syntax around these messages, no end delimiter. >>> >>> I'm able to flow messages in using filebeat, but I can't capture >>> multiline messages properly. I believe per a Graylog blog entry, I need a >>> regex that matches the entire message. I don't think this is feasible with >>> our widely-varied messages. We do have a well-defined phrase that starts >>> every message, but I'm not sure how I would define the end of and capture >>> the varied messages. >>> >>> I've tried NXLog outputting to the system input of GELF TCP. I suspect >>> NXLog has better multiline handling, but I can't flow messages reliably >>> using NXLog - I get shut down repeatedly by the string size limit error in >>> nxlog.log: >>> >>> 2017-02-16 17:13:06 INFO connecting to 10.100.15.196:12201 >>> 2017-02-16 17:13:06 INFO reconnecting in 1 seconds >>> 2017-02-16 17:13:06 ERROR oversized string, limit is 1048576 bytes >>> >>> Is there any way for me to correct this string size limit issue using >>> NXLog CE? >>> >>> Any other alternatives I'm not considering? Anything I'm doing obviously >>> wrong, or missed? >>> >>> Thanks in advance! >>> --ab >>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Graylog Users" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/graylog2/hhVs0N5d9tQ/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/graylog2/84085e67-c94c-4a41-a045-164452b77be7%40googlegroups.com >> <https://groups.google.com/d/msgid/graylog2/84085e67-c94c-4a41-a045-164452b77be7%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/graylog2/hhVs0N5d9tQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/graylog2/CAAD%3DdiqqeCrJhmuDkEcNXOjwsNUeYOWs > 7OVzE3hagLLxH8MCLA%40mail.gmail.com > <https://groups.google.com/d/msgid/graylog2/CAAD%3DdiqqeCrJhmuDkEcNXOjwsNUeYOWs7OVzE3hagLLxH8MCLA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > — > Jan Doberstein > Support Engineer > > Phone: +49 40 609452029 > Fax: +49 40 609452030 <+49%2040%20609452030> > > TORCH GmbH - A Graylog company <https://www.graylog.com/> > Poolstraße 21 > 20355 Hamburg, Germany > > Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 > Geschäftsführer: Lennart Koopmann (CEO) > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/graylog2/hhVs0N5d9tQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/graylog2/etPan.58a6e241.3de586f9.ad4%40graylog.com > <https://groups.google.com/d/msgid/graylog2/etPan.58a6e241.3de586f9.ad4%40graylog.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAAD%3DdiqVDk6OUaAw_ostOtFrj7OujMw255D_w1WDjWXOnC67wA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
