> In CSV a line break inside a field is easy to process for a parser, because (per https://tools.ietf.org/html/rfc4180): >"Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes"
Interesting, does that implicitly mean the whole log event would get transmitted as a "line" (with CRLF) in CSV. Nor do I know how to confirm of falsify that... In the affirmative scenario, this then would work for a true streaming aggregator (if CSV was supported). Wouldn't it continue to be resource expensive if for some reason someone wants to continue to do log tailing out of a log file, which as I understand it is lined by line semantics? I do not plan to do that, but there are people who prefer to also have a local copy of their logfile just in case. But if CSV, as emitted by postgres, would be supported by fluentbit I would be equally happy with that solution. The second order problem, after all is just that: of second order... El lun., 16 abr. 2018, 1:28 p.m., Daniel Verite <dan...@manitou-mail.org> escribió: > David Arnold wrote: > > > Not claiming this assumption does imply parsing of a *rolling* set > > of log lines with *previously unkown cardinality*. That's expensive > > on computing resources. I don't have actual numbers, but it doesn't > > seem too far fetched, neither. > > I filed a question to the author of fluent-bit to that extend which > > you can consult here: > > https://github.com/fluent/fluent-bit/issues/564 Let's see what > > Eduardo has to inform us about this... > > fluent-bit does not appear to support CSV, as mentioned in > https://github.com/fluent/fluent-bit/issues/459 > which got flagged as an enhancement request some time ago. > > In CSV a line break inside a field is easy to process for > a parser, because (per https://tools.ietf.org/html/rfc4180): > > "Fields containing line breaks (CRLF), double quotes, and commas > should be enclosed in double-quotes" > > So there is no look ahead to do. In a character-by-character loop, > when encountering a line break, either the current field did not > start with a double quote and the line break is part of the content, or it > did > start with a double quote and the line break ends the current record. > > What doesn't quite work is to parse CSV with a regex, it's > discussed in some detail here for instance: > > https://softwareengineering.stackexchange.com/questions/166454/can-the-csv-format-be-defined-by-a-regex > > > Best regards, > -- > Daniel Vérité > PostgreSQL-powered mailer: http://www.manitou-mail.org > Twitter: @DanielVerite >
