Ah, the parsing component doesn't include search, but it is a necessary part to enable searching compressed logs. CLP's searching of compressed logs is enabled by the storage format it uses (as it both compresses and indexes the logs). The parsing component is how we extract information from the logs necessary for the storage format.
We can write bindings for search as well---we only started with compression because that's what we had more requests for. To focus our effort, could you let us know more about what kind of queries you want to run? The open-source C++ code supports wildcard queries but some users prefer grep-like regex, SQL-like boolean expressions, etc. We haven't yet settled on one, especially for a binding that needs to be somewhat stable. On Friday, 18 November 2022 at 20:09:06 UTC-5 Jason E. Aten wrote: > Sorry for not being familiar with the terminology. I thought the search > through compressed logs was the most interesting part. Is that included in > the "parsing component"? > > On Fri, Nov 18, 2022 at 1:21 PM david lion <david...@gmail.com> wrote: > >> At the moment we're working on a Go binding around the parsing component >> of CLP. >> You're right, the current code is all C++, but we'll be moving the >> parsing component to be standalone and creating a C API for it. >> >> Right now, we're only creating bindings for the parsing component since >> it seems to be the common element in the use-cases we have come across, but >> we'd love to hear about any use-cases you're interested in and how we could >> support them. For example, with SeaweedFS it makes more sense to integrate >> most of the CLP logic directly into the filesystem itself. >> >> On Sunday, 13 November 2022 at 04:28:12 UTC-5 Jason E. Aten wrote: >> >>> CLP looks impressive. >>> >>> Is there is a C API then it would probably be easy to write Go bindings >>> to it. >>> >>> Since there is some kind of python integration, perhaps a C API is >>> already there. >>> I took a quick look but could only see C++ code; it would have to be an >>> actual C (not C++) API for cgo to be usable. >>> >>> Usually this is not too difficult to add to a C++ project. >>> >>> On Wednesday, November 9, 2022 at 2:14:03 AM UTC-6 Bharghava Varun Ayada >>> wrote: >>> >>>> I'm exploring CLP for a logging platform and I think either a cgo or >>>> native go library might be something that would help us. >>>> >>>> On Tuesday, 4 October 2022 at 22:03:34 UTC+5:30 david...@gmail.com >>>> wrote: >>>> >>>>> It would be awesome to help SeaweedFS, so I'm eager to hear more about >>>>> your ideas to try and understand anything we can do. >>>>> >>>>> Sorry if some of my questions are naive. At a high level I'm trying to >>>>> better understand two topics: >>>>> 1. the use cases (or features) we/CLP can help with >>>>> 2. the technical aspects/limitations that make a pure-go >>>>> implementation necessary/beneficial >>>>> >>>>> I saw that SeaweedFS currently can automatically compresses certain >>>>> file types with Gzip (very cool btw). >>>>> I can imagine that augmenting this feature with CLP for log file types >>>>> could be beneficial. >>>>> Are these the log files you're referring to? Does SeaweedFS itself >>>>> also produce log files that can benefit from CLP? Perhaps CLP could help >>>>> with both cases? >>>>> >>>>> As I imagine packaging the current form of CLP with SeaweedFS is >>>>> awkward, is packaging a main reason for a pure-go implementation? >>>>> Would a library with programmatic API be favourable for integration >>>>> rather than calling out to another tool/executable (as required in CLP's >>>>> current form)? >>>>> Would a go package using cgo (to reuse some existing cpp code) be >>>>> desirable or is there a requirement for strictly go code? >>>>> >>>>> -david >>>>> >>>>> On Sunday, 2 October 2022 at 22:36:38 UTC-4 ChrisLu wrote: >>>>> >>>>>> Thanks! CLP is great! >>>>>> >>>>>> I am working on a distributed file system, SeaweedFS, >>>>>> https://github.com/seaweedfs/seaweedfs >>>>>> I am interested in a pure-go implementation to store the log files >>>>>> more efficiently. >>>>>> >>>>>> Chris >>>>>> >>>>>> On Sun, Oct 2, 2022 at 6:45 PM david lion <david...@gmail.com> wrote: >>>>>> >>>>>>> Hi Chris, >>>>>>> >>>>>>> I'm one of the CLP developers. We'd be interested in hearing your >>>>>>> use case and any features you'd like implemented in CLP. >>>>>>> As for a Go implementation, are you asking about bindings to >>>>>>> directly call CLP functionality from a Go program or are you interested >>>>>>> in >>>>>>> a complete re-write in Go? >>>>>>> >>>>>>> -david >>>>>>> >>>>>>> On Friday, 30 September 2022 at 03:58:44 UTC-4 ChrisLu wrote: >>>>>>> >>>>>>>> Seems there are no Go implementation for Compressed Log Processor >>>>>>>> (CLP) yet? >>>>>>>> >>>>>>>> CLP is a tool capable of losslessly compressing text logs and >>>>>>>> searching the compressed logs without decompression. >>>>>>>> >>>>>>>> https://github.com/y-scope/clp >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "golang-nuts" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/golang-nuts/XeDIZfTMlX8/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> golang-nuts...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/golang-nuts/0c57905a-1096-4688-b268-93742c48316fn%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/golang-nuts/0c57905a-1096-4688-b268-93742c48316fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "golang-nuts" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/golang-nuts/XeDIZfTMlX8/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> golang-nuts...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/82c4077a-e6ce-4ac6-b944-cd5ef792a394n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/82c4077a-e6ce-4ac6-b944-cd5ef792a394n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a9e58ce0-0617-430c-80d9-263465a227a5n%40googlegroups.com.
