On Sat, Mar 16, 2019 at 11:10 PM <caijia...@bytedance.com> wrote:
>
> I'm not sure if the log produce more than consume by the func 'centrale' in
> the same period of time, will 'buff' be larger and larger and cost too much
> memory.
> For example, The log comes from network and write into a disk but network
> speed is greater than disk write speed. So is there any way to control the
> 'buff' length.
There might be a simpler solution, but it needs to be tuned and tested
to see if it behaves better: you can use a buffered string channel to
store log messages. Your writer goroutine would read from this
channel, and store the log messages in an internal bytes.Buffer{}
until it reaches a certain size, and then writes it out. This should
limit the memory usage based on the channel buffer size and buffer
threshold. You should also have a time.Ticker to write the buffer
periodically if log generation slows down.
>
> On Sunday, March 17, 2019 at 12:02:06 AM UTC+8, Thomas S wrote:
>>
>> Hello,
>>
>> I have a software needing a lot of logs when processing.
>>
>> I need to :
>> 1- Improve the processing time by doing non-blocking logs
>> 2- Be safe with goroutines parallel logs
>>
>> fmt package doesn't match with (1) & (2)
>> log package doesn't match with (1)
>>
>> Here is my proposal. What do you think ?
>>
>> Design :
>>
>> [Log functions] -----channel---->[Buffering function (goroutine)]
>> ----channel----> [Printing function (goroutine)]
>>
>>
>> package glog
>>
>> import (
>> "container/list"
>> "encoding/json"
>> "fmt"
>> )
>>
>> /*
>> ** Public
>> */
>>
>> func Println(data ...interface{}) {
>> bufferChan <- fmt.Sprintln(data...)
>> }
>>
>> func Print(data ...interface{}) {
>> bufferChan <- fmt.Sprint(data...)
>> }
>>
>> func Printf(s string, data ...interface{}) {
>> go func() {
>> r := fmt.Sprintf(s, data...)
>> bufferChan <- r
>> }()
>> }
>>
>> /*
>> ** Private
>> */
>>
>> var bufferChan chan string
>> var outChan chan string
>>
>> func init() {
>> bufferChan = make(chan string)
>> outChan = make(chan string)
>> go centrale()
>> go buffout()
>> }
>>
>> func centrale() {
>> var buff *list.List
>> buff = list.New()
>> for {
>> if buff.Len() > 0 {
>> select {
>> case outChan <- buff.Front().Value.(string):
>> buff.Remove(buff.Front())
>> case tmp := <-bufferChan:
>> buff.PushBack(tmp)
>> }
>>
>> } else {
>> tmp := <-bufferChan
>> buff.PushBack(tmp)
>> }
>> }
>> }
>>
>> func buffout() {
>> for {
>> data := <-outChan
>> fmt.Print(data)
>> }
>> }
>>
>>
>>
>> It works well for now, I want to be sure to not miss anything as it's a very
>> important part of the code.
>>
>> Thank you for your review.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
