[go-nuts] Re: How to write 100MB string to STDOUT

Why not upload your entire unit test to https://go.dev/play/ so we can have a look at what you are doing? On Sunday, 8 May 2022 at 23:45:15 UTC+1 Const V wrote: > In order to test I temporarily use a pipe in order to get the string from > the stdout. > > tempStdout := os.Stdout > r, w, _ := os.

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

So what happens when you run your program > /dev/null ? For testing I would write a function that reads from an io.Reader and writes to an io.Reader. Write a unit test which uses a bytes.Buffer to catch the output. On Monday, 9 May 2022 at 04:59:37 UTC+1 Const V wrote: > I'm using OSX. > > The

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

Why don't you try redirecting stdout to /dev/null and see how your program behaves. Also, which OS are you using? On Sun, May 8, 2022 at 11:36 PM Const V wrote: > reading 1 line '\n' delimited 100MB file > r1 := bufio.NewReader(file) > s := ReadWithReadLine(r1) > InputProcessing(strings.NewRead

Re: [go-nuts] Re: GO program's memory footprint is confusing

/proc//limits of the two processes are the same. But /proc//smaps of the two processes are quite different, I have uploaded them to the attachment. 在2022年5月8日星期日 UTC+8 16:08:50 写道： > I cannot say what the difference is, but I would do two things to solve > the mystery: > > 1) Compare /proc//li

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

On May 7, 2022, at 1:24 PM, Constantine Vassilev wrote: > > I need to write a program that reads STDIN and should output every line that > contains a search word "test" to STDOUT. > > How I can test that considering the problem is a line can be 100s of MB long > (\n is line end) and tens of M

[go-nuts] Re: How to write 100MB string to STDOUT

How did you reach the conclusion that it is not working? What is your stdout connected to? On Sunday, 8 May 2022 at 22:24:16 UTC+1 Const V wrote: > I'm submitting as a separate post to focus on the following problem I have: > > writing 100MB string to STDOUT > w io.Writer > w.Write([]byte(s)) > >

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

On Sun, May 8, 2022 at 10:41 PM Const V wrote: > write to stdout is not working for MB long strings > >> >> That is very surprising indeed. How do you reach the conclusion? How can we replicate that failure? -- You received this message because you are subscribed to the Google Groups "golang-n

Re: [go-nuts] Scheduler prioritization on time sensitive routines?

No. Go has not knowledge of priority. It does do preemption. > On May 8, 2022, at 3:24 PM, TH wrote: > > Hey, > > I have few time sensitive functions that are being executed from multiples > goroutines. I'm trying to minimize latency, or make the latency more > consistent. (Not using RT ker

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

Way over complicating this. Use a buffered reader. Keep track of the position the last newline was seen. It is a trivial state machine the find ‘test’ continue to next newline. Seek to stored last newline position and buffered read and write to stdout until next newline. > On May 8, 2022, at

[go-nuts] Re: Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it?

Using r.ReadLine repeatedly I was able to read the full line in memory. It is working very fast. for isPrefix && err == nil { line, isPrefix, err = r.ReadLine() ln = append(ln, line...) } if strings.Contains(s, "error") { finds the substring very fast Now is coming the last step - writing 100M

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

pre-allocating a buffer is not an option, it should be dynamic On Sunday, May 8, 2022 at 1:24:40 PM UTC-7 Barnim Dzwillo wrote: > I had a similar use case in the past and got the best performance when > using ReadSlice() instead of scanner.Scan(). > See sample code here: https://go.dev/play/p/Ef

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

reallocating a buffer is not an option, it should be dynamic On Sunday, May 8, 2022 at 1:26:34 PM UTC-7 Const V wrote: > Using r.ReadLine() I can successfully read 100 MB line in a string, using > the following conditional statement which is > increasing the buffer until '\n' is encountered. >

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

Using r.ReadLine() I can successfully read 100 MB line in a string, using the following conditional statement which is increasing the buffer until '\n' is encountered. for isPrefix && err == nil { line, isPrefix, err = r.ReadLine() ln = append(ln, line...) } Now the last problem is how to search

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

Using r.ReadLine() I can r=successfully read 100 MB line in a string, using the following conditional statement which is increasing the buffer until '\n' is encountered. for isPrefix && err == nil { line, isPrefix, err = r.ReadLine() ln = append(ln, line...) } Now the last problem is how to sear

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

I had a similar use case in the past and got the best performance when using ReadSlice() instead of scanner.Scan(). See sample code here: https://go.dev/play/p/EfvadCURcXt On Sunday, May 8, 2022 at 7:25:29 AM UTC+2 Amnon wrote: > So you raise a couple of questions: > > 1) How about handling rune

[go-nuts] Scheduler prioritization on time sensitive routines?

Hey, I have few time sensitive functions that are being executed from multiples goroutines. I'm trying to minimize latency, or make the latency more consistent. (Not using RT kernel and no lock contention) I'm wondering if scheduler can switch context in the middle of a running function (no bl

[go-nuts] Re: Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it?

Rather than overthink it at this stage, just use bufio.Scanner and strings.Contains() and see what performance is like. I suspect that for a plain string and a large-ish file it will be about as good as it gets. On Sunday, 8 May 2022 at 09:36:15 UTC+1 Tamás Gulácsi wrote: > If that "100s of MB

Re: [go-nuts] `go run: no go files listed` issue

Hi, I had the same issue! Though I specified the file to pick as an argument and it worked! go run ./filename.go Regards, Nkoi On Friday, 9 August 2013 at 15:00:07 UTC+2 Ian Lance Taylor wrote: > On Fri, Aug 9, 2013 at 5:38 AM, Jochen Voss wrote: > > Hi Rob, > > > > > > On Friday, 9 August 20

[go-nuts] Re: Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it?

You can try to use mmap. Store in memory the start and end/size of a line, so that when a line contains your word, you can pass the address directly to libc.write, using cgo. In alternative, you can create a slice backed by the mapped memory using https://pkg.go.dev/reflect#SliceHeader. Not te

[go-nuts] Re: Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it?

If that "100s of MB" is acceptable to be in memory, then bufio.Scanner with properly sized scanner.Buffer is the easiest and battle tested solution. If not, then you have to reimplement the same, with a temp file as a buffer: read input, store in file, seeking to the start on each '\n', copying

Re: [go-nuts] Re: GO program's memory footprint is confusing

I cannot say what the difference is, but I would do two things to solve the mystery: 1) Compare /proc//limits for differences. 2) Compare the output of egrep '^Rss:|^[0-9a-f]' /proc//smaps for the processes on the different machines. Report the results and share the outputs. On Friday, May 6,