Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Amnon
So you raise a couple of questions: 1) How about handling runes? The nice thing about utf8 is you don't have to care. If you are searching for the word ascii byte 'test', you can simply compare byte by byte - the letter t is represented by 0x74, and this byte in the search buffer can only repr

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread 'Dan Kortschak' via golang-nuts
On Sat, 2022-05-07 at 16:16 -0700, Const V wrote: > The question is will scanner.Scan handle a line of 100s MB? No, at least not by default (https://pkg.go.dev/bufio#Scanner.Buffer). But that that point you want to start questioning why you're doing what you're doing. Your invocation of grep can

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Const V
The question is will scanner.Scan handle a line of 100s MB? On Saturday, May 7, 2022 at 2:49:08 PM UTC-7 Amnon wrote: > How about something like > > func grep(pat []byte, r io.Reader, w io.Writer) error { > scanner := bufio.NewScanner(r) > for scanner.Scan() { > if (bytes.Contain

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Const V
Here is what came up withL func TestGrep1(t *testing.T) { cmd := exec.Command("./read.bash") fmt.Printf("%v\n", cmd) stdout, err := cmd.StdoutPipe() if err != nil { log.Fatal(err) } if err := cmd.Start(); err != nil { log.Fatal(err) } fmt.Printf("%v\n", stdout) find := []byte{'b', 'u', 'f', 'i', '

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Const V
Now the next question is if I have to handle runes. On Saturday, May 7, 2022 at 3:31:31 PM UTC-7 kortschak wrote: > On Sat, 2022-05-07 at 15:18 -0700, Amnon wrote: > > The other interesting question is what algorithm we use to find the > > pattern in each line. > > Generally bytes.Contains uses R

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread 'Dan Kortschak' via golang-nuts
On Sat, 2022-05-07 at 15:18 -0700, Amnon wrote: > The other interesting question is what algorithm we use to find the > pattern in each line. > Generally bytes.Contains uses Rabin-Karp. But as the pattern is the > word "test" which is only 4 bytes long, > a brute force search is used, using SSE typ

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Amnon
The other interesting question is what algorithm we use to find the pattern in each line. Generally bytes.Contains uses Rabin-Karp. But as the pattern is the word "test" which is only 4 bytes long, a brute force search is used, using SSE type instructions where available. So the naive Go approac

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Amnon
p.s. If you changed the above code to use strings rather than []byte it would run many times slower due to the cost of allocation. On Saturday, 7 May 2022 at 22:49:08 UTC+1 Amnon wrote: > How about something like > > func grep(pat []byte, r io.Reader, w io.Writer) error { > scanner := bufio

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Amnon
How about something like func grep(pat []byte, r io.Reader, w io.Writer) error { scanner := bufio.NewScanner(r) for scanner.Scan() { if (bytes.Contains(scanner.Bytes(), pat)) { w.Write(scanner.Bytes()) } } return scanner.Err() } and for extra speed, j

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Jan Mercl
On Sat, May 7, 2022 at 10:24 PM Constantine Vassilev wrote: > I need to write a program that reads STDIN and should output every line that > contains a search word "test" to STDOUT. Piping the data through grep(1) would be my first option. -- You received this message because you are subscrib

[go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it?

2022-05-07 Thread Const V
I need to write a program that reads STDIN and should output every line that contains a search word "test" to STDOUT. How I can test that considering the problem is a line can be 100s of MB long (\n is line end) and tens of MB info is passed to it. -- You received this message because you a

[go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

2022-05-07 Thread Constantine Vassilev
I need to write a program that reads STDIN and should output every line that contains a search word "test" to STDOUT. How I can test that considering the problem is a line can be 100s of MB long (\n is line end) and tens of MB info is passed to it. -- You received this message because you

Re: [go-nuts] Are pointers guaranteed to keep the entire pointee alive?

2022-05-07 Thread Jan Mercl
On Sat, May 7, 2022 at 2:53 PM Axel Wagner wrote: >> AFAICT, interior pointers do and must keep the entire allocation block >> alive, > > Is there any "official" documentation on the "must"? I don't think there is and I don't think it's necessary. I believe it's implied by the language semantics

Re: [go-nuts] Are pointers guaranteed to keep the entire pointee alive?

2022-05-07 Thread Robert Engels
The code is not compliant with the rules T2 is larger than T1 and no guarantee of the same layout either. > On May 7, 2022, at 7:54 AM, 'Axel Wagner' via golang-nuts > wrote: > >  > >> On Sat, May 7, 2022 at 1:20 PM Jan Mercl <0xj...@gmail.com> wrote: >> ^ Breaking the unsafe rules enables

Re: [go-nuts] Are pointers guaranteed to keep the entire pointee alive?

2022-05-07 Thread 'Axel Wagner' via golang-nuts
On Sat, May 7, 2022 at 1:20 PM Jan Mercl <0xj...@gmail.com> wrote: > ^ Breaking the unsafe rules enables the code to behave non predictably. > Sure, that's why I'm asking :) In the hope of either a) learning that it is, indeed, breaking the rules and thus should not be done, or b) learning that t

Re: [go-nuts] Are pointers guaranteed to keep the entire pointee alive?

2022-05-07 Thread Jan Mercl
On Sat, May 7, 2022 at 12:34 PM 'Axel Wagner' via golang-nuts wrote: > I am assuming this is true, but I couldn't find a definitive answer yet (or > rather, the answers seems bad): Does a pointer into any part of a value keep > the entire value alive? So, e.g. is this code safe? > > type A stru

[go-nuts] Are pointers guaranteed to keep the entire pointee alive?

2022-05-07 Thread 'Axel Wagner' via golang-nuts
Hi, I am assuming this is true, but I couldn't find a definitive answer yet (or rather, the answers seems bad): Does a pointer into any part of a value keep the entire value alive? So, e.g. is this code safe? type A struct { X int; Y int } func F() *int { a := A{42, 23} return &a.X } func