Hi Folks,
I encountered this error but was able to fix it after setting the
scanner.Buffer size to a rather large number.
*bufio.Scanner: token too long*
Here is my fixed function:
func scanFile(f *os.File) error {
scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 0, 819200), 819200)
splitFunc := func(data []byte, atEOF bool) (advance int, token []byte, err
error) {
for i := 0; i < len(data); i++ {
if data[i] == ',' {
return i + 1, data[:i], nil
}
if !atEOF {
return 0, nil, nil
}
}
return 0, data, bufio.ErrFinalToken
}
scanner.Split(splitFunc)
newfile, err := os.Create("test/new.txt")
if err != nil {
return err
}
writer := bufio.NewWriter(newfile)
for scanner.Scan() {
n, err := writer.Write(scanner.Bytes())
if err != nil {
return err
}
fmt.Printf("%d bytes written\n", n)
}
if err := scanner.Err(); err != nil {
return err
}
return nil
}
Now, the question is when I run this function successfully, the first token
that is generated from the split function is 637 bytes.
What is taking up the buffer that I am getting the error when I set the
buffer to smaller values?
Thanks.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/3b1cf995-d052-48f7-aaf6-63875fd704edo%40googlegroups.com.
