No worries - happy to help. One last thing base64 coding is fairly trivial - a cursory shows that the padded version uses = signs. I suspect you could write a decoder that handled either during the decoding.
> On Jan 12, 2025, at 3:29 PM, Rory Campbell-Lange <r...@campbell-lange.net> > wrote: > > Thanks very much for the links, pointers and possible solution. > > Trying to read base64 standard (padded) encoded data with > base64.RawStdEncoding can produce an error such as > > illegal base64 data at input byte <n> > > Reading base64 raw (unpadded) encoded data produces the EOF error. > > I'll go with trying to read the standard encoded data up to maybe 1MB and > then switch to base64.RawStdEncoding if I hit the "illegal base64 data" > problem, maybe with reference to bufio.Reader which has most of the methods > suggested below. > > Yes, the use of a "Rewind" method would be crucial. I guess this would need > to: > 1. error if more than one buffer of data has been read > 2. else re-read from byte 0 > > Thanks again very much for these suggestions. > > Rory > >> On 12/01/25, robert engels (reng...@ix.netcom.com) wrote: >> Also, see this >> https://stackoverflow.com/questions/69753478/use-base64-stdencoding-or-base64-rawstdencoding-to-decode-base64-string-in-go >> as I expected the error should be reported earlier than the end of stream >> if the chosen format is wrong. >> >>>> On Jan 12, 2025, at 2:57 PM, robert engels <reng...@ix.netcom.com> wrote: >>> >>> Also, this is what Gemini provided which looks basically correct - but I >>> think encapsulating it with a Rewind() method would be easier to understand. >>> >>> >>> >>> While Go doesn't have a built-in PushbackReader like some other languages >>> (e.g., Java), you can implement similar functionality using a custom struct >>> and a buffer. >>> >>> Here's an example implementation: >>> >>> package main >>> >>> import ( >>> "bytes" >>> "io" >>> ) >>> >>> type PushbackReader struct { >>> reader io.Reader >>> buffer *bytes.Buffer >>> } >>> >>> func NewPushbackReader(r io.Reader) *PushbackReader { >>> return &PushbackReader{ >>> reader: r, >>> buffer: new(bytes.Buffer), >>> } >>> } >>> >>> func (p *PushbackReader) Read(b []byte) (n int, err error) { >>> if p.buffer.Len() > 0 { >>> return p.buffer.Read(b) >>> } >>> return p.reader.Read(b) >>> } >>> >>> func (p *PushbackReader) UnreadByte() error { >>> if p.buffer.Len() == 0 { >>> return io.EOF >>> } >>> lastByte := p.buffer.Bytes()[p.buffer.Len()-1] >>> p.buffer.Truncate(p.buffer.Len() - 1) >>> p.buffer.WriteByte(lastByte) >>> return nil >>> } >>> >>> func (p *PushbackReader) Unread(buf []byte) error { >>> if p.buffer.Len() == 0 { >>> return io.EOF >>> } >>> p.buffer.Write(buf) >>> return nil >>> } >>> >>> func main() { >>> // Example usage >>> r := NewPushbackReader(bytes.NewBufferString("Hello, World!")) >>> buf := make([]byte, 5) >>> r.Read(buf) >>> r.UnreadByte() >>> r.Read(buf) >>> } >>> >>> Explanation: >>> PushbackReader struct: This struct holds the underlying io.Reader and a >>> buffer to store the pushed-back bytes. >>> NewPushbackReader: This function creates a new PushbackReader from an >>> existing io.Reader. >>> Read method: This method reads bytes from either the buffer (if it contains >>> data) or the underlying reader. >>> UnreadByte method: This method pushes back a single byte into the buffer. >>> Unread method: This method pushes back a slice of bytes into the buffer. >>> Important Considerations: >>> The buffer size is not managed automatically. You may need to adjust the >>> buffer size based on your use case. >>> This implementation does not handle pushing back beyond the initially read >>> data. If you need to support arbitrary pushback, you'll need a more complex >>> solution. >>> >>> Generative AI is experimental. >>> >>>> On Jan 12, 2025, at 2:53 PM, Robert Engels <reng...@ix.netcom.com> wrote: >>>> >>>> You can see the two pass reader here >>>> https://stackoverflow.com/questions/20666594/how-can-i-push-bytes-into-a-reader-in-go >>>> >>>> But yea, the basic premise is that you buffer the data so you can rewind >>>> if needed >>>> >>>> Are you certain it is reading to the end to return EOF? It may be >>>> returning eof once the parsing fails. >>>> >>>> Otherwise I would expect this is being decoded wrong - eg the mime type or >>>> encoding type should tell you the correct format before you start decoding. >>>> >>>>> On Jan 12, 2025, at 2:46 PM, Rory Campbell-Lange >>>>> <r...@campbell-lange.net> wrote: >>>>> >>>>> Thanks for the suggestion of a ReadSeeker to wrap an io.Reader. >>>>> >>>>> My google fu must be deserting me. I can find PushbackReader >>>>> implementations in Java, but the only similar thing for Go I could find >>>>> was https://gitlab.com/osaki-lab/iowrapper. If you have a specific >>>>> recommendation for a ReadSeeker wrapper to an io.Reader that would be >>>>> great to know. >>>>> >>>>> Since the base64 decoding error I'm looking for is an EOF, I guess the >>>>> wrapper approach will not work when the EOF byte position is > than the >>>>> io.ReadSeeker buffer size. >>>>> >>>>> Rory >>>>> >>>>> On 12/01/25, robert engels (reng...@ix.netcom.com) wrote: >>>>>> create a ReadSeeker that wraps the Reader providing the buffering (mark >>>>>> & reset) - normally the buffer only needs to be large enough to detect >>>>>> the format contained in the Reader. >>>>>> >>>>>> You can search Google for PushbackReader in Go and you’ll get a basic >>>>>> implementation. >>>>>> >>>>>>> On Jan 12, 2025, at 12:52 PM, Rory Campbell-Lange >>>>>>> <r...@campbell-lange.net> wrote: >>>>> ... >>>>>>> I'm attempting to rationalise the process [of avoiding reading email >>>>>>> parts into byte slices] by simply wrapping the provided io.Reader with >>>>>>> the necessary decoders to reduce memory usage and unnecessary >>>>>>> processing. >>>>>>> >>>>>>> The wrapping strategy seems to work ok. However there is a particular >>>>>>> issue in detecting base64.StdEncoding versus base64.RawStdEncoding, >>>>>>> which requires draining the io.Reader using base64.StdEncoding and >>>>>>> (based on the current implementation) switching to >>>>>>> base64.RawStdEncoding if an io.ErrUnexpectedEOF is found. >>>>>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to golang-nuts+unsubscr...@googlegroups.com >>>> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/golang-nuts/DD0C1480-D237-447A-B978-78FC8951FE05%40ix.netcom.com >>>> >>>> <https://groups.google.com/d/msgid/golang-nuts/DD0C1480-D237-447A-B978-78FC8951FE05%40ix.netcom.com?utm_medium=email&utm_source=footer>. >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/90FBDABA-BFEF-4C88-A305-2485866B35C1%40ix.netcom.com.
