[go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread Lealem Amedie 


Hi,

I’m trying to get an https server working by overwriting the 
accept/read/write methods in the http module. I’m using go-wolfssl for TLS.

The server will accept a TLS 1.2 connection, then send some random data to 
the client connects, then wait for another connection.

The issue that I’m seeing is that whenever the server sends large payloads, 
it’ll do that successfully but it then can’t send data to the next client 
that connects (the send errors out with the TCP error EBADF). If dealing 
with smaller payloads (<30k), the server can send data successfully to each 
client that connects. Then as soon as a large one is sent, the next 
transmission fails.

If I disable the GOGC with debug.SetGCPercent(-1), the issue goes away and 
I can send as many large payloads as required. From the debugging I’ve done 
so far, this looks like an issue with the GO garbage collector dropping C 
pointers. go-wolfssl relies on the wolfSSL C library so it uses CGO. Does 
anyone have any other ideas or input?

Code is below. See this repo to actually run it GitHub - 
lealem47/go-wolfssl-https-server  


Thank you!


```

package main import ( "bytes" "crypto/rand" "encoding/base64" "fmt" log 
"github.com/sirupsen/logrus" wolfSSL "github.com/wolfssl/go-wolfssl" "net" 
"net/http" "os" "strconv" "sync" "time" ) const defaultPort = "8443" type 
wolfSSLListener struct { listener net.Listener ctx *wolfSSL.WOLFSSL_CTX } 
// Accept waits for and returns the next connection to the listener. func 
(cl *wolfSSLListener) Accept() (net.Conn, error) { conn, err := 
cl.listener.Accept() if err != nil { return nil, err } 
fmt.Println("Accepted new connection from:", conn.RemoteAddr()) ssl := 
wolfSSL.WolfSSL_new(cl.ctx) if ssl == nil { fmt.Println("WolfSSL_new 
Failed") os.Exit(1) } file, err := conn.(*net.TCPConn).File() if err != nil 
{ panic(err) } fd := file.Fd() wolfSSL.WolfSSL_set_fd(ssl, int(fd)) ret := 
wolfSSL.WolfSSL_accept(ssl) if ret != wolfSSL.WOLFSSL_SUCCESS { 
fmt.Println("WolfSSL_accept error ", ret) } else { fmt.Println("Client 
Successfully Connected!") } return &wolfSSLConn{ conn: conn, ssl: ssl, }, 
nil } // Close closes the listener, making it stop accepting new 
connections. func (cl *wolfSSLListener) Close() error { 
fmt.Println("Closing listener...") return cl.listener.Close() } // Addr 
returns the listener's network address. func (cl *wolfSSLListener) Addr() 
net.Addr { return cl.listener.Addr() } type wolfSSLConn struct { conn 
net.Conn ssl *wolfSSL.WOLFSSL buffer bytes.Buffer mu sync.Mutex closed bool 
} func (w *wolfSSLConn) Read(b []byte) (int, error) { log.Infof("Calling 
read: %d", len(b)) ret := wolfSSL.WolfSSL_read(w.ssl, b, uintptr(len(b))) 
if ret < 0 { errCode := wolfSSL.WolfSSL_get_error(w.ssl, int(ret)) return 
0, fmt.Errorf("read error: %d", errCode) } log.Infof("Read bytes: %s", 
string(b[:ret])) return int(ret), nil } func (w *wolfSSLConn) Write(b 
[]byte) (int, error) { log.Infof("Calling write: %d", len(b)) sz := 
uintptr(len(b)) ret := wolfSSL.WolfSSL_write(w.ssl, b, sz) if ret < 0 { 
errCode := wolfSSL.WolfSSL_get_error(w.ssl, int(ret)) return 0, 
fmt.Errorf("write error: %d", errCode) } return int(ret), nil } func (w 
*wolfSSLConn) Close() error { log.Infof("Closing connection") 
wolfSSL.WolfSSL_shutdown(w.ssl) wolfSSL.WolfSSL_free(w.ssl) return 
w.conn.Close() } func (w *wolfSSLConn) LocalAddr() net.Addr { return 
w.conn.LocalAddr() } func (w *wolfSSLConn) RemoteAddr() net.Addr { return 
w.conn.RemoteAddr() } func (w *wolfSSLConn) SetDeadline(t time.Time) error 
{ return w.conn.SetDeadline(t) } func (w *wolfSSLConn) SetReadDeadline(t 
time.Time) error { return w.conn.SetReadDeadline(t) } func (w *wolfSSLConn) 
SetWriteDeadline(t time.Time) error { return w.conn.SetWriteDeadline(t) } 
// Handler for generating and base64 encoding 5KB of random data func 
randomDataHandler(w http.ResponseWriter, r *http.Request) { // Get the 
"size" query parameter from the request sizeParam := 
r.URL.Query().Get("size") size := 50 // default size // If the "size" 
parameter is provided, convert it to an integer if sizeParam != "" { 
parsedSize, err := strconv.Atoi(sizeParam) if err != nil || parsedSize <= 0 
{ http.Error(w, "Invalid size parameter", http.StatusBadRequest) return } 
size = parsedSize } // Generate random data of the specified size data := 
make([]byte, size) _, err := rand.Read(data) if err != nil { http.Error(w, 
"Could not generate random data", http.StatusInternalServerError) return } 
// Base64 encode the random data encodedData := 
base64.StdEncoding.EncodeToString(data) // Set content type and write the 
base64 encoded data w.Header().Set("Content-Type", "application/base64") 
w.Write([]byte(encodedData)) } func main() { port := defaultPort // Set 
logging level log.SetLevel(log.InfoLevel) 
log.SetFormatter(&log.TextFormatter{ DisableColors: false, FullTimestamp: 
true, }) // Set up the HTTP server and routes http.HandleFunc("

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread Ian Lance Taylor 
On Mon, Dec 2, 2024 at 9:32 AM Lealem Amedie  wrote:
>
> I’m trying to get an https server working by overwriting the 
> accept/read/write methods in the http module. I’m using go-wolfssl for TLS.
>
> The server will accept a TLS 1.2 connection, then send some random data to 
> the client connects, then wait for another connection.
>
> The issue that I’m seeing is that whenever the server sends large payloads, 
> it’ll do that successfully but it then can’t send data to the next client 
> that connects (the send errors out with the TCP error EBADF). If dealing with 
> smaller payloads (<30k), the server can send data successfully to each client 
> that connects. Then as soon as a large one is sent, the next transmission 
> fails.
>
> If I disable the GOGC with debug.SetGCPercent(-1), the issue goes away and I 
> can send as many large payloads as required. From the debugging I’ve done so 
> far, this looks like an issue with the GO garbage collector dropping C 
> pointers. go-wolfssl relies on the wolfSSL C library so it uses CGO. Does 
> anyone have any other ideas or input?
>
> Code is below. See this repo to actually run it GitHub - 
> lealem47/go-wolfssl-https-server

If the program uses cgo, and the problem is fixed by disabling the
garbage collector, then it is most likely the case that the cgo code
is not following the rules described at
https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers.  You could try
setting GOEXPERIMENT=cgocheck2 in the environment when building the
program; that will enable stricter checks of cgo pointers, though it
still won't catch all possible misuses in the C code.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVEMsv%3DLGXna6tR0AAtRr6615YtzqYN5g1wnwOgEg_TGA%40mail.gmail.com.

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread robert engels 
Apologies - didn’t look closely - it doesn’t appear the buffer in the 
connection is used at all. I suspect though if you read the wolfssl api that 
the implementation may be async - meaning the data must remain referenced until 
it is sent.

> On Dec 2, 2024, at 11:38 AM, robert engels  wrote:
> 
> You are using the same buffer for reading and writing - I suspect that is 
> your problem...
> 
>> On Dec 2, 2024, at 11:00 AM, Lealem Amedie  wrote:
>> 
>> Hi,
>> 
>> I’m trying to get an https server working by overwriting the 
>> accept/read/write methods in the http module. I’m using go-wolfssl for TLS.
>> 
>> The server will accept a TLS 1.2 connection, then send some random data to 
>> the client connects, then wait for another connection.
>> 
>> The issue that I’m seeing is that whenever the server sends large payloads, 
>> it’ll do that successfully but it then can’t send data to the next client 
>> that connects (the send errors out with the TCP error EBADF). If dealing 
>> with smaller payloads (<30k), the server can send data successfully to each 
>> client that connects. Then as soon as a large one is sent, the next 
>> transmission fails.
>> 
>> If I disable the GOGC with debug.SetGCPercent(-1), the issue goes away and I 
>> can send as many large payloads as required. From the debugging I’ve done so 
>> far, this looks like an issue with the GO garbage collector dropping C 
>> pointers. go-wolfssl relies on the wolfSSL C library so it uses CGO. Does 
>> anyone have any other ideas or input?
>> 
>> Code is below. See this repo to actually run it GitHub - 
>> lealem47/go-wolfssl-https-server  
>> 
>> Thank you!
>> 
>> 
>> 
>> ```
>> 
>> package main
>> 
>> import (
>>  "bytes"
>>  "crypto/rand"
>>  "encoding/base64"
>>  "fmt"
>>  log "github.com/sirupsen/logrus"
>>  wolfSSL "github.com/wolfssl/go-wolfssl"
>>  "net"
>>  "net/http"
>>  "os"
>>  "strconv"
>>  "sync"
>>  "time"
>> )
>> 
>> const defaultPort = "8443"
>> 
>> type wolfSSLListener struct {
>>  listener net.Listener
>>  ctx  *wolfSSL.WOLFSSL_CTX
>> }
>> 
>> // Accept waits for and returns the next connection to the listener.
>> func (cl *wolfSSLListener) Accept() (net.Conn, error) {
>>  conn, err := cl.listener.Accept()
>>  if err != nil {
>>  return nil, err
>>  }
>>  fmt.Println("Accepted new connection from:", conn.RemoteAddr())
>> 
>>  ssl := wolfSSL.WolfSSL_new(cl.ctx)
>>  if ssl == nil {
>>  fmt.Println("WolfSSL_new Failed")
>>  os.Exit(1)
>>  }
>> 
>>  file, err := conn.(*net.TCPConn).File()
>>  if err != nil {
>>  panic(err)
>>  }
>>  fd := file.Fd()
>>  wolfSSL.WolfSSL_set_fd(ssl, int(fd))
>> 
>>  ret := wolfSSL.WolfSSL_accept(ssl)
>>  if ret != wolfSSL.WOLFSSL_SUCCESS {
>>  fmt.Println("WolfSSL_accept error ", ret)
>>  } else {
>>  fmt.Println("Client Successfully Connected!")
>>  }
>> 
>>  return &wolfSSLConn{
>>  conn: conn,
>>  ssl:  ssl,
>>  }, nil
>> }
>> 
>> // Close closes the listener, making it stop accepting new connections.
>> func (cl *wolfSSLListener) Close() error {
>>  fmt.Println("Closing listener...")
>>  return cl.listener.Close()
>> }
>> 
>> // Addr returns the listener's network address.
>> func (cl *wolfSSLListener) Addr() net.Addr {
>>  return cl.listener.Addr()
>> }
>> 
>> type wolfSSLConn struct {
>>  conn   net.Conn
>>  ssl*wolfSSL.WOLFSSL
>>  buffer bytes.Buffer
>>  mu sync.Mutex
>>  closed bool
>> }
>> 
>> func (w *wolfSSLConn) Read(b []byte) (int, error) {
>>  log.Infof("Calling read: %d", len(b))
>> 
>>  ret := wolfSSL.WolfSSL_read(w.ssl, b, uintptr(len(b)))
>>  if ret < 0 {
>>  errCode := wolfSSL.WolfSSL_get_error(w.ssl, int(ret))
>>  return 0, fmt.Errorf("read error: %d", errCode)
>>  }
>> 
>>  log.Infof("Read bytes: %s", string(b[:ret]))
>>  return int(ret), nil
>> }
>> 
>> func (w *wolfSSLConn) Write(b []byte) (int, error) {
>>  log.Infof("Calling write: %d", len(b))
>> 
>>  sz := uintptr(len(b))
>> 
>>  ret := wolfSSL.WolfSSL_write(w.ssl, b, sz)
>>  if ret < 0 {
>>  errCode := wolfSSL.WolfSSL_get_error(w.ssl, int(ret))
>>  return 0, fmt.Errorf("write error: %d", errCode)
>>  }
>> 
>>  return int(ret), nil
>> }
>> 
>> func (w *wolfSSLConn) Close() error {
>>  log.Infof("Closing connection")
>> 
>>  wolfSSL.WolfSSL_shutdown(w.ssl)
>>  wolfSSL.WolfSSL_free(w.ssl)
>>  return w.conn.Close()
>> }
>> 
>> func (w *wolfSSLConn) LocalAddr() net.Addr {
>>  return w.conn.LocalAddr()
>> }
>> 
>> func (w *wolfSSLConn) RemoteAddr() net.Addr {
>>  return w.conn.RemoteAddr()
>> }
>> 
>> func (w *wolfSSLConn) SetDeadline(t time.Time) error {
>>  return w.conn.SetDeadline(t)
>> }

Re: [go-nuts] Scalar Replacement of Aggregates (SROA)

2024-12-02 Thread Ian Lance Taylor 
On Sat, Nov 30, 2024 at 7:37 PM Pepper Lebeck-Jobe  wrote:
>
> Summary: Would we be open to adding SROA as a compiler optimization to the go 
> compiler?
>
> I recently discovered via this rathole that SROA is something that Clang does 
> for the programming languages which compile with it. The implementation is 
> here.
>
> I also think that the lack of this compiler optimization is why there is a 
> notable performance difference in the two benchmarks mentioned in 
> https://github.com/golang/go/issues/49785
>
> On the one hand, I have heard that the go team generally favors fast 
> compilation times over compiler complexity and slower build times. On the 
> other hand, I suspect that this optimization could really speed up a lot of 
> go projects that exist in the wild.
>
> Before really being surprised by the differences between the C and go 
> performance in the bddicken/languages, I wouldn't have thought twice about 
> writing a loop that aggregates values in a slice or an array in go. And, now 
> that I've seen the performance difference, I'd much rather have the compiler 
> optimize this for me than to go combing through my projects and manually 
> allocating a local aggregation variable in the hopes of getting to use a 
> register.
>
> At this point, I'm just taking a "temperature" check to see if folks would 
> entertain the idea. I haven't really studied compilers, and would probably 
> need some guidance to successfully contribute it to gc.  But, I don't want to 
> go through the learning and heavy-lifting if there's a major chance that the 
> PR review would end up sounding like, "While it clearly produces more 
> efficient binaries, we don't think it's worth the additional compilation 
> time."
>
> Let me know if I should "dive in" or just remember to be very careful when 
> aggregating operations in loops to a location in memory.

As I understand it, the Go compiler already does scalar replacement of
aggregates for small structs and single element arrays.  See
decomposeStructPhi and decomposeArrayPhi, in
https://go.googlesource.com/go/+/refs/heads/master/src/cmd/compile/internal/ssa/decompose.go.
I don't think anybody would object to extending the code if there were
minimal effect on compile time.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcU_YtWiDz6XMBgR-Uv1qvWAAYwQ9QtQ4fQYh8hmnO6rgQ%40mail.gmail.com.

Re: [go-nuts] Is it possible to compile a Go program without jump tables ?

2024-12-02 Thread 'Keith Randall' via golang-nuts 
Ah, I see, there are still a few left in the runtime. The runtime forces 
optimizations on even when -N is used.

So no, I don't think you can do what you want (without hacking the compiler 
to turn jump tables off completely.)

On Monday, December 2, 2024 at 2:25:22 AM UTC-8 Karolina GORNA wrote:

> Hello Keith,
>
> Thank you for your curiosity.
>
> This is the procedure I follow :
>
>1. I have written the test program main.go below in the directory 
>switch-go
>2. I use the command *GOARCH=amd64 GOOS=linux go build 
>-gcflags=all='-N -l' . *with "go version go1.23.3 linux/amd64".
>3. I open the produced binary switch-go in the emulator tool of 
>Ghidra, selecting the Golang compiler and launching the default analysis.
>4. In the Symbol Tree, I search for "switch" key world, and I find 
>switch tables in the .rodata section, which are actually the jump tables I 
>don't want to have.
>
> Following the same procedure with go1.23.3 linux/amd64 and 
> GCCGO 14.2.0 (Ubuntu 14.2.0-4ubuntu2~24.04) , but with the command 
> *GOARCH=amd64 
> GOOS=linux go build -buildvcs=false -compiler=gccgo 
> -gccgoflags='-fno-jump-tables' . *, I don't have any switch tables (more 
> precisely, jump tables) in the .rodata section (where there are supposed to 
> be if they exist). But again, GCCGO currently doesn't support 
> functionalities for Go above 1.18. 
>
> Best,
> Karolina
>
> PS : I know I don't need to specify *GOARCH=amd64 GOOS=linux *in the 
> command but I have included them for the purpose of this debugging 
> discussion.
>
>
> *switch-go/main.go*--
> package main
>
> import "fmt"
>
> func main() {
> x := 10
> switch x {
> case 1:
> fmt.Println("One")
> case 5:
> fmt.Println("Five")
> case 10:
> fmt.Println("Ten")
> default:
> fmt.Println("Other")
> }
> }
> --
> On Saturday, November 30, 2024 at 2:16:20 AM UTC+1 Keith Randall wrote:
>
>> Hmm, that's strange. I am not seeing that behavior.
>> Could you paste the *exact* commands you used to build your program here? 
>> Include GOARCH, GOOS, Go version, and anything else relevant.
>> How are you determining that there are are still jump tables? Please post 
>> the exact commands you used to determine that also.
>>
>> One thing to try is to use "-gcflags=all=-N -l" instead of just 
>> "-gcflags=-N -l".
>>
>>
>> On Friday, November 29, 2024 at 2:58:01 AM UTC-8 Karolina GORNA wrote:
>>
>> Hello everyone,
>>
>> @Lin thank you for your feedback. However, I don't want to add a 
>> constraint on rewriting switch cases in if/else, I would be happy that the 
>> compiler can handle it at its end. For instance, GCC is able to remove jump 
>> tables even when there are switch cases in the source code with the option 
>> "-fno-jump-tables".
>>
>> @Keith, the example I took was the simple Go code below, so no cgo 
>> involved. I built it with the command *go build -gcflags='-N -l'  . . *I 
>> analyzed the binary in Ghidra and I saw the switch tables I mentioned 
>> (being actually jump tables) in the .rodata section.
>> --> However, I figured out that by using the command *go build -o 
>>  -compiler=gccgo -gccgoflags='-fno-jump-tables' . *, I 
>> successfully get a binary of the same code without jump tables in the 
>> .rodata section. The issue with GCCGO is that it doesn't handle Go 
>> functionalities about Go1.18 (see 
>> https://www.mail-archive.com/debian-b...@lists.debian.org/msg1983336.html 
>> )
>>  
>> so it's quite limiting.
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *package mainimport "fmt"func main() {x := 10switch x {case 
>> 1:fmt.Println("One")case 5:fmt.Println("Five")case 
>> 10:fmt.Println("Ten")default:fmt.Println("Other")}}*
>> --
>>
>> @Ian, I'm working on bug detection, and there seems to be more 
>> granularity and therefore more accuracy in analyzing low-level code than 
>> source code.
>>
>> Best,
>> Karolina
>> On Friday, November 29, 2024 at 6:39:02 AM UTC+1 Ian Lance Taylor wrote:
>>
>> On Thu, Nov 28, 2024, 1:13 AM 'Karolina GORNA' via golang-nuts <
>> golan...@googlegroups.com> wrote:
>>
>>
>> @Ian, ok I understand thank you. I am working on Go binary analysis and 
>> this feature would help a lot.
>>
>>
>> Out of curiosity, why do you want to do binary analysis on a program if 
>> you can ask for specific options to be used when building?  Why not just do 
>> source analysis, for which there are a bunch of good tools available? 
>>
>> Ian
>>
>>
>> --
>> Les informations contenues dans ce message électronique ainsi que celles 
>> contenues dans les documents attachés sont strictement confidentielles et

Re: [go-nuts] Is it possible to compile a Go program without jump tables ?

2024-12-02 Thread 'Karolina GORNA' via golang-nuts 
Hello Keith,

Thank you for your curiosity.

This is the procedure I follow :

   1. I have written the test program main.go below in the directory 
   switch-go
   2. I use the command *GOARCH=amd64 GOOS=linux go build -gcflags=all='-N 
   -l' . *with "go version go1.23.3 linux/amd64".
   3. I open the produced binary switch-go in the emulator tool of Ghidra, 
   selecting the Golang compiler and launching the default analysis.
   4. In the Symbol Tree, I search for "switch" key world, and I find 
   switch tables in the .rodata section, which are actually the jump tables I 
   don't want to have.

Following the same procedure with go1.23.3 linux/amd64 and 
GCCGO 14.2.0 (Ubuntu 14.2.0-4ubuntu2~24.04) , but with the command 
*GOARCH=amd64 
GOOS=linux go build -buildvcs=false -compiler=gccgo 
-gccgoflags='-fno-jump-tables' . *, I don't have any switch tables (more 
precisely, jump tables) in the .rodata section (where there are supposed to 
be if they exist). But again, GCCGO currently doesn't support 
functionalities for Go above 1.18. 

Best,
Karolina

PS : I know I don't need to specify *GOARCH=amd64 GOOS=linux *in the 
command but I have included them for the purpose of this debugging 
discussion.


*switch-go/main.go*--
package main

import "fmt"

func main() {
x := 10
switch x {
case 1:
fmt.Println("One")
case 5:
fmt.Println("Five")
case 10:
fmt.Println("Ten")
default:
fmt.Println("Other")
}
}
--
On Saturday, November 30, 2024 at 2:16:20 AM UTC+1 Keith Randall wrote:

> Hmm, that's strange. I am not seeing that behavior.
> Could you paste the *exact* commands you used to build your program here? 
> Include GOARCH, GOOS, Go version, and anything else relevant.
> How are you determining that there are are still jump tables? Please post 
> the exact commands you used to determine that also.
>
> One thing to try is to use "-gcflags=all=-N -l" instead of just 
> "-gcflags=-N -l".
>
>
> On Friday, November 29, 2024 at 2:58:01 AM UTC-8 Karolina GORNA wrote:
>
> Hello everyone,
>
> @Lin thank you for your feedback. However, I don't want to add a 
> constraint on rewriting switch cases in if/else, I would be happy that the 
> compiler can handle it at its end. For instance, GCC is able to remove jump 
> tables even when there are switch cases in the source code with the option 
> "-fno-jump-tables".
>
> @Keith, the example I took was the simple Go code below, so no cgo 
> involved. I built it with the command *go build -gcflags='-N -l'  . . *I 
> analyzed the binary in Ghidra and I saw the switch tables I mentioned 
> (being actually jump tables) in the .rodata section.
> --> However, I figured out that by using the command *go build -o 
>  -compiler=gccgo -gccgoflags='-fno-jump-tables' . *, I 
> successfully get a binary of the same code without jump tables in the 
> .rodata section. The issue with GCCGO is that it doesn't handle Go 
> functionalities about Go1.18 (see 
> https://www.mail-archive.com/debian-b...@lists.debian.org/msg1983336.html 
> )
>  
> so it's quite limiting.
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *package mainimport "fmt"func main() {x := 10switch x {case 
> 1:fmt.Println("One")case 5:fmt.Println("Five")case 
> 10:fmt.Println("Ten")default:fmt.Println("Other")}}*
> --
>
> @Ian, I'm working on bug detection, and there seems to be more granularity 
> and therefore more accuracy in analyzing low-level code than source code.
>
> Best,
> Karolina
> On Friday, November 29, 2024 at 6:39:02 AM UTC+1 Ian Lance Taylor wrote:
>
> On Thu, Nov 28, 2024, 1:13 AM 'Karolina GORNA' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
>
> @Ian, ok I understand thank you. I am working on Go binary analysis and 
> this feature would help a lot.
>
>
> Out of curiosity, why do you want to do binary analysis on a program if 
> you can ask for specific options to be used when building?  Why not just do 
> source analysis, for which there are a bunch of good tools available? 
>
> Ian
>
>
> --
> Les informations contenues dans ce message électronique ainsi que celles 
> contenues dans les documents attachés sont strictement confidentielles et 
> sont destinées à l'usage exclusif du (des) destinataire(s) nommé(s).
> Toute divulgation, distribution ou reproduction, même partielle, en est 
> strictement interdite sauf autorisation écrite et expresse de l’émetteur.
> Si vous recevez ce message par erreur, veuillez le notifier immédiatement 
> à son émetteur par retour, et le détruire ainsi que tous les documents qui 
> y sont attachés.
>
> The information contained in this email and in any document enclosed is 
> stric

Re: [go-nuts] Looking for a SQLite statement parser

2024-12-02 Thread 'Jacob Shandling' via golang-nuts 
Thanks for the references Julian and Jim!

On Tuesday, November 26, 2024 at 8:26:07 AM UTC-8 Jim Idle wrote:

I revamped the Go code generation not so long ago - performance is sound 
with reasonably well written grammars, and very good with good grammars. 
Unfortunately the sample grammars for SQL are poor and will give massive 
performance problems with all targets (Java, C++, etc). I decided not to 
spend too much time on optimizing bad grammar specifications. The Go target 
is used extensively within Google.

Writing grammars looks easy because ANTLR will make a parser out of almost 
anything - but when it is written with huge ambiguities, it will explore 
every possible way that an input could be parsed and the performance will 
suffer. Unfortunately, SQL is the type of input where it is very easy to 
write a very ambiguous grammar, and so people have done so. I do not 
recommend using them. The MySQL grammar by Mike Lischke is performant - the 
MySQL one in grammars-v4 should not be used.

I have this available though:

jimidle/tsqlparser: An implementation of an ANTLR4 TSQL grammar at the 
2008R2 spec 
github.com 
[image: apple-touch-icon-180x180-a80b8e11abe2.png] 
 


It should be a reasonable task to cut down to SQLlite from the full TSQL 
spec.

That said, I have not tried the sample grammar for sqlite so I do not know 
what state it is in. You might try it and see what the performance is like 
before changing an existing grammar... [EDIT] - I see that Mike has worked 
on the sqlite grammar, so you may well have a good starting point there,

Jim





On Nov 25, 2024, at 21:30, Julian Peterson  wrote:

Hi,

For lexer/parser generation, I've used antlr with some success. 
https://www.antlr.org/
The go code generation is a little rough atm, but not a show stopper.

Also, they have grammars for many languages, so they might be useful even 
on their own: https://github.com/antlr/grammars-v4 (sqlite is in there)


Cheers,

Julian



On Tue, 26 Nov 2024 at 05:44, 'Jacob Shandling' via golang-nuts <
golan...@googlegroups.com> wrote:

*> parse the SQLite command, create an AST*

Hello Mr. O'Toole,

This is precisely the task I am looking to carry out, followed by walking 
the AST. Did you ever end up finding a good Go parser for SQLite, or 
building one yourself? I see your rqlite/sql 
 package, but am unclear as to 
whether it has this functionality.

Thank you,
Jacob Shandling 

On Tuesday, January 31, 2017 at 10:33:38 PM UTC-8 Philip O'Toole wrote:

Great -- thanks for the reference, I'll take a look.

Philip

On Tue, Jan 31, 2017 at 10:02 AM, Vasiliy Tolstov  wrote:

cznic/ql have own lexer for SQL and it syntax very similar.

31 Янв 2017 г. 16:33 пользователь  написал:

Hello,

I am the creator of rqlite (https://github.com/rqlite/rqlite), a 
distributed relational database built on SQLite.

I'm looking for a lexer and parser for SQL -- specifically SQLite, to 
improve the ease-of-use of rqlite. I've thought about looking into using 
the C code exposed by the SQLite source itself (which is what 
https://github.com/mattn/go-sqlite3 does more generally). But someone might 
know of a lexer and parser written in Go. I know how to build it myself, 
but would rather avoid the effort if possible.

What I am really trying to do is to determine, for the end-user, whether 
the Exec() (https://golang.org/pkg/database/sql/#DB.Exec)  or Query() (
https://golang.org/pkg/database/sql/#DB.Query) method should be called on 
the standard Go SQL interface, given a user-supplied SQL statement. Right 
now the user must know what type of statement she is executing, and send 
the statements to the right endpoint. I'd really like rqlite to just do the 
right thing, and determine this automatically. Hence my idea to parse the 
SQLite command, create an AST, and work it out.

Or perhaps there is another way of thinking about this?

Thanks,

Philip

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.




-- 
Philip O'Toole
Director, Engineering
Get inspired. See what's new on the Percolate blog: blog.percolate.com


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/4b82f10e-1790-47f8-9506-9bd996f497bfn%40googlegroups.com
 

.


-- 
You

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread Amnon 
Why are you using CGO?
Why not just make life easy for yourself by sticking to the http.ServeTLS 
built in to the Go standard library.
On Monday, 2 December 2024 at 19:25:21 UTC Ian Lance Taylor wrote:

> On Mon, Dec 2, 2024 at 11:16 AM Jason E. Aten  wrote:
> >
> > ChatGPT seems to think that the go-wolfssl library is not following 
> proper CGO rules in
> > many places. The transcript below is long. I post it nonetheless, in 
> case such
> > analysis is new for readers. I would suggest that you (everyone) can and 
> should
> > be doing this kind of "code review by LLM" yourself to quickly resolve
> > these kinds of issues. Also, it might be wrong!?! since it comes from an 
> LLM.
> > But at least we can talk about specific code snippets now.
> >
> > Transcript using ChatGPT 4o from Github copilot:
> >
> > @workspace I suspect there are one or more bugs in the use of CGO in 
> this this Go library, go-wolfssl. That is, I suspect that the CGO rules for 
> passing C pointers back and forth between Go and C threads are not being 
> followed, and this is causing bugs in the use of the library. Please audit 
> all the Go and C files in this workspace, and point out places where the 
> CGO pointer rules are not being followed.
> >
> > Workspace: In CGO, there are strict rules for passing pointers between 
> Go and C. Specifically, Go pointers should not be passed to C code, and C 
> pointers should not be used in Go code without proper conversion. Here are 
> some potential violations of these rules in your workspace:
> >
> > 1. **`random.go`**:
> > ```go
> > func Wc_RNG_GenerateBlock(rng *C.struct_WC_RNG, b []byte, sz int) int {
> > return int(C.wc_RNG_GenerateBlock(rng, 
> (*C.uchar)(unsafe.Pointer(&b[0])), C.word32(sz)))
> > }
> > ```
> > - The Go slice
> >
> > b
> >
> > is being converted to a C pointer and passed to the C function. This is 
> a violation of the CGO rules.
>
> This code is in itself fine, and is a typical example of using cgo.
>
> I didn't read any further.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/19c73eec-48bc-4a16-bcc9-a850f4df4bd9n%40googlegroups.com.

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread robert engels 
You are using the same buffer for reading and writing - I suspect that is your 
problem...

> On Dec 2, 2024, at 11:00 AM, Lealem Amedie  wrote:
> 
> Hi,
> 
> I’m trying to get an https server working by overwriting the 
> accept/read/write methods in the http module. I’m using go-wolfssl for TLS.
> 
> The server will accept a TLS 1.2 connection, then send some random data to 
> the client connects, then wait for another connection.
> 
> The issue that I’m seeing is that whenever the server sends large payloads, 
> it’ll do that successfully but it then can’t send data to the next client 
> that connects (the send errors out with the TCP error EBADF). If dealing with 
> smaller payloads (<30k), the server can send data successfully to each client 
> that connects. Then as soon as a large one is sent, the next transmission 
> fails.
> 
> If I disable the GOGC with debug.SetGCPercent(-1), the issue goes away and I 
> can send as many large payloads as required. From the debugging I’ve done so 
> far, this looks like an issue with the GO garbage collector dropping C 
> pointers. go-wolfssl relies on the wolfSSL C library so it uses CGO. Does 
> anyone have any other ideas or input?
> 
> Code is below. See this repo to actually run it GitHub - 
> lealem47/go-wolfssl-https-server  
> 
> Thank you!
> 
> 
> 
> ```
> 
> package main
> 
> import (
>   "bytes"
>   "crypto/rand"
>   "encoding/base64"
>   "fmt"
>   log "github.com/sirupsen/logrus"
>   wolfSSL "github.com/wolfssl/go-wolfssl"
>   "net"
>   "net/http"
>   "os"
>   "strconv"
>   "sync"
>   "time"
> )
> 
> const defaultPort = "8443"
> 
> type wolfSSLListener struct {
>   listener net.Listener
>   ctx  *wolfSSL.WOLFSSL_CTX
> }
> 
> // Accept waits for and returns the next connection to the listener.
> func (cl *wolfSSLListener) Accept() (net.Conn, error) {
>   conn, err := cl.listener.Accept()
>   if err != nil {
>   return nil, err
>   }
>   fmt.Println("Accepted new connection from:", conn.RemoteAddr())
> 
>   ssl := wolfSSL.WolfSSL_new(cl.ctx)
>   if ssl == nil {
>   fmt.Println("WolfSSL_new Failed")
>   os.Exit(1)
>   }
> 
>   file, err := conn.(*net.TCPConn).File()
>   if err != nil {
>   panic(err)
>   }
>   fd := file.Fd()
>   wolfSSL.WolfSSL_set_fd(ssl, int(fd))
> 
>   ret := wolfSSL.WolfSSL_accept(ssl)
>   if ret != wolfSSL.WOLFSSL_SUCCESS {
>   fmt.Println("WolfSSL_accept error ", ret)
>   } else {
>   fmt.Println("Client Successfully Connected!")
>   }
> 
>   return &wolfSSLConn{
>   conn: conn,
>   ssl:  ssl,
>   }, nil
> }
> 
> // Close closes the listener, making it stop accepting new connections.
> func (cl *wolfSSLListener) Close() error {
>   fmt.Println("Closing listener...")
>   return cl.listener.Close()
> }
> 
> // Addr returns the listener's network address.
> func (cl *wolfSSLListener) Addr() net.Addr {
>   return cl.listener.Addr()
> }
> 
> type wolfSSLConn struct {
>   conn   net.Conn
>   ssl*wolfSSL.WOLFSSL
>   buffer bytes.Buffer
>   mu sync.Mutex
>   closed bool
> }
> 
> func (w *wolfSSLConn) Read(b []byte) (int, error) {
>   log.Infof("Calling read: %d", len(b))
> 
>   ret := wolfSSL.WolfSSL_read(w.ssl, b, uintptr(len(b)))
>   if ret < 0 {
>   errCode := wolfSSL.WolfSSL_get_error(w.ssl, int(ret))
>   return 0, fmt.Errorf("read error: %d", errCode)
>   }
> 
>   log.Infof("Read bytes: %s", string(b[:ret]))
>   return int(ret), nil
> }
> 
> func (w *wolfSSLConn) Write(b []byte) (int, error) {
>   log.Infof("Calling write: %d", len(b))
> 
>   sz := uintptr(len(b))
> 
>   ret := wolfSSL.WolfSSL_write(w.ssl, b, sz)
>   if ret < 0 {
>   errCode := wolfSSL.WolfSSL_get_error(w.ssl, int(ret))
>   return 0, fmt.Errorf("write error: %d", errCode)
>   }
> 
>   return int(ret), nil
> }
> 
> func (w *wolfSSLConn) Close() error {
>   log.Infof("Closing connection")
> 
>   wolfSSL.WolfSSL_shutdown(w.ssl)
>   wolfSSL.WolfSSL_free(w.ssl)
>   return w.conn.Close()
> }
> 
> func (w *wolfSSLConn) LocalAddr() net.Addr {
>   return w.conn.LocalAddr()
> }
> 
> func (w *wolfSSLConn) RemoteAddr() net.Addr {
>   return w.conn.RemoteAddr()
> }
> 
> func (w *wolfSSLConn) SetDeadline(t time.Time) error {
>   return w.conn.SetDeadline(t)
> }
> 
> func (w *wolfSSLConn) SetReadDeadline(t time.Time) error {
>   return w.conn.SetReadDeadline(t)
> }
> 
> func (w *wolfSSLConn) SetWriteDeadline(t time.Time) error {
>   return w.conn.SetWriteDeadline(t)
> }
> 
> // Handler for generating and base64 encoding 5KB of random data
> func randomDataHandler(w http.ResponseWriter, r *http.Request) {
>   // Get the "size" query

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread Jason E. Aten 
ChatGPT seems to think that the go-wolfssl library is not following proper 
CGO rules in 
many places.  The transcript below is long. I post it nonetheless, in case 
such 
analysis is new for readers. I would suggest that you (everyone) can and 
should 
be doing this kind of "code review by LLM" yourself to quickly resolve 
these kinds of issues. Also, it might be wrong!?! since it comes from an 
LLM.
But at least we can talk about specific code snippets now.

Transcript using ChatGPT 4o from Github copilot:

@workspace I suspect there are one or more bugs in the use of CGO in this 
this Go library, go-wolfssl. That is, I suspect that the CGO rules for 
passing C pointers back and forth between Go and C threads are not being 
followed, and this is causing bugs in the use of the library. Please audit 
all the Go and C files in this workspace, and point out places where the 
CGO pointer rules are not being followed.

Workspace: In CGO, there are strict rules for passing pointers between Go 
and C. Specifically, Go pointers should not be passed to C code, and C 
pointers should not be used in Go code without proper conversion. Here are 
some potential violations of these rules in your workspace:

1. **`random.go`**:
```go
func Wc_RNG_GenerateBlock(rng *C.struct_WC_RNG, b []byte, sz int) int {
return int(C.wc_RNG_GenerateBlock(rng, 
(*C.uchar)(unsafe.Pointer(&b[0])), C.word32(sz)))
}
```
- The Go slice 

b

 is being converted to a C pointer and passed to the C function. This is a 
violation of the CGO rules.

2. **`blake2.go`**:
```go
func Wc_Blake2sFinal(blake2s *C.struct_Blake2s, out []byte, requestSz 
int) int {
return int(C.wc_Blake2sFinal(blake2s, 
(*C.uchar)(unsafe.Pointer(&out[0])), C.word32(requestSz)))
}
```
- The Go slice 

out

 is being converted to a C pointer and passed to the C function. This is a 
violation of the CGO rules.

3. **`curve25519.go`**:
```go
func Wc_curve25519_shared_secret(privKey, pubKey 
*C.struct_curve25519_key, out []byte) int {
outLen := len(out)
return int(C.wc_curve25519_shared_secret_ex(privKey, pubKey, 
(*C.uchar)(unsafe.Pointer(&out[0])), (*C.word32)(unsafe.Pointer(&outLen)), 
C.EC25519_LITTLE_ENDIAN))
}
```
- The Go slice 

out

 is being converted to a C pointer and passed to the C function. This is a 
violation of the CGO rules.

4. **`chacha_poly.go`**:
```go
func Wc_ChaCha20Poly1305_Encrypt(inKey, inIv, inAAD, inPlain, 
outCipher, outAuthTag []byte) int {
var sanInAAD *C.uchar
if len(inAAD) > 0 {
sanInAAD = (*C.uchar)(unsafe.Pointer(&inAAD[0]))
} else {
sanInAAD = (*C.uchar)(unsafe.Pointer(nil))
}
var sanInPlain *C.uchar
if len(inPlain) > 0 {
sanInPlain = (*C.uchar)(unsafe.Pointer(&inPlain[0]))
} else {
sanInPlain = (*C.uchar)(unsafe.Pointer(nil))
}
var sanOutCipher *C.uchar
if len(outCipher) > 0 {
sanOutCipher = (*C.uchar)(unsafe.Pointer(&outCipher[0]))
} else {
emptyStringArray := []byte("")
sanOutCipher = (*C.uchar)(unsafe.Pointer(&emptyStringArray))
}
return 
int(C.wc_ChaCha20Poly1305_Encrypt((*C.uchar)(unsafe.Pointer(&inKey[0])), 
(*C.uchar)(unsafe.Pointer(&inIv[0])), sanInAAD, C.word32(len(inAAD)), 
sanInPlain, C.word32(len(inPlain)), sanOutCipher, 
(*C.uchar)(unsafe.Pointer(&outAuthTag[0]
}
```
- The Go slices 

inAAD

, 

inPlain

, 

outCipher

, and 

outAuthTag

 are being converted to C pointers and passed to the C function. This is a 
violation of the CGO rules.

5. **`examples/aes-encrypt/aes-encrypt.go`**:
```go
func AesEncrypt(aes wolfSSL.Aes, inFile string, outFile string, size 
int) {
ret := wolfSSL.Wc_InitRng(&rng)
if ret != 0 {
fmt.Println("Failed to initialize RNG")
os.Exit(1)
}

ret = wolfSSL.Wc_RNG_GenerateBlock(&rng, iv, wolfSSL.AES_BLOCK_SIZE)
if ret != 0 {
fmt.Println("Failed to generate RNG block")
os.Exit(1)
}

ret = wolfSSL.Wc_RNG_GenerateBlock(&rng, salt, SALT_SIZE)
if ret != 0 {
fmt.Println("Failed to generate RNG block")
os.Exit(1)
}
}
```
- The Go slices 

iv

 and 

salt

 are being converted to C pointers and passed to the C function. This is a 
violation of the CGO rules.

6. **`examples/ecc-sign-verify/ecc-sign-verify.go`**:
```go
func sign_verify(eccKeySz int, hash []byte, printSig int) {
var key wolfSSL.Ecc_key
var rng wolfSSL.WC_RNG
var sig []byte = nil
var max int= wolfSSL.ECC_MAX_SIG_SIZE

byteField := int((eccKeySz + (BYTE_SZ - 1)) / BYTE_SZ)

sig = make([]byte, max)

ret := wolfSSL.Wc_ecc_init(&key)
if ret != 0 {
fmt.Println("Failed to initialize ecc_key")
os.Exit

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread Ian Lance Taylor 
On Mon, Dec 2, 2024 at 11:16 AM Jason E. Aten  wrote:
>
> ChatGPT seems to think that the go-wolfssl library is not following proper 
> CGO rules in
> many places.  The transcript below is long. I post it nonetheless, in case 
> such
> analysis is new for readers. I would suggest that you (everyone) can and 
> should
> be doing this kind of "code review by LLM" yourself to quickly resolve
> these kinds of issues. Also, it might be wrong!?! since it comes from an LLM.
> But at least we can talk about specific code snippets now.
>
> Transcript using ChatGPT 4o from Github copilot:
>
> @workspace I suspect there are one or more bugs in the use of CGO in this 
> this Go library, go-wolfssl. That is, I suspect that the CGO rules for 
> passing C pointers back and forth between Go and C threads are not being 
> followed, and this is causing bugs in the use of the library. Please audit 
> all the Go and C files in this workspace, and point out places where the CGO 
> pointer rules are not being followed.
>
> Workspace: In CGO, there are strict rules for passing pointers between Go and 
> C. Specifically, Go pointers should not be passed to C code, and C pointers 
> should not be used in Go code without proper conversion. Here are some 
> potential violations of these rules in your workspace:
>
> 1. **`random.go`**:
> ```go
> func Wc_RNG_GenerateBlock(rng *C.struct_WC_RNG, b []byte, sz int) int {
> return int(C.wc_RNG_GenerateBlock(rng, 
> (*C.uchar)(unsafe.Pointer(&b[0])), C.word32(sz)))
> }
> ```
> - The Go slice
>
> b
>
>  is being converted to a C pointer and passed to the C function. This is a 
> violation of the CGO rules.

This code is in itself fine, and is a typical example of using cgo.

I didn't read any further.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX4a9AAA_btAsjF5AVheOaddQrqKsVnCN%2BM7v2rbwq2cg%40mail.gmail.com.

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread Jason E. Aten 
Sorry. I've deleted those unhelpful suggestions.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/455959a3-4ddf-42b9-9d06-4d651effd7e2n%40googlegroups.com.

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread robert engels 
I don’t think either suggestion was unhelpful. My first thought was the library 
was broken as well, as Ian provided some evidence of. And not using CGO should 
almost always be your first choice when there are Go only solutions available. 
Too many things to get right in a highly concurrent system when CGO is involved 
- and I suspect greater than 90% of all runtime bugs in Go programs.

> On Dec 2, 2024, at 4:43 PM, Jason E. Aten  wrote:
> 
> Sorry. I've deleted those unhelpful suggestions.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion visit 
> https://groups.google.com/d/msgid/golang-nuts/455959a3-4ddf-42b9-9d06-4d651effd7e2n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/FA1E7D0C-DD12-4FA1-9BCB-DF030FC22974%40ix.netcom.com.

Re: [go-nuts] Issue with CGO and GOGC when dealing with large buffers

2024-12-02 Thread robert engels 
You might want to run under asan, msan, and tsan - as maybe the library is 
corrupting it’s own data structures? Just an idea.

> On Dec 2, 2024, at 4:52 PM, robert engels  wrote:
> 
> I don’t think either suggestion was unhelpful. My first thought was the 
> library was broken as well, as Ian provided some evidence of. And not using 
> CGO should almost always be your first choice when there are Go only 
> solutions available. Too many things to get right in a highly concurrent 
> system when CGO is involved - and I suspect greater than 90% of all runtime 
> bugs in Go programs.
> 
>> On Dec 2, 2024, at 4:43 PM, Jason E. Aten  wrote:
>> 
>> Sorry. I've deleted those unhelpful suggestions.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> .
>> To view this discussion visit 
>> https://groups.google.com/d/msgid/golang-nuts/455959a3-4ddf-42b9-9d06-4d651effd7e2n%40googlegroups.com
>>  
>> .
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/ED21DF97-36E4-465C-B4B2-2228163A1878%40ix.netcom.com.

[go-nuts] Where should I define the errors in a Go project?

2024-12-02 Thread JUAN DIEGO LATORRE RAMIREZ 
I am trying to standardize an architecture for my Go projects, so I have a 
file structure like this:

├── go.mod
├── go.sum
├── internal
│   ├── domain
│   │   ├── models
│   │   │   └── user.go
│   │   └── services
│   │   └── user.go
│   └── repositories
│   └── mongodb
│   ├── base.go
│   ├── config.go
│   └── user.go

File services/user.go:

 https://go.dev/play/p/ZYWhgHRLU5V

In the services I have business logic and in another folder called 
repositories I have implementations in different technologies as needed. 
For example I can have an implementation of UserStore in Mongo DB... or 
Postgres etc And the idea is then to have a layer on top of these 
services... that can be, an APIRest, a serverless etc
I have doubts about where I should define the errors in my system. For 
example, a common error could be *ErrRecordNotFound*, but depending on the 
repository/technology, this error will be different. 
I was thinking of defining the errors inside the domain folder, and then in 
the repositories, the errors are mapped to my domain errors for example 
if it was in mongo, my code would look like this:

if errors.Is(err, mongo.ErrNoDocuments) {
return nil, domain.ErrDocumentNotFound
}
   
(or in some cases wrap the library specific error in my domain error). 
This way my business logic would not depend on specific technologies, but 
the implementations/libraries depend on my business logic. A possible 
disadvantage is that my repositories code may no longer be reusable because 
it is “smeared” with specific business logic.

Is this approach correct? Or should I approach it differently?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/golang-nuts/85b7fa7d-1c01-4281-be20-4837f6019743n%40googlegroups.com.