How to debug incomplete message on postgres 12.3 docker
Hello, I hope i reached the right mailing list, i have psql 12.3 container that is the db of apache guacamole container is connected to, the apache guacamole stopped working and i started to get this message on the postgres: LOG: incomplete message from client How can I debug it? i want to see when and who did the translation and what is the incomplete query ? because i did approached guacamole mailing list and they said my DB is closing connection in the middle of the transaction and its psql issue and i am not able to debug it because i only have this incomlit msg. *Is there a way to catch this transaction?* full logs : postgres_gqm_1| 2022-01-13T19:32:24.187916615Z 2022-01-13 19:32:24.187 UTC [4394] LOG: incomplete message from client guacamole_1 | 2022-01-13T19:32:24.214231454Z 19:32:24.214 [http-nio-8080-exec-6] ERROR o.a.g.rest.RESTExceptionMapper - Unexpected internal error: guacamole_1 | 2022-01-13T19:32:24.214398980Z ### Error rolling back transaction. Cause: org.postgresql.util.PSQLException: This connection has been closed. guacamole_1 | 2022-01-13T19:32:24.214445260Z ### Cause: org.postgresql.util.PSQLException: This connection has been closed. docker file : postgres_gqm: image: postgres:12.3 environment: PGDATA: /var/lib/postgresql/data/guacamole POSTGRES_DB: db POSTGRES_PASSWORD: XXX POSTGRES_USER: user restart: always volumes: - ./init:/docker-entrypoint-initdb.d:ro - ./data:/var/lib/postgresql/data:rw logging: driver: "json-file" options: max-size: "750m"
Re: How to debug incomplete message on postgres 12.3 docker
Dave Cramer www.postgres.rocks On Sun, 16 Jan 2022 at 10:13, R Batchen wrote: > > Hello, > > I hope i reached the right mailing list, i have psql 12.3 container that > is the db of apache guacamole container is connected to, > the apache guacamole stopped working and i started to get this message on > the postgres: > > LOG: incomplete message from client > > How can I debug it? > > i want to see when and who did the translation and what is the incomplete > query ? because i did approached guacamole mailing list and they said my DB > is closing connection in the middle of the transaction > > and its psql issue and i am not able to debug it because i only have this > incomlit msg. > > *Is there a way to catch this transaction?* > > full logs : > > postgres_gqm_1| 2022-01-13T19:32:24.187916615Z 2022-01-13 19:32:24.187 > UTC [4394] LOG: incomplete message from client > > guacamole_1 | 2022-01-13T19:32:24.214231454Z 19:32:24.214 > [http-nio-8080-exec-6] ERROR o.a.g.rest.RESTExceptionMapper - Unexpected > internal error: > guacamole_1 | 2022-01-13T19:32:24.214398980Z ### Error rolling back > transaction. Cause: org.postgresql.util.PSQLException: This connection has > been closed. > guacamole_1 | 2022-01-13T19:32:24.214445260Z ### Cause: > org.postgresql.util.PSQLException: This connection has been closed. > > > docker file : > > postgres_gqm: > > image: postgres:12.3 > environment: > PGDATA: /var/lib/postgresql/data/guacamole > POSTGRES_DB: db > POSTGRES_PASSWORD: XXX > POSTGRES_USER: user > restart: always > volumes: > - ./init:/docker-entrypoint-initdb.d:ro > - ./data:/var/lib/postgresql/data:rw > logging: > driver: "json-file" > options: > max-size: "750m" > > > My guess is that the container dropped the connection somehow. Either network error or application error. I would look at the logs on the container. There is no way to debug this on the postgres side (that I am aware of) Dave
Re: How to debug incomplete message on postgres 12.3 docker
R Batchen writes: > I hope i reached the right mailing list, i have psql 12.3 container that is > the db of apache guacamole container is connected to, > the apache guacamole stopped working and i started to get this message on > the postgres: > LOG: incomplete message from client > How can I debug it? This message indicates that the server collected a message header (length word) from the communication channel, but then detected EOF while trying to read that number of bytes. The two most plausible explanations for that are (1) some external force closed the network connection or (2) the client sent a invalid (far too large) length word. But to believe (2) you'd also have to believe that the client gave up and closed the connection before sending the number of bytes the server thought it had promised. That could happen if the client thought the command had timed out, but I'd expect it to log something about a timeout if it had. In short, I think something at the kernel or container level is closing the connection under you. You need to be looking into those system-level logs, as neither the postgres nor client logs would have any evidence of the primary cause. regards, tom lane
Re: How to debug incomplete message on postgres 12.3 docker
On 1/16/22 07:13, R Batchen wrote: Hello, I hope i reached the right mailing list, i have psql 12.3 container that is the db of apache guacamole container is connected to, the apache guacamole stopped working and i started to get this message on the postgres: |LOG: incomplete message from client | How can I debug it? i want to see when and who did the translation and what is the incomplete query ? because i did approached guacamole mailing list and they said my DB is closing connection in the middle of the transaction and its psql issue and i am not able to debug it because i only have this incomlit msg. The message is pretty clear, the error is coming from the client which is not Postgres. *Is there a way to catch this transaction?* full logs : |postgres_gqm_1 | 2022-01-13T19:32:24.187916615Z 2022-01-13 19:32:24.187 UTC [4394] LOG: incomplete message from client guacamole_1 | 2022-01-13T19:32:24.214231454Z 19:32:24.214 [http-nio-8080-exec-6] ERROR o.a.g.rest.RESTExceptionMapper - Unexpected internal error: guacamole_1 The above would seem to be the place to look. | 2022-01-13T19:32:24.214398980Z ### Error rolling back transaction. Cause: org.postgresql.util.PSQLException: This connection has been closed. guacamole_1 | 2022-01-13T19:32:24.214445260Z ### Cause: org.postgresql.util.PSQLException: This connection has been closed. | docker file : postgres_gqm: |image: postgres:12.3 environment: PGDATA: /var/lib/postgresql/data/guacamole POSTGRES_DB: db POSTGRES_PASSWORD: XXX POSTGRES_USER: user restart: always volumes: - ./init:/docker-entrypoint-initdb.d:ro - ./data:/var/lib/postgresql/data:rw logging: driver: "json-file" options: max-size: "750m" | -- Adrian Klaver adrian.kla...@aklaver.com
Re: WAL Archiving and base backup
On 1/16/22 01:23, Mladen Gogala wrote: On 1/14/22 16:00, David G. Johnston wrote: I still don't really understand what is so great about it. About its only redeeming feature is a declaration that "it is in core" and that newcomers can just default to it without thinking. I'd rather just play favorites and write "use pgbackrest" in our documentation. Or some hybrid approach where we don't just pick one but instead guide people to the community solutions that are out there. I don't think I really want the people responsible for core to spend time on writing end-user backup tooling. Their time is much more valuably spent working on the core product. David J. Well, the "without thinking" part of your post can be rephrased as "ease of use". Do database administrators really need to think about which backup software to use? What kind of knowledge will such an evaluation provide? All commercial databases have some form of backup software included into the core database. After all, backup and restore are extremely important functions which IMHO should be provided along with the database software. -- Mladen Gogala Database Consultant Tel: (347) 321-1217 https://dbwhisperer.wordpress.com Just to avoid any misunderstanding. I am perfectly happy using the backup/restore with pg_dump and we use it for at least a decade with success for our need of backups. My question is really in the context of WAL archiving and preparing a slave instance with a base 'non-exclusive' backup from a script (to be precise, on reboot of the server running the slave instance) since the exclusive way is deprecated. Thx for all your valuable comments -- Issa
Re: How to debug incomplete message on postgres 12.3 docker
Thanks all for the fast replay, the thing is i cannot get the transection query to know what really happened, ill try backing up db and rm containers and re build them as i dont have any other idea also server logs looks fine also cpu.. thanks! בתאריך יום א׳, 16 בינו׳ 2022 ב-19:05 מאת Adrian Klaver < adrian.kla...@aklaver.com>: > On 1/16/22 07:13, R Batchen wrote: > > > > Hello, > > > > I hope i reached the right mailing list, i have psql 12.3 container that > > is the db of apache guacamole container is connected to, > > the apache guacamole stopped working and i started to get this message > > on the postgres: > > > > |LOG: incomplete message from client | > > > > How can I debug it? > > > > i want to see when and who did the translation and what is the > incomplete query ? because i did approached guacamole mailing list and they > said my DB is closing connection in the middle of the transaction > > > > and its psql issue and i am not able to debug it because i only have > this incomlit msg. > > The message is pretty clear, the error is coming from the client which > is not Postgres. > > > > > *Is there a way to catch this transaction?* > > > > full logs : > > > > > |postgres_gqm_1 | 2022-01-13T19:32:24.187916615Z 2022-01-13 19:32:24.187 > > UTC [4394] LOG: incomplete message from client guacamole_1 | > > 2022-01-13T19:32:24.214231454Z 19:32:24.214 [http-nio-8080-exec-6] ERROR > > o.a.g.rest.RESTExceptionMapper - Unexpected internal error: guacamole_1 > > The above would seem to be the place to look. > > > | 2022-01-13T19:32:24.214398980Z ### Error rolling back transaction. > > Cause: org.postgresql.util.PSQLException: This connection has been > > closed. guacamole_1 | 2022-01-13T19:32:24.214445260Z ### Cause: > > org.postgresql.util.PSQLException: This connection has been closed. | > > > > > > docker file : > > > > postgres_gqm: > > > > |image: postgres:12.3 environment: PGDATA: > > /var/lib/postgresql/data/guacamole POSTGRES_DB: db POSTGRES_PASSWORD: > > XXX POSTGRES_USER: user restart: always volumes: - > > ./init:/docker-entrypoint-initdb.d:ro - > > ./data:/var/lib/postgresql/data:rw logging: driver: "json-file" options: > > max-size: "750m" | > > > > > > > > > > > -- > Adrian Klaver > adrian.kla...@aklaver.com >
Re: WAL Archiving and base backup
On 1/16/22 13:12, Issa Gorissen wrote: Just to avoid any misunderstanding. I am perfectly happy using the backup/restore with pg_dump and we use it for at least a decade with success for our need of backups. I am using pgbackrest. I used to use pg_rman, being well acquainted with rman backup tool belonging to another database system, but pg_rman was removing WAL archives before they were delivered to replica. I have replication slots from the OLTP database to the DW database with partitions, hash and bloom indexes and massive amount of processors, for parallel query. I need the replication to work. In addition to that, pgbackrest can employ parallelism while pg_rman cannot do that. I find pgbackrest by far the best Postgres backup system. -- Mladen Gogala Database Consultant Tel: (347) 321-1217 https://dbwhisperer.wordpress.com
