Can you point the mgmt server directly at a single Galera node (removing
haproxy), and see if the problem persists?
Is your Galera cluster healthy? Could you be seeing latency on commit?
________________________________________
From: Andrija Panic <andrija.pa...@gmail.com>
Sent: Monday, June 8, 2015 7:42 AM
To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: [HELP needed] haproxy vs mysql client timeout
Hi,
I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
seems it's loosing connection to the haproxy/mysql, since we are using
haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
the backend section...(2 others are backup etc).
haproxy timeout is 100sec.
mysql timeouts are all defaults, except innodb_lock_wait_timeout.
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 600 |
| innodb_rollback_on_timeout | ON |
| interactive_timeout | 28800 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
To make things more interesting, all 3 servers (acs mgmt, haproxy,
galera-node1) are on the same physical host, so I rule out network
issues... and I dont see anything interesting in the haproxy logs, also no
backend downtimes etc...
The errors are like folowing, from mgmt logs, but after that mgmt server
continues to work fine...
Any suggestions on tuning timeouts ?
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 137,744
milliseconds ago. The last packet sent successfully to the server was 1
milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
... 61 more
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
lost.
... 76 more
Thanks,
--
Andrija Panić
