#1 - No change #2 - All of grants are in the ip/mask form such as: 'nova'@' 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change. #3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing "nova-manage service list" works correctly on test2.
:( Thanks for the help! Sam On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza <sylvain.ba...@digimind.com>wrote: > When looking at MySQL 5.1 refman ( > http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would > suggest to follow the procedure : > 1. 'mysqladmin flush-hosts' > 2. replace DNS entries in mysql.user table by IP addresses instead > 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and > restart nova-api !) > > I wouldn't bet on it, but I would say this is due to some name resolution > which is incorrect. > > -Sylvain > > > Le 11/03/2013 17:00, Sylvain Bauza a écrit : > > Ok, lemme try to summarize. > You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to > Pacemaker. > This setup is relying on two hosts, test1 (10.21.0.1) and test2 > (10.21.0.2). > Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. > > Are you sure your my.cnf is actually the same in between both DRBD nodes ? > (I would recommend to symlink it to a physical file hosted on the DRBD > device). > > One thing is hurting me : you told me that nova is also pacemake'd. If so, > why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to > nova-ha (assuming 10.21.2.4 as per /etc/hosts). > > Also, as per my understanding of Pacemaker, DRBD partition is setup by > default on test2, correct ? > > > Sorry, as per my first reading, I can't see anything obvious. That said, > I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct > MySQL connection attempt. But maybe I'm wrong ? > > Anyway, are you sure you only have *one* MySQL engine running (either on > test1 or test2) and nova-manage trying to access this right one ? > > Perms look good to me. As it a test setup, you could try to unleash the > grants by deleting them and allowing nova@'%' to see if it's a basic dns > mapping issue. > > -Sylvain > > > > Le 11/03/2013 16:09, Samuel Winchenbach a écrit : > > I > enabled general_log in /etc/mysql/my.cnf Here are the results of > connecting from "test1", "test2" and using the client: > http://paste2.org/p/3115525 > I purposefully used the real password in case there is a problem with > it. > I changed before submitting post. > > here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an > attempted "nova-manage service list" from test1: > https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump > > I looked at it with wireshark and couldn't see anything that jumped out > at me as incorrect. I have not yet tried to recreate the salted password. > > > Here is my pacemaker configuration for mysql. I stripped out openstack > services, rabbitmq and others for clarity. All resources are currently > disabled (other than MySQL): > http://paste2.org/p/3115685 > > > Please don't yell at me for having STONITH disabled :P This is a > testing cluster and I am working on getting routed to the IPMI interface. > > /etc/hosts: > http://paste2.org/p/3115713 > /etc/nova/nova.conf: > http://paste2.org/p/3115739 > > > If there is anything else I can provide you, please let me know! I > have pulled out most of my hair at this point! > > Sam > > > > > On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza < > sylvain.ba...@digimind.com> wrote: > >> So as to reproduce the nova-manage SQL command, I would recommand to >> tcpdump -A port 3306 on the host and get the SQL trace on what's failing. >> >> Could you please explain further what is your HA config ? Are you using >> pacemaker/heartbeat or any VIP ? >> >> -Sylvain >> >> Le 11/03/2013 14:23, Samuel Winchenbach a écrit : >> >> Does anyone think this could be an openstack bug? I just want to check >> before submitting a bug report. >> >> Sam >> >> >> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypi...@gmail.com> wrote: >> >>> Sorry, I really can't think of anything :( >>> >>> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: >>> > I dropped those users and no change. >>> > >>> > I also set up general logging in mysql but it really doesn't provide >>> any >>> > additional information. Any idea for a next step I could take? >>> > >>> > I am almost at the point of taking a tcpdump and trying to recreate the >>> > salted password. :/ >>> > >>> > Thanks for the help >>> > >>> > Sam >>> > >>> > >>> > >>> > >>> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes <jaypi...@gmail.com >>> > <mailto:jaypi...@gmail.com>> wrote: >>> > >>> > I'm stumped :( Looks like everything is set up correctly to me. >>> What is >>> > interested is that your nova user access works from test2, but >>> there is >>> > no nova@test2 user in the mysql.user table. What about doing a >>> DROP USER >>> > nova@test1; FLUSH PRIVILEGES; and then see if that fixes >>> things... since >>> > the nova@10.21.0.0/255.255.0.0 <http://nova@10.21.0.0/255.255.0.0 >>> > >>> > user is clearly working for the access >>> > from test2. >>> > >>> > Also, I'd recommend highly removing the nova@% user. >>> > >>> > Best, >>> > -jay >>> > >>> > On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: >>> > > >>> > > http://paste2.org/p/3085807 >>> > > >>> > > >>> > > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes <jaypi...@gmail.com >>> > <mailto:jaypi...@gmail.com> >>> > > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com>>> wrote: >>> > > >>> > > Please paste the results of SELECT User, Host, Password >>> FROM >>> > mysql.user >>> > > when running as root... >>> > > >>> > > Thanks! >>> > > -jay >>> > > >>> > > On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: >>> > > > Here are my grants. I don't know if this helps, but I did >>> > verify that >>> > > > the password was identical for each grant: >>> > > http://paste2.org/p/3085361 >>> > > > >>> > > > >>> > > > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach >>> > > <swinc...@gmail.com <mailto:swinc...@gmail.com> >>> > <mailto:swinc...@gmail.com <mailto:swinc...@gmail.com>> >>> > > > <mailto:swinc...@gmail.com <mailto:swinc...@gmail.com> >>> > <mailto:swinc...@gmail.com <mailto:swinc...@gmail.com>>>> wrote: >>> > > > >>> > > > root@test1:/var/log# mysql -hmysql-ha -unova >>> > > > -p******************************** -e"SELECT User, >>> Host, >>> > Password >>> > > > FROM mysql.user;" >>> > > > ERROR 1142 (42000) at line 1: SELECT command denied to >>> user >>> > > > 'nova'@'test1' for table 'user' >>> > > > >>> > > > >>> > > > On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes >>> > <jaypi...@gmail.com <mailto:jaypi...@gmail.com> >>> > > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com>> >>> > > > <mailto:jaypi...@gmail.com <mailto: >>> jaypi...@gmail.com> >>> > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com>>>> wrote: >>> > > > >>> > > > What does this show? >>> > > > >>> > > > mysql -hmysql-ha -unova -p<PASS> -e"SELECT User, >>> Host, >>> > > Password FROM >>> > > > mysql.user" >>> > > > >>> > > > -jay >>> > > > >>> > > > On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: >>> > > > > Sorry, that must have been a copy and paste >>> error. >>> > Here >>> > > is what I >>> > > > > actually ran: >>> > > > > >>> > > > > http://paste2.org/p/3084996 >>> > > > > >>> > > > > >>> > > > > On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes >>> > > <jaypi...@gmail.com <mailto:jaypi...@gmail.com> >>> > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com>> >>> > > > <mailto:jaypi...@gmail.com >>> > <mailto:jaypi...@gmail.com> <mailto:jaypi...@gmail.com >>> > <mailto:jaypi...@gmail.com>>> >>> > > > > <mailto:jaypi...@gmail.com >>> > <mailto:jaypi...@gmail.com> <mailto:jaypi...@gmail.com >>> > <mailto:jaypi...@gmail.com>> >>> > > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com> >>> > <mailto:jaypi...@gmail.com <mailto:jaypi...@gmail.com>>>>> wrote: >>> > > > > >>> > > > > On 03/08/2013 12:19 PM, Samuel Winchenbach >>> wrote: >>> > > > > > Hi All, >>> > > > > > >>> > > > > > I have two nodes (test1 and test2) that I >>> am >>> > trying to >>> > > > set up in a >>> > > > > > highly available configuration. >>> > > > > > >>> > > > > > During the setup process I tried running >>> > "nova-manage >>> > > > service list" on >>> > > > > > both nodes. It worked fine on test2, but >>> > fails on >>> > > > test1 even >>> > > > > though I >>> > > > > > can connect to the database with the mysql >>> > client from >>> > > > test1. >>> > > > > > >>> > > > > > Here is a screen capture that shows the >>> setup on >>> > > the two >>> > > > nodes are >>> > > > > > basically identical: >>> > http://paste2.org/p/3084223 >>> > > > > >>> > > > > In the above paste you are doing: >>> > > > > >>> > > > > mysql -unova - hmysql-ha -u root >>> nova >>> > > > > -p******************************** >>> > > > > >>> > > > > Note you are supplying 2 -u arguments, and >>> mysql >>> > > will take >>> > > > the second >>> > > > > (root). >>> > > > > >>> > > > > -jay >>> > > > > >>> > > > > >>> _______________________________________________ >>> > > > > Mailing list: >>> https://launchpad.net/~openstack >>> > > > > Post to : openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net> >>> > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net>> >>> > > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net> >>> > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net>>> >>> > > > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net> >>> > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net>> >>> > > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net> >>> > > <mailto:openstack@lists.launchpad.net >>> > <mailto:openstack@lists.launchpad.net>>>> >>> > > > > Unsubscribe : >>> https://launchpad.net/~openstack >>> > > > > More help : >>> https://help.launchpad.net/ListHelp >>> > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > >>> > >>> >> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> >> > > >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
