Hi Roey, I made this change and have been running this fix on 4 different servers. I believe this fix works. Things are working very smoothly.
I think we need to incorporate this change into devstack scripts or capture it in the documentation so that it saves some grief to the next person. Thanks -Sukhdev On Tue, Mar 11, 2014 at 3:06 AM, Roey Chen <ro...@mellanox.com> wrote: > Forwarding the answer to the relevant mailing lists: > > > > --- > > > > Hi, > > > > Hope this could help, > > > > I've encountered this issue myself not to long ago on Ubuntu 12.04 host, > > it didn't happen again after messing with the Kernel Semaphore Limits > parameters [1]: > > > > Adding this [2] line to `/etc/sysctl.conf` seems to do the trick. > > > > > > - Roey > > > > > > [1] http://paste.openstack.org/show/73086/ > > [2] http://paste.openstack.org/show/73082/ > > > > > > *From:* Sukhdev Kapur [mailto:sukhdevka...@gmail.com] > *Sent:* Monday, March 10, 2014 5:56 PM > *To:* Dane Leblanc (leblancd) > *Cc:* OpenStack Development Mailing List (not for usage questions); > openstack-in...@lists.openstack.org; openstack...@lists.openstack.org > > *Subject:* Re: [OpenStack-Infra] tgt restart fails in Cinder startup > "start: job failed to start" > > > > I see the same issue. This issue has crept in during the latest flurry of > check-ins. I started noticing this issue a day or two before the Icehouse > Feature Freeze deadline. > > > > I tried restarting tgt as well, but, it does not help. > > > > However, rebooting the VM helps clear it up. > > > > Has anybody else seen it as well? Does anybody have a solution for it? > > > > Thanks > > -Sukhdev > > > > > > > > > > On Mon, Mar 10, 2014 at 8:37 AM, Dane Leblanc (leblancd) < > lebla...@cisco.com> wrote: > > I don't know if anyone can give me some troubleshooting advice with this > issue. > > I'm seeing an occasional problem whereby after several DevStack > unstack.sh/stack.sh cycles, the tgt daemon (tgtd) fails to start during > Cinder startup. Here's a snippet from the stack.sh log: > > 2014-03-10 07:09:45.214 | Starting Cinder > 2014-03-10 07:09:45.215 | + return 0 > 2014-03-10 07:09:45.216 | + sudo rm -f /etc/tgt/conf.d/stack.conf > 2014-03-10 07:09:45.217 | + _configure_tgt_for_config_d > 2014-03-10 07:09:45.218 | + [[ ! -d /etc/tgt/stack.d/ ]] > 2014-03-10 07:09:45.219 | + is_ubuntu > 2014-03-10 07:09:45.220 | + [[ -z deb ]] > 2014-03-10 07:09:45.221 | + '[' deb = deb ']' > 2014-03-10 07:09:45.222 | + sudo service tgt restart > 2014-03-10 07:09:45.223 | stop: Unknown instance: > 2014-03-10 07:09:45.619 | start: Job failed to start > jenkins@neutronpluginsci:~/devstack$ 2014-03-10 07:09:45.621 | + exit_trap > 2014-03-10 07:09:45.622 | + local r=1 > 2014-03-10 07:09:45.623 | ++ jobs -p > 2014-03-10 07:09:45.624 | + jobs= > 2014-03-10 07:09:45.625 | + [[ -n '' ]] > 2014-03-10 07:09:45.626 | + exit 1 > > If I try to restart tgt manually without success: > > jenkins@neutronpluginsci:~$ sudo service tgt restart > stop: Unknown instance: > start: Job failed to start > jenkins@neutronpluginsci:~$ sudo tgtd > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > CMA: unable to get RDMA device list > (null): iser_ib_init(3263) Failed to initialize RDMA; load kernel modules? > (null): fcoe_init(214) (null) > (null): fcoe_create_interface(171) no interface specified. > jenkins@neutronpluginsci:~$ > > The config in /etc/tgt is: > > jenkins@neutronpluginsci:/etc/tgt$ ls -l > total 8 > drwxr-xr-x 2 root root 4096 Mar 10 07:03 conf.d > lrwxrwxrwx 1 root root 30 Mar 10 06:50 stack.d -> > /opt/stack/data/cinder/volumes > -rw-r--r-- 1 root root 58 Mar 10 07:07 targets.conf > jenkins@neutronpluginsci:/etc/tgt$ cat targets.conf > include /etc/tgt/conf.d/*.conf > include /etc/tgt/stack.d/* > jenkins@neutronpluginsci:/etc/tgt$ ls conf.d > jenkins@neutronpluginsci:/etc/tgt$ ls /opt/stack/data/cinder/volumes > jenkins@neutronpluginsci:/etc/tgt$ > > I don't know if there's any missing Cinder config in my DevStack localrc > files. Here's one that I'm using: > > MYSQL_PASSWORD=nova > RABBIT_PASSWORD=nova > SERVICE_TOKEN=nova > SERVICE_PASSWORD=nova > ADMIN_PASSWORD=nova > > ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,rabbit > enable_service mysql > disable_service n-net > enable_service q-svc > enable_service q-agt > enable_service q-l3 > enable_service q-dhcp > enable_service q-meta > enable_service q-lbaas > enable_service neutron > enable_service tempest > VOLUME_BACKING_FILE_SIZE=2052M > Q_PLUGIN=cisco > declare -a Q_CISCO_PLUGIN_SUBPLUGINS=(openvswitch nexus) > declare -A > Q_CISCO_PLUGIN_SWITCH_INFO=([10.0.100.243]=admin:Cisco12345:22:neutronpluginsci:1/9) > NCCLIENT_REPO=git://github.com/CiscoSystems/ncclient.git > PHYSICAL_NETWORK=physnet1 > OVS_PHYSICAL_BRIDGE=br-eth1 > TENANT_VLAN_RANGE=810:819 > ENABLE_TENANT_VLANS=True > API_RATE_LIMIT=False > VERBOSE=True > DEBUG=True > LOGFILE=/opt/stack/logs/stack.sh.log > USE_SCREEN=True > SCREEN_LOGDIR=/opt/stack/logs > > Here are links to a log showing another localrc file that I use, and the > corresponding stack.sh log: > > http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_console_log.txt > > http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_stack_sh_log.txt > > Does anyone have any advice on how to debug this, or recover from this > (beyond rebooting the node)? Or am I missing any Cinder config? > > Thanks in advance for any help on this!!! > Dane > > > > _______________________________________________ > OpenStack-Infra mailing list > openstack-in...@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
