Hey Rafael, If it helps any, I have noticed that this issue generally occurs after several runs of tempest tests. I run all smoke tests. I have been always suspicious that some new test got added which is not cleaning up afterwards. Can you point me to which tempest test corresponds to test_boot_pattern? I can do some investigation as well.
Thanks -Sukhdev On Mon, Mar 10, 2014 at 10:10 AM, Rafael Folco <rafaelfo...@gmail.com>wrote: > This looks very familiar.... > https://bugzilla.redhat.com/show_bug.cgi?id=848942 > I'm not sure this is exactly the same bahavior, try turning debug messages > on by adding -d to the tgtd start. > I'm investigating a similar issue with tgt on test_boot_pattern test. It > turns out that the file written to the volume in the first instance is not > found by the second instance. But that's a separate issue. > > -rfolco > > > On Mon, Mar 10, 2014 at 12:54 PM, Sukhdev Kapur <sukhdevka...@gmail.com>wrote: > >> I see the same issue. This issue has crept in during the latest flurry of >> check-ins. I started noticing this issue a day or two before the Icehouse >> Feature Freeze deadline. >> >> I tried restarting tgt as well, but, it does not help. >> >> However, rebooting the VM helps clear it up. >> >> Has anybody else seen it as well? Does anybody have a solution for it? >> >> Thanks >> -Sukhdev >> >> >> >> >> >> On Mon, Mar 10, 2014 at 8:37 AM, Dane Leblanc (leblancd) < >> lebla...@cisco.com> wrote: >> >>> I don't know if anyone can give me some troubleshooting advice with this >>> issue. >>> >>> I'm seeing an occasional problem whereby after several DevStack >>> unstack.sh/stack.sh cycles, the tgt daemon (tgtd) fails to start during >>> Cinder startup. Here's a snippet from the stack.sh log: >>> >>> 2014-03-10 07:09:45.214 | Starting Cinder >>> 2014-03-10 07:09:45.215 | + return 0 >>> 2014-03-10 07:09:45.216 | + sudo rm -f /etc/tgt/conf.d/stack.conf >>> 2014-03-10 07:09:45.217 | + _configure_tgt_for_config_d >>> 2014-03-10 07:09:45.218 | + [[ ! -d /etc/tgt/stack.d/ ]] >>> 2014-03-10 07:09:45.219 | + is_ubuntu >>> 2014-03-10 07:09:45.220 | + [[ -z deb ]] >>> 2014-03-10 07:09:45.221 | + '[' deb = deb ']' >>> 2014-03-10 07:09:45.222 | + sudo service tgt restart >>> 2014-03-10 07:09:45.223 | stop: Unknown instance: >>> 2014-03-10 07:09:45.619 | start: Job failed to start >>> jenkins@neutronpluginsci:~/devstack$ 2014-03-10 07:09:45.621 | + >>> exit_trap >>> 2014-03-10 07:09:45.622 | + local r=1 >>> 2014-03-10 07:09:45.623 | ++ jobs -p >>> 2014-03-10 07:09:45.624 | + jobs= >>> 2014-03-10 07:09:45.625 | + [[ -n '' ]] >>> 2014-03-10 07:09:45.626 | + exit 1 >>> >>> If I try to restart tgt manually without success: >>> >>> jenkins@neutronpluginsci:~$ sudo service tgt restart >>> stop: Unknown instance: >>> start: Job failed to start >>> jenkins@neutronpluginsci:~$ sudo tgtd >>> librdmacm: couldn't read ABI version. >>> librdmacm: assuming: 4 >>> CMA: unable to get RDMA device list >>> (null): iser_ib_init(3263) Failed to initialize RDMA; load kernel >>> modules? >>> (null): fcoe_init(214) (null) >>> (null): fcoe_create_interface(171) no interface specified. >>> jenkins@neutronpluginsci:~$ >>> >>> The config in /etc/tgt is: >>> >>> jenkins@neutronpluginsci:/etc/tgt$ ls -l >>> total 8 >>> drwxr-xr-x 2 root root 4096 Mar 10 07:03 conf.d >>> lrwxrwxrwx 1 root root 30 Mar 10 06:50 stack.d -> >>> /opt/stack/data/cinder/volumes >>> -rw-r--r-- 1 root root 58 Mar 10 07:07 targets.conf >>> jenkins@neutronpluginsci:/etc/tgt$ cat targets.conf >>> include /etc/tgt/conf.d/*.conf >>> include /etc/tgt/stack.d/* >>> jenkins@neutronpluginsci:/etc/tgt$ ls conf.d >>> jenkins@neutronpluginsci:/etc/tgt$ ls /opt/stack/data/cinder/volumes >>> jenkins@neutronpluginsci:/etc/tgt$ >>> >>> I don't know if there's any missing Cinder config in my DevStack localrc >>> files. Here's one that I'm using: >>> >>> MYSQL_PASSWORD=nova >>> RABBIT_PASSWORD=nova >>> SERVICE_TOKEN=nova >>> SERVICE_PASSWORD=nova >>> ADMIN_PASSWORD=nova >>> >>> ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,rabbit >>> enable_service mysql >>> disable_service n-net >>> enable_service q-svc >>> enable_service q-agt >>> enable_service q-l3 >>> enable_service q-dhcp >>> enable_service q-meta >>> enable_service q-lbaas >>> enable_service neutron >>> enable_service tempest >>> VOLUME_BACKING_FILE_SIZE=2052M >>> Q_PLUGIN=cisco >>> declare -a Q_CISCO_PLUGIN_SUBPLUGINS=(openvswitch nexus) >>> declare -A >>> Q_CISCO_PLUGIN_SWITCH_INFO=([10.0.100.243]=admin:Cisco12345:22:neutronpluginsci:1/9) >>> NCCLIENT_REPO=git://github.com/CiscoSystems/ncclient.git >>> PHYSICAL_NETWORK=physnet1 >>> OVS_PHYSICAL_BRIDGE=br-eth1 >>> TENANT_VLAN_RANGE=810:819 >>> ENABLE_TENANT_VLANS=True >>> API_RATE_LIMIT=False >>> VERBOSE=True >>> DEBUG=True >>> LOGFILE=/opt/stack/logs/stack.sh.log >>> USE_SCREEN=True >>> SCREEN_LOGDIR=/opt/stack/logs >>> >>> Here are links to a log showing another localrc file that I use, and the >>> corresponding stack.sh log: >>> >>> >>> http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_console_log.txt >>> >>> http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_stack_sh_log.txt >>> >>> Does anyone have any advice on how to debug this, or recover from this >>> (beyond rebooting the node)? Or am I missing any Cinder config? >>> >>> Thanks in advance for any help on this!!! >>> Dane >>> >>> >>> >>> _______________________________________________ >>> OpenStack-Infra mailing list >>> OpenStack-Infra@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> >> >> >> _______________________________________________ >> OpenStack-Infra mailing list >> OpenStack-Infra@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >> >> >
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
