Jaya,
Could you please supply logs for Sodium apex jobs with TRACE enabled for
org.opendaylight.openflowjava.protocol.impl.core.OpenflowDiagStatusProviderImpl
?
Thanks,
Faseela
-----Original Message-----
From: SOMASHEKHAR MANOHARA JAVALAGI
Sent: Tuesday, April 2, 2019 9:05 PM
To: Jaya Priyadarshini <jaya.priyadarsh...@ericsson.com>; Jamo Luhrsen
<jluhr...@gmail.com>; Faseela K <faseel...@ericsson.com>;
integration-...@lists.opendaylight.org; netvirt-...@lists.opendaylight.org; D
Arunprakash <d.arunprak...@ericsson.com>
Cc: openflowplugin-dev <openflowplugin-dev@lists.opendaylight.org>
Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in
jenkins - openflow stuck in starting state
Hi Jaya,
We are sending status report of openflow service being operational to
infrautils diagstatus service, as and when we receive systemBootReady
notification.
This happened in 2 mins from the start of controller. We also report the status
to be ERROR in case of some issues.
As I am not able to see any such logs, issue has to be checked at infrautils
diagstatus service, whether it received status report properly.
Regards,
Somashekhar
-----Original Message-----
From: Jaya Priyadarshini
Sent: Tuesday, April 2, 2019 4:46 PM
To: Jamo Luhrsen <jluhr...@gmail.com>; SOMASHEKHAR MANOHARA JAVALAGI
<somashekhar.manohara.javal...@ericsson.com>; Faseela K
<faseel...@ericsson.com>; integration-...@lists.opendaylight.org;
netvirt-...@lists.opendaylight.org; D Arunprakash <d.arunprak...@ericsson.com>
Cc: openflowplugin-dev <openflowplugin-dev@lists.opendaylight.org>
Subject: RE: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in
jenkins - openflow stuck in starting state
Hi Som,
As per info which u asked,
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/71/console-timestamp.log.gz
ODL is started at
07:22:10 2019-03-31 07:22:10,517 INFO: Starting Opendaylight on node
overcloud-controller-0.opnfvlf.org
And it is checking if all bundles came up at around this time.
08:32:41
==============================================================================
08:32:41 01_l2 :: Test suite to verify packet flows between vm instances.
08:32:41
==============================================================================
Regards
Jaya
-----Original Message-----
From: Jamo Luhrsen <jluhr...@gmail.com>
Sent: Monday, April 1, 2019 9:34 PM
To: SOMASHEKHAR MANOHARA JAVALAGI <somashekhar.manohara.javal...@ericsson.com>;
Faseela K <faseel...@ericsson.com>; Jaya Priyadarshini
<jaya.priyadarsh...@ericsson.com>; integration-...@lists.opendaylight.org;
netvirt-...@lists.opendaylight.org; D Arunprakash <d.arunprak...@ericsson.com>
Cc: openflowplugin-dev <openflowplugin-dev@lists.opendaylight.org>
Subject: Re: [integration-dev] [openflowplugin-dev] Netvirt Apex jobs in
jenkins - openflow stuck in starting state
something is broken.
Looks like the failures are only there in sodium. here is a karaf log from job
#71 failure:
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/71/controller_sos/sosreport-controllerreport-20190331083711/opt/opendaylight/data/log/karaf.log.gz
you can see some WARN messages in TracingBroker which seem to just repeat.
the controller is up for at least 10m before the actual failure is finally
noted. In a neon job I just checked, the same validation is completed in 3s.
JamO
On 4/1/19 4:44 AM, SOMASHEKHAR MANOHARA JAVALAGI wrote:
> Hi Faseela,
>
> Karaf is started at Mar 31, 2019 7:23:59 AM.
>
> Once all the bundles are up, we are registering to SystemReadyMonitor
> by 2019-03-31T07:25:31,172
>
> After 16 seconds we are receiving onSystemBootready notification at
> 2019-03-31T07:25:47,219
>
> Immediately within a second we are marking openflow service status as
> operational at 2019-03-31T07:25:48,709
>
> As bundles coming up is taking more time, we are finding delay in operational
> status change of openflow service.
>
> Regards,
>
> Somashekhar
>
> *From:* openflowplugin-dev-boun...@lists.opendaylight.org
> <openflowplugin-dev-boun...@lists.opendaylight.org> *On Behalf Of
> *Faseela K
> *Sent:* Monday, April 1, 2019 12:43 PM
> *To:* Jaya Priyadarshini <jaya.priyadarsh...@ericsson.com>;
> integration-...@lists.opendaylight.org;
> netvirt-...@lists.opendaylight.org; D Arunprakash
> <d.arunprak...@ericsson.com>
> *Cc:* openflowplugin-dev <openflowplugin-dev@lists.opendaylight.org>
> *Subject:* Re: [openflowplugin-dev] Netvirt Apex jobs in jenkins -
> openflow stuck in starting state
>
> Arun,
>
> Any idea when this can happen?
>
> System Ready State says all bundles have come up, but OFP is still in
> starting state.
>
> Thanks,
>
> Faseela
>
> *From:* netvirt-dev-boun...@lists.opendaylight.org
> <mailto:netvirt-dev-boun...@lists.opendaylight.org>
> <netvirt-dev-boun...@lists.opendaylight.org
> <mailto:netvirt-dev-boun...@lists.opendaylight.org>> *On Behalf Of
> *Jaya Priyadarshini
> *Sent:* Monday, April 1, 2019 12:41 PM
> *To:* integration-...@lists.opendaylight.org
> <mailto:integration-...@lists.opendaylight.org>;
> netvirt-...@lists.opendaylight.org
> <mailto:netvirt-...@lists.opendaylight.org>
> *Subject:* [netvirt-dev] Apex jobs in jenkins
>
> Here is the apex jobs status.
>
> Success
>
>
>
> 100%
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin
> e/lastBuild>
>
>
>
> netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-
> fluorine
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin
> e/>
>
>
>
> 8 hr 19 min - #142
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin
> e/lastSuccessfulBuild/>
>
>
>
> 18 days - #123
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-fluorin
> e/lastFailedBuild/>
>
>
>
> 1 hr 0 min
>
>
>
> 25 / 25 passed
> https://jenkins.opendaylight.org/releng/plugin/robot/robot.png
>
>
>
> Success
>
>
>
> 100%
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/la
> stBuild>
>
>
>
> netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-
> neon
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/>
>
>
>
> 15 hr - #158
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/la
> stSuccessfulBuild/>
>
>
>
> 11 days - #148
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-neon/la
> stFailedBuild/>
>
>
>
> 52 min
>
>
>
> 25 / 25 passed
> https://jenkins.opendaylight.org/releng/plugin/robot/robot.png
>
>
>
> In progress
>
>
>
> 0%
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/
> lastBuild>
>
>
>
> netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-
> sodium
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/
> >
>
>
>
> 23 hr - #71
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/
> lastSuccessfulBuild/>
>
>
>
> 16 days - #40
> <https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt
> -csit-1node-0cmb-1ctl-2cmp-apex-queens-upstream-snat-conntrack-sodium/
> lastFailedBuild/>
>
>
>
> 1 hr 36 min
>
>
>
> 0 / 25 passed
> https://jenkins.opendaylight.org/releng/plugin/robot/robot.png
>
>
>
> 3^rd failure is due to following reason
>
>
>
> [?1l>[?2004lTimestamp: Sun Mar 31 08:33:15 UTC 2019 Node IP Address:
> /127.0.0.1 System is operational: false System ready state: ACTIVE
>
> OPENFLOW : STARTING (INITIALIZING) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<It never
> comes to operational.
>
> IFM : OPERATIONAL
>
> ITM : OPERATIONAL
>
> ELAN : OPERATIONAL (Service started)
>
> OVSDB : OPERATIONAL (OVSDB initialization complete)
>
> DATASTORE : OPERATIONAL [?1h=[90m~[0m
>
> So Suite setup itself has failed and hence all failures.
>
> Now how do we go ahead with apex, what should be next step?
>
> Regards
>
> Jaya
>
>
> _______________________________________________
> integration-dev mailing list
> integration-...@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
>
_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
