Hi Michael,
I had work pipeline job. But after add more environments (envList) job
broke with log:
[Pipeline] End of Pipeline
an exception which occurred:
in field delegate
in field closures
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@1845f26
Caused: java.io.NotSerializableException: java.util.LinkedHashMap$Entry
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at
org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at
org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at
org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at
org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.LinkedHashMap.internalWriteEntries(LinkedHashMap.java:333)
at java.util.HashMap.writeObject(HashMap.java:1362)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
I use NonCPS function sortSuites() and it always return list. I don't
have more ideas how to fix. Can you help?
Thank you!
My test pipeline:
#!groovy
def envList = []
envList.add('NODE=behat3 ORO_PHP=7.1 ORO_APP=application/commerce-crm-ee
ORO_TEST_SUITE=behat ORO_DB=pgsql BEHAT_MODE=standalone')
envList.add('NODE=docker-doc ORO_TEST_SUITE=documentation
ORO_APP=documentation/crm ORO_DOC=OroCRM_Documentation')
envList.add('NODE=docker-doc ORO_TEST_SUITE=documentation
ORO_APP=documentation/commerce ORO_DOC=OroCommerce_Documentation')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_1.12 ORO_SE=ES ES_VER=2.4.3
UPGRADE=true TEST_RUNNER_OPTIONS=--group=search,dist')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_2.0.0 ORO_SE=ES ES_VER=2.4.3')
envList.add('NODE=func_2cpu ORO_PHP=7.0
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_1.12 UPGRADE=true')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.3.0 ORO_SE=ES ES_VER=2.3.5')
envList.add('NODE=func ORO_PHP=7.0
ORO_APP=application/commerce-enterprise ORO_TEST_SUITE=functional
ORO_DB=MYSQL ORO_INSTALLED=commerce_1.0.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.0.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm ORO_TEST_SUITE=functional
ORO_DB=PG ORO_ENABLE_PRICE_SHARDING=true')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.3.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=functional
ORO_DB=MYSQL ORO_INSTALLED=crm-enterprise_1.12 UPGRADE=true')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/platform ORO_TEST_SUITE=functional
ORO_DB=PG')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm ORO_TEST_SUITE=functional
ORO_DB=MYSQL')
envList.add('NODE=func_2cpu ORO_PHP=7.1 ORO_TEST_SUITE=behat_wiring')
envList.add('NODE=func ORO_PHP=7.1 ORO_TEST_SUITE=javascript JS=YES CS=YES')
envList.add('NODE=func_2cpu ORO_PHP=7.0
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=unit')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=unit')
def behatList = []
def enviroments = [:]
for (int i = 0; i < envList.size() ; i++) {
int index=i, e = i+1
enviroments["TestEnv_${e}"] = {
stage ("TestEnv_${e}"){
withEnv(["NETWORK=${index}"] + ["TESTENV=TestEnv_${e}"] +
envList[index].tokenize()) {
if (env.ORO_TEST_SUITE == 'behat') {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Install"
"""
behatStr = sh (
script: 'echo AutoSuiteSet_1 AutoSuiteSet_2
AutoSuiteSet_3',
returnStdout: true
).trim()
}
}
behatList=behatStr.tokenize()
echo "SUITES=${behatList}"
behatList=sortSuites(behatList)
echo "SUITES SORTED=${behatList}"
echo "behatList TYPE=${behatList.getClass()}"
def enviroments_b = [:]
for (int j = 0; j < behatList.size() ; j++) {
int index_b=j
enviroments_b["TestEnv_${e}
behat=${behatList[index_b]}"] = {
withEnv(["BEHAT_SUIT=${behatList[index_b]}"]) {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run BEHAT_SUIT=$BEHAT_SUIT"
"""
} //workspace
} //node
} //withEnv
} //enviroments_b
} //for
try {
parallel enviroments_b
} catch (error) {
throw (error)
}
} else if (env.ORO_TEST_SUITE == 'functional' ||
env.ORO_TEST_SUITE == 'unit' || env.ORO_TEST_SUITE == 'behat_wiring' ||
env.ORO_TEST_SUITE == 'javascript'){
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run $ORO_TEST_SUITE"
"""
}
}
} else {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run $ORO_TEST_SUITE"
"""
}
}
}
}
}
}
}
try {
parallel enviroments
} catch (error) {
throw (error)
}
@NonCPS
def sortSuites(suites){
def suitesStat=[AutoSuiteSet_1:435, AutoSuiteSet_2:323,
AutoSuiteSet_3:363]
def suites_time = suitesStat
// def suites_time = [:]
def suitesMap = [:]
if (suites_time.size()>0){
def timeLimit = suites_time.values().max() //find max value
for(s in suites){
x = suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
}else{
def suitesList = suites.collect { t -> [t] }
return suitesList
}
tasks = divideList(suitesMap)
tasks = sortListInMap(tasks)
tasksList = tasks.collect { t -> t.keySet()}
return tasksList
}
@NonCPS
def divideList(inputMap){
// Sort an incoming array in descending order
inputMap=sortMap(inputMap)
maxValue = inputMap.values().max()
// Resulting set of maps
def outputSet = []
// Variable for storing an individual map
def outputMap = [:]
for (inputElement in inputMap) {
// We add to the temporary map and consider the resulting sum of
the maps items
possibleMap = outputMap + [inputElement]
possibleMapSum = possibleMap.values().sum()
// If the amount in the temporary map is less than maxValue, go
to the next iteration, we will try to supplement the map with one more
element
if (possibleMapSum < maxValue) {
outputMap = possibleMap
// If the amount in the temporary map is maxValue, add a
temporary map to the result set and zero the temporary map
}else if(possibleMapSum == maxValue) {
outputSet.add(possibleMap)
outputMap = [:]
// If the amount in the temporary map exceeded maxValue, add
the previous version of the temporary map to the result set (without the
current element), and create a new temporary map with the current element
}else if(possibleMapSum > maxValue) {
outputSet.add(outputMap)
outputMap = [:]
outputMap.put(inputElement.key, inputElement.value)
}
}
if (outputMap) {
outputSet.add(outputMap)
}
return outputSet
}
@NonCPS
def sortMap(map) {
map.sort{a, b -> b.value <=> a.value }
}
@NonCPS
def sortListInMap(map) {
map.sort { a, b -> b.values().sum() <=> a.values().sum() }
}
06.08.2017 17:47, Michael Pailloncy пишет:
It seems like there are some variables used out of their scope and
some misused Groovy methods.
Here is a working version, IIUC your pipeline :-)
#!groovy
def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2',
'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2,
AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]
stage("test") {
node('master') {
behatList2=sortSuites(behatList, suitesStat)
echo "SUITES2=${behatList2}"
}
}
@NonCPS
def sortSuites(suites, suites_time){
def timeLimit = suites_time.values().max()
def suitesMap= [:]
for(s in suites){
def x = suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
def tasks = [suitesMap]
timeLimit = suitesMap.values().max()
while(canSplit(tasks, timeLimit)) {
tasks = tasks.collect { t ->
if(checkLimit(t, timeLimit)){
t = splitTo2(t)
}
t
}.flatten()
}
tasks.sort { a, b -> b.values().sum() <=> a.values().sum() }
// tasks = tasks.collect { t -> t.keySet()} // not working, multiple
elements are collected here
tasks = tasks.collectMany { t -> t.keySet()}
return tasks
}
@NonCPS
def checkLimit(t, timeLimit) {
if(t.values().sum()>timeLimit && t.size()>1){
return true
}else{
return false
}
}
@NonCPS
def canSplit(tasks, timeLimit) {
for(t in tasks) {
if(checkLimit(t, timeLimit)){
return true
}
}
return false
}
@NonCPS
def splitTo2(int_map) {
A=[:]
B=[:]
// int_map.sort{it.value} not working
// for(n in int_map.sort{it.value}) {
for(n in int_map) {
if (A.size() < B.size()) {
A.put(n.key, n.value)
}else{
B.put(n.key, n.value)
}
}
return [A, B]
}
However, what do you want to achieve exactly ? One of the main
advantage of Pipeline is its resumability after restart. But if you
use too many @NonCPS, you avoid this feature and this could lead to
unexpected behaviour sometimes.
Hopefully it helps :-)
2017-08-04 19:17 GMT+02:00 Slava Dubrovskiy <[email protected]
<mailto:[email protected]>>:
Hi.
I use a special algorithm to pre-sort the steps for parallel start.
Here is my test pipeline:
|
#!groovy
defbehatList
=['AutoSuiteSet_0','AutoSuiteSet_1','AutoSuiteSet_2','AutoSuiteSet_3','AutoSuiteSet_4','AutoSuiteSet_5']
defsuitesStat=[AutoSuiteSet_0:0,AutoSuiteSet_1:1,AutoSuiteSet_2:2,AutoSuiteSet_3:3,AutoSuiteSet_4:4,AutoSuiteSet_5:5]
stage("test"){
node('master'){
behatList2=sortSuites(behatList,suitesStat)
echo "SUITES2=${behatList2}"
}
}
@NonCPS
defsortSuites(suites,suites_time){
timeLimit =suites_time.values().max()
defsuitesMap=[:]
for(s insuites){
x=suites_time.find{artifact ->artifact.key ==s}
if(x){
suitesMap.put(x.key,x.value)
}else{
suitesMap.put(s,timeLimit)
}
}
tasks =[suitesMap]
timeLimit =suitesMap.values().max()
while(canSplit()){
tasks =tasks.collect {t ->
if(checkLimit(t)){
t =splitTo2(t)
}
t
}.flatten()
}
tasks.sort {a,b ->b.values().sum()<=>a.values().sum()}
tasks =tasks.collect {t ->t.keySet()}
returntasks
}
@NonCPS
defcheckLimit(t){
if(t.values().sum()>timeLimit &&t.size()>1){
returntrue
}else{
returnfalse
}
}
@NonCPS
defcanSplit(){
for(t intasks){
if(checkLimit(t)){
returntrue
}
}
returnfalse
}
@NonCPS
defsplitTo2(int_map){
A=[:]
B=[:]
for(n inint_map.sort{it.value}){
if(A.size()<B.size()){
A.put(n.key,n.value)
}else{
B.put(n.key,n.value)
}
}
return[A,B]
}
|
If I run it, I get the error:
|
an exception which occurred:
infield delegate
infield closures
inobjectorg.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7fd2cde1
Caused:java.io.NotSerializableException:java.util.LinkedHashMap$Entry
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at
org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at
org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at
org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOu
...
|
Please, help me what is wrong?
All methods under @NonCPS directive.
--
WBR,
Slava.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com
<https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
WBD,
Viacheslav Dubrovskyi
--
You received this message because you are subscribed to the Google Groups "Jenkins
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/2493fd53-6fd8-e528-462f-663457813e98%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
