The problem occurs when a function having a threaded loop calling another
function with
a threaded loop inside.
here is a code sample computing the mean of a random matrix.
expected result 0.5
here is the code:
==========================================================================
debuglock=Threads.RecursiveSpinLock()
using Distances
n=100
A=rand(n,n)
vmat=Array{Array{Float64,2},1}(2)
vmat[1]=rand(n,n)
vmat[2]=rand(n,n)
#compute sum of column i
function fbidon(A::Array{Float64,2}, i::Int64, D::Distances.SemiMetric)
lock(debuglock)
@printf STDOUT "\n fbidon thread %d " Threads.threadid()
unlock(debuglock)
# to get cpu doing something
evaluate(D, A[:,i], A[:,1])
#
v=sum(A[:,i])
#
lock(debuglock)
@printf STDOUT "\n fbidon thread %d end " Threads.threadid()
unlock(debuglock)
#
return v
end
function meanMatThread(A::Array{Float64,2})
lock(debuglock)
@printf STDOUT "\n meanMatThreadedF thread %d " Threads.threadid()
unlock(debuglock)
#
n=size(A)[2]
v=zeros(n)
D=Jaccard()
vid=Array{Int64,1}(n)
Threads.@threads for i in 1:n
vid[i] = Threads.threadid()
v[i] = fbidon(A,i,D)
end
lock(debuglock)
@printf STDOUT "\n meanMatThreadedF thread %d end " Threads.threadid()
unlock(debuglock)
sum(v)/(n*n)
end
function meanMatThreadThread(vm::Array{Array{Float64,2},1})
res=Array{Float64,1}(2)
Threads.@threads for i in 1:2
res[i] = meanMatThread(vm[i])
end
return res
end
===============================================================================
the result in detail is given in file bugThread.txt
summary :
call to meanMat(vmat[1]) OK 0.49068859745047
call to meanMat(vmat[2]) OK 0.5044064092497642
call to res=meanMatThreadThread(vmat)
result is false
2-element Array{Float64,1}:
0.124943
0.128193
I ran with 4 threads, and I get the result of 1/4 threads.
With n = 50000 it I get really 0.125 !!
Moreover the interpreter gets in a fuzzy state as described in details in
joined bugThread.txt.
==============================================================
My configuration
julia> versioninfo(true)
Julia Version 0.5.0-rc3+0
Commit e6f843b
<https://github.com/JuliaLang/julia/commit/e6f843b0737bdffddbe4cf5c2f8b06278103fe64>
(2016-08-22 23:43 UTC)
Platform Info:
System: Linux (x86_64-unknown-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
WORD_SIZE: 64
Debian GNU/Linux testing (stretch)
uname: Linux 4.6.0-1-amd64 #1 <https://github.com/JuliaLang/julia/issues/1>
SMP Debian 4.6.4-1 (2016-07-18) x86_64 unknown
Memory: 125.84801864624023 GB (108943.20703125 MB free)
Uptime: 357928.0 sec
Load Avg: 0.0703125 0.17724609375 0.1572265625
Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz:
speed user nice sys idle irq
#1 <https://github.com/JuliaLang/julia/issues/1>-32 1200 MHz 5257848 s
6463890 s 1669061 s 1131089210 s 0 s
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.7.1 (ORCJIT, haswell)
Environment:
MANPATH =
TERM = xterm-256color
LD_LIBRARY_PATH =
/usr/local/qwt-6.1.2/lib:/usr/local/lib:/home.2/jpboth/Spectro/Spectviewer/lib/x86_64:/home.2/jpboth/lib:/usr/local/qwt-6.1.2/lib:/usr/local/lib:/home.2/jpboth/Spectro/Spectviewer/lib/x86_64:/home.2/jpboth/lib:
JULIA_NUM_THREADS = 4
julia> meanMat(vmat[1])
0.49068859745047
OK
===============================================
julia> meanMat(vmat[2])
0.5044064092497642
OK
==============================================
julia> res= meanMatThread(vmat[1])
meanMatThreadedF thread 1
fbidon thread 2
fbidon thread 1
fbidon thread 3
fbidon thread 2 end
fbidon thread 1 end
fbidon thread 4
fbidon thread 3 end
fbidon thread 2
fbidon thread 4 end
fbidon thread 2 end
fbidon thread 1
fbidon thread 3
fbidon thread 2
fbidon thread 3 end
fbidon thread 1 end
.....
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
meanMatThreadedF thread 1 end 0.49068859745047
OK
=====================================================================
julia> res= meanMatThread(vmat[2])
meanMatThreadedF thread 1
fbidon thread 1
fbidon thread 3
fbidon thread 2
fbidon thread 1 end
fbidon thread 4
fbidon thread 1
fbidon thread 4 end
fbidon thread 1 end
fbidon thread 3 end
fbidon thread 2 end
fbidon thread 3
.....................
fbidon thread 3 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
fbidon thread 4
fbidon thread 4 end
meanMatThreadedF thread 1 end 0.5044064092497642
Result OK
=========================================================
julia> res=meanMatThreadThread(vmat)
meanMatThreadedF thread 1
meanMatThreadedF thread 2
fbidon thread 1
fbidon thread 2
fbidon thread 1 end
fbidon thread 2 end
fbidon thread 1
fbidon thread 2
fbidon thread 1 end
fbidon thread 2 end
fbidon thread 1
fbidon thread 2
fbidon thread 1 end
fbidon thread 2 end
fbidon thread 1
.................................
fbidon thread 2
fbidon thread 2 end
fbidon thread 2
fbidon thread 2 end
fbidon thread 2
fbidon thread 2 end
fbidon thread 2
fbidon thread 2 end
fbidon thread 2
fbidon thread 2 end
fbidon thread 2
fbidon thread 2 end
2-element Array{Float64,1}:
0.124943
0.128193
RESULT NOT OK it should be roughly 0.5 0.5
if I increase n to 50000 I get 0.12489..... i.e 0.125 id est 1/4 of the
expected result.
and i used 4 threads see versioninfo log
It seems that the not all the thread results are collected , a barrier
problem ???
Sometimes I get also thinbgs like that...
2-element Array{Float64,1}
0.375061
6.94823e-310
roughly 3 * 0.125 , 0
One more detail
============
often after the call res=meanMatThreadThread(vmat)
I get also
julia> meanMatThread(vmat[2])
....
meanMatThreadedF thread 1 end 0.12483100788814704
but if insert a standard call
julia> sum(vmat[1])
4999.966884561036
then
julia> meanMatThread(vmat[2])
.....
meanMatThreadedF thread 1 end 0.4997901873733239
So the call to meanMatThreadThread(vmat)
seems to leave the julia interptreter a in a fuzzy state.
=========================================================================================
ulia> versioninfo(true)
Julia Version 0.5.0-rc3+0
Commit e6f843b (2016-08-22 23:43 UTC)
Platform Info:
System: Linux (x86_64-unknown-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
WORD_SIZE: 64
Debian GNU/Linux testing (stretch)
uname: Linux 4.6.0-1-amd64 #1 SMP Debian 4.6.4-1 (2016-07-18) x86_64 unknown
Memory: 125.84801864624023 GB (108943.20703125 MB free)
Uptime: 357928.0 sec
Load Avg: 0.0703125 0.17724609375 0.1572265625
Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz:
speed user nice sys idle irq
#1-32 1200 MHz 5257848 s 6463890 s 1669061 s 1131089210 s 0
s
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.7.1 (ORCJIT, haswell)
Environment:
MANPATH =
TERM = xterm-256color
LD_LIBRARY_PATH =
/usr/local/qwt-6.1.2/lib:/usr/local/lib:/home.2/jpboth/Spectro/Spectviewer/lib/x86_64:/home.2/jpboth/lib:/usr/local/qwt-6.1.2/lib:/usr/local/lib:/home.2/jpboth/Spectro/Spectviewer/lib/x86_64:/home.2/jpboth/lib:
JULIA_NUM_THREADS = 4
PATH =
/home.1/Opt/julia-0.5.0.rc3/bin:/home.2/jpboth/Spectro/Spectviewer/bin/x86_64:/home.1/Opt/julia-0.5.0.rc3/bin:/home.2/jpboth/Spectro/Spectviewer/bin/x86_64:/home.1/Opt/julia/bin:/home.2/jpboth/Spectro/Spectviewer/bin/x86_64:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
SPECTVIEWER_HOME = /home.2/jpboth/Spectro/Spectviewer
HOME = /home.2/jpboth
CLASSPATH = /usr/local/jdk1.8.0_45/jre/bin:/home.2/jpboth/lib/jclasses
INFOPATH = /usr/share/info:/usr/local/info:/usr/local/share/info:
WINDOWPATH = 1
JAVAPATH = /usr/local/jdk1.8.0_45/bin
Package Directory: /home.2/jpboth/.julia/v0.5
42 required packages:
- Cairo 0.2.34
- Calculus 0.1.15
- Cbc 0.2.3
- Clp 0.2.2
- Clustering 0.6.0
- Convex 0.4.0
- DReal 0.1.0
- DSP 0.1.1
- DataArrays 0.3.8
- DecisionTree 0.4.2
- Distances 0.3.2
- DualNumbers 0.2.2
- ExcelReaders 0.6.0
- ForwardDiff 0.2.5
- GSL 0.3.4
- Graphs 0.7.0
- Ipopt 0.2.4
- JSON 0.7.0
- JuMP 0.14.0
- JuMPChance 0.3.0
- KDTrees 0.6.0
- LARS 0.0.3
- Lasso 0.0.4
- LightXML 0.3.0 7467163e (dirty)
- Lora 0.5.4
- ManifoldLearning 0.1.0
- MultipleTesting 0.0.2
- MultivariateStats 0.3.0
- NLopt 0.3.3
- NLsolve 0.7.3
- NearestNeighbors 0.1.0
- NumericFuns 0.2.4
- ParallelAccelerator 0.1.8
- PolynomialRoots 0.0.4
- PyCall 1.7.1
- PyPlot 2.2.1
- QuickShiftClustering 0.1.1
- Rif 0.0.12
- SortingAlgorithms 0.1.0
- Stan 1.0.0 c7ba6a63 (dirty)
- Vega 0.6.8
- Wavelets 0.5.1
63 additional packages:
- AbstractDomains 0.1.0
- ArrayViews 0.6.4
- AxisAlgorithms 0.1.5
- Benchmark 0.1.0
- BinDeps 0.4.5
- Codecs 0.2.0
- ColorBrewer 0.3.0
- ColorTypes 0.2.6
- Colors 0.6.6
- Compat 0.8.8
- CompilerTools 0.1.6
- Compose 0.4.4
- Conda 0.2.3
- Contour 0.2.0
- DataFrames 0.7.8
- DataStructures 0.4.5
- Dates 0.4.4
- Devectorize 0.4.2
- Distributions 0.10.2
- ECOS 0.6.7
- FactCheck 0.4.3
- FixedPointNumbers 0.1.4
- FixedSizeArrays 0.2.3
- Formatting 0.1.5
- FunctionalCollections 0.1.3
- FunctionalData 0.1.2
- FunctionalDataUtils 0.1.0
- GLM 0.5.4
- GZip 0.2.20
- Gadfly 0.4.3
- Graphics 0.1.3
- Hexagons 0.0.4
- Interpolations 0.3.6
- Iterators 0.1.10
- KernelDensity 0.3.0
- LaTeXStrings 0.2.0
- Lazy 0.11.0
- Loess 0.0.7
- MacroTools 0.3.2
- Mamba 0.10.0
- MathProgBase 0.5.4
- Measures 0.0.3
- NaNMath 0.2.1
- NoveltyColors 0.1.0
- Optim 0.6.1
- PDMats 0.4.2
- Patchwork 0.2.2
- Polynomials 0.1.0
- PositiveFactorizations 0.0.2
- ProgressMeter 0.3.3
- Ratios 0.0.4
- Reexport 0.0.3
- ReverseDiffSource 0.2.3
- ReverseDiffSparse 0.5.8
- Rmath 0.1.2
- SHA 0.1.2
- ScikitLearnBase 0.1.1
- Showoff 0.0.7
- StaticArrays 0.0.4
- StatsBase 0.9.0
- StatsFuns 0.3.0
- URIParser 0.1.6
- WoodburyMatrices 0.2.0
julia>
