Aurélien Bouteiller wrote:
> From a pretty old experiment I made, compression was giving good
> results on 10Mbps network but was actually decreasing RTT on 100Mbs
> and more. I played with all the zlib settings from 1 to 9, and
> actually even the low compression setting was unable to reach decent
> performance. I don't believe that the computing/bandwidth ratio has
> changed to favor compression.
>
Ok.
So it is only useful to use compression in narrow special cases.
Thanks for info.
> Aurelien.
>
> Le 24 avr. 08 à 11:06, George Bosilca a écrit :
>
>
>> Actually, even in this particular condition (over internet)1
>> compression make sense only for very specific data. The problem is
>> that usually the compression algorithm is very expensive if you want
>> to really get a interesting factor of size reduction. And there is
>> the tradeoff, what you save in terms of data transfer you lose in
>> terms of compression time. In other terms, the compression became
>> interesting in only 2 scenarios: you have a very congested network
>> (really very very congested) or you have a network with a limited
>> bandwidth.
>>
>> The algorithm use in the paper you cited is fast, but unfortunately
>> very specific for MPI_DOUBLE and only works if the data exhibit the
>> properties I cited in my previous email. The generic compression
>> algorithms are at least one order of magnitude slower. And then
>> again, one needs a very slow network in order to get any benefits
>> from doing the compression ... And of course slow networks is not
>> exactly the most common place where you will find MPI applications.
>>
>> But as Jeff stated in his email, contributions are always welcomed :)
>>
>> george.
>>
>>
>> On Apr 24, 2008, at 8:26 AM, Tomas Ukkonen wrote:
>>
>>
>>> George Bosilca wrote:
>>>
>>>> The paper you cited, while presenting a particular implementation
>>>> doesn't bring present any new ideas. The compression of the data
>>>> was studied for long time, and [unfortunately] it always came back
>>>> to the same result. In the general case, not worth the effort !
>>>>
>>>> Now of course, if one limit itself to very regular applications
>>>> (such as the one presented in the paper), where the matrices
>>>> involved in the computation are well conditioned (such as in the
>>>> paper), and if you only use MPI_DOUBLE (\cite{same_paper}), and
>>>> finally if you only expect to run over slow Ethernet (1Gbs)
>>>> (\cite{same_paper_again})... then yes one might get some benefit.
>>>>
>>>>
>>> Yes, you are probably right that its not worth the effort in
>>> general and
>>> especially not in HPC environments where you have very fast network.
>>>
>>> But I can think of (rather important) special cases where it is
>>> important
>>>
>>> - non HPC environments with slow network: beowulf clusters and/or
>>> internet + normal PCs where you use existing workstations and
>>> network
>>> for computations.
>>> - communication/io-bound computations where you transfer
>>> large redundant datasets between nodes
>>>
>>> So it would be nice to be able to turn on the compression (for spefic
>>> communicators and/or data transfers) when you need it.
>>>
>>> --
>>> Tomas
>>>
>>>> george.
>>>>
>>>> On Apr 22, 2008, at 9:03 AM, Tomas Ukkonen wrote:
>>>>
>>>>
>>>>> Hello
>>>>>
>>>>> I read from somewhere that OpenMPI supports
>>>>> some kind of data compression but I couldn't find
>>>>> any information about it.
>>>>>
>>>>> Is this true and how it can be used?
>>>>>
>>>>> Does anyone have any experiences about using it?
>>>>>
>>>>> Is it possible to use compression in just some
>>>>> subset of communications (communicator
>>>>> specific compression settings)?
>>>>>
>>>>> In our MPI application we are transferring large
>>>>> amounts of sparse/redundant data that compresses
>>>>> very well. Also my initial tests showed significant
>>>>> improvements in performance.
>>>>>
>>>>> There are also articles that suggest that compression
>>>>> should be used [1].
>>>>>
>>>>> [1] J. Ke, M. Burtcher and E. Speight.
>>>>> Runtime Compression of MPI Messages to Improve the
>>>>> Performance and Scalability of Parallel Applications.
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>> Tomas
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
--
Tomas Ukkonen, tutkija
Finnish Geodetic Institute
PL 15 (Geodeetinrinne 2)
02431 MASALA, FINLAND
