The project I’m working on is a larger toolchain (written in Python) to run regression tests. The part that does the data comparison is fairly small. Speed is not crucial but doing the data comparison in python was incredibly slow. So we went with a C++ extension. For everything else python works just fine, though.
The software that is tested recently switched from netcdf to parallel netcdf. So the extension also needs to switch. I got it working now using the same trick the mpi4py project is using and it works (for now). Thanks for your help :) Best, Joel On 17 Sep 2015, at 21:49, Nick Papior <nickpap...@gmail.com> wrote: > Depending on your exact usage and the data contained in the CDF-5 files I > guess netcdf4-python would work for reading the files (if the underlying > netcdf library is compiled against pnetcdf). > However, this will not immediately yield mpi features. Yet, reading different > segments of files could be made embarrassingly parallel which might be OK, > but defeat the purpose of your code. > > Yet, why do you use python on-top of C++ for data comparison? If you need the > speed (you mentioned speed) why not do it in simple C++, C or fortran? > Data comparison can be made extremely easy in fortran. Sorry for the blurp ;) > > > 2015-09-17 18:20 GMT+00:00 Joel Hermanns <joel.herma...@gmail.com>: > > > FYI, you can also see what they have done in mpi4py to by-pass this problem. > > Could you elaborate on this or give me some pointer to other resources? > > > I would actually highly recommend you to use mpi4py rather than > > implementing this from scratch your-self ;) > > I fully agree that it is a bad idea to implement something like mpi4py from > scratch. However, I don’t plan to do this and > I’m not sure if mpi4py will work for us. This problem initially came up when > working on a thin layer around some parallel netcdf functionality to request > and compare data from NetCDF (especially CDF-5) files. > It is written in C++ due to performance reasons. Additionally, I’m not sure > if there is any up-to-date python library for parallel netcdf that could help > here. > As you can see, we don’t need full blown MPI features in python, and so I’m > not really sure if mpi4py can help us. > > Please correct me if I’m wrong! > > Best, > Joel > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/09/27613.php > > > > -- > Kind regards Nick > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/09/27615.php
