Re: Calling python from C with OpenMP
wrote: > Second and most important question: When I run this code it sometimes > segementation faults, and sometimes some threads run normal and some > other threads says "Cannot call 'do_multiply'". Sometimes I get the > message: Fatal Python error: GC object already tracked. And some times it > even runs normally... > I understand there is some kind of race condition here, where python > tries to refer to some memory that has been already released. But how can > I avoid this? What am I doing wrong? (or, less likely, is this a bug?) You must own the GIL before you can safely use the Python C API, object creation and refcounting in particular. Use the "Simplified GIL API" to grab the GIL and release it when you are done. -- https://mail.python.org/mailman/listinfo/python-list
Re: python and matlab
Sam wrote: > is it able to utilize functions written in Python in Matlab? Yes, if you embed the Python interpreter in a MEX-file. -- https://mail.python.org/mailman/listinfo/python-list
Re: python and matlab
Sam Adams wrote:
> Thanks Sturla, could you please explain in more details, I am new to Python :)
All the information you need to extend or embed Python is in the docs.
Apart from that, why do you need Matlab? A distro like Enthought Canopy or
Anaconda has all the tools you will ever need for scientific programming.
Embedding Python is almost never the right solution to a problem. But if
you really need Matlab, you will be far better off by calling Matlab engine
from Python (e.g. using ctypes or Cython).
There are also packages like mlab and matlabwrap that will help you embed
Matlab in Python:
https://pypi.python.org/pypi/mlab
http://mlabwrap.sourceforge.net
Apart from that, you can also invoke Python from Matlab like this:
arg1 = 'script.py';
s = system(sprintf('python %s', arg1));
or
arg1 = 'some Python code';
s = system(sprintf('python -c "%s"', arg1));
Or if you are on Windows, you can use pywin32 to create an ActiveX server
with Python, and invoke that from Matlab.
Sturla
--
https://mail.python.org/mailman/listinfo/python-list
Re: python and matlab
Rustom Mody wrote: > What Sturla is probably saying is that the matmab-python imp-mismatch is > so high that jumping across is almost certainly not worth the trouble. I am saying that the abundance of Python packages for numerical and scientific computing (NumPy et al.) and their quality is now so good that binding Python to Matlab is not worth the effort. The only reason to do this would be if Matlab is needed for a very special reason. E.g. there might be a toolbox only available for Matlab, or there might be an administrative decision to use Matlab albeit a Python package is needed. But if it is just a general feeling that Python lacks the tools needed for numerical computing, then it is a false assumption and not worth it. Note that hardly any of the tools used for numerical computing with Python is in the standard library. They are all focused around NumPy as the central package. See scipy.org for further information. > And BTW have you seen sage? > http://www.sagemath.org/ Sage is supposed to be a computer algebra system, i.e. a free alternative to Maple or Mathematica. Matlab is not a CAS system but a scripting environment for numerical computing. Enthought Canopy and Anaconda are similar environments based on Python. enthought.com continuum.io While they require a payed license, it is also possible to hand-pick the needed packages and install them ourselves. But for libraries like NumPy to be liked against high-performance libraries like Intel MKL, we must build them ourselves or use one of the commercial Python distros for scientific computing. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Sort one sequence by O(n) in time and O(1) in space
Wesley wrote: > [Wesley] This is not homework:-) > And actually I am new to algorithm, so you guys can feel free to say anything > you want In general, we cannot sort a sequence in O(n) time. O(n log n) is the lower bound on the complexity. -- https://mail.python.org/mailman/listinfo/python-list
Re: Sort one sequence by O(n) in time and O(1) in space
Chris Angelico wrote: > That's assuming it really is a sort operation. The problem description > isn't entirely clear on this point, but if it's actually a zip, then > it can definitely be done in O(n). Ah, I didn't read it carefully enough. :-) Granted, a zip can be done in O(n) time and O(1) memory using a generator, which by the way is what itertools.izip does. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python has arrived!
Grant Edwards wrote: > According to > http://www.theregister.co.uk/2014/11/06/hackers_use_gmail_drafts_as_dead_drops_to_control_malware_bots: > > "Attacks occur in two phases. Hackers first infect a targeted >machine via simple malware that installs Python onto the device, >[...]" > A virus that runs on Python. It had to happen sooner or later. -- https://mail.python.org/mailman/listinfo/python-list
Re: "**" in python
Abdul Abdul wrote: > Wxy**2 > > What do ** mean here? Exponentiation. Same as ** means in Fortran. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Proposed new conditional operator: "or else"
Zachary Ware wrote: > On Tue, Dec 2, 2014 at 11:18 AM, Roy Smith wrote: >> Wouldn’t it be neat to write: >> >>foo == 42 or else >> >> and have that be an synonym for: >> >> assert foo == 42 >> >> :-) > > Never going to happen, but I like it! Perhaps raise IntimidationError > instead of AssertionError when it fails? I guess the "or else" statement should do this: or else where the statement is executed if the statement evaluates to false. In absence of an explicit threat or else it should raise IntimidationError if evaluates to false false. If evaluates to true it should just return . Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Proposed new conditional operator: "or else"
Dennis Lee Bieber wrote: >> foo == 42 or else >> > > Has a PERL stink to it... like: foo == 42 or die I think this statement needs to take ellipsis as well foo == 42 or else ... Sturls -- https://mail.python.org/mailman/listinfo/python-list
Re: Python, C++ interaction
Cython is nearly always the answer to scientific computing in Python, including wrapping C++. Sturla Michael Kreim wrote: > Hi, > > we are working on a small scientific program that helps us in developing > and testing of new numerical methods for a certain type of biochemical > problems. I spare you the math ;-) > > We code our new methods in Python and compare them with the existing > methods. Unfortunately, these existing methods are quite slow and need a > lot of runtime. Therefor we implemented them in C++. > > Now, we like to combine these two approaches. So we can develop a new > method in Python and compare it easily with the results from the C++ > functions. > > I did some googleing on extending Python by C++ code but I did not find > something that satisfies me. I gave SWIG a try, but several webpages > disadvised me of using it. Also my small experiments did not work. Now, > I read about ctypes. But as far as I understood it, I have to write a C > wrapper for my C++ code and then a Python wrapper for my C code. That > seems a little complicated. > > My C++ code is structured as a class that contains several variables and > functions. Then everything is compiled as a shared library. I can write > things like: > > MyClass.initialvalue = 5; > MyClass.timestep = 0.1; > ... > MyClass.solve(); > > I would like to keep this structure in python, but if I understand > ctypes correctly I would loose the class approach. > > So after the long writeup I come to my questions: > > What are you using to wrap C++ classes for Python? > Can you recommend swig? Should I give it another try? > Did I misunderstood ctypes? > > Also later on in our code development we would like to pass python > lambda functions to C++. So far I understood that this seems not to be > possible. Do you have any ideas or hints on this topics? > > Thank you very much. > > Cheers, > > Michael -- https://mail.python.org/mailman/listinfo/python-list
Re: Python, C++ interaction
Dan Stromberg wrote: > 1) writing in Cython+CPython (as opposed to wrapping C++ with Cython) That is an option, but it locks the code to Cython and CPython forever. C and C++ are at least semi-portable. > 2) using numba+CPython (It's a pretty fast decorator - I've heard it's > faster than Cython) Numba is usually not "faster than Cython" (Cython can be as fast as C), but it can be pretty fast. Sometimes it is comparable to -O2 in C for the subset of Python it supports, but usually a bit slower. But if you can use it, it is easier to use than Cython. There are no extra compilation steps, etc. Just add a couple of decorators to the Python code and it takes off like a rocket. For anyone who are familiar with PyPy and Psyco, Numba is far better than those. It is a Python JIT compiler that often can perform better than the Java VM. Numba will also JIT-compile Python code that uses ctypes or cffi to call external libraries down to almost zero overhead. You forgot to mention using Fortran and f2py. Many scientists and engineers prefer Fortran to C and C++ because it is easier to use. And Fortran 90 and later standards are not anything like the loathed Fortran 66 and 77 languages. Fortran is a high-level language particularly suited for numerical computing, C is a semi-portable high-level assembler. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: jitpy - Library to embed PyPy into CPython
Albert-Jan Roskam wrote: > Interesting, but it is not clear to me when you would use jitpy instead > of pypy. Too bad pypy alone was not included in the benchmarks (cython > would have also been nice). And Numba can JIT compile this far better than PyPy and jitpy. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python, C++ interaction
On 05/12/14 23:17, wesleiram...@gmail.com wrote: m'giu vous êtès nom souris, pseudo nom cha'rs out oiu êtès, i'ret egop c'hâse I have not idea what that means, but I am sure it would be interesting if I knew French (or whatever it is). Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing process termination
Why not use a multiuser database server instead of trying to make one? You do not have the resources to a better job on your own. You know where to find Firebird SQL, MariaDB, MySQL, PostegreSQL, IBM DB2, Oracle, etc. Personally I prefer Firebird because like SQLite the database is stored in a file. Another nice way to get the database in a file is to run it in an Oracle VirtualBox VM. SQLite also allows multiple connections, by the way, but it does not scale very well. Regards, Sturla Charles Hixson wrote: > In order to allow multiple processes to access a database (currently > SQLite) I want to run the process in a separate thread. Because it will > be accessed from multiple processes I intent to use Queues for shifting > messages back and forth. But none of the individuals processes will > know when all of the others are through. It there a way to properly > close the database, or does it need to be done from the parent process? > > What I want to do is detect that a request to shutdown the Process has > been received, and then execute a close procedure that cleans things up > and shutdown. Terminate is the obvious procedure to use, but that comes > with a warning not to use it if there is an queue attached (and without > defining attached). > OTOH, since the queue will never be read after the process has been > shutdown, perhaps rendering it unusable is the proper thing. > > Could someone comment on this step of the design? > > The basic idea is: > > class handledb(Process): > def __init__(self, q, otherstuff): > self.q = q > >def run(self, msg): > while (true): > if self.q.empty(): > sleep(someamountoftime) > else: > while (not self.q.empty()): > read and parse message > if message says to shutdown: self.shutdown() > >def shutdown(self): > close things down > ?? terminate ??<<-- should this be done > while q is live? -- https://mail.python.org/mailman/listinfo/python-list
Re: Trees
On 20/01/15 01:49, Dan Stromberg wrote: I think probably the most common need for a tree is implementing a cache, That is probably true, at least if you're a squirrel. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
Steven D'Aprano wrote: > Remember too that type-hinting will *absolutely* remain *completely* > optional for Python. Developers can choose to use it or not, No! Developers have to do what managers and customers tell them to do. They will start to require type hinting everywhere. And then the question is what Python has to offer over Java or Swift. Python will no longer be dynamic, it will just be a slow static language. Yes, Python could still be used as a dynamic language, but nobody will allow you to do it. Even packages in widespread use will be banned because they don't typehint. And then there will be complaint about lack of such packages. And in 5 years every textbook read by new Python programmers will require type hinting as a "best practice". Forget about compatibility with Python 2. People will upgrade from Python 2 to Swift. And then goodbye. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
Marko Rauhamaa wrote: > I think the SATAN is in the optional type declarations, not in the > particular syntax chosen. Yes. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
Skip Montanaro wrote: > FUD? What evidence do you have that this will be the way things shake out? I don't underestimate the stupidity of those who are not writing the code themselves. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
Chris Angelico wrote: > Uhh... if your managers and customers are stipulating non-Pythonic > coding styles, then it's time to find new managers/customers. If > they're not writing the code, code quality shouldn't be their concern. I am saying the day someone requires me to write a type hint, I will no longer be using Python or recommending it. Python is great, but not that great. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
Mark Lawrence wrote: > If they're too stupid to know the > meaning of the word "hint" that's their problem. It will also be Python's problem, because people are that stupid. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
On 22/01/15 20:10, Mario Figueiredo wrote: Customers don't have access to static analysis output and project managers should know better than to demand static analysis without properly contextualize it. I just don't see a project manager having no idea what static analysis means. I don't know how it works in you country, but here projects managers are selected for having any kind of skill other than technical knowledge. I know of one who got the job because he was a professional hockey player. So yes, they can very well be unaware what static analysis means. But what they will do is pick up "buzzwords" from the kind of journals that project managers read. And then they want that buzzword implemented. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
On 22/01/15 20:43, Skip Montanaro wrote: The way you couched your opinion as a certainty, as if you could see the future, How do you know I cannot? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
On 22/01/15 21:03, Mario Figueiredo wrote: That is fine. But then the problem isn't type hinting, is it? Neither I think you are suggesting we don't introduce language because there are bad project managers out there. The problem is then bad project managers. That has nothing to do with type hinting, or Python for that matter. It has everything to do with Python if we facilitate them. Type hinting will be mandatory because of bad managers. But then someone is going to ask what benefit Python has to offer: Readabiliy? No ... down the drain. Dynamic typing? No ... down the drain. Speed? No ... still 200x slower than Swift. Why go for Python? There is no benefit to it any longer. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
On 22/01/15 23:08, Ian Kelly wrote:
T = TypeVar('T')
def adder(a: T, b: T) -> T:
return a + b
I'm not thrilled about having to actually declare T in this sort of
situation, but I don't have a better proposal.
Here is a better proposal:
def adder(a, b):
return a + b
Sturla
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python is DOOMED! Again!
On 23/01/15 04:53, Steven D'Aprano wrote: If your manager is so bad, why isn't he insisting that you program in PHP or Java or Algol 68 [insert name of some language you dislike] instead of Python? Is your bad manager forcing you to write Java-style code in Python, or insisting on Hungarian Notation? Nobody is forcing me to do anything. But without a thriving community, Python has no value to me. If we add type hinting, then that community will go away, because other languages are better options. Who will win? Julia? Ruby? They are all waiting in line for Python to fail. Then Python will be like Smalltalk. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing module backport from 3 to 2.7 - spawn feature
Skip Montanaro wrote: > Can you explain what you see as the difference between "spawn" and "fork" > in this context? Are you using Windows perhaps? I don't know anything > obviously different between the two terms on Unix systems. spawn is fork + exec. Only a handful of POSIX functions are required to be "fork safe", i.e. callable on each side of a fork without an exec. An example of an API which is not safe to use on both sides of a fork is Apple's GCD. The default builds of NumPy and SciPy depend on it on OSX because it is used in Accelerate Framework. You can thus get problems if you use numpy.dot in a process started with multiprocessing. What will happen is that the call to numpy.dot never returns, given that you called any BLAS or LAPACK function at least once before the instance of multiprocessing.Process was started. This is not a bug in NumPy or in Accelerate Framework, it is a bug in multiprocessing because it assumes that BLAS is fork safe. The correct way of doing this is to start processes with spawn (fork + exec), which multiprocessing does on Python 3.4. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing module backport from 3 to 2.7 - spawn feature
Andres Riancho wrote: > Spawn, and I took that from the multiprocessing 3 documentation, will > create a new process without using fork(). > This means that no memory > is shared between the MainProcess and the spawn'ed sub-process created > by multiprocessing. If you memory map a segment with MAP_SHARED it will be shared, even after a spawn. File descriptors are also shared. -- https://mail.python.org/mailman/listinfo/python-list
Re: RAII vs gc (was fortran lib which provide python like data type)
Rustom Mody wrote:
> The case of RAII vs gc is hardly conclusive:
>
> http://stackoverflow.com/questions/228620/garbage-collection-in-c-why
The purpose of RAII is not to be an alternative to garbage collection
(which the those answers imply), but to ensure deterministc execution of
setup and tear-down code. The Python equivalent of RAII is not garbage
collection but context managers.
Those answers is a testimony to how little the majority of C++ users
actually understand about the language.
A C++ statement with RAII like
{
Foo bar();
// suite
}
is not equivalent to
bar = Foo()
in Python. It actually corresponds to
with Foo() as bar:
Sturla
--
https://mail.python.org/mailman/listinfo/python-list
Re: [OT] fortran lib which provide python like data type
Michael Torrie wrote: > Yes I can tell you haven't used C++. Compared to C, I've always found > memory management in C++ to be quite a lot easier. The main reason is > that C++ guarantees objects will be destroyed when going out of scope. > So when designing a class, you put any allocation routines in the > constructor, and put deallocation routines in the destructor. And it > just works. This is something I miss in other languages, even Python. Python has context managers for that. -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing module backport from 3 to 2.7 - spawn feature
On 30/01/15 23:25, Marko Rauhamaa wrote: Sturla Molden : Only a handful of POSIX functions are required to be "fork safe", i.e. callable on each side of a fork without an exec. That is a pretty surprising statement. Forking without an exec is a routine way to do multiprocessing. I understand there are things to consider, but all system calls are available and safe. POSIX says this: - No asynchronous input or asynchronous output operations shall be inherited by the child process. - A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. - Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. - When the application calls fork() from a signal handler and any of the fork handlers registered by pthread_atfork() calls a function that is not asynch-signal-safe, the behavior is undefined. Hence you must be very careful which functions you use after calling forking before you have called exec. Generally never use an API above POSIX, e.g. BLAS or Apple's CoreFoundation. Apple said this when the problem with multiprocessing and Accelerate Framework first was discovered: -- Forwarded message -- From: Date: 2012/8/2 Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate / GCD after in a forked process To: **@** Hi Olivier, Thank you for contacting us regarding Bug ID# 11036478. Thank you for filing this bug report. This usage of fork() is not supported on our platform. For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX. We recommend that you either restrict usage of blas to the parent or the child process but not both, or that you switch to using GCD or pthreads rather than forking to create parallelism. Also see this: http://bugs.python.org/issue8713 https://mail.python.org/pipermail/python-ideas/2012-November/017930.html Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Monte Carlo probability calculation in Python
Paul Moore wrote: > > Yes. And a number of other variations. None gave anything that seemed to > relate. It's quite likely though that I'm simply not understanding how > things like pymc (which came up in the searches) might help me, or how to > convert my problem into a Monte Carlo integration problem (another topic > that came up a lot, for example). So if there are specific links from > such a search that match well to the problem as I described it above, I'd > be really grateful for pointers. PyMC and emcee are used for solving complicated integrals that often turn up in Bayesian statistics. They have the same usecase as the R-like application BUGS (WinBUGS). If you don't know what Markov Chain Monte Carlo, Gibbs sampler or Metropolis-Hastings means you probably don't want this. It is rather hardcore numerical mathematics for solving a particular problem (multidimensional integrals that are not analytically tractable), not something you would use for any kind of Monte Carlo simulation. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python discussed in Nature
On 12/02/15 15:39, Marko Rauhamaa wrote: I write both Py2 and Py3 code, but I keep the two worlds hermetically separated from each other. In SciPy world we run the same code on Python 2 and Python 3. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug in timsort!?
On 24/02/15 22:34, Roy Smith wrote: http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/ This is awful. It is broken for arrays longer than 2**49 elements. With 8 bytes per PyObject* pointer this is >4096 terabytes of RAM. I don't see how we can fix this in time. Oh yes, and they mention that TimSort is used on billions of devices due to Android mobile phones. This is clearly very relevant for mobile phones. Next thing you know your litte Samsung Galaxy with more than 4096 terabytes breaks down from a stack overflow in TimSort. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug in timsort!?
On 25/02/15 15:33, Chris Angelico wrote: It's even worse than that. Unless you have a list of 2**49 references to the same few objects, you're going to need to have some actual content for each one. The absolute best you could do is to sort integers, which would take 32 bytes each [1]; if you're sorting strings, they'll be 90 bytes each, so the integers are our best bet. So add another *five* powers of two to the RAM requirements. In that case you also need to add the PyObject_HEAD overhead for each object. ;-) Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug in timsort!?
On 25/02/15 17:04, Peter Otten wrote: These guys found a bug that is subtler than what most of us have dealt with in a widely used piece of code originally developed by one of the smarter members of the python "community". I bow my head to them and say thank you. I am not joking about that. It is more the hype this gets that indicates TimSort is already broken today, and even on your cell phone. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Bug in timsort!?
On 25/02/15 18:22, Mario Figueiredo wrote: And also presented a solution. Which also was incorrect :-D But now Benjamin Peterson has finally fixed it, it appears: http://bugs.python.org/issue23515 Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: which async framework?
Chris Withers wrote: > Hi All, > > I see python now has a plethora of async frameworks and I need to try > and pick one to use from: > > - asyncio/tulip > - tornado > - twisted Looking at Tornado's examples on the web I find this: tornado.ioloop.IOLoop.instance().start() This single line of code says more than thousand words. But it boils down to: (1) This was written by some Java guys. (2) Someone used Python to write Java. And that's all I need to know about Tornado. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: which async framework?
Chris Withers wrote: > Hi All, > > I see python now has a plethora of async frameworks and I need to try > and pick one to use from: > > - asyncio/tulip > - tornado > - twisted I'd go for using iocp, epoll and kqueue/kevent directly. Why bother to learn a framework? You will find epoll and kqueue/kevent in the select module and iocp in pywin32. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: which async framework?
Marko Rauhamaa wrote: > Now, I've taken a brief look at the new asyncio and it looks as if it > has everything one would hope for (and then some). You'd still need to > supply the protocol implementations yourself. Tulip (the new async module) is nice. But I am a bit confused as to how it combines the IOCP model on Windows with the "readyness" signalling (epoll, kqueue) on Linux, *BSD and Apple. Because these paradigms are so inherently different, I don't see how both can be used effectively with the same client code. But Guido/BDFL is a smart guy and probably knows this better than me :) Another thing is that there is no IOCP support on Solaris, AIX and z/OS using Tulip, only Windows. But Windows is not the only OS with IOCP. I'd prefer that over /dev/poll any day (does Tulip use /dev/poll by the way?) Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: which async framework?
Antoine Pitrou wrote: > Yes, why use a library when you can rewrite it all yourself? This assumes something equivalent to the library will have to be written. But if it can be replaced with something very minimalistic it is just bloat. I would also like to respond that the select module and pywin32 are libraries. So what we are talking about here is not using a library versus issuing low-level system calls, but using a cross-platform library instead of having separate modules with system calls for Linux, *BSD/Apple and Windows. Another thing is that co-routines and "yield from" statements just makes it hard to follow the logic of the program. I still have to convince myself that a library for transforming epoll function calls into co-routines is actually useful. If it just turns otherwise readable code into something most Python programmers cannot read it is better left out of the project. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: which async framework?
Antoine Pitrou wrote: > This is the usual assumption that high-level libraries are made of useless > cruft piled up by careless programmers. It often is the case, particularly in network programming. But in this case the programmer is Guido, so it doesn't apply. :) > What irks me with your response is that you phrased it as though writing > a good event loop was an almost trivial thing to do, which it is not > once you start considering multiple use cases and constraints. Right. But in my programs an event loops does not have multiple usecases. I know all the details about my usecase. I am more concerned that multiple frameworks have their own event loops. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Examples of modern GUI python programms
Metallicow wrote: > My opinion would be wxPython if not actually using for a mobile, or > PySide if you are. Both of these have acceptable licenses if you want to > go commercial also without having to pay for commercial library usage. If you are to distribute a program using LGPL software on AppStore or Gopgle Play, then remember that the user must be allowed to relink the program with anpther version of the library. That is an LGPL requirement. I don't see how this requirement can be satisfied in this case. Thus, LGPL on AppStore or Google Play is probably put of the question. At least on iOS, the user cannot change what you put in an App bundle. This excludes wxPython and PySide. Thus, the only viable cross-platform choices are commercial PyQt + commercial Qt or tkinter. In case of iOS, PyObjC is also an option. py2app will also be useful for creating App bundles on iOS. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Examples of modern GUI python programms
Metallicow wrote: > One would have to tool through the PySide agreement for their specifics, > but as I recall it is exactly the same as Qt is, which makes sense. According to their web page, PySide is only LGPL. Qt is LGPL or commercial. > Just because a library is LGPL doesn't mean the authors code has to be > depending on the circumstances. That just means usually you have to be > able to provide the library code(and your mods to it) used. No, that would be MPL (Mozilla Public License). Many believe LGPL implies the same freedom as MPL. It does not. LGPL also means you also have to give the user a means to "relink the program with a different version of the library". That is a less known restriction of LGPL. Usually this is done by providing the LGPL library as a DLL. But a DLL is actually not sufficient, in case a different version of the library breaks the application binary interface (ABI). In case of ABI breakage, LGPL means the user be given access to the program source code to recompile and relink the program. Because of the closed nature of app bundles on iOS, the user cannot do anything with an .so or .dylib file. Thus, the DLL solution to LGPL infestation is not possible on iOS, even if it were sufficient. MPL is basically a version og LGPL that has removed the requirement to make relinkage possible. That is e.g. why a library like Eigen is released as MPL instead of LGPL. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Line of best fit
Roy Smith wrote: > Please tell us more about the environment you're working in. I'm > guessing from the fact that you're calling plt.plot(), that you've > already got some add-on modules loaded. Pandas, maybe? Probably matplotlib.pyplot -- https://mail.python.org/mailman/listinfo/python-list
Re: Examples of modern GUI python programms
Wolfgang Keller wrote: >> Id like to ask.. do you know any modern looking GUI examples of > Judging from the example screenshots on their website, Kivy might be > adequate. If you want to build something from scratch, libSDL is excellent and free (zlib license). Official supported platforms are: Windows XP/Vista/7/8 Mac OS X 10.5+ Linux 2.6+ iOS 5.1.1+ Android 2.3.3+ libSDL can be used from Python using ctypes or Cython. There is no GUI, but you can draw whatever you like. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Examples of modern GUI python programms
Wolfgang Keller wrote: > Judging from the example screenshots on their website, Kivy might be > adequate. Kivy depends on PyGame which is GPL, and can only be used to build GPL software. -- https://mail.python.org/mailman/listinfo/python-list
Re: Examples of modern GUI python programms
On 03/04/14 16:02, Robert Kern wrote: Kivy depends on PyGame which is GPL, and can only be used to build GPL software. It is not. http://www.pygame.org/LGPL Their web paged says GPL, but I assume that is an error. Is Python allowed on iOS anyway? Apple used to ban any code not written in C, C++, Objective-C and Objective-C++. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Examples of modern GUI python programms
Chris Angelico wrote: > Where? I can't see it. The home page redirects me to /news.html which > doesn't say anything about GPL (other than in its collection of tags, > which seem to be for finding other people's projects - that is, > clicking that link takes you to a list of all pygame-using projects > that are GPL'd); on the Documentation page, the license is clearly > LGPL. > http://www.pygame.org/wiki/about "Pygame is free. Released under the GPL License, you can create open source, free, freeware, shareware, and commercial games with it. See the licence for full details." But as pointed out, it seems to be a typo. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python streaming media server
Wesley wrote: > Anyone knows open source streaming media server written by Python? > > I am trying to setup a streaming media server in python, wanna find an > existing one and have a look. Not open source, but there is a famous closed-source one called YouTube. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python streaming media server
Wesley wrote: >> Not open source, but there is a famous closed-source one called YouTube. > > Are you kidding? > I know youtube, but do you think we can use it setup our own streaming media > server? Obviously not. Before YouTube was bought by Google, it was common knowledge that it ran on Stackless Python. So a streaming media server on Python is absolutely possible. But no, I don't know of one you can set up and use on your own. You can make a highly scalable server with PyZMQ and Tornado or Twisted. NumPy is great for storing binary data like media streams. HDF5 (PyTables or h5py) might be a better database than some SQL server, as it is capable of highly scalable parallel binary i/o. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Dennis Lee Bieber wrote: > That's been my experience too... Threading works for me... My attempts > at so called asyncio (whatever language) have always led to my having to > worry about losing data if some handler takes too long to return. > > To me, asyncio is closer to a polling interrupt handler, and I still > need a thread to handle the main processing. On a 32 bit system, the virtual address space with limit the scalabiliy for multi-threading in parallel i/o. That is, the stack space for each thread can quickly become a problem. Memory is not a problem on 64-bit servers. Multithreading can be used to solve the C10K problem, contrary to common belief. Before you dissmiss threads, take a look at this: http://www.mailinator.com/tymaPaulMultithreaded.pdf http://stackoverflow.com/questions/17593699/tcp-ip-solving-the-c10k-with-the-thread-per-client-approach My personal feeling is that asynchronous i/o is mostly useful on 32-bit systems, and the problem it actually solves is the limited virtual address space. On a 64 bit system we can just throw more RAM at it and threads be fine. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Marko Rauhamaa wrote: > The main problems with threads include: > > * Thread-safety is rarely done right. Also, when it's done wrong, it >can be virtually impossible to fix it without a significant rewrite. >This is not a theoretical concern: I have had to deal with the >resulting nightmares in my work. > > * There is no accepted, taught, industry-wide discipline on proper >thread-safety practices so every developer has to improvise. I have >come up with a "bullet-proof" way of developing with threads, but >even that methodology has nasty corner cases. > > * Thread-safety cannot be abstracted out. IOW, divide and conquer >doesn't work. You can't hide the locking inside a class and forget >about it. The entire application must be aware low-level thread >synchronization needs. The problem here is the belief that "thread-safety cannot be abstracted out". It can. The solution is to share nothing and send messages through queues. If you start to use mutexes and conditions all over your code, you might shoot yourself in the foot, eventually. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
alister wrote: > As my only professional coding experience has been with embedded 8 bit > processors with limited resources i naturally abhorrent to the process of > "Just throw more RAM (Or any other resource for that matter)at it". I understand. I do not advocate threads for parallel i/o on 8, 16 or 32 bit systems – nor on 64 bit systems without sufficient RAM. But I do advocate is that buying RAM is cheaper than buying developer time. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Roy Smith wrote: > Let's say I've got a program which consumes 60 GB of RAM, so I'm renting > the 2xlarge instance to run it. My software architect could recode the > program to be more efficient, and fit into just 30 GB, saving me > $3000/year. How much of his time is it worth to do that? He's costing > me about $600/day, so if he can do it in a week, it'll take a year to > recoup my investment. Exactly. That is what I said "just throw more RAM at it". We see this in scientific HPC too. What does it cost of optimizing software compared to just using a bigger computer? It virtually never pays off. Or Python related: Python might be slow, but how much should we value our own time? A simulation which took one week to complete, how long did it take to write the code? When should we use C++ or Fortran instead of Python? Ever? There is a reason scientists are running Python on even the biggest supercomputers today. Hardware might be expensive, but not compared to human resources. And that gap just increases as hardware is getting cheaper. So we should optimize for human resources rather than for hardware. And with 64 bit that is finally possible. (It isn't always possible with 32 bit, so that is a different story.) Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Roy Smith wrote: > Thread 1 and Thread 2 use a pair of queues to communicate. T1 sends > work to T2 using Q1, and T2 sends back results using Q2. > > T1 pushes Item1 onto Q1, and waits for Result1 to come back on Q2. > > T2 reads Item1 from its end of Q1, and waits to read Item2, which it > needs to compute Result1. > > Sounds like a deadlock to me. As it turns out, if you try hard enough, you can always construct a race condition, deadlock or a livelock. If you need to guard against it, there is paradigms like BSP, but not everything fits in. a BSP design. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
alister wrote: > My main point was that when you have only 8K of ROM & 128 byte of ram you > have to think about your algorithms first. And that also happens if you have a 32 bit system, with just 2 GB of memory available to user space (the operating system reserves the upper half of the 4 GB address space). Using threads to solve a C10K problem means you must limit yourself to e.g. 150KB of memory (including stack space) per thread. Is that doable? Perhaps not. Can you put 10,000 threads in the kernel and hope for good performance? On Windows, yes, at least since Windows 2000. On Linux? Only recently. That I think was the real argument for asynchronous i/o. Today this is not so relevant, but idea that asynchronous i/o is fundamentally better than threads is still with us. So what we are left with, then, in favor of non-blocking i/o is ability to time-out a non-responsive i/o operation. But that does not mean we need an asynchronous event-driven server. (Actually we can time-out blocking i/o with threads, but it involves using two threads per connection instead of one.) Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Marko Rauhamaa wrote: > Queues are fine if you hermetically insulate your objects. IOW, you > group your objects in single-threaded pseudoprocesses that don't make > any direct method calls to each other. If you do that, you might as well > use real processes. That way, you can even naturally enforce your > insulation assumption against accidents. No, 10,000 processes will not do. First, they use more resources than threads, at least on Windows. Second, if something fails, you have 10,000 worker processes to kill. You also risk flooding your system with zombies. So thanks, but no thanks. I would consider a small pool of processes to utilize all cores on the CPU, though. But each process would have hundreds or thousands of threads. .NET AppDomains is an attempt to solve this. Python do not support AppDomains in the interpreter. It would be nice if it did, though, e.g. a thread with an isolated interpreter instance (similar to tcl). But currently the Python interpreter is not designed for that, as it has states which are lobal to the process, not just the interpreter. If we insulate objects completely, we must also serialize (pickle) them before sending them off as messages on a queue. That can be a major bottleneck. At least it is in numerical computing with Python. Avoiding serialization is also an important consideration. But IPC itself is very fast, so the important part is packing a object into a byte string and unpacking, not the overhead involved in sending it. Unix domain sockets and Windows named pipes are close to a memcpy in performance, with very little overhead. Pickle on the other hand is "very slow". Serialization is hard ot get around if we use processes for multi-core CPUs, but in a pure multithread design it is not an issue. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Paul Rubin wrote: > Sturla Molden writes: >> When should we use C++ or Fortran instead of Python? Ever? > > When performance matters? Sometimes, but usually performance only matters in certain types of calculations like matrix algebra or FFTs. But we always use specialized libraries for that, such as Intel MKL, OpenBLAS, AMD ACML, or Apple's Accelerate Framework. "When performance matters" is not really a good answer, because performance might not matter where we think or as much as we think. For examle if I were to write a Levenberg-Marquardt solver, I would e.g. find that 95% of the time is spent solving the linearized least-squares equations (J' J + lambda diag(J' J)) dx = J' error using QR or SVD. If I used LAPACK method DGELS or DGELSS to solve these equations with QR or SVD, I would find that my otherwise idential Fortran and Python code would perform roughly the same. None of the performance libraries I mentioned cares if I am calling them from Python or Fortran. The differences in runtime would thus be in the 5 % spent outside the performance libraries. And if i/o is more efficient in Python, which is very likely, it might even be that the Python version runs faster. So the question is, when does it matter for me? Now and then, perhaps, but very rarely. For example it did matter when we wrote the cKDTree object for SciPy. But it never matters when I use it. I will make the bold claim that 90 % of the use of C++ or Fortran code in scientific computing stems from the misconception that "writing everything" in a compiled language will be faster. I have done this mistake myself, and lost valuable time. I have promised myself I will not do it again, and then realized I have done the same mistake yet again. It is a very dangerous misconception. Always prototype in Python first, then profile, then choose between optimization or using a bigger computer. Starting out by writing C, C++, Cython or Fortran is a trap. Yet many do, myself included, because we errorneously trust ourselves to know when we should just use Python. But the thing is, nobody does, we just think we do. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
On 07/04/14 05:56, Chris Angelico wrote: On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith wrote: There is (or at least, was) another reason. Creating a new process used to be far more expensive than creating a new thread. In modern Unix kernels, however, the cost difference has become much less, so this is no longer a major issue. Unix maybe, but what about Windows? Is it efficient to create processes under Windows? Processes are very heavy-weight on Windows. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
On 08/04/14 22:30, Grant Edwards wrote: Unix maybe, but what about Windows? Is it efficient to create processes under Windows? Processes are very heavy-weight on Windows. Not surprising given its VMS heritage. I remember running shell scripts under VMS on a VAX-11/780 that took hours to do what would have taken minutes on an LSI-11 running Unix. The whole Unix "small tools working together" paradigm is based on the assumption that process creation and death are fast and cheap. That is one reason software tend to be monolithic on Windows, including build tools. Running a configure script used to take forever, but thankfully computers are getting faster. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
"Frank Millman" wrote: > If I have understood correctly, then is there any benefit at all in my going > async? I might as well just stick with threads for the request handling as > well as the database handling. 1. There is a scalability issue with threads, particularly if you don't have enough RAM or use a 32 bit system. 2. Earlier Linux kernels did not perform well if they had to schedule 10,000 threads. 3. It is nice to be able to abort a read or write that hangs (for whatever reason). Killing a thread with pthread_cancel or TerminateThread is not recommended. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Sturla Molden wrote: > 3. It is nice to be able to abort a read or write that hangs (for whatever > reason). Killing a thread with pthread_cancel or TerminateThread is not > recommended. While "graceful timeout" is easy to do on Unix, using fcntl.fcntl or signal.alarm, on Windows it requires overlapped I/O. This means the normal Python file objects cannot be used for this purpose on Windows. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Chris Angelico wrote: > People with a fear of threaded programming almost certainly never grew > up on OS/2. :) I learned about GUI programming thus: Write your > synchronous message handler to guarantee that it will return in an > absolute maximum of 0.1s, preferably a lot less. If you have any sort > of heavy processing to do, spin off a thread. It was simply the normal > way to do things. That is still the best way to do it, IMHO. As I recall, on BeOS the operating system would even spawn a new thread to handle each GUI event. Pervasive multithreading is great for creating responsive user interfaces and running multimedia. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Marko Rauhamaa wrote: > Other points: > > * When you wake up from select() (or poll(), epoll()), you should treat >it as a hint. The I/O call (accept()) could still raise >socket.error(EAGAIN). > > * The connections returned from accept() have to be individually >registered with select() (poll(), epoll()). > > * When you write() into a connection, you may be able to send only part >of the data or get EAGAIN. You need to choose a buffering strategy -- >you should not block until all data is written out. Also take into >account how much you are prepared to buffer. > > * There are two main modes of multiplexing: level-triggered and >edge-triggered. Only epoll() (and kqueue()) support edge-triggered >wakeups. Edge-triggered requires more discipline from the programmer >but frees you from having to tell the multiplexing facility if you >are interested in readability or writability in any given situation. > >Edge-triggered wakeups are only guaranteed after you have gotten an >EAGAIN from an operation. Make sure you keep on reading/writing until >you get an EAGAIN. On the other hand, watch out so one connection >doesn't hog the process because it always has active I/O to perform. > > * You should always be ready to read to prevent deadlocks. > > * Sockets can be half-closed. Your state machines should deal with the >different combinations gracefully. For example, you might read an EOF >from the client socket before you have pushed the response out. You >must not close the socket before the response has finished writing. >On the other hand, you should not treat the half-closed socket as >readable. > > * While a single-threaded process will not have proper race conditions, >you must watch out for preemption. IOW, you might have Object A call >a method of Object B, which calls some other method of Object A. >Asyncio has a task queue facility. If you write your own main loop, >you should also implement a similar task queue. The queue can then be >used to make such tricky function calls in a safe context. > > * Asyncio provides timers. If you write your own main loop, you should >also implement your own timers. > >Note that modern software has to tolerate suspension (laptop lid, >virtual machines). Time is a tricky concept when your server wakes up >from a coma. > > * Specify explicit states. Your connection objects should have a data >member named "state" (or similar). Make your state transitions >explicit and obvious in the code. In fact, log them. Resist the >temptation of deriving the state implicitly from other object >information. > > * Most states should be guarded with a timer. Make sure to document for >each state, which timers are running. > > * In each state, check that you handle all possible events and >timeouts. The state/transition matrix will be quite sizable even for >seemingly simple tasks. And exactly how is getting all of this correct any easier than just using threads and blocking i/o? I'd like to see the programmer who can get all of this correct, but has no idea how to use a queue og mutex without deadlocking. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
On 10/04/14 21:44, Marko Rauhamaa wrote: I'm happy that asyncio is happening after all these long years. It would be nice if it supported edge-triggered wakeups, but I suppose that isn't supported in all operating systems. I have an issue with the use of coroutines. I think they are to evil. When a man like David Beazley says this https://twitter.com/dabeaz/status/440214755764994048 there is something crazy going on. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
On 11/04/14 01:51, Sturla Molden wrote: I have an issue with the use of coroutines. I think they are to evil. When a man like David Beazley says this https://twitter.com/dabeaz/status/440214755764994048 there is something crazy going on. And why did Python get this Tulip beast, instead of a simple libuv wrapper? Node.js has already proven what libuv can do. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
Ian Kelly wrote: > The only reliable way to prevent a customer from reverse-engineering > your software is to not give them the software. Not really. You just need to make it so difficult that it is not worth the effort. In that case they will go away and do something else instead. At least if the threat is other companies out to make money. Dropbox is an example. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
Wesley wrote: > Does python has any good obfuscate? > > Currently our company wanna release one product developed by python to > our customer. But dont's wanna others see the py code. > > I googled for a while but mostly just say using pyc. Any better one? It depends on the threat and how competent persons you want to protect your code from. If this comes from your boss, chances are he does not know that even x86 machine code can be decompiled. So as many has said, this is mostly futile business. The only way to protect your code is never to ship anything. Hacking the interpreter might be satisfactory to calm your boss: - Run a script that strips comments and make variable names incomprehensible - Swap .pyc byte codes so they don't mean the same as in vanilla Python - Make the compiler spit out scrambled bytes and make the .pyc loader unencrypt Any of these measures can be circumvented, though. But it is hardly easier to read than compiled C++. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
Joshua Landau wrote: > However, if this really is your major blocker to using Python, I > suggest compiling with Cython. Cython restains all the code as text, e.g. to readable generate exceptions. Users can also still steal the extension modules and use them in their own code. In general, Cython is not useful as an obfuscation tool. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: threading
Steven D'Aprano wrote: >> I have an issue with the use of coroutines. I think they are to evil. > > They are to evil ... as what? To evil as chalk is to cheese? Black is to > white? Bees are to honey? I think coroutines are one of those things that don't fit the human mind. A class with a couple of queues for input and output is far easier to comprehend. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
Ian Kelly wrote: > How is that last statement different from the one I made above, that > you disagreed with? Who says I disagreed? But to answer you question, it depends on the level of safety you need: Total secrecy or just enough protection to make it not worthwhile to access the code? Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
wrote: >> It's worth noting, as an aside, that this does NOT mean you don't >> produce or sell anything. You can keep your code secure by running it >> on a server and permitting users to access it; that's perfectly safe. >> > Perfectly? :-) Unless you have a heartbleed :) Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
alister wrote: > Concentrate on making the product (even) better rather than trying to > hide the unhideable. I think the number one reason for code obfuscation is an ignorant boss. Another reason might be to avoid the shame of showing crappy code to the customer. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
Mark H Harris wrote: > This is the age of open source in computer science. > > It is far better to develop a strategy and culture of openness. > Everyone benefits; especially your customers. I recommend the GPLv3 > license. I also advocate for copyleft. I would not use GPL in a commercial product, but "Open Source" might still be beneficial. E.g. one can get better feedback and even bug fixes or code improvements from customers and other interested parties. It is a win-win situation. If I ran a software business (I don't currently do), I would gladly discount customers or pay anyone who help to improve my software. Open Source does not mean that software has to be free, that copyright is lost, or that copyleft is implied. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
Mark H Harris wrote: > Obfuscation (hiding) of your source is *bad*, usually done for one > of the following reasons: > 1) Boss is paranoid and fears loss of revenues due to intellectual > property theft. > 2) Boss is ignorant of reverse engineering strategies available to > folks who want to get to the heart of the matter. > 3) Boss and|or coders are embarrassed for clients (or other coders) > to see their art, or lack thereof. Sometimes this is also wanting to > hide the fact that the product really isn't "worth" the price being > charged for it?!? You can also add fear of patent trolls to this list. Particularly if you are in a startup and cannot afford a long battle in court. You can quickly go bankrupt on attorney fees. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: python obfuscate
CM wrote: > You're saying that fear of patent trolls is yet another bad reason to > obfuscate your code? But then it almost sounds like you think it is a > justifiable reason. So I don't think I understand your point. Whether a > patent troll has your original code or not has no bearing on the patent > infringement. There might be no infringment. Patent trolls usually possess invalid patents, as they constitute no real invention. These are usually not engineers who have invented something, but lawyers who have been granted patent on vague thoughts for the purpose of "selling protection". The US patent office has allowed this to happen, by believing that any invalid patent can be challenged in court, so their review process is close to non-existent. If patent trolls have your code they are in a better position to blackmail. They can use your code to generate bogus "legal documents" in the thousands, and thereby turn up your legal expenses. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: What is the difference between 32 and 64 bit Python on Windows 7 64 bit?
On 11/05/14 08:56, Ross Gayler wrote: It looks to me as though 32 and 64 bit versions of Python on 64 bit Windows are both really 32 bit Python, differing only in how they interact with Windows. No! Pointers are 64 bit, Python integers (on Python 2.x) are 32 bit. Microsoft decided to use a 32 bit long in MSVC for backwards compatiblity, but also because the AMD64 (x86-64) architecture was designed to use a 64 address with a 32 bit offset. (A 64 bit long was originally slightly less efficient.) You can see the value of 64 bit Python e.g. if you allocate a lot of objects or if you try to mmap a huge file. With 32 bit Python you are limited to only 2 GB of virtual memory. In 64 bit Python you can in practice mmap as much as you want. The element size of what you try to index also matters. While a C long and a Python int is 32 bit on Windows, 64-bit Python will use a 64-bit offset internally (Py_ssize_t and Py_intptr_t) even on Windows. The 32 bit Python int just limits how many objects you can index from Python space before Python roll over to using long instead of int. It does not limit the amount of memory a Python int can index. In is only when you index an array of bytes you will see the roll-over from Python int to Python long at the 2 GB limit. Typically, object will be much larger than one byte. Here are two examples: - A one-dimensional NumPy array with dtype np.float64 can keep 16 GB of data before a 32 bit index is too small and Python starts to use long. A two-dimensional NumPy array with dtype np.float64 can keep 256 GB of data before a 32 bit index is too small. - A Python list stores internally an array of pointers, each of which is 64 bit. So just indexing those goes up to 16 GB of pointer data before the int rolls over. Then each of these pointers point to a Python object. A Python float on my computer (not Windows) is 24 bytes, which I got from sys.getsizeof(1.) So 2**32 of those are another 383 GB. So if I indexed a list of Python floats on this computer, Python could handle an almost 400 GB data structure with a 32 bit int as indexer without rolling over to long. This is obviously way beyond anything the 2 GB limit on 32 bit Python allows. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: NumPy, SciPy, & Python 3X Installation/compatibility issues
wrote: > 4.In the long run, would it be better to use UNIX instead of Windows, if > I were to use Python for all of my research? > Thanks in advance. EK For scientific computing, a UNIX or Linux system is clearly preferable. Most of the scientific computing software is built around the UNIX ecosystem. This is the reason many scientists prefer to work on a Mac. I have a UNIX system ready to use in the terminal, developer tools are free, and it stills runs all the deskop apps I need (even Microsoft Office). Apple even provides us with highly optimized LAPACK, BLAS and FFT libraries as a part of the operating system (Accelerate Framework). Even the free NumPy and SciPy installers can link to Accelerate on a Mac. Matlab runs on Linux and Mac as well. That is not s reason to stay on Windows. An alternative to a Mac is to install Oracle Virtualbox and use it to run Windows or Linux. Windows as host tends to work best on a laptop. On a workstation it does not matter. If you are using Windows now and are happy with it, I would suggest you just install Ubuntu into an VM with Virtualbox and forget about Windows installers for Python, NumPy, SciPy, et al. Spend some money to buy an SSD drive and more RAM if you need. The performance with Virtualbox is excellent. Get a Python distro that uses MKL for faster linear algebra, such as Enthought or Anaconda. Windows can be a great desktop OS because of all the available applications. It sucks rocks for any coding or scientific computing. But there is no law that says you need to use either. You can have the best of both world's if you like. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: What is the difference between 32 and 64 bit Python on Windows 7 64 bit?
On 12/05/14 15:42, Sturla Molden wrote: - A one-dimensional NumPy array with dtype np.float64 can keep 16 GB of data before a 32 bit index is too small and Python starts to use long. A two-dimensional NumPy array with dtype np.float64 can keep 256 GB of data before a 32 bit index is too small. Oops, the latter should be 34359738336 GB (that is, 32767 pentabytes) :) Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: What is the difference between 32 and 64 bit Python on Windows 7 64 bit?
On 11/05/14 08:56, Ross Gayler wrote: Is that true?I have spent a couple of hours searching for a definitive description of the difference between the 32 and 64 bit versions of Python for Windows and haven't found anything. Why do you care if a Python int object uses 32 or 64 bits internally? Python 2.x will automatically switch to long when needed. The size of the Python integer is an internal implementation detail you will not notice. Python knows when to use a long instead of an int. Python 3.x does not even have a fixed-size integer. 64 bit Python is 64 bit Python, even on Windows. The difference between 32 bit and 64 bit Python is what you would expect: The size of a C pointer is 64 bits, and the virtual address space is much larger (in general not 2**63-1 bytes, but some OS dependent value). Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: What is the difference between 32 and 64 bit Python on Windows 7 64 bit?
On 13/05/14 02:09, Chris Angelico wrote: Sometimes you just want to confirm. :) Or maybe you want your program to be able to detect which it's on. There are ways of doing both, but sys.maxint isn't one of them, as it's specific to the int->long promotion of Py2. The OPs main mistake, I guess, was to assume that sys.maxint is the biggest integer value Python 2.x can use. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Using threads for audio computing?
On 12/05/14 07:33, lgabiot wrote: But AFAIK the python GIL (and in smaller or older computers that have only one core) does not permit true paralell execution of two threads. I believe it is quite like the way multiple processes are handled by an OS on a single CPU computer: process A has x CPU cycles, then process B has y CPU cycles, etc... Python threads are native OS threads. The GIL serializes access to the Python interpreter. If your thread is waiting for i/o or running computations in C or Fortran (e.g. with NumPy), it does not need the Python interpreter. Scientists and engineers use Python threads for "true parallel processing" all the time. The FUD you will find about the GIL is written by people who don't fully understand the issue. So in my case, I must have a way to make sure that: thread 1 (which gets audio from Pyaudio and put() it in the Queue) is not interrupted long enough to miss a sample. Here you are mistaken. The DMA controller takes care of the audio i/o. Your audio acquisition thread is asleep while its buffer fills up. You don't miss a sample because your thread is interrupted. You do, however, have to make sure your thread don't block on the write to the Queue (use block=False in the call to Queue.put), but it is not a "GIL issue". In your case you basically have on thread waiting for the DMA controller to fill up a buffer and another doing computations in NumPy. Neither needs the GIL for most of their work. If you are worried about the GIL you can always use processes (multiprocessing, subprocess, or os.fork) instead of threads. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Fortran
Ian Kelly wrote: > Also numba, which is reminiscent of psyco, but with more features and > Python 3 support. For numerical computing with NumPy, Numba tends to give performance comparable to -O2 in C. This is because it is very easy to do type inference in most scientific array computing. Numba is still a bit immature, though, compared to e.g. Cython. Sturla -- https://mail.python.org/mailman/listinfo/python-list
OT: This Swift thing
Dear Apple, Why should I be exited about an illegitmate child of Python, Go and JavaScript? Because it has curly brackets, no sane exception handling, and sucks less than Objective-C? Because init is spelled without double underscores? Because it faster than Python? Computers and smart phones are slow these days. And I guess Swift makes my 3g connection faster. It's ok to use in iOS apps. That would be it, I guess. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: OT: This Swift thing
Nicholas Cole wrote: > Of course, I wish they had picked Python rather than inventing their > own language. But Apple put a huge stock in the ability of their > libraries to make full use of multiple cores. The GIL is not relevant if they stick to the Objective-C runtime and LLVM. > The GIL is surely the > sticking point here. It is also clear (reading the Swift > documentation) that they wanted a script-like language but with strict > typing. A Python with static typing would have been far better, IMHO. It seems they have created a Python-JavaScript bastard with random mix of features. Unfortunately they retained the curly brackets from JS... Are Python apps still banned from AppStore, even if we bundle an interpreter? If not, I see no reason to use Swift instead of Python and PyObjC – perhaps with some Cython if there is "need for speed". Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: OT: This Swift thing
On 04/06/14 01:39, Kevin Walzer wrote: On 6/3/14, 4:43 PM, Sturla Molden wrote: Are Python apps still banned from AppStore, even if we bundle an interpreter? Python apps are not banned from the App Store. See https://itunes.apple.com/us/app/quickwho/id419483981?mt=12. Mac AppStore yes, iOS AppStore as well? There used to be a ban on interpreted languages to keep out Java and Flash, but it also hurt Python. On Mac there has never been a policy against Java or Flash. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallelization of Python on GPU?
If you are doing SVM regression with scikit-learn you are using libSVM. There is a CUDA accelerated version of this C library here: http://mklab.iti.gr/project/GPU-LIBSVM You can presumably reuse the wrapping code from scikit-learn. Sturla John Ladasky wrote: > I've been working with machine learning for a while. Many of the > standard packages (e.g., scikit-learn) have fitting algorithms which run > in single threads. These algorithms are not themselves parallelized. > Perhaps, due to their unique mathematical requirements, they cannot be > paralleized. > > When one is investigating several potential models of one's data with > various settings for free parameters, it is still sometimes possible to > speed things up. On a modern machine, one can use Python's > multiprocessing.Pool to run separate instances of scikit-learn fits. I > am currently using ten of the twelve 3.3 GHz CPU cores on my machine to > do just that. And I can still browse the web with no observable lag. :^) > > Still, I'm waiting hours for jobs to finish. Support vector regression > fitting is hard. > > What I would REALLY like to do is to take advantage of my GPU. My NVidia > graphics card has 1152 cores and a 1.0 GHz clock. I wouldn't mind > borrowing a few hundred of those GPU cores at a time, and see what they > can do. In theory, I calculate that I can speed up the job by another > five-fold. > > The trick is that each process would need to run some PYTHON code, not > CUDA or OpenCL. The child process code isn't particularly fancy. (I > should, for example, be able to switch that portion of my code to static > typing.) > > What is the most effective way to accomplish this task? > > I came across a reference to a package called "Urutu" which may be what I > need, however it doesn't look like it is widely supported. > > I would love it if the Python developers themselves added the ability to > spawn GPU processes to the Multiprocessing module! > > Thanks for any advice and comments. -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallelization of Python on GPU?
GPU computing is great if you have the following: 1. Your data structures are arrays floating point numbers. 2. You have a data-parallel problem. 3. You are happy with single precision. 4. You have time to code erything in CUDA or OpenCL. 5. You have enough video RAM to store your data. For Python the easiest solution is to use Numba Pro. Sturla Jason Swails wrote: > On Thu, 2015-02-26 at 14:02 +1100, Steven D'Aprano wrote: >> John Ladasky wrote: >> >> >>> What I would REALLY like to do is to take advantage of my GPU. >> >> I can't help you with that, but I would like to point out that GPUs >> typically don't support IEE-754 maths, which means that while they are >> likely significantly faster, they're also likely significantly less >> accurate. Any any two different brands/models of GPU are likely to give >> different results. (Possibly not *very* different, but considering the mess >> that floating point maths was prior to IEEE-754, possibly *very* different.) > > This hasn't been true in NVidia GPUs manufactured since ca. 2008. > >> Personally, I wouldn't trust GPU floating point for serious work. Maybe for >> quick and dirty exploration of the data, but I'd then want to repeat any >> calculations using the main CPU before using the numbers anywhere :-) > > There is a *huge* dash toward GPU computing in the scientific computing > sector. Since I started as a graduate student in computational > chemistry/physics in 2008, I watched as state-of-the-art supercomputers > running tens of thousands to hundreds of thousands of cores were > overtaken in performance by a $500 GPU (today the GTX 780 or 980) you > can put in a desktop. I went from running all of my calculations on a > CPU cluster in 2009 to running 90% of my calculations on a GPU by the > time I graduated in 2013... and for people without as ready access to > supercomputers as myself the move was even more pronounced. > > This work is very serious, and numerical precision is typically of > immense importance. See, e.g., > http://www.sciencedirect.com/science/article/pii/S0010465512003098 and > http://pubs.acs.org/doi/abs/10.1021/ct400314y > > In our software, we can run simulations on a GPU or a CPU and the > results are *literally* indistinguishable. The transition to GPUs was > accompanied by a series of studies that investigated precisely your > concerns... we would never have started using GPUs if we didn't trust > GPU numbers as much as we did from the CPU. > > And NVidia is embracing this revolution (obviously) -- they are putting > a lot of time, effort, and money into ensuring the success of GPU high > performance computing. It is here to stay in the immediate future, and > refusing to use the technology will leave those that *could* benefit > from it at a severe disadvantage. (That said, GPUs aren't good at > everything, and CPUs are also here to stay.) > > And GPU performance gains are outpacing CPU performance gains -- I've > seen about two orders of magnitude improvement in computational > throughput over the past 6 years through the introduction of GPU > computing and improvements in GPU hardware. > > All the best, > Jason -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallelization of Python on GPU?
On 26/02/15 18:34, John Ladasky wrote: Hi Sturla, I recognize your name from the scikit-learn mailing list. If you look a few posts above yours in this thread, I am aware of gpu-libsvm. I don't know if I'm up to the task of reusing the scikit-learn wrapping code, but I am giving that option some serious thought. It isn't clear to me that gpu-libsvm can handle both SVM and SVR, and I have need of both algorithms. My training data sets are around 5000 vectors long. IF that graph on the gpu-libsvm web page is any indication of what I can expect from my own data (I note that they didn't specify the GPU card they're using), I might realize a 20x increase in speed. A GPU is a "floating point monster", not a CPU. It is not designed to run things like CPython. It is also only designed to run threads in parallel on its cores, not processes. And as you know, in Python there is something called GIL. Further the GPU has hard-wired fine-grained load scheduling for data-parallel problems (e.g. matrix multiplication for vertex processing in 3D graphics). It is not like a thread on a GPU is comparable to a thread on a CPU. It is more like a parallel work queue, with the kind of abstraction you find in Apple's GCD. I don't think it really doable to make something like CPython run with thousands of parallel instances on a GPU. A GPU is not designed for that. A GPU is great if you can pass millions of floating point vectors as items to the work queue, with a tiny amount of computation per item. It would be crippled if you passed a thousand CPython interpreters and expect them to do a lot of work. Also, as it is libSVM that does the math in you case, you need to get libSVM to run on the GPU, not CPython. In most cases the best hardware for parallel scientific computing (taking economy and flexibility into account) is a Linux cluster which supports MPI. You can then use mpi4py or Cython to use MPI from your Python code. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallelization of Python on GPU?
On 26/02/15 18:48, Jason Swails wrote: On Thu, 2015-02-26 at 16:53 +, Sturla Molden wrote: GPU computing is great if you have the following: 1. Your data structures are arrays floating point numbers. It actually works equally great, if not better, for integers. Right, but not complicated data structures with a lot of references or pointers. It requires data are laid out in regular arrays, and then it acts on these arrays in a data-parallel manner. It is designed to process vertices in parallel for computer graphics, and that is a limitation which is always there. It is not a CPU with 1024 cores. It is a "floating point monster" which can process 1024 vectors in parallel. You write a tiny kernel in a C-like language (CUDA, OpenCL) to process one vector, and then it will apply the kernel to all the vectors in an array of vectors. It is very comparable to how GLSL and Direct3D vertex and fragment shaders work. (The reason for which should be obvious.) The GPU is actually great for a lot of things in science, but it is not a CPU. The biggest mistake in the GPGPU hype is the idea that the GPU will behave like a CPU with many cores. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: (Still OT) Nationalism, language and monoculture [was Re: Python Worst Practices]
Steven D'Aprano wrote: > Variations in idiom and spelling are a good thing. They open our minds to > new possibilities, remind us that we aren't all the same, and keep life > fresh. I remember the first time I realised that when Indians talk about "a > code" they aren't using "wrong English", they are using a regional > variation. In British and American English, "code" in the programming > sense[2] is a mass or uncountable noun, like air[3], milk, music and > housework. I can assure you that in a veterinary sence, Yersey cows will produce a milk with higher fat content. In a lingustic sence the "a" is not a count -- that would be the word "one" --, it is the indefinite article. Here is the difference: The Enigma machine produced a code that only Alan Turing could break. If I say the Enigma machine produced one code that only Alan Turing could break, it means all the other codes could be broken by someone else. What if I say "this file contains a long Fortran code"? Or what if I say "this file contains one long Fortran code"? There is a subtile difference in meaning here. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: (Still OT) Nationalism, language and monoculture [was Re: Python Worst Practices]
Mark Lawrence wrote: >> I can assure you that in a veterinary sence, Yersey cows will produce a >> milk with higher fat content. > > Yersey? Eh, Jersey. -- https://mail.python.org/mailman/listinfo/python-list
Re: Lockfile hanling
Ian Kelly wrote: > As long as there's not *also* some other external process that needs > to access the file occasionally. :-) Then there is multiprocessing.Lock :) -- https://mail.python.org/mailman/listinfo/python-list
Re: No ttk in 2.7
Zachary Ware wrote: > The way I would do it is as follows: > >try: >import tkinter as tk >from tkinter import ttk >except ImportError: >import Tkinter as tk >import ttk > > If I may suggest, just write it in Python3 first, then when it does > what you want tack on whatever you need to make 2.7 happy. I find it > easier to do things that way, though you may find that the only thing > you have to adjust is the imports. This is a good advice. And yes, it is easier than most would think. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Question About Image Processing in Python
Serge Christian Ibala wrote: > Or what is the recommendation of Python for image processing? Basic setup everyone should have: Python NumPy SciPy (e.g. scipy.ndimage) Cython C and C++ compiler matplotlib scikit-image scikit-learn pillow Also consider: mahotas tifffile (by Christoph Gohlke) OpenCV PyOpenGL VTK mayavi pyglet PyGame PyQt Abandonware: PIL -- https://mail.python.org/mailman/listinfo/python-list
Re: Hi. I have questions about Python
이현상 wrote: > Hi.Please note that do not speak english well. > Do you know Python 2 vs Python3 MultiProcessing the difference > ?Multiprocessing is better performance? The main difference is that multiprocessing on Python 3.4 (and later) will allow you to use APIs that are not "forksafe" on Linux or Mac. One example is Accelerate Framework on Mac OSX (e.g. used by numpy.linalg). -- https://mail.python.org/mailman/listinfo/python-list
Re: What is considered an "advanced" topic in Python?
Mike Driscoll wrote: > Hi, > > I've been asked on several occasions to write about intermediate or > advanced topics in Python and I was wondering what the community > considers to be "intermediate" or "advanced". I realize we're all growing > in our abilities with the language, so this is going to be very > subjective, but I am still curious what my fellow Python developers think > about this topic. Coroutines, metaclasses and decorators is probably high up on the obscurity list. Writing C extension modules is high up there as well, not to mention embedding Python and using subinterpreters. Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Random vs. Cython C Rand for Dice Rolls
On 08/06/15 19:33, Laura Creighton wrote: Better C random number generator. http://www.pcg-random.org/download.html Or for something less minimalistic, just grab randomkit.c and randomkit.h from NumPy, which implements the same Mersenne Twister as Python. That is what I usually do to get fast random numbers from Cython https://github.com/numpy/numpy/tree/master/numpy/random/mtrand Sturla -- https://mail.python.org/mailman/listinfo/python-list
Re: Should iPython Notebook replace Idle
Jason Swails wrote: > Everything gets swallowed into Python. I can't imagine this ever happening. IPython's successor Jupyter is also an REPL environment for Julia and R, and many other languages will also be supported (e.g. Java and C++). Having this swallowed into Python is probably never going to happen. IIRC, the current release is the last to be named IPython. Sturla -- https://mail.python.org/mailman/listinfo/python-list
