Re: N-grams
On Thursday 10 November 2016 17:53, Wolfram Hinderer wrote: [...] > 1. The startup looks slightly ugly to me. > 2. If n is large, tee has to maintain a lot of unnecessary state. But n should never be large. If practice, n-grams are rarely larger than n=3. Occasionally you might use n=4 or even n=5, but I can't imagine using n=20 in practice, let alone the example you show of n=500. See, for example: http://stackoverflow.com/a/10382221 In practice, large n n-grams run into three problems: - for word-based n-grams, n=3 is about the maximum needed; - for other applications, n can be moderately large, but as n-grams are a kind of auto-correlation function, and few data sets are auto-correlated *that* deeply, you still rarely need large values of n; - there is the problem of sparse data and generating a good training corpus. For n=10, and just using ASCII letters (lowercase only), there are 26**10 = 141167095653376 possible 10-grams. Where are you going to find a text that includes more than a tiny fraction of those? -- Steven 299792.458 km/s — not just a good idea, it’s the law! -- https://mail.python.org/mailman/listinfo/python-list
Re: N-grams
srinivas devaki wrote: Interesting approach. > def myngrams(iterable, n=2): > t = list(tee(iterable, 1)) I don't think I've seen tee(iterable, 1) before. Did you do this for aesthetic reasons or is there an advantage over t = [iter(iterable)] ? > for _ in range(n - 1): > t.extend(tee(t.pop())) > next(t[-1], None) > return zip(*t) -- https://mail.python.org/mailman/listinfo/python-list
Re: N-grams
Paul Rubin wrote: > This can probably be cleaned up some: > > from itertools import islice > from collections import deque > > def ngram(n, seq): > it = iter(seq) > d = deque(islice(it, n)) > if len(d) != n: > return > for s in it: > yield tuple(d) > d.popleft() > d.append(s) > if len(d) == n: > yield tuple(d) > > def test(): > xs = range(20) > for a in ngram(5, xs): > print a > > test() I started with def ngrams2(items, n): items = iter(items) d = deque(islice(items, n-1), maxlen=n) for item in items: d.append(item) yield tuple(d) and then tried a few dirty tricks, but nothing except omitting tuple(d) brought performance near Steven's version. Just for fun, here's the obligatory oneliner: def ngrams1(items, n): return zip(*(islice(it, i, None) for i, it in enumerate(tee(items, n Be aware that the islice() overhead is significant (I wonder if the islice() implementation could be tweaked to reduce that). -- https://mail.python.org/mailman/listinfo/python-list
Re: is modulefinder.ModuleFinder working at all?
On 10.11.2016 01:02, Steve D'Aprano wrote:
On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:
Hi,
I just used the stdlib's modulefinder.ModuleFinder (intended to find
modules used by a script) for the first time in my life and it just
doesn't seem to work like documented at all.
Not sure what is going on, but if I try the usage example from
https://docs.python.org/3/library/modulefinder.html
it's reporting every single module from the stdlib whether imported or
not!
I see what you mean, but it's not quite *every* module. After I run the
example from the docs, I get 197 modules, and here's at least two that
aren't included:
py> len(finder.modules)
197
py> 'statistics' in finder.modules
False
py> 'cmath' in finder.modules
False
Curiously, it includes modules that aren't cached in sys.modules:
py> len(finder.modules.keys() - sys.modules.keys())
107
So I'm not sure how that's possible.
According to the example module, it imports re and itertools. Both of those
have already been imported, and so will be cached in sys.modules, as will
all their dependencies. So I don't see how it is possible that ModuleFinder
can find dependencies that aren't cached.
Theoretically, of course some dependency might import a bunch of modules,
then delete them from sys.modules. If it were only one or two, I'd believe
that. But not 107 of them.
After running the module finder on the sample file, I ran the report()
method to get a nicer display of the imported modules:
py> finder = ModuleFinder()
py> finder.run_script('/tmp/bacon.py')
py> finder.report()
Name File
m __future__/usr/local/lib/python3.5/__future__.py
m __main__ /tmp/bacon.py
m _ast
m _bootlocale /usr/local/lib/python3.5/_bootlocale.py
m _bz2
/usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
[...]
which shows the unittest package being loaded, which is pretty dubious.
On the other hand, here's a simpler example which seems to work fine:
py> with open('/tmp/do_little.py', 'w') as f:
... f.write('import math\n')
...
12
py> finder = ModuleFinder()
py> finder.run_script('/tmp/do_little.py')
py> finder.report()
Name File
m __main__ /tmp/do_little.py
m math
/usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so
So I'm not really sure what's going on.
Thanks for the detailed analysis.
You are right that it seems to work with certain imports. I verified
your example with import math and found a few additional examples.
Things seem to work with imports of any of the following (alone or in
combination):
io, itertools, math, sys
OTOH, I doubt that it reports correctly with:
os, heapq, operator, re
Personally, I fail to see a pattern here, but maybe somebody else can
explain that behaviour.
I'd really like to report this to the bug tracker, but at the moment I'm
not sure as what exactly. It seems that at least the documentation is
incorrect since the usage example there is not working.
Best,
Wolfgang
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python rules!
On Wed, 09 Nov 2016 20:45:45 -0600, Skip Montanaro wrote: > On Wed, Nov 9, 2016 at 7:53 PM, Chris Angelico wrote: >> It's called Jython. :) > > Well, sure, but that didn't look enough like Python, so no chance that I > would mistake it for Jython. I suspect that whoever worked out that > arrangement of semicolons and braces had some help from her tools. > > Skip i think whoever did that WAS a tool -- How long does it take a DEC field service engineer to change a lightbulb? It depends on how many bad ones he brought with him. -- https://mail.python.org/mailman/listinfo/python-list
Re: Help me cythonize a python routine!
On 09/11/2016 21:25, breamore...@gmail.com wrote:
On Wednesday, November 9, 2016 at 7:34:41 PM UTC, BartC wrote:
However according to your mindset nothing matters provided it's fast,
> accuracy does not matter to users.
Hence your recent comment on another thread about converting invalid integer
entries into zero.
That's by design. And it's not unprecedented; using C's 'atoi()'
function (convert a string to an integer), then inputs of "" and "ABC"
both return 0. Input of "123ABC" returns 123.
And in the context that was being discussed, it was desired that hitting
end-of-line while attempting to read more items should return null
values such as 0, 0.0 or "".
For any more refined behaviour, values can be read differently (read as
strings then converted with routines that do more validation).
Effectively, the conversion looks at the entire line buffer contents
remaining, and converts the first integer encountered, if any, then
removes that and any terminator from the buffer.
If I try something like that in Python, trying to convert an integer at
the beginning of a string, it fails: int("123,456"), int("123 456"),
int("63924 the"). It will need processing first so that the parameter
comprises only the thing we're trying to read.
Anyway the comparison between Python and that particular /lower-level/
language is moot as Python apparently doesn't have a way of directly
reading items from an input line, either from a console or from a file.
You have to read a line as one string then apply DIY conversions. Which
the other language could equally do.
So it's a detail. The other language could also have chosen to use
exceptions to signal such events (no integer following, end of input,
invalid terminator etc). I chose to keep it simple and to make it work
the same way it's always done.
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Why I took October off from OSS volunteering
http://www.snarky.ca/why-i-took-october-off-from-oss-volunteering -- https://mail.python.org/mailman/listinfo/python-list
Re: Why I took October off from OSS volunteering
On 11/09/2016 11:35 PM, breamore...@gmail.com wrote: http://www.snarky.ca/why-i-took-october-off-from-oss-volunteering Good article, Mark, thanks for sharing. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: I want to insert beacon scan result in to a database using python and mysql
On 11/10/2016 06:15 AM, Dennis Lee Bieber wrote: > On Wed, 9 Nov 2016 21:05:50 -0800 (PST), sudeeratechn...@gmail.com > declaimed the following: > >> >> sql = "insert into beacon VALUES(null, '%s')" % \ >> (beacon) >> > DON'T DO THAT... Wouldn't hurt to include a brief why on this, and the right way to do this. The why is, of course, that this operation is vulnerable to SQL injection. This should be avoided as a matter of practice, even if you're not taking input from anyone but yourself. The correct way to do this is to use a prepared statement. And of course the relevant xkcd is: https://xkcd.com/327/ -- https://mail.python.org/mailman/listinfo/python-list
Re: N-grams
On Thu, Nov 10, 2016 at 2:26 PM, Peter Otten <__pete...@web.de> wrote: > > I don't think I've seen tee(iterable, 1) before. Did you do this for > aesthetic reasons or is there an advantage over > > t = [iter(iterable)] Yeah just to be aesthetic, there's no extra advantage over that as with n=1 tee just returns a wrapper around the iterable. Regards Srinivas Devaki Senior (4th year) student at Indian Institute of Technology (ISM), Dhanbad Computer Science and Engineering Department phone: +91 9491 383 249 telegram: @eightnoteight -- https://mail.python.org/mailman/listinfo/python-list
Re: Python rules!
Alister> i think whoever did that WAS a tool Perhaps, but maybe she is a Python programmer forced to write Java (not Jython). If so, props to her for making the best of a bad situation. :-) Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: I want to insert beacon scan result in to a database using python and mysql
On Fri, Nov 11, 2016 at 2:36 AM, Michael Torrie wrote:
> On 11/10/2016 06:15 AM, Dennis Lee Bieber wrote:
>> On Wed, 9 Nov 2016 21:05:50 -0800 (PST), sudeeratechn...@gmail.com
>> declaimed the following:
>>
>>>
>>> sql = "insert into beacon VALUES(null, '%s')" % \
>>> (beacon)
>>>
>> DON'T DO THAT...
>
> Wouldn't hurt to include a brief why on this, and the right way to do
> this. The why is, of course, that this operation is vulnerable to SQL
> injection. This should be avoided as a matter of practice, even if
> you're not taking input from anyone but yourself. The correct way to do
> this is to use a prepared statement. And of course the relevant xkcd
> is: https://xkcd.com/327/
The easiest way is to use a parameterized query:
cur.execute("insert into beacon VALUES(null, %s)", (beacon,))
I don't understand why so many people conflate parameterized with
prepared. "Prepared statements" have a two-step execution.
"Parameterized queries" needn't.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: I want to insert beacon scan result in to a database using python and mysql
On 11/10/2016 11:32 AM, Chris Angelico wrote:
> The easiest way is to use a parameterized query:
>
> cur.execute("insert into beacon VALUES(null, %s)", (beacon,))
>
> I don't understand why so many people conflate parameterized with
> prepared. "Prepared statements" have a two-step execution.
> "Parameterized queries" needn't.
Good point. Yes that's what I was going for but forgot the term.
--
https://mail.python.org/mailman/listinfo/python-list
Fwd: Error message
To whom this may concern, I am continuously receiving this error after the installation of Python 3.5.2. The purpose of using this program is for a class I am currently enrolled in at a University. (I am running Windows 7 Home Premium 64-bit paired with an i3-2100 processor and 6 gb of ram. I don't believe the specifications are a problems but I have listed them just in case.) Although this computer is not new, it has recently been factory reset. I have tried multiple times installing and uninstalling this program and restarting the computer. If you would be able to help me figure this out, I would very much appreciate it. Thank you, Keenan Chu [image: Inline image 1] -- https://mail.python.org/mailman/listinfo/python-list
Windows: subprocess won't run different Python interpreter
Hi, I'm trying to run a script with a different Python version by extending the path variable and executing "python.exe". It looks like subprocess will always run the current executing Python. The following snippet demonstrates the problem: """ import os, subprocess os.environ['PATH'] = '' print(subprocess.check_output(['python.exe', '-V'])) """ It outputs the version of the current executing Python interpreter although it should generate an error because Python is not in the PATH. Thorsten -- https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
On 11/10/2016 04:58 PM, Thorsten Kampe wrote: Hi, I'm trying to run a script with a different Python version by extending the path variable and executing "python.exe". It looks like subprocess will always run the current executing Python. > [...] > Thorsten Have you tried using the full path to the other binary? Cheers, Thomas -- https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
* Thomas Nyberg (Thu, 10 Nov 2016 17:07:35 -0500) > > On 11/10/2016 04:58 PM, Thorsten Kampe wrote: > > Hi, > > > > I'm trying to run a script with a different Python version by > > extending the path variable and executing "python.exe". It looks like > > subprocess will always run the current executing Python. > > > > [...] > > > > Thorsten > > > > Have you tried using the full path to the other binary? Yes. That works. But it's not like subprocess should work. Thorsten -- https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
On 11/10/2016 05:32 PM, Thorsten Kampe wrote: Yes. That works. But it's not like subprocess should work. It certainly is odd. I can at least confirm that when I try to run your code I get the error that you're expecting, but I run debian. Have you tried using os.unsetenv()? https://docs.python.org/2/library/os.html#os.unsetenv Cheers, Thomas -- https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
On Thu, Nov 10, 2016 at 9:58 PM, Thorsten Kampe
wrote:
>
> I'm trying to run a script with a different Python version by
> extending the path variable and executing "python.exe". It looks like
> subprocess will always run the current executing Python.
WinAPI CreateProcess checks the application directory, current
directory (an insecure legacy default), %SystemRoot%\System32,
%SystemRoot%\System, %SystemRoot%, and then the directories in %PATH%.
This is listed in the documentation of the lpCommandLine parameter on
MSDN [1].
Let's take a look with a breakpoint set on WinAPI SearchPath [2]:
>>> subprocess.Popen('python -c 42').wait()
Breakpoint 0 hit
KERNELBASE!SearchPathW:
7fff`4b344f60 488bc4 mov rax,rsp
In the x64 ISA, the lpPath parameter is passed in register rcx:
0:000> du @rcx L4D
0241`9a74dbe0 "C:\Program Files\Python35;.;C:\W"
0241`9a74dc20 "indows\SYSTEM32;C:\Windows\syste"
0241`9a74dc60 "m;C:\Windows;"
0:000> g
0
Note that the second entry is ".", which is the current directory.
Setting %NoDefaultCurrentDirectoryInExePath% [3] removes the current
directory from the search path:
>>> os.environ['NoDefaultCurrentDirectoryInExePath'] = '1'
>>> subprocess.Popen('python -c 42').wait()
Breakpoint 0 hit
KERNELBASE!SearchPathW:
7fff`4b344f60 488bc4 mov rax,rsp
0:000> du @rcx L4B
0241`99e43f90 "C:\Program Files\Python35;C:\Win"
0241`99e43fd0 "dows\SYSTEM32;C:\Windows\system;"
0241`99e44010 "C:\Windows;"
[1]: https://msdn.microsoft.com/en-us/library/ms682425
[2]: https://msdn.microsoft.com/en-us/library/aa365527
[3]: https://msdn.microsoft.com/en-us/library/ms684269
--
https://mail.python.org/mailman/listinfo/python-list
Re: Error message
On Thu, Nov 10, 2016 at 9:37 PM, Keenan C wrote: > > I am continuously receiving this error after the installation of Python > 3.5.2. The purpose of using this program is for a class I am currently > enrolled in at a University. (I am running Windows 7 Home Premium 64-bit > paired with an i3-2100 processor and 6 gb of ram. I don't believe the > specifications are a problems but I have listed them just in case.) > Although this computer is not new, it has recently been factory reset. I > have tried multiple times installing and uninstalling this program and > restarting the computer. > > [image: Inline image 1] python-list is text-only and drops message attachments. Please copy and paste or transcribe the error message text and error code. -- https://mail.python.org/mailman/listinfo/python-list
Re: Fwd: Error message
On 2016-11-10 21:37, Keenan C wrote: To whom this may concern, I am continuously receiving this error after the installation of Python 3.5.2. The purpose of using this program is for a class I am currently enrolled in at a University. (I am running Windows 7 Home Premium 64-bit paired with an i3-2100 processor and 6 gb of ram. I don't believe the specifications are a problems but I have listed them just in case.) Although this computer is not new, it has recently been factory reset. I have tried multiple times installing and uninstalling this program and restarting the computer. If you would be able to help me figure this out, I would very much appreciate it. Thank you, Keenan Chu [image: Inline image 1] This list doesn't support attachments, so I'm guessing that it's complaining that it can't find "api-ms-win-crd-runtime-l1-1-0.dll". You should already have it if you've kept your PC up to date via Windows Update, but if you haven't, read this: https://support.microsoft.com/en-us/kb/3118401 -- https://mail.python.org/mailman/listinfo/python-list
Re: I want to insert beacon scan result in to a database using python and mysql
On 11/10/2016 06:10 PM, Dennis Lee Bieber wrote:
> {I could swear I'd included an example of a parameterized query in my
> response... I didn't want to go into the details of "SQL injection attack"
> as, based on the rest of the OPs post, it would have needed a large
> explanation... And the biggest flaw was improper indentation for the
> database access}
You did indeed. My bad.
--
https://mail.python.org/mailman/listinfo/python-list
what is the procedure or how to plan how many nodes of dispy need for dsolve differential system in amazon cloud in limited time such as 1 hour, 2 hours.etc
what is the procedure or how to plan how many nodes of dispy need for dsolve
differential system in amazon cloud in limited time such as 1 hour, 2 hours.etc
?
#For Amazon Linux, the user name is ec2-user. For RHEL5, the user name is
either root or ec2-user.
#For Ubuntu, the user name is ubuntu. For Fedora, the user name is either
fedora or ec2-user.
#For SUSE Linux, the user name is either root or ec2-user.
#Otherwise, if ec2-user and root don't work, check with your AMI provider.
import random, dispy
import ast
from __future__ import division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
import inspect
def compute(n): # executed on nodes
import random, time, socket
name = socket.gethostname()
cur_best = 1
for ii in range(n[0],n[0]):
for jj in range(n[1],n[1]):
for kk in range(n[2],n[3],100):
#assume dsolve with sympy for differential system
dispy_provisional_result((name, r))
cur_best = r
time.sleep(0.1)
# final result
return (name, cur_best)
def job_callback(job): # executed at the client
if job.status == dispy.DispyJob.ProvisionalResult:
#if job.result[1] < 0.005:
# acceptable result; terminate jobs
print('%s computed: %s %s %s %s' % (job.result[0], job.result[1],
job.result[2], job.result[3], job.result[4]))
# 'jobs' and 'cluster' are created in '__main__' below
for j in jobs:
if j.status in [dispy.DispyJob.Created, dispy.DispyJob.Running,
dispy.DispyJob.ProvisionalResult]:
cluster.cancel(j)
if __name__ == '__main__':
#cluster = dispy.JobCluster(compute, callback=job_callback)
cluster = dispy.JobCluster(compute,
nodes=['ec2-35-162-137-237.us-west-2.compute.amazonaws.com'],
ip_addr='127.0.0.1', port=51347, node_port=51348, callback=job_callback,
keyfile=r"C:\Users\hello\datacenterusekey.ppk")
jobs = []
prevk = 1
count = 0
for ii in range(1,2):
for jj in range(1,2000):
for kk in range(1,2000,100):
if ii < jj and jj < kk:
job = cluster.submit([ii,jj,prevk,kk])
prevk = kk
if job is None:
print('creating job %s failed!' % n)
continue
job.id = count
count = count + 1
jobs.append(job)
cluster.wait()
cluster.print_status()
cluster.close()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
* eryk sun (Thu, 10 Nov 2016 23:04:02 +) > > On Thu, Nov 10, 2016 at 9:58 PM, Thorsten Kampe > wrote: > > > > I'm trying to run a script with a different Python version by > > extending the path variable and executing "python.exe". It looks like > > subprocess will always run the current executing Python. > > WinAPI CreateProcess checks the application directory, current > directory (an insecure legacy default), %SystemRoot%\System32, > %SystemRoot%\System, %SystemRoot%, and then the directories in %PATH%. > This is listed in the documentation of the lpCommandLine parameter on > MSDN [1]. I'm aware of that. Python is in F:\PortableApps\Python3x and neither the current directory nor the PATH points to that. Thorsten -- https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
* Thomas Nyberg (Thu, 10 Nov 2016 17:46:06 -0500) > > On 11/10/2016 05:32 PM, Thorsten Kampe wrote: > > Yes. That works. But it's not like subprocess should work. > > > > It certainly is odd. I can at least confirm that when I try to run your > code I get the error that you're expecting, but I run debian. > > Have you tried using os.unsetenv()? I think you are kind of misunderstanding my intention. I'm actually trying the opposite: running python.exe by setting F:\PortableApps \Python3x as the first element in PATH. No Python interpreter is in my PATH. So unsetting PATH is just a way to demonstrate that this should never ever work. I'm going to open a bug report. Thorsten -- https://mail.python.org/mailman/listinfo/python-list
Re: Windows: subprocess won't run different Python interpreter
On Fri, Nov 11, 2016 at 6:01 AM, Thorsten Kampe wrote: > * eryk sun (Thu, 10 Nov 2016 23:04:02 +) >> >> On Thu, Nov 10, 2016 at 9:58 PM, Thorsten Kampe >> wrote: >> > >> > I'm trying to run a script with a different Python version by >> > extending the path variable and executing "python.exe". It looks like >> > subprocess will always run the current executing Python. >> >> WinAPI CreateProcess checks the application directory, current >> directory (an insecure legacy default), %SystemRoot%\System32, >> %SystemRoot%\System, %SystemRoot%, and then the directories in %PATH%. >> This is listed in the documentation of the lpCommandLine parameter on >> MSDN [1]. > > I'm aware of that. Python is in F:\PortableApps\Python3x and neither > the current directory nor the PATH points to that. That's the application directory, which is the first place CreateProcess looks (via the SearchPath call), as both of my examples shows. In my case python.exe is located in the standard 3.5 system installation path, "C:\Program Files\Python35". I showed how to remove the current directory from the search. But, in hindsight, I should have clarified that the working directory was not related to your problem. It's just the only directory that can be removed from the implicit search path (AFAIK). I thought about replying with a clarification, but it seemed clear in context that the source of the behavior was the application directory. I know of no API, environment variable, or manifest setting to remove the application directory from the head of the search path used by CreateProcess. What you can do is call SearchPath explicitly (via PyWin32 or ctypes) with just %Path%, or do the search in pure Python via shutil.which. Note also that LoadLibrary uses a separate DLL search path that also defaults to preferring the application directory, but this can be controlled by calling SetDefaultDllDirectories [1], among other strategies. [1]: https://msdn.microsoft.com/en-us/library/hh310515 -- https://mail.python.org/mailman/listinfo/python-list
