Thorough Python 2.7.3 Windows Build Documentation?
Hello fellow Python programmers, I'm building a 32-bit CPython 2.7.3 distro for Windows using the MS Visual Studio Professional 2008 SP1 (and all hotfixes) MSVC 9 compiler. My build works, technically, but it also happens to benchmark over 30% slower than the precompiled binaries in the distributed Python 2.7.3 MSI. Can anyone point me in the direction of some thoroughly detailed build documentation so that I can figure out how to get that 30% back with my build? The only documentation that I can find just says MSVC 9, period. There's no mention of SP1 or not, hotfixes, nor of any specific compiler/linker optimizations used to build the official distro. Something, somewhere, has to be significantly different between our builds for a 30% performance difference, and it'd probably be handy for the Python community to know how to avoid the same pitfall that cost me performance so that we can all get the most out of Python. Any and all help will be greatly appreciated. Thanks. Sincerely, Arah Leonard Arah Leonard Software Development Engineer Bruker AXS Inc. 5465 East Cheryl Parkway Madison, WI 53711 US Phone: +1 608-276-3812 Phone: +1 800-234-XRAY(9729) Fax: arah.leon...@bruker-axs.com<mailto:arah.leon...@bruker-axs.com> www.bruker.com<http://www.bruker.com/> The information contained in this email is confidential. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any form of disclosure, reproduction, distribution or any action taken or refrained from in reliance on it, is prohibited and may be unlawful. Please notify the sender immediately. -- http://mail.python.org/mailman/listinfo/python-list
RE: Thorough Python 2.7.3 Windows Build Documentation?
> I think the official binaries use the PGO build. Be sure to run all tests > thoroughly, we've had problems in with the PGO build in 3.3. Thanks Stefan. I hope that's not the case. PGO seems something of an anathema from well-documented builds. Unless I'm missing something, if PGO is used it means that without knowing the specific CPU of the build machine there would be no way to reliably duplicate the build except by accident. Even with knowing, unless you have the same CPU... But after benchmarking a PGO build made by running the build_pgo.bat it turns out that it made no difference whatsoever to my performance loss. Within an acceptable statistical variation in the benchmark tool itself my PGO build performed identically to my regular build. Which means that I'm still running 30% slower than the precompiled binaries somehow even though I'm theoretically using the right compiler. And which also means that I can neither confirm nor deny if the released precompiled binaries are PGO builds or not. It's one of those days I guess. But thanks anyway. I appreciate the useful input. And yes, absolutely thorough testing is on the agenda. :) Sincerely, Arah Leonard -- http://mail.python.org/mailman/listinfo/python-list
RE: Thorough Python 2.7.3 Windows Build Documentation?
> I remember that some versions of Visual Studio silently completed the PGO > build without actually having PGO capabilities. :) > I think for VS 2008 at least "Professional" is needed, for VS 2010 "Ultimate". Well, that certainly sounds like Microsoft. Fortunately I'm using VS 2008 Professional, so I guess that's not it. :( But thanks! By the way, do you happen to know how tricky it is to get Python 2.7.3 to build with VS 2010? Or have any tips there? It doesn't seem to be officially supported, but it sure would be nice to get out of the dark ages of MS compilers and be only one version behind the latest. -- http://mail.python.org/mailman/listinfo/python-list
RE: Thorough Python 2.7.3 Windows Build Documentation?
>Try dumping the build configuration parameters:
>
> >>> import pprint, sysconfig
> >>> pprint.pprint(sysconfig.get_config_vars())
>
>Then you can compare the existing version with yours.
I would absolutely love to be able to do that and have it work. Most
unfortunately that only works on *nix OSes. The Python sysconfig module's
config vars are almost non-existent on Windows with a really tiny and almost
useless subset. :( I really wish someone would fix that, but hey, it's free
so I'm not complaining. Just wishing. In case anyone else didn't know that,
here's the results containing the complete and total list of config vars on
Windows:
>>> import pprint, sysconfig
>>> pprint.pprint(sysconfig.get_config_vars())
{'BINDIR': 'C:\\Python27',
'BINLIBDEST': 'C:\\Python27\\Lib',
'EXE': '.exe',
'INCLUDEPY': 'C:\\Python27\\Include',
'LIBDEST': 'C:\\Python27\\Lib',
'SO': '.pyd',
'VERSION': '27',
'base': 'C:\\Python27',
'exec_prefix': 'C:\\Python27',
'platbase': 'C:\\Python27',
'prefix': 'C:\\Python27',
'projectbase': 'C:\\Python27',
'py_version': '2.7.3',
'py_version_nodot': '27',
'py_version_short': '2.7',
'srcdir': 'C:\\Python27',
'userbase': 'C:\\Documents and Settings\\Administrator\\Application
Data\\Python'}
That's it. It's actually also a common cause of problems in distutils and the
likes when people write their module installers for Linux and never test those
scriptlets on Windows, not realizing that in Python on Windows the sysconfig /
distutils.sysconfig config vars are hamstrung like that.
--
http://mail.python.org/mailman/listinfo/python-list
RE: Thorough Python 2.7.3 Windows Build Documentation?
Hello Python programmers, Here's an update on my conundrum: When compiling and building CPython 2.7.3 for Win32 from source I see a 30% performance loss from the precompiled binaries in the Python 2.7.3 MSI. Everything that I do gets the same results. I've tried a standard release build. I've tried a PGO build by running build_pgo.bat. I've tried changing every compile and link optimization flag manually. I've tried VS 2008 Professional with SP1 and hotfixes applied, and then without. I've even tried building on two different PCs just in case. I always get the same results: 30% less performance than the precompiled binaries. Frankly, I'm out of things to try. All that I can imagine at this point is that maybe a different edition of VS 2008 creates a 30% faster executable than the Professional edition I have access to? If anyone has specific knowledge of the exact build machine hardware and compiler version used to create the Win32 Python 2.7.3 precompiled binaries I would greatly appreciate knowin g how to duplicate that verbatim. Thanks. Sincerely, Arah Leonard -- http://mail.python.org/mailman/listinfo/python-list
RE: Thorough Python 2.7.3 Windows Build Documentation?
Hello again Python programmers, Final update on my Python 2.7.3 build issues. After much insanity I finally used a process sniffer when running Python and tracked down that because of a path environment variable and the stupidity of Microsoft, Windows was loading another python27.dll rather than the one in the directory with the python.exe that I built, thus rendering my former tests useless. Sorting this out using the .exe.local trick, I was able to re-test everything and come to reliable conclusions. Based on performance results, it is without a doubt in my mind that the Python 2.7.3 precompiled binaries in the MSI are a PGO build. I can now build my own binaries with the same performance as the released precompiled binaries. And on my machine at least, these PGO built binaries do perform 30-36% faster than a normal release build. Useful information if you're doing your own Python builds. I'd still like to see Python start including some thorough build documentation on each release, but I'm certainly not going to be holding my breath on that. ;) Cheers. Sincerely, Arah Leonard -- http://mail.python.org/mailman/listinfo/python-list
RE: Slightly OT: What metro areas are best for a software development career?
> I am looking for a position as a software development engineer. I'm > currently learning to develop Android apps > (http://www.jasonhsu.com/android-apps), and I use > Python for implementing Doppler Value Investing > (http://www.dopplervalueinvesting.com) and for developing Swift Linux > (http://www.swiftlinux.org). NOTE: Thanks > to those of you who answered the questions I asked as I developed Doppler > Value Investing. > > I currently live in Minnesota about 50 miles west of Minneapolis, and I am > considering moving. What are the best metro areas (Silicon Valley, Los > Angeles, San Diego, > Chicago, Twin Cities, Boston, NYC, DC, etc.) for a software development > career, how would you rank them, and why? > > The Twin Cities metro area has a technical community portal called > http://tech.mn . Are there analogous technical community portals for other > metro areas? Well, for what it's worth, I've literally moved from coast to coast and from my experience, that really depends entirely on what kind of programming you want to do. Nevada has lots of jobs for casino-style gaming software development. Texas has a lot of industrial programming for oil rigs. Washington (state) has a lot of Microsoft-related jobs. Portland, Oregon has a lot of Intel-related jobs. Etc. (Pennsylvania just kind of sucked.) In most cases it's about what major business is in the area. So I'd say if you're not looking to be near your family or anything non-work related, then just ask yourself what you want to do, look up who does it best and where their main offices are located, and then apply for jobs there. Even if you don't get the dream job on the get-go, just getting a job that helps you move into the area makes it that much easier to continue applying for your dream job over the years. So you say that you're currently developing Android apps. Google is Android, so I'd suggest looking more around the Mountain View, CA area. (I wouldn't count on a Python-specific career anywhere though. Those kinds of jobs are ones that you have to chase to the oddest ends of the Earth because they're so rare.) Or, if you're just happy being a general programmer or switching things up a lot, then aim for the more heavily populated areas like Silicon Valley and prepare to be chewed up and spat out by random companies while you grow in experience. Just make sure to look into the cost of living in and around the area that you want to move to, keeping various forms of commuting in mind, so that you can plan for how you'll pay the rent once you find a job you like. ;) Fortunately, most major metro areas have good forms of public transportation. (Maybe not enjoyable, but dependable and affordable at least.) That, and if you have any allergies or medical conditions, keep in mind the area. Portland, OR was a lovely place to be my first year there, but my second spring there the pollen counts shot through the roof so I had to leave after a really bad bout of bronchitis. So if you have any health issues, research the area well. Even a whole year of living there isn't always enough to prove the area safe. And good luck! Sincerely, Arah Leonard -- http://mail.python.org/mailman/listinfo/python-list
RE: Using filepath method to identify an .html page
> Thank you but the number needs to be a 4-digit integer only, if its to be > stored in the database table correctly. Okay, I think we need to throw the flag on the field at this point. What you're asking for has gone into a realm where you clearly don't even appear to understand what you're asking for. What is the reason for your integer being limited to only 4 digits? Not even databases are limited in such a way. So what are you doing that imposes that kind of a limit, and why? -- http://mail.python.org/mailman/listinfo/python-list
RE: Converting a string to a number by using INT (no hash method)
> I just need a way to CONVERT a string(absolute path) to a 4-digit unique > number with INT!!! That's all i want!! But i cannot make it work :( > > And the best part is that "that" number must be able to turn back into a path. > > This way i DON'T EVEN HAVE TO STORE THE ACTUAL HTML PAGE'S ABSOLUTE PATH > > 1. User requests a specific html page( .htaccess gives my script the absolute > path for that .html page) 2. I turn the path into a 4-digitnumber 3. i store > that number to the database. I DONT EVEN HAVE TO STORE THE PATH TO THE > DATABASE ANYMORE!!! this is just great! Without involving some kind of lookup table/map service to store the paths (which would entirely defeat the purpose) what you are ranting about is technically impossible. If you tried really really hard you *might* be able to convert a string that long into some kind of 4-digit integer checksum, but you would *never* be able to convert that back into a file path. Nor would it be guaranteed to be unique. -- http://mail.python.org/mailman/listinfo/python-list
RE: Converting a string to a number by using INT (no hash method)
> No need, to turn the number back to a path anymore, just the path to a
> number, to identify the specific .html page
>
> Can this be done?
Guaranteed to be unique? Not even remotely possible. Even with a lookup table
approach (which defeats your purpose of not storing the path) with 4 digits
you're looking at a maximum 1 unique file paths before your system
duplicates numbers. And that's the best-case scenario. Anything else would be
worse.
Not guaranteed to be unique? Easy. Just take then previously given example of
pin = int( htmlpage.encode("hex"), 16 ) and mod it to your limit, to make:
pin = int( htmlpage.encode("hex"), 16 ) % 1
It'll give you your number, but there are no guarantees of uniqueness. You're
looking at more blind random luck using that.
--
http://mail.python.org/mailman/listinfo/python-list
RE: Converting a string to a number by using INT (no hash method)
> The perl code will produce the same hash for "abc.html" as for "bca.html" > That's probably one reason Leonard didn't try to transliterate the buggy code. > Actually, to give credit where it's due, it wasn't me. I just modified someone else's interesting solution in this thread and added the silly limit of 1 to it. > In any case, the likelihood of a hash collision for any non-trivial website > is substantial. > Exactly. Four digits is hardly enough range for it to be even remotely safe. And even then range isn't really the issue as technically it just improves your odds. The results of a modulus operator are still non-unique no matter how many digits are there to work with ... within reason. Statistically anyone who buys a ticket could potentially win the lottery no matter how bad the odds are. ;) And now back to the OP, I'm still confused on this four-digit limitation. Why isn't the limitation at least adhering to a bytelength like byte/short/long? Is this database storing a string of characters instead of an actual number? (And if so, then why not just block out 255 characters instead of 4 to store a whole path? Or at the very least treat 4 characters as 4 bytes to greatly increase the numeric range?) -- http://mail.python.org/mailman/listinfo/python-list
RE: Failed to import a "pyd: File When python intepreter embed in C++ project
> I create a pyd File named "testPyd" with boostPython,and then I import the > testPyd module into "test.py", it works perfect! > But when I embeded the python interpreter into my C++ project and run the > "test.py", it comes out a "ImportErr: no module named testPyd". > It has confused me for two days and I googled for long time,but I can't find > the answer! > Anybody here can help me ? > Thank you! > Ah, that sounds familiar. I have a small bit of experience with Boost. I could be wrong, because once you start mixing it up with Boost all sorts of weird things can happen (especially on a MS compiler, because no one tests for Windows, let alone a pay-for compiler) but my experience has shown that if you get that specific import error, what it actually means is just that the PYD import failed for ANY reason. It has nothing to do with the name or that the PYD couldn't be found. Just that somewhere, at some time during import, it failed to fully load. In my use a lot of times it was either that a DLL that module depended on wasn't in the path, or my favorite kicker, that some compile/link flags between the PYD and the Python interpreter don't match well enough. (Usually from mixing debug and release builds together.) From what I've seen anyway, the Python interpreter really doesn't like being built in a traditional debug mode, so I always do a release build of it. It's a little inconvenient, but in the linker flags you can still set your PYDs to generate debug information even in release builds, so you can still run the debugger on them when you attach to the process of the python interpreter EXE. And especially be sure to use the RELEASE runtime library flag (such as /MD) instead of the debug flag (such as /MDd). That's as much as I know anyway. Though depending, if you add any new templates/libraries into Boost (such as for NumPy ndarray), you also may need to use the /DBOOST_ALL_NO_LIB compiler macro on an MS compiler because MS doesn't adhere to template standards correctly and you often end up with multiply-defined functions if you don't use that macro. If I remember correctly. (It's been a while.) That and you may be picking up variable length arrays out of your teeth, replacing chunks of code with the use of new and delete operators. No one tests for Microsoft and the MS compiler is way behind in adhering to C/C++ standards, and VLAs pop up a lot. Hopefully something in all of this helped. Boost can be ... daunting. I get it, in theory. But in practice it often hurts my head. ;) Sincerely, Arah Leonard -- http://mail.python.org/mailman/listinfo/python-list
RE: Using filepath method to identify an .html page
> "his quote string is Cyrillic"? > > If you're referring to the "Τη Τρίτη, 22 Ιανουαρίου 2013 6:23:16 μ.μ. > UTC+2, ο χρήστης Leonard, Arah έγραψε", that's Greek. > Cyrillic or not, it's all Greek to me. ;) -- http://mail.python.org/mailman/listinfo/python-list
RE: Using filepath method to identify an .html page
> =
> my @i = split(//,$url); # put each letter in it's own bin
> my $j=0; # Initailize our
> my $k=1; # hashing increment values
> my @m=(); # workspace
> foreach my $n(@i){
>my $q=ord($n); # ASCII for character
>$k += $j; # Increment our hash offset
>$q += $k; # add our "old" value
>$j = $k;# store that.
>push @m,$q; # save the offsetted value
> }
>
> my $hashval=0; #initialize our hash value # Generate that map { $hashval =
> ($hashval + $_) % 10} @m;
> =
>
>
> Is this the solution i seek to turn an 'absolute path' <=> '5-digit number'
> in a bi-directional way?
>
1) There is NO solution to turn a complete path into a 4-digit number in a
bi-directional way, no matter what language you write it in. Nor does adding
one more digit make it any more plausible. It is NOT possible. Which is why
EVERYONE keeps telling you that. The only way to store a complete path in a
unique and bi-directional way is to STORE THE COMPLETE PATH. Even if you
compress the path data in some way, you would still need to store the complete
path.
2) Within reason and sanity, any use of a modulus operator to chop a large
checksum value into a small value means that the results are not unique and not
reversible. This was plainly stated from the beginning, and is still true no
matter how many programming languages you write it in.
3) This is a Python-specific resource and that's not even Python code. What
next? Javascript? Ada? Fortran? COBOL? 8-bit x86 assembly with minimal
comments written in Esperanto?
4) The novelty of the entertainment resulting from this perversity has waned,
even for me. The educational aspect to novice programmers has likewise run
dry. I've now officially grown bored of your game and am joining everyone else
who already has already gotten off of this kiddie ride. Congratulations on
beating a dead horse into mince-meat and successfully milking the one-uddered
cow until the pale is full. I hope that you enjoyed your meal.
Or to borrow a phrase, "I say GOOD DAY, sir!"
--
http://mail.python.org/mailman/listinfo/python-list
RE: The best, friendly and easy use Python Editor.
> All true (especially the holy wars bit!). OP didn't (as far as I can see) > even say which OS he is using. Anyway, my suggestion is generally that people > use the editor with which they are already comfortable. > Sound advice. Most of the time I still use Visual Studio for editing Python because I also use it for C++, so it's just what I'm used to. No big deal, really. Whatever works is what works. It's just a text file after all. -- http://mail.python.org/mailman/listinfo/python-list
RE: The best, friendly and easy use Python Editor.
>> Sound advice. Most of the time I still use Visual Studio for editing >> Python because I also use it for C++, so it's just what I'm used to. >> No big deal, really. Whatever works is what works. It's just a text >> file after all. >> > > I assume you're using PyTools (http://pytools.codeplex.com/)? > Would that I could! To my knowledge they still haven't backported PyTools to VS2008, which I'm presently stuck on because Python 2.7 still doesn't compile on VS2010. :( Kind of makes you think... -- http://mail.python.org/mailman/listinfo/python-list
RE: Dict comp help
> Hi,
> Slightly different take on an old problem, I have a list of dicts, I need to
> build one dict from this based on two values from each dict in the list. Each
> of the dicts in the list have similar key names, but values of course differ.
>
>
> [{'a': 'xx', 'b': 'yy', 'c': 'zz'}, {'a': 'dd', 'b': 'ee', 'c': 'ff'}]
>
>
> { 'xx': 'zz', 'dd': 'ff'}
>
>
> Anyone have insight on how to pull this off?
>
>
> Thanks!
> jlc
>
listy = [{'a':'xx', 'b':'yy', 'c':'zz'}, {'a':'dd', 'b':'ee','c':'ff'}]
kryten = {}
keys = []
for l in listy:
for key in l.keys():
if key not in keys: keys.append(key)
for key in keys:
kryten[key] = ''
for l in listy:
kryten[key] += l.has_key(key) and l[key] or ''
print kryten
--
http://mail.python.org/mailman/listinfo/python-list
RE: The best, friendly and easy use Python Editor.
>> >> It's just a text file after all. >> > > True indeed, let's not worry about trivial issues like indentation, mixing > tabs and spaces or whatever. Notepad anybody? :) > Hey, I didn't say Notepad was the *best* tool for the job, just that Python scripts are merely text files. Though, that said, I have used Notepad and Wordpad any number of times in the past to edit Python files, all without bringing the universe to an untimely end. Even used DOS Edit once. You use what you have at the time. A good craftsman never blames the tools. ;) -- http://mail.python.org/mailman/listinfo/python-list
