Re: memory control in Python
On Tuesday, August 18, 2015 at 3:40:11 AM UTC+5:30, Ping Liu wrote: > Hi, Dieter, > > If I move from Python to Jython or IronPython, do I need to retool whatever I > have done? If so, that may take quite a long time. This may make the > reimplementation impossible. Hi Ping There is a message from Laura Creighton that may be useful to you that googlegroups shows in the thread: "python implementation of a new integer encoding algorithm." Thought I'd mention in case you are not seeing other threads. [How she (her mail client) manages to befuddle googlegroups thusly is quite a mystery... ] -- https://mail.python.org/mailman/listinfo/python-list
Re: memory control in Python
In a message of Tue, 18 Aug 2015 01:56:16 -0700, Rustom Mody writes: >[How she (her mail client) manages to befuddle googlegroups thusly is >quite a mystery... >] For me as well, as all I am doing is just replying to the mail ... And I haven't changed my mail client at all in years and years ... Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: memory control in Python
On 15 August 2015 at 00:41, Ping Liu wrote: > Dear All, Hi Ping Liu, > I am working on an optimization problem, where we are trying to minimize > some indicators like energy usage, energy cost, CO2 emission. In this > problem, we have a bunch of energy conversion technologies for electricity > and thermal purpose, such as heat pump, boiler, chiller, etc.. We are trying > to model it with a one year time period. the current time step is one hour. I guess this means that you are running a simulation over a period of 1 year. If the timestep is one hour then the number of timesteps is 24*365 = 8760. > We have preselected the candidate technologies to exclude those we don't > want to study so that the problem size could be reduced with a limited > candidate technologies. In the current case study, we only analyze the > electric chiller and heat pump to supply the cooling load, while power grid > will supply the electricity for all electric loads. There are binary > variables regarding installation decisions of technologies and continuous > variables related to installation capacity and hourly operational decisions. How many binary variables do you have? I ask because it may be preferable to consider each case of the binary variables separately. Suppose that there are 5 binary variables. Then that makes 2**5 = 32 possible cases. It might be simpler to solve the continuous optimisation problem for each of the 32 cases separately. On the other hand if you have 20 binary variables then you'll have 1e6 cases to consider in which case it is less likely to be feasible to consider each one. > For small cases, Python works well. But if we consider longer time period. > then it would fail due to the memory usage issues. We have tested several > case studies to check the memory use for different time period, including 1) > 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours each > day, as well as 4) 30 days with 24 hours each day. The first 3 cases are > feasible while the last case gives out the memory error. If I understand correctly what you mean is that you are attempting different length simulations. In the first case you have 2 timesteps meaning a simulation of 2 hours, then 24, then 480, and then 720. Your target is to get to 8760 timesteps. Is that correct? > When we are testing the forth case, the memory error comes out while > creating the inequality constraints. The problem size is 1) Aeq: 12 * 26, > Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq: > 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error. I assume that this refers to the constraints that you pass to the solver which varies depending on the number of timesteps in your simulation: Aeq refers to the number of equality constraints and Aineq to the number of inequality constraints? The number of constraints is (timesteps, equality, inequality): 2 12*26 30*26 24 144*268 316*268 480 2880*5284 6244*5284 720 4320*7924 unknown So the pattern for N timesteps seems to be: N (6*N)*(11*N+4) (13*N+4)*(11*N+4) So I guess the number of inequality constraints in the case of memory error is 9364*7924. Essentially what this shows is that your problem is complexity is N**2 where N is the number of timesteps. Even if you can get the memory to go up to 720 timesteps you still need 10 times more steps to get to your target of 8760. Since the problem is growing quadratically you'll need to have 100 times more computer memory again to reach your target. Your problem scales very badly and is probably badly posed. Is it really necessary to impose so many constraints or can the problem be formulated in a way that uses fewer? > The solver is CPLEX (academic). It turns out that the solver is taking a lot > of memory as you can see in the memory test report. for the first three > cases, different memory usage is observed, and it grows up dramatically with > the increase of the time period. 1) solver memory usage: 25.6 MB, 2) 19.5 > MB; 3) solver memory usage: 830.0742 MB. This is unsurprising since the size of your problem grows quadratically. In the 720 timestep case you are trying to optimise with 100 million constraints. For your target of 1 year the number of constraints would be 16 billion. The amount of memory required to store these constraints is proportional to the number of constraints but the time taken to optimise with the constraints will probably grow even faster so even if you had the memory you're unlikely to have the CPU power to solve this. > Based on my observations, I have some of the following questions regarding > 1) In order to create the optimization problem (Aeq, Aineq), how can we > reduce the memory usage in python programming? 2) how can I reduce the > memory usage of different solvers in Python? Why would the memory decrease > for the CPLEX solver within the 24-hours-in-one-day case compared with the > case 1? Others have already pointed out that your p
Dave Angel RIP
Folks, Many of us will remember the long-time list regular Dave Angel. Dave was active on this list for many years, a keen participant in many discussions, and very helpful to Python users of all levels of expertise. Unfortunately, I am saddened to say that Dave passed away on May 25th, and the far too young an age of 66. He will be missed. http://www.monaghanfunerals.com/obits/obituary.php?id=552157 Thanks to Alan Gould on the tutor mailing list for passing this information on. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: execute commands as su on remote server
Hi,
try to use http://www.fabfile.org/
look at:
http://docs.fabfile.org/en/latest/api/core/operations.html
sudo() call has the posibility to change to another user besides root, with:
with settings(sudo_user='mysql'):
sudo("whoami") # prints 'mysql'
El lun., 17 de ago. de 2015 a la(s) 11:07 p. m., Chris Angelico <
ros...@gmail.com> escribió:
> On Tue, Aug 18, 2015 at 12:57 PM, wrote:
> > I need to execute commands after doing su to other user on remote
> server(not sudo which doesn't require password) how i can achieve this
> using python?
> > I googled and came to know that its not possible, so just for
> confirmation asking again, is it possible ?
>
> Ultimately, this isn't a Python question, it's a systems
> administration one. You want a way to execute commands as a specific
> user, triggering them remotely. There are basically two ways you can
> do this: either you provide some form of credentials (this is what
> sudo does), or you elevate the entire management process. The latter
> option is far FAR easier (running it as root if you might need to go
> to any different user, or as that specific user if you'll only ever
> use one), but then you have to ensure, in some way, that your Python
> program can't be compromised.
>
> Python has nothing to do with any of this. If you want to manage
> elevation using sudo, Python can invoke sudo in a subprocess. If you
> want to elevate the Python process and then simply invoke something
> directly, Python won't even be aware of it.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Dave Angel RIP
On Tue, Aug 18, 2015 at 10:25 PM, Steven D'Aprano wrote: > Many of us will remember the long-time list regular Dave Angel. Dave was > active on this list for many years, a keen participant in many discussions, > and very helpful to Python users of all levels of expertise. Aww :( He was notable to me personally for having a similar surname and similar signature style to mine, which we'd both been using for years before ever coming across each other. Vale DaveA. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Dave Angel RIP
Folks, Many of us will remember the long-time list regular Dave Angel. Dave was active on this list for many years, a keen participant in many discussions, and very helpful to Python users of all levels of expertise. Unfortunately, I am saddened to say that Dave passed away on May 25th, and the far too young an age of 66. He will be missed. He will be dearly missed. His contributions have been very helpful and will be even more valued now that they are all we have left to remind us of him here on the list. -- https://mail.python.org/mailman/listinfo/python-list
regex recursive matching (regex 2015.07.19)
Trying regex 2015.07.19
I'd like to match recursive parenthesized expressions, with groups such that
'(a(b)c)'
would give
group(0) -> '(a(b)c)'
group(1) -> '(b)'
but that's not what I get
import regex
#r = r'\((?>[^()]|(?R))*\)'
r = r'\(([^()]|(?R))*\)'
#r = r'\((?:[^()]|(?R))*\)'
m = regex.match (r, '(a(b)c)')
m.groups()
Out[28]: ('c',)
--
https://mail.python.org/mailman/listinfo/python-list
Re: regex recursive matching (regex 2015.07.19)
On 8/18/2015 10:25 AM, Neal Becker wrote:
Trying regex 2015.07.19
I'd like to match recursive parenthesized expressions, with groups such that
'(a(b)c)'
Extended regular expressions can only match strings in extended regular
languages. General nested expressions are too general for that. You
need a context-free parser. You can find them on pypi or write your
own, which in this case is quite simple.
---
from xploro.test import ftest # my personal function test function
io_pairs = (('abc', []), ('(a)', [(0, '(a)')]), ('a(b)c', [(1, '(b)')]),
('(a(b)c)', [(0, '(a(b)c)'), (2, '(b)')]),
('a(b(cd(e))(f))g', [(1, '(b(cd(e))(f))'), (3, '(cd(e))'),
(6, '(e)'), (10, '(f)')]),)
def parens(text):
'''Return sorted list of paren tuples for text.
Paren tuple is start index (for sorting) and substring.
'''
opens = []
parens = set()
for i, char in enumerate(text):
if char == '(':
opens.append(i)
elif char == ')':
start = opens.pop()
parens.add((start, text[start:(i+1)]))
return sorted(parens)
ftest(parens, io_pairs)
---
all pass
would give
group(0) -> '(a(b)c)'
group(1) -> '(b)'
but that's not what I get
import regex
#r = r'\((?>[^()]|(?R))*\)'
r = r'\(([^()]|(?R))*\)'
#r = r'\((?:[^()]|(?R))*\)'
m = regex.match (r, '(a(b)c)')
m.groups()
Out[28]: ('c',)
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: regex recursive matching (regex 2015.07.19)
On 2015-08-18 15:25, Neal Becker wrote:
Trying regex 2015.07.19
I'd like to match recursive parenthesized expressions, with groups such that
'(a(b)c)'
would give
group(0) -> '(a(b)c)'
group(1) -> '(b)'
but that's not what I get
import regex
#r = r'\((?>[^()]|(?R))*\)'
r = r'\(([^()]|(?R))*\)'
#r = r'\((?:[^()]|(?R))*\)'
m = regex.match (r, '(a(b)c)')
m.groups()
Out[28]: ('c',)
You can't capture them into different groups in the general case; it
won't create capture groups dynamically.
Capture into 1 group and then use the .captures method:
import regex
r = r'(\([^()]*(?:(?R)[^()]*)*\))'
m = regex.match(r, '(a(b)c)')
print(m.captures(1))
--
https://mail.python.org/mailman/listinfo/python-list
I2C protocol
Is there any library for I2C protocol so that I can control a device connected to Windows? -- https://mail.python.org/mailman/listinfo/python-list
Re: I2C protocol
On 18/08/2015 20:34, Johny wrote: Is there any library for I2C protocol so that I can control a device connected to Windows? Go to https://pypi.python.org/pypi, put i2c in the search box and take your pick. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Regular expression and substitution, unexpected duplication
Hello,
I want to make a replacement in a string, to ensure that ellipsis are
surrounded by spaces (this is not a typographycal problem, but a preparation
for late text chunking).
I tried with regular expressions and the SRE_Pattern.sub() method, but I
have an unexpected duplication of the replacement pattern:
The code:
ellipfind_re = re.compile(r"((?=\.\.\.)|…)", re.IGNORECASE|re.VERBOSE)
ellipfind_re.sub(' ... ',
"C'est un essai... avec différents caractères… pour voir.")
And I retrieve:
"C'est un essai ... ... avec différents caractères ... pour voir."
^^^
I tested with/without group capture, same result.
My Python version:
Python 3.4.3 (default, Mar 26 2015, 22:03:40)
[GCC 4.9.2] on linux
Any idea ?
Thanks.
Laurent.
--
https://mail.python.org/mailman/listinfo/python-list
Re: memory control in Python
Hi, Oscar, Your feedback is very valuable to me since you dig into the problem itself. Basically, we are trying to develop an open source software with multiple interface to several free solvers so that we can switch among them in case one of them is not working or so efficient. The optimization problem is developed on the basis of OPENOPT. the number of binary variables grows with the change of time period. The problem size (Aeq: equality constraints, Aineq: inequality constraints)and binary variables are as listed below. As you can imagine, it is hard to test each case of binary variables separately. Aeq Aineq Binary 1 hr6*1517*15 6 2 hr12*26 30*26 10 24 hr 144*268 316*26898 480 hr 2880*5284 6244*5284 1922 720 hr 4320*7924 9364*7924 unknown Our final goal is to test 8760 time steps in our problem. We can reduce it according to the purpose of optimization. For example, for planning, the time period would be one year with longer time step than 1 hr; for operational purpose, the time period would be 7 days, with 15 min time step. In the second case, the total time step comes down to 402. Actually, we have tried several solvers, including cplex, LPSOLVE, GLPK. We don't need to stick to CPLEX for the MIP problem. The performance of each solver is compared regarding memory usage(MB). 2hr24hr240hr 480hr GLPK 26 27.6222.6806 LPSOLVE 11.6 15.5 421.61557.3 CPLEX25.6 19.5 192.06 719.1 I think one way to reduce the problem size probably would be to categorize it into planning and operation problem. Another way would be to look into Cython for which I am not quite sure whether it would help. But I would like to try to reformulate the problem itself as you advised. -- https://mail.python.org/mailman/listinfo/python-list
Re: regex recursive matching (regex 2015.07.19)
Neal Becker writes: > Trying regex 2015.07.19 > > I'd like to match recursive parenthesized expressions, with groups such that > '(a(b)c)' > > would give > group(0) -> '(a(b)c)' > group(1) -> '(b)' > > but that's not what I get > > import regex > > #r = r'\((?>[^()]|(?R))*\)' > r = r'\(([^()]|(?R))*\)' > #r = r'\((?:[^()]|(?R))*\)' > m = regex.match (r, '(a(b)c)') The (?R) syntax is Perl -- it's no implemented in Python. Python and Perl regexs are very similar in syntax (for very good reasons) but neither (?R) nor the numbered or named versions of it are in Python. -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Regular expression and substitution, unexpected duplication
On 2015-08-18 22:42, Laurent Pointal wrote:
Hello,
I want to make a replacement in a string, to ensure that ellipsis are
surrounded by spaces (this is not a typographycal problem, but a preparation
for late text chunking).
I tried with regular expressions and the SRE_Pattern.sub() method, but I
have an unexpected duplication of the replacement pattern:
The code:
ellipfind_re = re.compile(r"((?=\.\.\.)|…)", re.IGNORECASE|re.VERBOSE)
ellipfind_re.sub(' ... ',
"C'est un essai... avec différents caractères… pour voir.")
And I retrieve:
"C'est un essai ... ... avec différents caractères ... pour voir."
^^^
I tested with/without group capture, same result.
My Python version:
Python 3.4.3 (default, Mar 26 2015, 22:03:40)
[GCC 4.9.2] on linux
Any idea ?
(?=...) is a lookahead; a non-capture group is (?:...).
The regex should be r"((?:\.\.\.)|…)", which can be simplified to just
r"\.\.\.|…" for your use-case. (You don't need the
re.IGNORECASE|re.VERBOSE either!)
--
https://mail.python.org/mailman/listinfo/python-list
testing, please disregard
I have had some problems with another python.org ML. I am sending this to see if it is received. Please disregard. thanks -- Tim http://www.akwebsoft.com, http://www.tj49.com -- https://mail.python.org/mailman/listinfo/python-list
Re: regex recursive matching (regex 2015.07.19)
On 2015-08-18 22:55, Ben Bacarisse wrote: Neal Becker writes: Trying regex 2015.07.19 I'd like to match recursive parenthesized expressions, with groups such that '(a(b)c)' would give group(0) -> '(a(b)c)' group(1) -> '(b)' but that's not what I get import regex #r = r'\((?>[^()]|(?R))*\)' r = r'\(([^()]|(?R))*\)' #r = r'\((?:[^()]|(?R))*\)' m = regex.match (r, '(a(b)c)') The (?R) syntax is Perl -- it's no implemented in Python. Python and Perl regexs are very similar in syntax (for very good reasons) but neither (?R) nor the numbered or named versions of it are in Python. He's using the regex module from PyPI: https://pypi.python.org/pypi/regex -- https://mail.python.org/mailman/listinfo/python-list
pysqlite 2.8.0 released
NEW FEATURES
- No new features, but tons of bugfixes. These mean that things now work
that
didn't before:
- Transactional DDL now works
- You can use SAVEPOINTs now
BUILD PROCESS
- Python 2.7.x is now required. If trying to use it with Python 3, print a
useful error message. Integrated all fixes from the sqlite3 module in
Python
2.7.10.
MAJOR IMPROVEMENTS
- Completety got rid of statement parsing. We now use SQLite functions to
determine if a statement modifies the database or not. If a statement
modifies the database, then we implicitly start a transaction. For
backwards
compatibility reasons, we do NOT implicitly start a transaction if we
encounter a DDL statement.
You can, however, now have transactional DDL if you want to:
cur = con.cursor()
cur.execute("begin")
cur.execute("create table foo(bar)")
con.rollback()
This also means that people can now finally use SAVEPOINTS.
- Use sqlite3_get_autocommit() to determine if we are within a transaction
instead of trying to be smart.
- Switch to v2 statement API. This simplified the code and will increase
stability.
MINOR IMPROVEMENTS
- You can use unicode strings as index for Row objects.
BUGFIXES
- Fixed a regression: statements should not be reset after a commit.
GENERAL CLEANUP AND DEPRECATIONS
- Since december 2005, row_factory is a feature of the Connection class
instead
of the Cursor class. It was kept in the Cursor class for backwards
compatibility. Now it was time to finally remove it from the Cursor class.
- DEPRECATE converters and adapters.
- DEPRECATE text_factory.
- Remove compatibility workarounds for old Python versions.
- Remove workarounds for old SQLite versions.
- Remove apsw related code.
--
https://mail.python.org/mailman/listinfo/python-list
Logging to a file from a C-extension
If a logging file is opened at the level of a Python application, how would the log file name be communicated to a C-extension so that logging from the extension would be sent to the same log file? -- https://mail.python.org/mailman/listinfo/python-list
Re: How to parse XML file and save results as .txt or .csv?
On Tuesday, 18 August 2015 22:20:32 UTC+5:30, ryguy7272 wrote: > I am trying to parse this XML file. > http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml > > I tried a bunch of things in R, and almost got it working, but it seems like > there was some issue with the rows and columns being out of sync, or some > such thing. Can someone here share some Python code that will parse the XML > file mentioned above, possibly clean up any garbage, and save the results as > .txt or .csv? > > Thanks!! Hey, you can use argparse or xml tree in python and you can save it any where in any format. its really work.. see the examples and try to figure out the code -- https://mail.python.org/mailman/listinfo/python-list
