netCDF4 variable manipulation
Hi, I'm trying to read a netCDF4 variable from a file (no problem) and then scale it before writing over the original variable in the file. I'm using python 2.7 and the latest netCDF4 module. It's not that I keep getting an error message but I want to do this without using for loops and all the manuals seems to be skipping this task as if it is never done. I'm new to python and coming over from matlab. Does anyone know how to modify the netCDF4 variable in python and then write it back without creating a new variable, or using for loops? Appreciate the help, /S -- http://mail.python.org/mailman/listinfo/python-list
Re: netCDF4 variable manipulation
On Feb 21, 12:53 am, Steven D'Aprano wrote:
> On Mon, 20 Feb 2012 12:37:22 -0800, Sheldon wrote:
> > Hi,
>
> > I'm trying to read a netCDF4 variable from a file (no problem) and then
> > scale it before writing over the original variable in the file.
>
> > I'm using python 2.7 and the latest netCDF4 module. It's not that I keep
> > getting an error message but I want to do this without using for loops
> > and all the manuals seems to be skipping this task as if it is never
> > done. I'm new to python and coming over from matlab. Does anyone know
> > how to modify the netCDF4 variable in python and then write it back
> > without creating a new variable, or using for loops?
>
> There is no such thing as "the netCDF4 variable" in Python -- it is not a
> built-in part of the language, and therefore there is no built-in feature
> for manipulating it.
>
> You are going to have to be more specific about what you want than just
> "how do I modify a variable with for loops?".
>
> My wild guess is that you have some sort of config file which includes a
> field called "netCDF4" and you want to update it in place. We can't tell
> you how to do this without knowing what the config file is -- is it an
> INI file, XML, JSON, YAML, Unix-style rc file, a binary pickle, or
> something else?
>
> --
> Steven
Hi,
My apologies. I didn't realize this module was not popular. I'm using
the python module "netCDF4"
that I installed using PIP. This module gives me the ability to read
and write netcdf4 files.
The module is an interface to the netcdf4.1.3 and HDF51.8.8 libraries.
What I'm trying to do is open an existing netcdf file, extract an
variable, modify the variable, and then
write it again to the file - effectively replacing the old variable.
rgrp = netCDF4.Dataset('ODIN_NWP_2001_01_01_00.NC','r+')
pv = rgrp.groups['Data_3D'].variables['PV']
print pv
float32 PV(u'level', u'lat', u'lon')
longname: Potential Vorticity
_FillValue: -999.0
units: K m**2 kg**-1 s**-1
code: 60
scale_factor: 1.0
add_offset: 0.0
path = /Data_3D
unlimited dimensions = ()
current size = (60, 181, 360)
As you can see, pv is described here as a netCDF4.Variable. I'm trying
to modify this pv and then write it back to the file
but the manual is vague on this part. I tried this:
pv = pv*2
---
TypeError Traceback (most recent call
last)
> 1 pv = pv*2
TypeError: unsupported operand type(s) for *: 'netCDF4.Variable' and
'int'
I'm unsure how this is suppose to work.
/S
--
http://mail.python.org/mailman/listinfo/python-list
syntax error
Hi Everyone, I am new here but I am pretty good at python programming but I am not exert. I have been away from programming for about a year and now I am programming in python again in combination with IDL. I came across a error that puzzles me and I need some help. It is pretty simple error but I suspect that the problem is not what is python says it is. While parsing a module my python reported a synthax error that, to me, seemed perfectly ok: Traceback (most recent call last): File "/local_disk/opt/MSG_PostProc/scr/sheldon_cmsaf_remap.py", line 2, in ? from msgpp_config import * File "/local_disk/opt/MSG_PostProc/cfg/msgpp_config.py", line 73 from smhi_safnwc_legends import * ^ SyntaxError: invalid syntax Now I know that there is no synthax error with that line of code. I have also checked for indentations errors. I think that the error is something else. Can anyone point me in the right direction? What triggers such erroneous errors in Python? Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: syntax error
Hi Rod, This sounds very interesting. I am checking the previous lines and will get back to you. Thanks, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: syntax error
Hi Rod, You were right. The error was on the previous line. I will remember that. Thanks for your help! Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: syntax error
Hi Fredrik, I am using python 2.3.3 I am checking now the previous lines for errors. Thanks, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading from input file.
Hi Kyrstian, Try reading the file with ope(file,'r').read() and then split the string using '\n' as the delimiter (this will break the file down into each individual line consisting of several columns). Then string split each line of the file (unsing ' ' as the delimiter), they should now be 3, or more, columns per line, and store each column in a vector - adding each line to the vector with append function. Then place each vector in a list. Now you should have a file to compare with the next one. But watch out for files of varying lengths as they cannot be compared on a one-to-one basis. One you have each file in a list or vector or dictionary or array style, comparison and calculations should be straight forward. Hope this points you in the right direction. I haven't tried this myself but I have done a similar thing earlier but I strongly believe that it should work. Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading from input file.
Hi,
after you have read the file then split it like this:
file = open('inputfile.txt', 'r').read()
import string
file = string.split(file,'\n')
now if you print file[0] you should only get the first line.
Be careful and examine the file for non-continuous sections where a
line is empty. That is to say where file[x] = ' '. You should remove
these lines from the txt files before you read it with python. It maybe
so that python reads the entire file, empty spaces and all, and then
you have to remove them after the split with a WHERE statement. This is
a much easier way so lets hope that this is so.
I hope this was more clear and helpful this time around.
Cheers
Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exception Handling
Hi, This is a non-trivial thing that you are trying to do. You can use some of python's built-in exceptions, like RuntimeError or IOError and if so then: try: call C except IOError, e: print e But this will return and print only IOErrors if they occur. You can define your own error handling using the function RAISE: try: Call C except: raise my_error. A catch-all error is RuntimeError; try this first. try: call C except RuntimeError, r: print r You can read up on it here: http://docs.python.org/api/standardExceptions.html Cheers, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading from input file.
So Mike if you can do better then do it then! There are many ways do solve a problem, perhaps you have not learned that yet. At first this guy didn't know what to do, so he had to begin somewhere. Now you can take him much further, I am sure but the journey might not be so pleasant. Your attitude really needs major adjustment but then again, I have to consider the source. FYI, the file[0] part was simply to check if it worked. Don't waste your time with me and my "hard way", it is Krystian you should be helping! -- http://mail.python.org/mailman/listinfo/python-list
Re: Exception Handling
Hi, I am not a c-expert, thank God! That code drives people mad :) But I do know that you can redirect the data stream into a variable of sorts. Maybe you should try writing the error to a file and then read the file with python. Python will not catch all the errors and if there is one that you need specifically, then you will have to make your own exception. I sent the link to you earlier. I use a large and complex script that is an unholy elixer of C, FORTRAN, and Python. When there is an error in the C code the command returned to python is the UNIX process code, i.e. 0 if the c-code was executed without error or -1 for an error. We have not tried to catch the error from the C-code. Sorry. Perhaps some else knows how. cheers, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading from input file.
Sorry Mike, after seeing so many "experts" beat up others for not being as "smart" as they are, I intrepreted your words incorrectly - my apologies. I am not in the least bit against impproving my programming. I liked what you did and thanks for the pointers. Sheldon -- http://mail.python.org/mailman/listinfo/python-list
C wrapper
Hi, Can anyone give me some idea as to what this error means? "ImportError: dynamic module does not define init function " I am new at this and there is still a lot to learn. Any help is appreciated, /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: C wrapper
Farshid Lashkari skrev: > Sheldon wrote: > > Can anyone give me some idea as to what this error means? > > > > "ImportError: dynamic module does not define init function " > > > > I am new at this and there is still a lot to learn. > > > > Any help is appreciated, > > Take a look at the documentation for creating extension modules, > especially the following page: > > http://docs.python.org/ext/methodTable.html > > "The initialization function must be named initname(), where name is the > name of the module, and should be the only non-static item defined in > the module file" > > -Farshid This function is there and is called init_mymodule() but I have other functions that are not static. Could this be the cause? /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: C wrapper
Robert Kern skrev:
> Sheldon wrote:
>
> > This function is there and is called init_mymodule() but I have other
> > functions that are not static.
>
> Is the module's name "_mymodule"? Or is it "mymodule"?
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
> that is made terrible by our own mad attempt to interpret it as though it had
> an underlying truth."
> -- Umberto Eco
Here is the file/module name: _msgpps_functions.c
Here is the initfunction:
PyMODINIT_FUNC init_msgpps_functions(void) {
PyObject* m;
m=Py_InitModule("_msgpps_functions",_msgpps_functionsMethods);
ErrorObject = PyString_FromString("_msgpps_functions.error");
if(ErrorObject == NULL || \
PyDict_SetItemString(PyModule_GetDict(m),"error",ErrorObject)!=0)
{
Py_FatalError("Can't define _msgpps_functions.error");
import_array();
} /* access to Numeric PyArray functions */
}
I have not main() function in the file. Instead the main function is
called the same name:
static PyObject* _msgpps_functions(PyObject* self, PyObject* args)
Now I am new at this and I have been reading anything I can find. The
only thing that is out of place is the part which I didn't include:
/* Initialize the Python interpreter. Required. */
Py_Initialize();
/* Add a static module */
initspam();
because I still don't understand this part.
/sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Re: C wrapper
Hi, > For a module called foo.c the initialization function must be called > initfoo (*not* init_foo) Ok, I fixed this part. Thanks > And what are those non-static functions used for? The *only* purpose > of your module should be to provide the Python bindings... I wrote the C module to do some number crunching. Now I just need to "connect" it to my python program. Should the initmsgpps_functions() be the only function in the file? Then how do I "connect" my C module to my Python program? /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: C wrapper
Gabriel Genellina skrev:
> At Tuesday 7/11/2006 17:27, Sheldon wrote:
>
> >Here is the file/module name: _msgpps_functions.c
> >Here is the initfunction:
> >
> >PyMODINIT_FUNC init_msgpps_functions(void) {
> > PyObject* m;
> > m=Py_InitModule("_msgpps_functions",_msgpps_functionsMethods);
> > ErrorObject = PyString_FromString("_msgpps_functions.error");
> > if(ErrorObject == NULL || \
> > PyDict_SetItemString(PyModule_GetDict(m),"error",ErrorObject)!=0)
> >{
> > Py_FatalError("Can't define _msgpps_functions.error");
> > import_array();
> > } /* access to Numeric PyArray functions */
> >}
> >
> >
> >I have not main() function in the file. Instead the main function is
> >called the same name:
> >
> >static PyObject* _msgpps_functions(PyObject* self, PyObject* args)
> >
> >Now I am new at this and I have been reading anything I can find. The
> >only thing that is out of place is the part which I didn't include:
> >
> > /* Initialize the Python interpreter. Required. */
> > Py_Initialize();
> >
> > /* Add a static module */
> > initspam();
> >because I still don't understand this part.
>
> Are you *extending* Python with a new module written in C (you should
> be using the first part),
> or *embedding* python inside your application, mainly written in C
> (you would use something like the last code).
>
>
> --
> Gabriel Genellina
> Softlab SRL
>
> __
> Correo Yahoo!
> Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
> ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar
Ok,
This I have done but still, the same error message. :(
--
http://mail.python.org/mailman/listinfo/python-list
C extensions and memory leakage
Hi, I have a program that I have extended with a C function. When the program is run once or twice everything is ok but in a loop of 5 to 12 iterations, the memory runs out and the program crashes. Now I have gone through this program thoroughly to check if all arrays have been deallocated prior to exiting and they have, but still the problem exists. Now I am wondering if the problem is in Python and the wrapper? Does anybody have any idea or experience with this? I am running on Mandrake10 using python 2.3. I am not exactly sure which C wrapper I am using as I have copied it from another person. thanks in advance, /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: C extensions and memory leakage
Marc 'BlackJack' Rintsch wrote: > In <[EMAIL PROTECTED]>, Sheldon wrote: > > > Now I am wondering if the problem is in Python and the wrapper? Does > > anybody have any idea or experience with this? I am running on > > Mandrake10 using python 2.3. I am not exactly sure which C wrapper I am > > using as I have copied it from another person. > > What do you mean by "C wrapper"? You know that Python uses reference > counters to manage memory and that you are responsible for these counters > if you are dealing with Python objects in your C extension? > > Ciao, > Marc 'BlackJack' Rintsch I am very new at this C extensions in Python so my term wrapper was probably a misnomer. Perhaps glue is better or the interface that allows the exchange of data between Python and C. Yes, I am using python objects in my C extension. Tell me where I can find out more about this reference counters? Or perhaps you can tell something about it. /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
printing all variables
Good day, I would like to know if there is a way to print all the variables set in a python program with having to write "print variable" on all? sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: printing all variables
Duncan Booth skrev:
> Sheldon wrote:
>
> > Good day,
> >
> > I would like to know if there is a way to print all the variables set
> > in a python program with having to write
> > "print variable" on all?
> >
> Not all the variables in a program (that would be rather more than you
> want), but you can print all the variables in a specific namespace easily
> enough:
>
> >>> from pprint import pprint
> >>> def f(x):
> pprint(locals())
>
>
> >>> f(2)
> {'x': 2}
> >>> pprint(globals())
> {'__builtins__': ,
> '__doc__': None,
> '__name__': '__main__',
> 'f': ,
> 'pprint': }
> >>> class C:
> classvar = []
> def __init__(self, n):
> self.n = n
>
>
> >>> c = C(3)
> >>> pprint(vars(c))
> {'n': 3}
> >>> pprint(vars(C))
> {'__doc__': None,
> '__init__': ,
> '__module__': '__main__',
> 'classvar': []}
> >>>
Thanks Duncan! This really helps!
/Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
python and HDF
Hi, I am using a HLHDF C wrapper to extract data from HDF (version 5) files. I can view the contents with IDL, for example, and read the attributes in the HDF file. There is one attribute that I extracted and only of part of the string was returned. In the HDF file, the attribute data is a name which is called "baseline" but when python reads it, it returns "baselin" instead. Does anyone have any idea why this is happening? Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: python and HDF
Marc 'BlackJack' Rintsch skrev: > In <[EMAIL PROTECTED]>, Sheldon wrote: > > > I am using a HLHDF C wrapper to extract data from HDF (version 5) > > files. > > I can view the contents with IDL, for example, and read the attributes > > in the HDF file. There is one attribute that I extracted and only of > > part of the string was returned. In the HDF file, the attribute data is > > a name which is called "baseline" but when python reads it, it returns > > "baselin" instead. > > Does anyone have any idea why this is happening? > > Not from this description. How do you read the data with Python? Which C > library/wrapper do you use? > > Ciao, > Marc 'BlackJack' Rintsch Hi I an using HLHDF developed locally. I found someone who said that if the string was written to the HDF file with being NULL+1 terminated then this would happen. Sounds good to me. Thanks for your help. /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
comparing two arrays
Hi, I have two arrays that are identical and contain 1s and zeros. Only the ones are valid and I need to know where both arrays have ones in the same position. I thought logical_and would work but this example proves otherwise: >>> a = [0,1,2,5,6,6] >>> b = [5,4,1,6,4,6] >>> Numeric.logical_and(a==6,b==6) 0 >>> Numeric.where(a==b,1,0) 0 >>> Numeric.where(a==6 and b==6,1,0) 0 The where() statement is also worhtless here. Does anyone have any suggestion on how to do this? Thanks in advance, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: comparing two arrays
Diez B. Roggisch skrev: > Diez B. Roggisch wrote: > > > print [i for i, _ in enumerate((None for v in zip(a, b) where v == > > (1,1)))] > > > > should give you the list of indices. > > I musunderstood your question. Use > > > print [i for i, _ in enumerate((None for x, y in zip(a, b) where x == y))] > > instead. > > Diez Hi Diez, I wish I say that I understood what you wrote here but I can't. Do you mind explaining a little more? /sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: comparing two arrays
Thanks Diez, It will take a little while for this one to sink in but it gets the job done now and will for future cases. /Sheldon Diez B. Roggisch skrev: > >> print [i for i, _ in enumerate((None for x, y in zip(a, b) where x == > >> y))] > >> > >> instead. > >> > >> Diez > > > > Hi Diez, > > > > I wish I say that I understood what you wrote here but I can't. > > Do you mind explaining a little more? > > I actually made a typo, instead of "where" in the above use "if". and the > whole thing won't compute what you are after. This will: > > [i for i, equals in enumerate((x == y for x, y in zip(a, b))) if equals] > > Sorry for that. > > Its a nested list-comprehension. Actually, the outer thingy is a > list-comprehension while the inner is a generator expression. There is of > course a difference, but for now this doesn't matter too much. > > a list comprehension of form > > [ for in if ] > > is a more compact form for > > result = [] > for in : > if : >result.append() > > > A generator expression works the same with () instead of []. > > Now let's decompose the above statment: > > (x == y for x, y in zip(a, b) )) > > = x, y > = zip(a, b) > = x == y > > So: the iterable is returned by the function zip. That is a built-in which > will take 2 or more lists and return a list of tuples, where the first > element is from the first list, then the second and so on. So your example > lists become: > > a = [0,1,2,5,6,6] > b = [5,4,1,6,4,6] > zip(a, b) = [(0,5), (1,4), (2,1), (5,6), (6,4), (6,6)] > > So iterating over this yields the pairs of (0,5) and so forth. > > Next thing is that with > > x, y = p > > python unpacks a tuple and refers to its contents by name x for the first > one and y for the second. So overall, we loop in sync over both lists, and > getting x and y to point to the corresponding elements. > > Now the expression x == y will result True if x == y - False otherwise. > > So the result of the inner listcomp/genexp looks like this: > > res = [False, False, False, False, False, True] > > As you can see: for each pair x,y in the original lists we come up with the > result of comparing them. > > Now the outer listcomp using "res" instead of the genexps for clarity reads > like this: > > > > [i for i, equals in enumerate(res) if equals] > > = i, equals > = enumerate(res) > = i > = equals > > enumerate is another built-in that takes an iterable and returns a tuple of > > (pos, element) > > for each element in the iterable. > > So for our list, it will return: > > [(0, False), (1, False), (2, False), (3, False), (4, False), (5, True)] > > These tupkes values are assigened to i and equals for each element of the > above list. The condtion > > equals > > will then guarantee that only those expressions are evaluated where equals > is True - the last pair, so to speak. The expression then only stores the > index i. Which will give us the desired result. > > Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: comparing two arrays
Thanks Diez, It will take a little while for this one to sink in but it gets the job done now and will for future cases. /Sheldon Diez B. Roggisch skrev: > >> print [i for i, _ in enumerate((None for x, y in zip(a, b) where x == > >> y))] > >> > >> instead. > >> > >> Diez > > > > Hi Diez, > > > > I wish I say that I understood what you wrote here but I can't. > > Do you mind explaining a little more? > > I actually made a typo, instead of "where" in the above use "if". and the > whole thing won't compute what you are after. This will: > > [i for i, equals in enumerate((x == y for x, y in zip(a, b))) if equals] > > Sorry for that. > > Its a nested list-comprehension. Actually, the outer thingy is a > list-comprehension while the inner is a generator expression. There is of > course a difference, but for now this doesn't matter too much. > > a list comprehension of form > > [ for in if ] > > is a more compact form for > > result = [] > for in : > if : >result.append() > > > A generator expression works the same with () instead of []. > > Now let's decompose the above statment: > > (x == y for x, y in zip(a, b) )) > > = x, y > = zip(a, b) > = x == y > > So: the iterable is returned by the function zip. That is a built-in which > will take 2 or more lists and return a list of tuples, where the first > element is from the first list, then the second and so on. So your example > lists become: > > a = [0,1,2,5,6,6] > b = [5,4,1,6,4,6] > zip(a, b) = [(0,5), (1,4), (2,1), (5,6), (6,4), (6,6)] > > So iterating over this yields the pairs of (0,5) and so forth. > > Next thing is that with > > x, y = p > > python unpacks a tuple and refers to its contents by name x for the first > one and y for the second. So overall, we loop in sync over both lists, and > getting x and y to point to the corresponding elements. > > Now the expression x == y will result True if x == y - False otherwise. > > So the result of the inner listcomp/genexp looks like this: > > res = [False, False, False, False, False, True] > > As you can see: for each pair x,y in the original lists we come up with the > result of comparing them. > > Now the outer listcomp using "res" instead of the genexps for clarity reads > like this: > > > > [i for i, equals in enumerate(res) if equals] > > = i, equals > = enumerate(res) > = i > = equals > > enumerate is another built-in that takes an iterable and returns a tuple of > > (pos, element) > > for each element in the iterable. > > So for our list, it will return: > > [(0, False), (1, False), (2, False), (3, False), (4, False), (5, True)] > > These tupkes values are assigened to i and equals for each element of the > above list. The condtion > > equals > > will then guarantee that only those expressions are evaluated where equals > is True - the last pair, so to speak. The expression then only stores the > index i. Which will give us the desired result. > > Diez -- http://mail.python.org/mailman/listinfo/python-list
OverflowError: math range error...
Hi, I have a written a script that will check to see if the divisor is zero before executing but python will not allow this: if statistic_array[0:4] > 0.0: statistic_array[0,0:4] = int(multiply(divide(statistic_array[0,0:4],statistic_array \ [0,4]),1.0))/100.0 Does anyone know why Python is complaining: "statistic_array[0,0:4] = int(multiply(divide(statistic_array[0,0:4],statistic_array[0,4]),1.0))/100.0 OverflowError: math range error" and how do I get around this problem? This stupid because there is a if statement preventing this "dividing by zero". Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: OverflowError: math range error...
Thanks for the tips! I am going to look into this some more. /Sheldon Simon Forman skrev: > Sheldon wrote: > > Hi, > > > > I have a written a script that will check to see if the divisor is zero > > before executing but python will not allow this: > > > > if statistic_array[0:4] > 0.0: > > statistic_array[0,0:4] = > > int(multiply(divide(statistic_array[0,0:4],statistic_array \ > > [0,4]),1.0))/100.0 > > > > Does anyone know why Python is complaining: > > > > "statistic_array[0,0:4] = > > int(multiply(divide(statistic_array[0,0:4],statistic_array[0,4]),1.0))/100.0 > > > > OverflowError: math range error" > > > > and how do I get around this problem? This stupid because there is a if > > statement preventing this "dividing by zero". > > > > Sincerely, > > Sheldon > > I don't know what special math modules you're using, but python usually > raises ZeroDivisionError for divide-by-zero problems. > > Try printing the intermediate values of each step in your problem code. > > d = divide(statistic_array[0,0:4], statistic_array[0,4]) > print d > > m = multiply(d, 1.0) > print m > > i = int(m) > print i > > statistic_array[0,0:4] = i > > > That might help you track down what's wrong. -- http://mail.python.org/mailman/listinfo/python-list
Re: OverflowError: math range error...
Thanks for the tips! I am going to look into this some more. I am not used to using Numeric and the logical functions. I didn't think about what you pointed out and somewhere the returned values from these logical methods are not what I expect. I will rewrite the whole thing using alltrue instead. /Sheldon Robert Kern skrev: > Sheldon wrote: > > Hi, > > > > I have a written a script that will check to see if the divisor is zero > > before executing but python will not allow this: > > > > if statistic_array[0:4] > 0.0: > > statistic_array[0,0:4] = > > int(multiply(divide(statistic_array[0,0:4],statistic_array \ > > [0,4]),1.0))/100.0 > > > > Does anyone know why Python is complaining: > > > > "statistic_array[0,0:4] = > > int(multiply(divide(statistic_array[0,0:4],statistic_array[0,4]),1.0))/100.0 > > > > OverflowError: math range error" > > > > and how do I get around this problem? This stupid because there is a if > > statement preventing this "dividing by zero". > > What kind of arrays are you using? If it's Numeric (and I think it is because > numarray and numpy would throw an error at the if: statement), then your test > is > incorrect. > > Comparisons yield arrays of boolean values. When a Numeric boolean array is > used > as a truth value (like in an if: statement), then it will return True is *any* > of the values are True. Use Numeric.alltrue(statistic_array[:4] > 0.0) > instead. > > Both numarray and numpy throw an exception when one attempts to use arrays as > truth values since the desired meaning (alltrue or sometrue) is ambiguous. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." >-- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
array manipulation without for loops
Hi, I have two arrays that are of the same dimension but having 3 different values: 255, 1 or 2. I would like to set all the positions in both arrays having 255 to be equal, i.e., where one array has 255, I set the same elements in the other array to 255 and visa versa. Does anyone know how to do this without using for loops? Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: array manipulation without for loops
Hi Gary, I am really trying to cut the time down as I have 600+ arrays with dimensions (1215,1215) to compare and I do a lot more things with the arrays. If I understand you correctly, there is no way around a for loop? /Sheldon Gary Herron wrote: > Sheldon wrote: > > Hi, > > > > I have two arrays that are of the same dimension but having 3 different > > values: 255, 1 or 2. > > I would like to set all the positions in both arrays having 255 to be > > equal, i.e., where one array has 255, I set the same elements in the > > other array to 255 and visa versa. Does anyone know how to do this > > without using for loops? > > > > Sincerely, > > Sheldon > > > > > Whatever for? Have you got something against for loops? > > However... > > You could roll you own loop: > i=0 > while i < whatever: > # ... do something with i > i += 1 > > But what's the point? This does the same as a for loop but slower. > > If you don't want any kind of a loop (again, What's the point?) you > could write something recursive: > > def proc(i, ...): > # ... do something with i > if i < whatever: > proc(i+1, ...) > > But this would be even slower. > > Gary Herron -- http://mail.python.org/mailman/listinfo/python-list
Re: array manipulation without for loops
Alex, I am using Numeric and have created 3 arrays: zero((1215,1215),Float) Two arrays are compared and one is used to hold the mean difference between the two compared arrays. Then I compare 290 or 340 pairs of arrays. I know that memory is a problem and that is why I don't open all of these arrays at the same time. I cannot install Numpy due to my working conditions. Sorry I should have made it clear that is was Numeric I was working with. /Sheldon Alex Martelli wrote: > Sheldon <[EMAIL PROTECTED]> wrote: > > > Hi Gary, > > > > I am really trying to cut the time down as I have 600+ arrays with > > dimensions (1215,1215) to compare and I do a lot more things with the > > arrays. If I understand you correctly, there is no way around a for > > loop? > > In pure Python (w/o extension packages) there are no 2-D arrays; so > either you're using lists of lists (and I wonder how you fit even one of > them in memory, if they're 1215 by 1215, much less 600!) or you're > already using some extension (Numeric, numarray, numpy) and aren't > telling us which one. If you're using pure Python add your extension of > choice, if you're using an extension already tell us which one, and in > each case there will be ways to perform your manipulation tasks faster > than Python-level for loops would afford. > > > Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: array manipulation without for loops
Hi Alex, I will code this in a little while and get back to you. Terrific! I saw this function but I skipped over it without realizing what it could do. The Numeric doc is not very good and I am just getting into Python so your book sounds great especially since it covers Numeric. I will look into it when I get back to work tomorrow. Bye for now, Sheldon Alex Martelli wrote: > Sheldon <[EMAIL PROTECTED]> wrote: > > > Alex, > > > > I am using Numeric and have created 3 arrays: zero((1215,1215),Float) > > Two arrays are compared and one is used to hold the mean difference > > between the two compared arrays. Then I compare 290 or 340 pairs of > > arrays. I know that memory is a problem and that is why I don't open > > all of these arrays at the same time. I cannot install Numpy due to my > > working conditions. Sorry I should have made it clear that is was > > Numeric I was working with. > > It's OK, even if the hard-core numeric-python people are all > evangelizing for migration to numpy (for reasons that are of course > quite defensible!), I think it's quite OK to stick with good old Numeric > for the moment (and that's exactly what I do for my own personal use!). > > So, anyway, I'll assume you mean your 1215 x 1215 arrays were created by > calling Numeric.zeros, not "zero" (with no trailing s) which is a name > that does not exists in Numeric. > > Looking back to your original post, let's say that you have two such > arrays, a and b, both 1215x1215 and of Numeric.Float type, and the > entries of each array are all worth 1, 2, or 255 (that's how I read your > original post; if that's not the case, please specify). We want to > write a function that alters both a and b, specifically setting to 255 > all entries in each array whose corresponding entries are 255 in the > other array. > > Now that's pretty easy -- for example: > > import Numeric > > def equalize(a, b, v=255): > Numeric.putmask(a, b==v, v) > Numeric.putmask(b, a==v, v) > > if __name__ == '__main__': > a = Numeric.zeros((5,5), Numeric.Float) > b = Numeric.zeros((5,5), Numeric.Float) > a[1,2]=a[2,1]=b[3,4]=b[0,2]=255 > a[3,0]=a[0,0]=1 > b[0,3]=b[4,4]=2 > print "Before:" > print a > print b > equalize(a, b) > print "After:" > print a > print b > > > brain:~/pynut alex$ python ab.py > Before: > [[ 1.0.0.0.0.] > [ 0.0. 255.0.0.] > [ 0. 255.0.0.0.] > [ 1.0.0.0.0.] > [ 0.0.0.0.0.]] > [[ 0.0. 255.2.0.] > [ 0.0.0.0.0.] > [ 0.0.0.0.0.] > [ 0.0.0.0. 255.] > [ 0.0.0.0.2.]] > After: > [[ 1.0. 255.0.0.] > [ 0.0. 255.0.0.] > [ 0. 255.0.0.0.] > [ 1.0.0.0. 255.] > [ 0.0.0.0.0.]] > [[ 0.0. 255.2.0.] > [ 0.0. 255.0.0.] > [ 0. 255.0.0.0.] > [ 0.0.0.0. 255.] > [ 0.0.0.0.2.]] > brain:~/pynut alex$ > > Of course I'm using tiny arrays here, for speed of running and ease of > display and eyeball-checking, but everything should work just as well in > your case. Care to check and let us know? > > Numeric has pretty good documentation (numpy's is probably even better, > but it is not available for free, so I don't know!), and if you don't > find that documentation sufficient you might want to have a look to my > book "Python in a Nutshell" which devotes a chapter to Numeric (it also > is not available for free, but you can get a subscription to O'Reilly's > Safari online-books repository, which is free for the first two weeks, > and lets you look at many books including Python in a Nutshell -- if you > don't want to pay monthly subscription fees, make sure you cancel your > trial subscription before two weeks have passed!!!). > > I strongly recommend that, in some way or other, you DO get a taste of > the huge amount of functionality that Numeric provides for you -- with > the size of computational tasks you're talking about, an investment of > 2-3 hours spent becoming deeply familiar with everything Numeric offers > may well repay itself in savings of ten times as much execution time, > and what other investments offer such ROI as 1000%?-) > > > Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: array manipulation without for loops
The following script (using your function) raised no exception so it worked! Elegant Alex, thanx. res = equalize_arrays(msgtmp,ppstmp,255) # class (ppstmp,msgtmp) = res.equalize() # class method for i in range(int(main.xsize)): for j in range(int(main.ysize)): if msgtmp[i,j] == 255 and ppstmp[i,j] != 255: raise "equalize error!" if ppstmp[i,j] == 255 and msgtmp[i,j] != 255: raise "equalize error!" I read up on the putmask function and I don't understand this part: >>> print x [10 1 30 3 50] >>> putmask(x, [1,0,1,0,1], [-1,-2]) >>> print x [-1 1 -1 3 -1] Can you explain why the -2 didn't factor in? /Sheldon Sheldon wrote: > Hi Alex, > > I will code this in a little while and get back to you. Terrific! I saw > this function but I skipped over it without realizing what it could do. > > The Numeric doc is not very good and I am just getting into Python so > your book sounds great especially since it covers Numeric. I will look > into it when I get back to work tomorrow. > > Bye for now, > Sheldon > > Alex Martelli wrote: > > > Sheldon <[EMAIL PROTECTED]> wrote: > > > > > Alex, > > > > > > I am using Numeric and have created 3 arrays: zero((1215,1215),Float) > > > Two arrays are compared and one is used to hold the mean difference > > > between the two compared arrays. Then I compare 290 or 340 pairs of > > > arrays. I know that memory is a problem and that is why I don't open > > > all of these arrays at the same time. I cannot install Numpy due to my > > > working conditions. Sorry I should have made it clear that is was > > > Numeric I was working with. > > > > It's OK, even if the hard-core numeric-python people are all > > evangelizing for migration to numpy (for reasons that are of course > > quite defensible!), I think it's quite OK to stick with good old Numeric > > for the moment (and that's exactly what I do for my own personal use!). > > > > So, anyway, I'll assume you mean your 1215 x 1215 arrays were created by > > calling Numeric.zeros, not "zero" (with no trailing s) which is a name > > that does not exists in Numeric. > > > > Looking back to your original post, let's say that you have two such > > arrays, a and b, both 1215x1215 and of Numeric.Float type, and the > > entries of each array are all worth 1, 2, or 255 (that's how I read your > > original post; if that's not the case, please specify). We want to > > write a function that alters both a and b, specifically setting to 255 > > all entries in each array whose corresponding entries are 255 in the > > other array. > > > > Now that's pretty easy -- for example: > > > > import Numeric > > > > def equalize(a, b, v=255): > > Numeric.putmask(a, b==v, v) > > Numeric.putmask(b, a==v, v) > > > > if __name__ == '__main__': > > a = Numeric.zeros((5,5), Numeric.Float) > > b = Numeric.zeros((5,5), Numeric.Float) > > a[1,2]=a[2,1]=b[3,4]=b[0,2]=255 > > a[3,0]=a[0,0]=1 > > b[0,3]=b[4,4]=2 > > print "Before:" > > print a > > print b > > equalize(a, b) > > print "After:" > > print a > > print b > > > > > > brain:~/pynut alex$ python ab.py > > Before: > > [[ 1.0.0.0.0.] > > [ 0.0. 255.0.0.] > > [ 0. 255.0.0.0.] > > [ 1.0.0.0.0.] > > [ 0.0.0.0.0.]] > > [[ 0.0. 255.2.0.] > > [ 0.0.0.0.0.] > > [ 0.0.0.0.0.] > > [ 0.0.0.0. 255.] > > [ 0.0.0.0.2.]] > > After: > > [[ 1.0. 255.0.0.] > > [ 0.0. 255.0.0.] > > [ 0. 255.0.0.0.] > > [ 1.0.0.0. 255.] > > [ 0.0.0.0.0.]] > > [[ 0.0. 255.2.0.] > > [ 0.0. 255.0.0.] > > [ 0. 255.0.0.0.] > > [ 0.0.0.0. 255.] > > [ 0.0.0.0.2.]] > > brain:~/pynut alex$ > > > > Of course I'm using tiny arrays here, for speed of running and ease of > > display and eyeball-checking, but everything should work just as well in > > your case. Care to check and let us know? > > > > Numeric has pretty good documentation (numpy
Numeric help!
Hi, I have the following loop that I think can be written to run faster in Numeric. I am currently using Numeric. range_va = [60,61,62,63,64,65,66,67,68,69,70,71,72] main.xsize= 600 main.ysize= 600 #msgva is an (600x600) Numeric array with mutiple occurrences of the values in range_va #sat_id is an (600x600) Numeric array with values ranging from -2 to 2 for z in range_va: count = 0 mbias = 0 for i in range(main.xsize): for j in range(main.ysize): if msgva[i,j] == z: mbias += sat_id[i,j] # take the sum of the values count += 1 # count the occurrences tmp_array[0,index] = round(mbias/count,1) # store the mean tmp_array[1,index] = z index += 1 Any help would be greatly appreciated! Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Numeric help!
Hi Carl, My days as a student is over for the most part. I am learning python on my own and Numeric is not properly documented so I am learning by doing and copying from others. I thought about the problem and a solution another problem given to me earlier using "putmask" is the solution but there is a bug that I cannot figure out: index = 0 for z in range_va: wk = msgva # working arrary sattmp = sat_id # working array mask = where(equal(wk,z),1,0) # creating a mask of valid data pixs putmask(sattmp,mask==0,-999) # since zero is valid data, -999 is used instead rdata = compress(ravel(not_equal(sattmp,-999)),ravel(sattmp)) if sum(sum(rdata)) == 0: av = 0 else: av = average(rdata,axis=None) tmparray[0,index] = av tmparray[1,index] = z index += 1 print tmparray *** But the tmparray is returning zeros as averages. When I try just one value for z everything works. I can't see where I going wrong. I am not using the original arrays, only the copies and when a new z is chosen then these are recreated. Care to help out with this? /Sheldon Carl Banks wrote: > Sheldon wrote: > > Hi, > > > > I have the following loop that I think can be written to run faster in > > Numeric. I am currently using Numeric. > > range_va = [60,61,62,63,64,65,66,67,68,69,70,71,72] > > main.xsize= 600 > > main.ysize= 600 > > #msgva is an (600x600) Numeric array with mutiple occurrences of the > > values in range_va > > #sat_id is an (600x600) Numeric array with values ranging from -2 to 2 > > for z in range_va: > > count = 0 > > mbias = 0 > > for i in range(main.xsize): > > for j in range(main.ysize): > > if msgva[i,j] == z: > > mbias += sat_id[i,j] # take the sum of the > > values > > count += 1 # count the occurrences > > tmp_array[0,index] = round(mbias/count,1) # store the mean > > tmp_array[1,index] = z > > index += 1 > > > > Any help would be greatly appreciated! > > I'm not sufficiently sure this isn't a homework problem, so here's a > partial answer. > > Your intuition is correct--there's almost always a faster way to do it > when you're cycling through Numeric array indices in a Python for loop. > Numeric and successors exist mostly to get rid of them. > > Given z, you can calculate mbias in one line: > > mbias = Numeric.sum( > Numeric.ravel(Numeric.where(msgva==z,sat_id,0))) > > Here, the Numeric.where function allows you to filter out entries: it > returns an array with the elements of sat_id, but only where the > corresponding element of msvga==z; otherwise the entry is zero. > Numeric.ravel flattens an multidimensional array into one dimension, > and, of course, Numeric.sum adds up all the elements in the array. > > How to get count and to work this into your loop are left as an > exercise. > > > Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Numeric help!
Carl Banks wrote: > Sheldon wrote: > >Carl Banks wrote: > >> I'm not sufficiently sure this isn't a homework problem, so here's a > >> partial answer. > [snip] > > > > My days as a student is over for the most part. I am learning python on > > my own and Numeric is not properly documented so I am learning by doing > > and copying from others. > > Are you aware of this guide? > > http://numeric.scipy.org/numpydoc/numdoc.htm > > I've seen better documents but it's fairly complete--I wouldn't call it > improper. > > > > I thought about the problem and a solution > > another problem given to me earlier using "putmask" is the solution but > > there is a bug that I cannot figure out: > > > > index = 0 > > for z in range_va: > > wk = msgva # working arrary > > sattmp = sat_id # working array > > This appears to be doing something you don't expect. In Python all > names are references; assignments don't create new objects but rather > new names for the same object. In the above, wk and msvga are the same > array: any changes in wk also appear in msvga. So wk[1,1] = 4 also > causes msgva[1,1] to be 4. > > Numeric arrays even share data when slicing. If you were to take a > slice of wk, for example, a = wk[0:10,0:10], then modifying a would > also show up in wk (and msvga). (However, slicing most other objects > copies rather than shares the data.) > > It's completely different from Matlab, which always copies data. > > Here's what you should do: > > wk = array(msgva) # this creates a new array for wk > sattmp = array(sat_id) > > > mask = where(equal(wk,z),1,0) # creating a mask of > > valid data pixs > > Note that the "where" is unnecessary here. equal creates an array of > ones and zeros. > > > putmask(sattmp,mask==0,-999) # since zero is valid > > data, -999 is used instead > > This would overwrite sat_id unless you copy the array as I've shown > above. > > > rdata = > > compress(ravel(not_equal(sattmp,-999)),ravel(sattmp)) > > I believe you could avoid the above putmask above step and just use > ravel(mask) as the first argument of compress here, or even just > equal(wk,z). > > "equal(wx,z)", "mask", and "not_equal(sattmp,-999)" are all equal > arrays. > > > if sum(sum(rdata)) == 0: > > av = 0 > > rdata is one-dimensional, so you only need one sum call; but why do > this? average will still work if the sum is zero. > > > else: > > av = average(rdata,axis=None) > > axis argument isn't necessary here since rdata is one-dimesional. > > > tmparray[0,index] = av > > tmparray[1,index] = z > > index += 1 > > print tmparray > > *** > > But the tmparray is returning zeros as averages. When I try just one > > value for z everything works. I can't see where I going wrong. I am not > > using the original arrays, only the copies and when a new z is chosen > > then these are recreated. > > Care to help out with this? > > /Sheldon > > Other than the array sharing mistake, it looks like it should work. > > Also, note that this group disdains top-posting; replies should go > below the quoted text because in these sorts of discussions it's best > to keep things in conversational order, especially for the sake of > other interested readers. > > Good luck. > > > Carl Banks Thanks for the tips about the array and how it is copied. I figured this out late yesterday and wrote a dirty solution to this problem but your is more elegant and concise. Yes, where statement is not needed and I removed it. The replacement is the following: average(compress(ravel(equal(wk,z)),ravel(sattmp)),axis=None) This is much more compact and elegant. Thanks for pointing this out. I don't know why average() returned a divide by zero error and to avoid this I inserted this if statement. Now it works much better ! Much obliged, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Numeric help!
Carl Banks wrote: > Sheldon wrote: > > average(compress(ravel(equal(wk,z)),ravel(sattmp)),axis=None) > > This is much more compact and elegant. Thanks for pointing this out. > > I don't know why average() returned a divide by zero error and to avoid > > this I inserted this if statement. Now it works much better ! > > Probably you had a case where the array length was zero, but it > wouldn't happen in the present case unless your input arrays are zero > by zero. > > > Carl Banks Thanks for you help kind Sir! /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Classes and global statements
Hi, I have a series of classes that are all within the same file. Each is called at different times by the main script. Now I have discovered that I need several variables returned to the main script. Simple, right? I thought so and simply returned the variables in a tuple: (a,b,c,d,e) = obj.method() Now I keep getting this error: "ValueError: unpack tuple of wrong size" I think this is because I am trying to return some Numeric arrays as well as list and I didn't declare these prior to calling the class method. The problem is that some of these arrays are set within the class and cannot be set in the calling script. I removed these arrays and tried it again and still I get this error. So I have another idea: I have one class that sets a bunch of varibles like this: myclass: def __init__(self,var1,var2,var3): self.var1 = var1 self.var2 = var2 . . . etc. Then I use the following script to make these variable global: global main main = myclass(var1,var2,var3) I am thinking that I should be able to "insert" other variable into this main from within other classes like this: otherclass: def __init__(self,a,b,c,d): self.a = a. etc. def somemethod(self): self.newvar = main.newvar = self.newvar return self.a This looks wierd but I am wondering if it will work? This would be a wonderful way to store variables that will be needed later instead of passing them back and forth. After trying this it did work! My question is why? Another way to solve this problem is to make the variable I need global in the class that they are created. Does anyone have a better way in mind? Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Classes and global statements
Simon Forman skrev:
> Sheldon wrote:
> > Hi,
> >
> > I have a series of classes that are all within the same file. Each is
> > called at different times by the main script. Now I have discovered
> > that I need several variables returned to the main script. Simple,
> > right? I thought so and simply returned the variables in a tuple:
> > (a,b,c,d,e) = obj.method()
> > Now I keep getting this error: "ValueError: unpack tuple of wrong size"
> >
>
> You're likely getting that error because the method() call is returning
> more or less than 5 values.. Try something like this to check:
>
> results = obj.method()
> print results
> a, b, c, d, e = results
>
> That way you'll be able to see exactly what the method is returning.
>
> >
> > I think this is because I am trying to return some Numeric arrays as
> > well as list and I didn't declare these prior to calling the class
> > method. The problem is that some of these arrays are set within the
> > class and cannot be set in the calling script. I removed these arrays
> > and tried it again and still I get this error. So I have another idea:
> >
> > I have one class that sets a bunch of varibles like this:
> > myclass:
> > def __init__(self,var1,var2,var3):
> > self.var1 = var1
> > self.var2 = var2
> > .
> > .
> > .
> > etc.
> > Then I use the following script to make these variable global:
> >
> > global main
> > main = myclass(var1,var2,var3)
>
> In this case, where the main var is being set at the "module level",
> you needn't use the global statement. If I understand it correctly,
> global is for use inside functions and methods to indicate that a
> variable is being reused from the outer "global" scope, rather than
> being temporarily "shadowed" by a var of the same name local to the
> function or method.
>
> > I am thinking that I should be able to "insert" other variable into
> > this main from within other classes like this:
> >
> > otherclass:
> > def __init__(self,a,b,c,d):
> >self.a = a. etc.
> > def somemethod(self):
> >self.newvar =
> >main.newvar = self.newvar
> >return self.a
> >
> > This looks wierd but I am wondering if it will work? This would be a
> > wonderful way to store variables that will be needed later instead of
> > passing them back and forth.
>
> You *can* use an object as a convenient, if unusual, storage place for
> variables because you can assign to attributes after object
> instantiation
>
> >>> class foo: pass
> ...
> >>> bar = foo()
> >>> bar.a = 23
> >>> bar.b = 'Hi there'
> >>> dir(bar)
> ['__doc__', '__module__', 'a', 'b']
> >>> bar.__dict__
> {'a': 23, 'b': 'Hi there'}
>
> There's nothing really wrong with this, it's like using a dict except
> that you can access variables using the attribute notation obj.var
>
> But keep in mind, if you're already passing around your 'main' object,
> then you already know how to use and pass around any other object.
>
>
> > After trying this it did work! My question is why? Another way to solve
> > this problem is to make the variable I need global in the class that
> > they are created. Does anyone have a better way in mind?
>
> I'm not sure, but I think there's no such thing as "global to a class",
> although you can make class attributes just like for objects. There
> are some complexities to doing this though so you should probably stick
> to instance objects rather than mucking about with class attributes..
>
> HTH,
> ~Simon
Thanks for sound advice!
Will give it another go.
/Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Re: Classes and global statements
Scott David Daniels skrev: > Sheldon wrote: > > Hi, > > > > I have a series of classes that are all within the same file. Each is > > called at different times by the main script. Now I have discovered > > that I need several variables returned to the main script. Simple, > > right? I thought so and simply returned the variables in a tuple: > > (a,b,c,d,e) = obj.method() > > Now I keep getting this error: "ValueError: unpack tuple of wrong size" > What you need to do at this point is read the _whole_ error message > (including the ugly traceback stuff). Think hard about what is going > on. "ValueError: unpack tuple of wrong size" comes from returning > a tuple of a different size than your assignment is producing: > > def triple(v): > return v, v*v > > x, y, z = triple(13) > > Break up the assignment and insert prints: > > # x, y, z = triple(13) > temp = triple(13) > print "I'm going to expand the triple:", temp > x, y, z = temp > > > > I think this is because I am trying to return some Numeric arrays as > > well as list and I didn't declare these prior to calling the class > > method. > Don't guess and hypothesize; create experiments and test. Your > reasoning is off in the ozone. You waste a lot of time creating > great theories; make tiny theories and _test_ them. > > <> > > --Scott David Daniels > [EMAIL PROTECTED] Thanks for sound advice! Will give it another go. /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
concatenate Numeric
Hi, I am trying to build a large array using concatenate function in python. So as I loop over the number of arrays, of which there are 12 (4 down and 3 across), I create 3 long arrays by concatenating them at the bottom and then concatenating them side by side: for ind in range(num_arrays): if ind == 0: bias_down1 = array(bias) ppslon1= array(ppslon) ppslat1= array(ppslat) elif ind > 0 and <= 3: bias_down1 = concatenate((bias_down1,bias),0) ppslon1= concatenate((ppslon1,ppslon),0) ppslat1= concatenate((ppslat1,ppslat),0) elif ind == 4: bias_down2 = array(bias) ppslon2= array(ppslon) ppslat2= array(ppslat) elif ind > 4 and ind <= 7: bias_down2 = concatenate((bias_down2,bias),0) ppslon2= concatenate((ppslon2,ppslon),0) ppslat2= concatenate((ppslat2,ppslat),0) elif ind == 8: bias_down3 = array(bias) ppslon3= array(ppslon) ppslat3= array(ppslat) elif ind > 8: bias_down3 = concatenate((bias_down3,bias),0) ppslon3= concatenate((ppslon3,ppslon),0) ppslat3= concatenate((ppslat3,ppslat),0) bias_a = concatenate((bias_down1,bias_down2),1) # joining the arrays side by side bias_all = concatenate((bias_a,bias_down3),1) ppslat_a = concatenate((ppslat1,ppslat2),1) ppslat_all = concatenate((ppslat_a,ppslat3),1) ppslon_a = concatenate((ppslon1,ppslon2),1) ppslon_all = concatenate((ppslon_a,ppslon3),1) print 'Max lat', max(max(ppslat_all)), '\t','Min lat', min(min(ppslat_all)) print 'Max lon', max(max(ppslon_all)), '\t','Min lon', min(min(ppslon_all)) * Now this works, the array size is correct but the longitude values given for max and min are wrong. What is even stranger to me is that when I write the array in binary format to a file and read it with Matlab, the max and min are correct but when I read it back with python the max and min are again incorrect for only the longitude data. I saved the max and min for the longitude for each array and then check it in the python program and they are correct at the end but the max(max(ppslon)) values is incorrect. Does anyone knows why this is so? If I was doing something wrong then Matlab would not have returned correct values. Any help is appreciated! /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate Numeric
Sheldon skrev: > Hi, > > I am trying to build a large array using concatenate function in > python. > So as I loop over the number of arrays, of which there are 12 (4 down > and 3 across), I create 3 long arrays by concatenating them at the > bottom and then concatenating them side by side: > for ind in range(num_arrays): > if ind == 0: > bias_down1 = array(bias) > ppslon1= array(ppslon) > > ppslat1= array(ppslat) > > elif ind > 0 and <= 3: > bias_down1 = concatenate((bias_down1,bias),0) > > ppslon1= concatenate((ppslon1,ppslon),0) > > ppslat1= concatenate((ppslat1,ppslat),0) > elif ind == 4: > bias_down2 = array(bias) > ppslon2= array(ppslon) > ppslat2= array(ppslat) > elif ind > 4 and ind <= 7: > bias_down2 = concatenate((bias_down2,bias),0) > > ppslon2= concatenate((ppslon2,ppslon),0) > > ppslat2= concatenate((ppslat2,ppslat),0) > > elif ind == 8: > bias_down3 = array(bias) > > ppslon3= array(ppslon) > > ppslat3= array(ppslat) > > elif ind > 8: > bias_down3 = concatenate((bias_down3,bias),0) > ppslon3= concatenate((ppslon3,ppslon),0) > ppslat3= concatenate((ppslat3,ppslat),0) > > bias_a = concatenate((bias_down1,bias_down2),1) # joining the arrays > side by side > bias_all = concatenate((bias_a,bias_down3),1) > > ppslat_a = concatenate((ppslat1,ppslat2),1) > ppslat_all = concatenate((ppslat_a,ppslat3),1) > > ppslon_a = concatenate((ppslon1,ppslon2),1) > ppslon_all = concatenate((ppslon_a,ppslon3),1) > > print 'Max lat', max(max(ppslat_all)), '\t','Min lat', > min(min(ppslat_all)) > print 'Max lon', max(max(ppslon_all)), '\t','Min lon', > min(min(ppslon_all)) > > * > Now this works, the array size is correct but the longitude values > given for max and min are wrong. What is even stranger to me is that > when I write the array in binary format to a file and read it with > Matlab, the max and min are correct but when I read it back with python > the max and min are again incorrect for only the longitude data. I > saved the max and min for the longitude for each array and then check > it in the python program and they are correct at the end but the > max(max(ppslon)) values is incorrect. Does anyone knows why this is > so? > If I was doing something wrong then Matlab would not have returned > correct values. > > Any help is appreciated! > > /Sheldon Sorry, there is a small error when I wrote this part of the program: should say "ind <= 3" /sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate Numeric
Robert Kern skrev: > Sheldon wrote: > > Hi, > > > > I am trying to build a large array using concatenate function in > > python. > > So as I loop over the number of arrays, of which there are 12 (4 down > > and 3 across), I create 3 long arrays by concatenating them at the > > bottom and then concatenating them side by side: > > [snip] > > > print 'Max lat', max(max(ppslat_all)), '\t','Min lat', > > min(min(ppslat_all)) > > print 'Max lon', max(max(ppslon_all)), '\t','Min lon', > > min(min(ppslon_all)) > > > > * > > Now this works, the array size is correct but the longitude values > > given for max and min are wrong. What is even stranger to me is that > > when I write the array in binary format to a file and read it with > > Matlab, the max and min are correct but when I read it back with python > > the max and min are again incorrect for only the longitude data. I > > saved the max and min for the longitude for each array and then check > > it in the python program and they are correct at the end but the > > max(max(ppslon)) values is incorrect. Does anyone knows why this is > > so? > > If I was doing something wrong then Matlab would not have returned > > correct values. > > Don't use min() and max() on multidimensional arrays. They won't give sensible > answers. > > > In [11]: a = RA.random([3,5]) > > In [12]: a > Out[12]: > array([[ 0.01721657, 0.64291363, 0.33210659, 0.89887972, 0.24437849], > [ 0.88205348, 0.00839329, 0.35999039, 0.9966411 , 0.54957126], > [ 0.59983864, 0.18983323, 0.13727718, 0.8987289 , 0.05425076]]) > > In [13]: min(a) > Out[13]: array([ 0.59983864, 0.18983323, 0.13727718, 0.8987289 , > 0.05425076]) > > > The builtin min() and max() compare the values in the sequence. In this case, > those values are the rows of the arrays. Numeric uses rich comparisons, so the > result of a comparison is a boolean array. Numeric also has the convention > that > if any of the elements of an array are considered to be True, then the whole > array is. > > > In [16]: a[1] < a[2] > Out[16]: array([0, 1, 0, 0, 0]) > > In [17]: bool(_) > Out[17]: True > > In [18]: a[2] < a[1] > Out[18]: array([1, 0, 1, 1, 1]) > > In [19]: bool(_) > Out[19]: True > > > This makes min(a) incorrect when len(a.shape) > 1. Instead, use the minimum > and > maximum ufuncs provided with Numeric: > > > In [21]: N.minimum.reduce(a.flat) > Out[21]: 0.0083932917161983426 > > In [22]: N.maximum.reduce(a.flat) > Out[22]: 0.99664110397663608 > > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." >-- Umberto Eco Thanks a million!!! Now I can sleep better. /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate Numeric
Robert Kern wrote: > Sheldon wrote: > > Hi, > > > > I am trying to build a large array using concatenate function in > > python. > > So as I loop over the number of arrays, of which there are 12 (4 down > > and 3 across), I create 3 long arrays by concatenating them at the > > bottom and then concatenating them side by side: > > [snip] > > > print 'Max lat', max(max(ppslat_all)), '\t','Min lat', > > min(min(ppslat_all)) > > print 'Max lon', max(max(ppslon_all)), '\t','Min lon', > > min(min(ppslon_all)) > > > > * > > Now this works, the array size is correct but the longitude values > > given for max and min are wrong. What is even stranger to me is that > > when I write the array in binary format to a file and read it with > > Matlab, the max and min are correct but when I read it back with python > > the max and min are again incorrect for only the longitude data. I > > saved the max and min for the longitude for each array and then check > > it in the python program and they are correct at the end but the > > max(max(ppslon)) values is incorrect. Does anyone knows why this is > > so? > > If I was doing something wrong then Matlab would not have returned > > correct values. > > Don't use min() and max() on multidimensional arrays. They won't give sensible > answers. > > > In [11]: a = RA.random([3,5]) > > In [12]: a > Out[12]: > array([[ 0.01721657, 0.64291363, 0.33210659, 0.89887972, 0.24437849], > [ 0.88205348, 0.00839329, 0.35999039, 0.9966411 , 0.54957126], > [ 0.59983864, 0.18983323, 0.13727718, 0.8987289 , 0.05425076]]) > > In [13]: min(a) > Out[13]: array([ 0.59983864, 0.18983323, 0.13727718, 0.8987289 , > 0.05425076]) > > > The builtin min() and max() compare the values in the sequence. In this case, > those values are the rows of the arrays. Numeric uses rich comparisons, so the > result of a comparison is a boolean array. Numeric also has the convention > that > if any of the elements of an array are considered to be True, then the whole > array is. > > > In [16]: a[1] < a[2] > Out[16]: array([0, 1, 0, 0, 0]) > > In [17]: bool(_) > Out[17]: True > > In [18]: a[2] < a[1] > Out[18]: array([1, 0, 1, 1, 1]) > > In [19]: bool(_) > Out[19]: True > > > This makes min(a) incorrect when len(a.shape) > 1. Instead, use the minimum > and > maximum ufuncs provided with Numeric: > > > In [21]: N.minimum.reduce(a.flat) > Out[21]: 0.0083932917161983426 > > In [22]: N.maximum.reduce(a.flat) > Out[22]: 0.99664110397663608 > > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." >-- Umberto Eco Hi Robert, Thanks again for showing me this. I have been trying to read up on reduce() as I have never used it before. I would like to know what it does. So far my search has found nothing that I can grasp. The reference library notes are estoteric at best. Can you enlighten me on this matter?' /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate Numeric
Robert Kern skrev: > Sheldon wrote: > > Thanks again for showing me this. I have been trying to read up on > > reduce() as I have never used it before. I would like to know what it > > does. So far my search has found nothing that I can grasp. The > > reference library notes are estoteric at best. > > Can you enlighten me on this matter?' > > The .reduce() method on ufuncs works pretty much like the reduce() builtin > function. It applies the binary ufunc along the given axis of the array (the > first one by default) cumulatively. > >a = [3, 2, 1, 0] >minimum.reduce(a) == minimum(minimum(minimum(a[0], a[1]), a[2]), a[3]) > > I will note, in the form of enticement to get you to try the currently active > array package instead of Numeric, that in numpy, arrays have methods to do > minimums and maximum rather more conveniently. > > >>> import numpy as N > >>> a = N.rand(3, 5) > >>> a > array([[ 0.49892358, 0.11931907, 0.37146848, 0.07494308, 0.91973863], > [ 0.92049698, 0.35016683, 0.01711571, 0.59542456, 0.49897077], > [ 0.57449315, 0.99592033, 0.20549262, 0.25135288, 0.04111402]]) > >>> a.min() > 0.017115711878847639 > >>> a.min(axis=0) > array([ 0.49892358, 0.11931907, 0.01711571, 0.07494308, 0.04111402]) > >>> a.min(axis=1) > array([ 0.07494308, 0.01711571, 0.04111402]) > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." >-- Umberto Eco Thanks for the explanation! Super. I am trying to get my bosses to purchase the Numpy documentation and upgrade to Numpy as well as matplotlib and other necessary scientific modules. But it is not entirely up to me. Still I need to learn more about Python and installing these modules myself. -- http://mail.python.org/mailman/listinfo/python-list
Concatenating arrays
Good day everyone, I would like to know if anyone has a fast and concise way of concatenating two 2D arrays of same dimensions? Whenever I try: a = concatenate(b,c) I get erroneous data as if the axises were incorrectly chosen. As far as I can see concatenate((b,c),0) does it vertically while a 1 does it horizontally. But the results does not reflect this. What am I missing here? I am using Numeric. Any help is appreciated, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Continuing after error
Hi, Does anyone know if it is possible to resume the execution of a program after it stops at an error in the code and the error was corrected? Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Resuming a program's execution after correcting error
Hi. Does anyone know if one can resume a python script at the error point after the error is corrected? I have a large program that take forever if I have to restart from scratch everytime. The error was the data writing a file so it seemed such a waste if all the data was lost and must be recalculated again. Sincerely, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Resuming a program's execution after correcting error
MonkeeSage wrote: > Georg Brandl wrote: > > As I said before, this can be done by finding out where the error is raised, > > what the cause is and by inserting an appropriate try-except-statement in > > the code. > > I could be mistaken, but I *think* the OP is asking how to re-enter the > stack at the same point as the exception exited from and continue with > the execution as if the exception never happened. AFAIK, that isn't > possible; however, given that he has a file to work from that indicates > a portion of the state at the time of the exception, I think he may be > able simulate that kind of functionality by reading in the file on > exception and then returning a call to the function where the exception > occured with the data from the file. Something like this mockup: > > def faulty_function(a, b, c=None): > if not c: > c = 0 > try: > # modify c, write c to file... > # oops hit an exception > c += a / b > except: > # read from the file here > # c = ... > # and fix the error > b += 1 > return faulty_function(a, b, c) > return c > > print faulty_function(2, 0) # => 2 > > Of course, it's probably much better to just fix the code and avoid the > exception in the first place. ;) > > Regards, > Jordan Thanks Jordon, I think you understood my problem best. I know now that this is not possible but I would like to create an exception that saves all the current variables when there is an error. I think pickle is the answer but I never got it to work. My program is very large and it is being modified often. Any advice on how to save the variables. /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Resuming a program's execution after correcting error
MRAB wrote: > Sheldon wrote: > > Hi. > > > > Does anyone know if one can resume a python script at the error point > > after the error is corrected? > > I have a large program that take forever if I have to restart from > > scratch everytime. The error was the data writing a file so it seemed > > such a waste if all the data was lost and must be recalculated again. > > > You could modify the program while you're debugging it so that instead > of, say: > > calculate data > write data > > you have: > > if saved data exists: > load data > else: > calculate data > save data > write data > > The pickle module would be useful here. > > Matthew I like your idea Matthew but I don't know how to pickle the many variables in one file. Do I need to pickle each and every variable into a seperate file? var1,var2 pickle.dump(var1,f) pickle.dump(var2,f2) /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: Resuming a program's execution after correcting error
MRAB wrote: > Sheldon wrote: > > MRAB wrote: > > > Sheldon wrote: > > > > Hi. > > > > > > > > Does anyone know if one can resume a python script at the error point > > > > after the error is corrected? > > > > I have a large program that take forever if I have to restart from > > > > scratch everytime. The error was the data writing a file so it seemed > > > > such a waste if all the data was lost and must be recalculated again. > > > > > > > You could modify the program while you're debugging it so that instead > > > of, say: > > > > > > calculate data > > > write data > > > > > > you have: > > > > > > if saved data exists: > > > load data > > > else: > > > calculate data > > > save data > > > write data > > > > > > The pickle module would be useful here. > > > > > > Matthew > > > > I like your idea Matthew but I don't know how to pickle the many > > variables in one file. Do I need to pickle each and every variable into > > a seperate file? > > var1,var2 > > pickle.dump(var1,f) > > pickle.dump(var2,f2) > > > Using the 'pickle' module: > > # To store: > f = open(file_path, "wb") > pickle.dump(var1, f) > pickle.dump(var2, f) > f.close() > > # To load > f = open(file_path, "rb") > var1 = pickle.load(f) > var2 = pickle.load(f) > f.close() > > A more flexible alternative is to use the 'shelve' module. This behaves > like a dict: > > # To store > s = shelve.open(file_path) > s["var1"] = "first" > s["var2"] = [2, 3] > s.close() > > # To load > s = shelve.open(file_path) > print s["var1"] # This prints "first" > print s["var2"] # This prints [2, 3] > s.close() > > Hope that helps > Matthew Perfect Matthew! Much obliged! /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: C extensions and memory leakage
Fredrik Lundh wrote: > Sheldon wrote: > > > I am very new at this C extensions in Python so my term wrapper was > > probably a misnomer. Perhaps glue is better or the interface that > > allows the exchange of data between Python and C. > > Yes, I am using python objects in my C extension. > > Tell me where I can find out more about this reference counters? Or > > perhaps you can tell something about it. > > http://docs.python.org/ext/refcounts.html > > Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: C extensions and memory leakage
Fredrik Lundh wrote:
> Sheldon wrote:
>
> > I am very new at this C extensions in Python so my term wrapper was
> > probably a misnomer. Perhaps glue is better or the interface that
> > allows the exchange of data between Python and C.
> > Yes, I am using python objects in my C extension.
> > Tell me where I can find out more about this reference counters? Or
> > perhaps you can tell something about it.
>
> http://docs.python.org/ext/refcounts.html
>
>
Ok, I understand that I need to allocate and deallocate memory for my
PyObject:
static PyObject* pack_datastruct_to_pyobject(int Row, int Col, int Cat)
{
PyObject *outobj_lat=NULL;
PyObject *outobj_lon=NULL;
PyObject *outobj_msgzenithangle=NULL;
PyObject *outobj_valconcen=NULL;
PyObject *outobj_lightcon=NULL;
PyObject *outobj_fracofland=NULL;
PyObject *outobj_bias100=NULL;
PyObject *outobj_bias75=NULL;
PyObject *outobj_bias50=NULL;
PyObject *outobj_bias25=NULL;
PyObject *outobj_stats=NULL;
PyObject *outobj_percentage=NULL;
outobj_lat=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.lat15km,Row/res,Col/res,outobj_lat))
goto fail;
outobj_lon=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.lon15km,Row/res,Col/res,outobj_lon))
goto fail;
outobj_msgzenithangle=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.msgzenithangle15km,Row/res,Col/res,outobj_msgzenithangle))
goto fail;
outobj_valconcen=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.valconcen15km,Row/res,Col/res,outobj_valconcen))
goto fail;
outobj_lightcon=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.lightcon15km,Row/res,Col/res,outobj_lightcon))
goto fail;
outobj_fracofland=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.fracofland15km,Row/res,Col/res,outobj_fracofland))
goto fail;
outobj_bias100=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.bias10015km,Row/res,Col/res,outobj_bias100))
goto fail;
outobj_bias75=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.bias7515km,Row/res,Col/res,outobj_bias75))
goto fail;
outobj_bias50=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.bias5015km,Row/res,Col/res,outobj_bias50))
goto fail;
outobj_bias25=PyTuple_New(Row/res*Col/res);
if
(!createPythonObject_f_2D(work.bias2515km,Row/res,Col/res,outobj_bias25))
goto fail;
outobj_stats=PyTuple_New(Cat);
if (!createPythonObject_f_1D(work.stats,Cat,outobj_stats))
goto fail;
outobj_percentage=PyTuple_New(Cat);
if (!createPythonObject_f_1D(work.percentage,Cat,outobj_percentage))
goto fail;
return Py_BuildValue("()",
outobj_lat,outobj_lon,outobj_msgzenithangle,outobj_valconcen,
outobj_lightcon,outobj_fracofland,outobj_bias100,outobj_bias75,
outobj_bias50,outobj_bias25,outobj_stats,outobj_percentage);
fail:
return NULL;
}
But just how it is done still eludes me. Can you give some help here?
/Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
function for allocating memory for string array
Hi,
Can someone tell me how to remove the conflicts here ?
#include
static char*** Memory2DStr(int R, int C);
static void MemoryFreeStr(int C, char*** array);
void main() {
unsigned char ***arr_scenes=NULL;
int R = 15;
int C = 15;
arr_scenes = Memory2DStr(nscenes,2);
MemoryFreeStr(2,arr_scenes);
}
static
char*** Memory2DStr(int R, int C) {
char*** array;
int i;
array = calloc(C,sizeof(char **));
if(array == NULL) {
fprintf(stderr, "Failed to allocate memory\n");
exit(EXIT_FAILURE);
}
for (i = 0; i < C; i++) {
array[i] = calloc(R,sizeof(char*));
if(array == NULL) { // Memory for the Col
fprintf(stderr, "Failed to allocate memory\n");
exit(EXIT_FAILURE);
}
}
return array;
}
static
void MemoryFreeStr(int Col, char*** array) {
int i;
for (i = 0; i < 15; i++) {
free((*array)[i]);
}
free(*array);
}
Thanks,
Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Core dump revisited
Hi, I have a python script that uses a C extention. I keep getting a recurring problem that causes a core dump a few lines after the C extention return data back tp python. I tried using pbd and gdb but I have not succeeded in understanding what went wrong and where. I post the python script here is the error message and gdb output after reading the core file: . printout from with C extention Completed freeing 2D arrays. Now freeing 1D arrays freed 1D arrays freeing 15km arrays Complete Returning data back to Python! In python: assigning tuple data to numeric arrays! Time to do the coastal effects Segmentation fault (core dumped) .. Now there next step after "Time to do the coastal effects" was never executed. And even if I put another print statement there the same thing happens. I am using python 2.3 and I have set my stacksize to unlimited. When reading the core dump file with gdb I get: gdb msgppscomp.py core.3203 GNU gdb 6.0-2mdk (Mandrake Linux) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i586-mandrake-linux-gnu"..."/data/proj_ns1/safworks/sheldon/msgppscomp.py": not in executable format: File format not recognized Core was generated by `python msgppscomp.py'. Program terminated with signal 11, Segmentation fault. #0 0x40079dfa in ?? () (gdb) . I am not very familiar with using gdb with python and C extentions. I would like to get some ideas about where the problem might be. I have made sure that all the arrays in the C extention are freed before exiting so I doubt that this might be the problem. Here is the python script: #!/usr/bin/env python #!-*-coding: UTF-8 -*- class WriteHdfFile: def __init__(self,outfile): self.outfile = outfile self.COMPRESS_LVL = 6 def writeAreainfo(self): import _pyhl import sys sys.float_output_precision = 2 aList = _pyhl.nodelist() aNode = _pyhl.node(_pyhl.GROUP_ID,"/PPS_MSG_COMP") aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/LAT") aNode.setArrayValue(1,lat.shape,lat,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/LON") aNode.setArrayValue(1,lon.shape,lon,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/VALCONCEN") aNode.setArrayValue(1,valconcen.shape,valconcen,"int",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/LIGHTCON") aNode.setArrayValue(1,lightcon.shape,lightcon,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/BIAS100") aNode.setArrayValue(1,bias100.shape,bias100,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/BIAS75") aNode.setArrayValue(1,bias75.shape,bias75,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/BIAS50") aNode.setArrayValue(1,bias50.shape,bias50,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/BIAS25") aNode.setArrayValue(1,bias25.shape,bias25,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/COAST") aNode.setArrayValue(1,coast.shape,coast,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/STATISTICS") aNode.setArrayValue(1,stats.shape,stats,"int",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/PERCENTAGE") aNode.setArrayValue(1,percentage.shape,percentage,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/VA_vs_BIAS") aNode.setArrayValue(1,va_area.shape,va_area,"float",-1) aList.addNode(aNode) aNode=_pyhl.node(_pyhl.DATASET_ID,"/VANUM") aNode.setArrayValue(1,vanum.shape,vanum,"float",-1) aList.addNode(aNode) aNode = _pyhl.node(_pyhl.ATTRIBUTE_ID,"/NSCENES") aNode.setScalarValue(-1,areascenes,"int",-1) aList.addNode(aNode) aNode = _pyhl.node(_pyhl.ATTRIBUTE_ID,"/SATELLITE") aNode.setScalarValue(-1,satid,"string",-1) aList.addNode(aNode) self.status = aList.write(self.outfile,self.COMPRESS_LVL) return self.status #
Re: Core dump revisited
Nick Craig-Wood skrev: > Sheldon <[EMAIL PROTECTED]> wrote: > > gdb msgppscomp.py core.3203 > > Run > > gdb python > > Then type > > run msgppscomp.py > > at the gdb prompt. > > When it crashes, type "bt" for a back trace. This will pinpoint > exactly where it crashes. > > Hopefully that will make things clearer. If not post the output. > > -- > Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick Wonderful! Now I know how to used gdb with python. The are results area posted below. Since I am new at this I could used some help in interpreting the problem. What I do know is this: my allocation of the array is good but when freeing the array, a problem arises. The problem is given below but since I am new as well to C I sure could use some help. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1077321856 (LWP 32710)] 0x40297b83 in mallopt () from /lib/tls/libc.so.6 (gdb) bt #0 0x40297b83 in mallopt () from /lib/tls/libc.so.6 #1 0x402958ba in free () from /lib/tls/libc.so.6 #2 0x405e070a in MemoryFreeOrd () at msgppsmodule_area.c:1675 #3 0x405dae0a in msgppsarea (self=0x0, args=0x4042aa7c, kwargs=0x0) at msgppsmodule_area.c:328 #4 0x40073b16 in PyCFunction_Call () from /usr/lib/libpython2.3.so.1.0 I appreciate your help so far and could use some more. /S -- http://mail.python.org/mailman/listinfo/python-list
Re: Core dump revisited
Nick Craig-Wood skrev: > Sheldon <[EMAIL PROTECTED]> wrote: > > gdb msgppscomp.py core.3203 > > Run > > gdb python > > Then type > > run msgppscomp.py > > at the gdb prompt. > > When it crashes, type "bt" for a back trace. This will pinpoint > exactly where it crashes. > > Hopefully that will make things clearer. If not post the output. > > -- > Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick Wonderful! Now I know how to used gdb with python. The are results area posted below. Since I am new at this I could used some help in interpreting the problem. What I do know is this: my allocation of the array is good but when freeing the array, a problem arises. The problem is given below but since I am new as well to C I sure could use some help. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1077321856 (LWP 32710)] 0x40297b83 in mallopt () from /lib/tls/libc.so.6 (gdb) bt #0 0x40297b83 in mallopt () from /lib/tls/libc.so.6 #1 0x402958ba in free () from /lib/tls/libc.so.6 #2 0x405e070a in MemoryFreeOrd () at msgppsmodule_area.c:1675 #3 0x405dae0a in msgppsarea (self=0x0, args=0x4042aa7c, kwargs=0x0) at msgppsmodule_area.c:328 #4 0x40073b16 in PyCFunction_Call () from /usr/lib/libpython2.3.so.1.0 I appreciate your help so far and could use some more. /S -- http://mail.python.org/mailman/listinfo/python-list
Re: Core dump revisited
Nick Craig-Wood skrev:
> Sheldon <[EMAIL PROTECTED]> wrote:
> > Sheldon skrev:
> > > Wonderful! Now I know how to used gdb with python.
>
> Good!
>
> > > The are results area posted below. Since I am new at this I could
> > > used some help in interpreting the problem. What I do know is
> > > this: my allocation of the array is good but when freeing the
> > > array, a problem arises. The problem is given below but since I am
> > > new as well to C
>
> Ambitious!
>
> > > I sure could use some help.
> > >
> > > Program received signal SIGSEGV, Segmentation fault.
> > > [Switching to Thread 1077321856 (LWP 32710)]
> > > 0x40297b83 in mallopt () from /lib/tls/libc.so.6
> > > (gdb) bt
> > > #0 0x40297b83 in mallopt () from /lib/tls/libc.so.6
> > > #1 0x402958ba in free () from /lib/tls/libc.so.6
> > > #2 0x405e070a in MemoryFreeOrd () at msgppsmodule_area.c:1675
> > > #3 0x405dae0a in msgppsarea (self=0x0, args=0x4042aa7c, kwargs=0x0) at
> > > msgppsmodule_area.c:328
> > > #4 0x40073b16 in PyCFunction_Call () from /usr/lib/libpython2.3.so.1.0
>
> Typing "up" and "down" to move up and down the call stack is a good
> thing. Type "l" to see a list of code at the point that things went
> wrong.
>
> You can type "print " to see the values of variables.
> gdb understands most C syntax so you can use "print
> this->member[1].data" etc.
>
> This all assumes you compiled and linked with debugging (-g flag to
> compiler and linker). Turning off optimisation may make the debugger
> more accurate also.
>
> I can't tell exactly where your program crashed because of line
> wrapping it your post, but is it certainly within MemoryFreeOrd().
>
> What I would do next is run the program under gdb, wait for it to
> crash then print the values of things in the MemoryFreeOrd() function.
> You'll find one of the pointers has been corrupted, ie doesn't point
> to valid memory or memory allocated with malloc() I expect. 0 or NULL
> is an acceptable input to most free() implementations but not all.
>
> Finding out exactly where that pointer got corrupted is harder. Maybe
> it was never initialised, or maybe you initialised it with a PyObject
> or maybe you have a memory scribble somewhere.
>
> A memory scribble is the hardest bug to find because it happened
> elsewhere in your program. Look at the data that was overwritten.
> Maybe it is a string you can identify... You'll also want to start
> reading the gdb manual on breakpoints and watchpoints at this moment!
>
> Find memory corruptions can be tricky and time consuming.
>
> Valgrind can help also.
>
> Good luck!
> --
> Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
Man. You are good. This is most insight I have had from anyone. I did
initialize the arrays with PyObjects and today, after hours of
debugging and now with your insight, I think the problem lies here:
/* here a python list of strings is read into a C string array */
static int readPythonObject(void) {
int i;
PyObject *msgop;
PyObject *ppsop;
PyObject *tileop;
PyObject *sceneop;
for (i = 0; i < work.sumscenes; i++) {
msgop = PyList_GetItem(work.msgobj, i);
work.msg_scenes[i] = PyString_AsString(msgop);
ppsop = PyList_GetItem(work.ppsobj, i);
work.pps_scenes[i] = PyString_AsString(ppsop);
}
for (i = 0; i < NumberOfTiles; i++) {
tileop = PyList_GetItem(work.tileobj, i);
work.tiles[i] = PyString_AsString(tileop);
sceneop = PyList_GetItem(work.nscenesobj, i);
work.nscenes[i] = PyInt_AsLong(sceneop);
}
return 1;
} /*end readPythonObject*/
I am new to this and copied this code from a colleague. So, it corrupts
the pointer. How do I do this properly?
Your help is truly appreciated!
/S
--
http://mail.python.org/mailman/listinfo/python-list
Cpoying a PyList to a C string array
Hi,
The code below is a rookie attempt to copy a python list of strings to
a string array in C. It works to some extent but results in memory
problems when trying to free the C string array. Does anyone know how
to do this properly?
***
/*Read an list of strings from a python object*/
static int readPythonObject(void) {
int i;
PyObject *msgop;
PyObject *ppsop;
PyObject *tileop;
PyObject *sceneop;
for (i = 0; i < work.sumscenes; i++) {
msgop = PyList_GetItem(work.msgobj, i);
work.msg_scenes[i] = PyString_AsString(msgop);
ppsop = PyList_GetItem(work.ppsobj, i);
work.pps_scenes[i] = PyString_AsString(ppsop);
}
for (i = 0; i < NumberOfTiles; i++) {
tileop = PyList_GetItem(work.tileobj, i);
work.tiles[i] = PyString_AsString(tileop);
sceneop = PyList_GetItem(work.nscenesobj, i);
work.nscenes[i] = PyInt_AsLong(sceneop);
}
return 1;
} /*end readPythonObject*/
***
/S
--
http://mail.python.org/mailman/listinfo/python-list
Re: Core dump revisited
Duncan Booth skrev: > "Sheldon" <[EMAIL PROTECTED]> wrote: > > > I am new to this and copied this code from a colleague. So, it > > corrupts the pointer. How do I do this properly? > > > Here is at least part of your problem: > > msgop = PyList_GetItem(work.msgobj, i); > work.msg_scenes[i] = PyString_AsString(msgop); > ppsop = PyList_GetItem(work.ppsobj, i); > work.pps_scenes[i] = PyString_AsString(ppsop); > ... > free(work.pps_scenes[i]); > free(work.msg_scenes[i]); > > You initialised msg_scenes and pps_scenes with a malloc'ed block but you > then just overwrote the pointer with the result of PyString_AsString. You > don't own the memory for the string returned from PyString_AsString, so > freeing it will cause a corruption. You should copy the string data into > the malloc'ed block (with appropriate length checks). Do you mean with: PyString_FromStringAndSize() and PyString_Size(PyObject *string) /S -- http://mail.python.org/mailman/listinfo/python-list
Re: Core dump revisited
Nick Craig-Wood skrev:
> Sheldon <[EMAIL PROTECTED]> wrote:
> > Man. You are good. This is most insight I have had from anyone.
>
> :-)
>
> > I did initialize the arrays with PyObjects and today, after hours of
> > debugging and now with your insight, I think the problem lies here:
>
> Good!
>
> You need to release some python references otherwise you'll have a
> memory leak and copy some strings you don't own
>
> It is worth reading the definitions for those functions in the Python
> API docs
>
> http://docs.python.org/api/api.html
>
> In particular read about reference counts.
>
> With the Python C API you have to know at all time (by reading the
> doc) whether you own the reference or not, and whether you own the
> memory or not.
>
> > /* here a python list of strings is read into a C string array */
> > static int readPythonObject(void) {
> >
> >int i;
> >PyObject *msgop;
> >PyObject *ppsop;
> >PyObject *tileop;
> >PyObject *sceneop;
> >
> >for (i = 0; i < work.sumscenes; i++) {
> > msgop = PyList_GetItem(work.msgobj, i);
> > work.msg_scenes[i] = PyString_AsString(msgop);
>
>work.msg_scenes[i] = strdup(PyString_AsString(msgop));
>Py_DECREF(msgop);
>
> > ppsop = PyList_GetItem(work.ppsobj, i);
> > work.pps_scenes[i] = PyString_AsString(ppsop);
>
> work.pps_scenes[i] = strdup(PyString_AsString(ppsop));
> Py_DECREF(ppsop);
>
> >}
> >for (i = 0; i < NumberOfTiles; i++) {
> > tileop = PyList_GetItem(work.tileobj, i);
> > work.tiles[i] = PyString_AsString(tileop);
>
>work.tiles[i] = strdup(PyString_AsString(tileop));
>Py_DECREF(tileop);
>
> > sceneop = PyList_GetItem(work.nscenesobj, i);
> > work.nscenes[i] = PyInt_AsLong(sceneop);
>
> Py_DECREF(sceneop);
>
> >}
> >return 1;
> > } /*end readPythonObject*/
>
> You free() the strings later which is fine.
>
> The above ignores errors which PyList_GetItem may return and strdup()
> returning 0, but it should get you out of trouble hopefully.
>
> ...
>
> I've written lots of quite similar code, here is a snippet. Note the
> comments about who owns the reference, and the error checking. Also
> note xstrdup() which is a strdup() which blows up if no memory is
> available.
>
> [snip]
> PyObject *item = PySequence_GetItem(value, i); /* real ref */
> if (item == 0)
> {
> fprintf(stderr, "Failed to read '%s[%d]' attribute\n", name, i);
> goto err;
> }
> item_cstr = PyString_AsString(item); /* borrowed */
> if (item_cstr == 0)
> {
> fprintf(stderr, "Failed to read '%s[%d]' as string\n", name, i);
> goto err;
> }
> label[i] = xstrdup(item_cstr);
> Py_DECREF(item);
> item = 0;
> [snip]
> err:;
> PyErr_Print();
> out:;
> if (value)
> Py_DECREF(value);
> if (item)
> Py_DECREF(item);
> return rc;
>
>
> --
> Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
Thanks Nick! Man, I really appreciate this. I did dereference before
but the program crashed and I did understand why so I kept on learning.
Now I have your snip. Much obliged.
I will be adding Numpy to my new PC in Jan and there I can pass arrays
instead of list. Still, I would like to save your email address i case
I have some mre questions later - if that ok with you?
I will make the changes and try to run the code.
/S
--
http://mail.python.org/mailman/listinfo/python-list
Re: Cpoying a PyList to a C string array
Klaas skrev:
> Sheldon wrote:
> > The code below is a rookie attempt to copy a python list of strings to
> > a string array in C. It works to some extent but results in memory
> > problems when trying to free the C string array. Does anyone know how
> > to do this properly?
>
> You have numerous problems in this code. The most important problem is
> that you are referring to global variables which appear to be c structs
> but you don't provide the definition (e.g., "work"). However, I can
> guess some of the issues:
>
> > for (i = 0; i < work.sumscenes; i++) {
> > msgop = PyList_GetItem(work.msgobj, i);
> > work.msg_scenes[i] = PyString_AsString(msgop);
> > ppsop = PyList_GetItem(work.ppsobj, i);
> > work.pps_scenes[i] = PyString_AsString(ppsop);
> > }
>
> PyString_AsString returns a pointer to the internal buffer of the
> python string. If you want to be able to free() it (or indeed have it
> exist for beyond the lifetime of the associated python string), you
> need to malloc() memory and strcpy() the data. If the strings contain
> binary data, you should be using PyString_AsStringAndSize. see
> http://docs.python.org/api/stringObjects.html.
>
> I notice that you are doing no error checking or ref counting, but my
> (inexperienced python c programming) opinion is that it should work
> (neither api could potentially call python code, so I don't think
> threading is an issue).
>
> > for (i = 0; i < NumberOfTiles; i++) {
> > tileop = PyList_GetItem(work.tileobj, i);
> > work.tiles[i] = PyString_AsString(tileop);
> > sceneop = PyList_GetItem(work.nscenesobj, i);
> > work.nscenes[i] = PyInt_AsLong(sceneop);
> > }
> > return 1;
>
> Similarly.
>
> -Mike
Thanks Mike,
I am rewriting the code but I don't understand the part about the c
struct variable called work. The function I posted is a part of a
larger script and I just posted that part that was problamatic. I was
under the impression that if I declared the structure as global with
the variable in tow:
struct my_struct {
int var;
} work;
then this is visible everywhere in the function as long as everything
is in one file. Did I miss something?
/S
--
http://mail.python.org/mailman/listinfo/python-list
Py_BuildValue or PyList_SetItem()
Hi,
I have a function that creates python objects out of C arrays and
returns them to Python. Below is a working example that I later want to
expand to return 12 arrays back to Python. The problem is that when I
print out the values in Python I get undesired reults. See below. Does
anyone know what is going on here?
The array values are good before the conversion.
**
int createPythonObject(void) {
int i,j,k;
PyObject *Rva=PyList_New(12);
for (i = 0; i < 12; i++) {
PyObject *op = PyFloat_FromDouble((double)va[i]);
if (PyList_SetItem(Rva,i,op) !=0) {
fprintf(stderr,"Error in creating python va object\n");
exit(EXIT_FAILURE);
}
Py_DECREF(op);
op = 0;
return Py_BuildValue("N",Rva);
}
Results in Python:
, , , , , , , ,
, , ,
Any help is appreciated!
/Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Re: Py_BuildValue or PyList_SetItem()
Jack Diederich skrev:
> On Wed, Jan 03, 2007 at 01:16:38PM -0800, Sheldon wrote:
> > I have a function that creates python objects out of C arrays and
> > returns them to Python. Below is a working example that I later want to
> > expand to return 12 arrays back to Python. The problem is that when I
> > print out the values in Python I get undesired reults. See below. Does
> > anyone know what is going on here?
> > The array values are good before the conversion.
> >
> > **
> > int createPythonObject(void) {
> > int i,j,k;
> > PyObject *Rva=PyList_New(12);
> >
> > for (i = 0; i < 12; i++) {
> > PyObject *op = PyFloat_FromDouble((double)va[i]);
> > if (PyList_SetItem(Rva,i,op) !=0) {
> > fprintf(stderr,"Error in creating python va object\n");
> > exit(EXIT_FAILURE);
> > }
> > Py_DECREF(op);
> > op = 0;
> > return Py_BuildValue("N",Rva);
> > }
> >
> > Results in Python:
> >
> > , , > 0x80d885c>, , , > at 0x80d885c>, , ,
> > , , > 0x80d885c>,
> >
>
> PyList_SetItem steals a reference to "op" so DECREF'ing it reduces
> the refcount to zero.
>
> http://docs.python.org/api/listObjects.html
>
> -Jack
Thanks Jack,
It works now!
/Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
PyType_IsSubtype ()
Hi, Can anyone tell me what this error mean: #0 0x4008376e in PyType_IsSubtype () from /usr/lib/libpython2.3.so.1.0 The core file backtrace can be found below. I am using Numeric and Python2.3 on a Linux OS. Any help is appreciated! * Core was generated by `/usr/bin/python msgppscomp.py'. Program terminated with signal 11, Segmentation fault. warning: current_sos: Can't read pathname for load map: Input/output error Symbols already loaded for /usr/lib/libpython2.3.so.1.0 Symbols already loaded for /lib/tls/libpthread.so.0 Symbols already loaded for /lib/libdl.so.2 Symbols already loaded for /lib/libutil.so.1 Symbols already loaded for /usr/lib/libstdc++.so.5 Symbols already loaded for /lib/tls/libm.so.6 Symbols already loaded for /lib/libgcc_s.so.1 Symbols already loaded for /lib/tls/libc.so.6 Symbols already loaded for /lib/ld-linux.so.2 Symbols already loaded for /usr/lib/python2.3/site-packages/Numeric/multiarray.so Symbols already loaded for /usr/lib/python2.3/site-packages/Numeric/_numpy.so Symbols already loaded for /usr/lib/python2.3/site-packages/Numeric/umath.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/strop.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/math.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/struct.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/binascii.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/cStringIO.so Symbols already loaded for /usr/lib/python2.3/site-packages/Numeric/ranlib.so Symbols already loaded for /usr/lib/python2.3/site-packages/Numeric/lapack_lite.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/time.so Symbols already loaded for /data/proj_ns1/safworks/sheldon/MSG_PPS_COMP/c_codes/msgppsarea.so Symbols already loaded for /usr/lib/libhdf5.so.0 Symbols already loaded for /lib/libz.so.1 Symbols already loaded for /usr/lib/python2.3/lib-dynload/cPickle.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/_bsddb.so Symbols already loaded for /usr/lib/libdb-4.1.so Symbols already loaded for /usr/lib/python2.3/lib-dynload/gdbm.so Symbols already loaded for /usr/lib/libgdbm.so.2 Symbols already loaded for /usr/lib/python2.3/lib-dynload/dbm.so #0 0x4008376e in PyType_IsSubtype () from /usr/lib/libpython2.3.so.1.0 ****** /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: PyType_IsSubtype ()
Martin v. Löwis skrev: > Sheldon schrieb: > > Can anyone tell me what this error mean: > > #0 0x4008376e in PyType_IsSubtype () from /usr/lib/libpython2.3.so.1.0 > > It's not an error. It is just a frame from the backtrace. > > To understand the crash better, one would need to see more frames from > the backtrace. Ideally, one would run the program in a debugger to > analyse it further. > > Regards, > Martin I ran the program in GDB and it crashed giving the following backtrace: #0 0x4008376e in PyType_IsSubtype () from /usr/lib/libpython2.3.so.1.0 #1 0x4047730c in ?? () from /usr/lib/python2.3/site-packages/Numeric/_numpy.so #2 0x00f2 in ?? () #3 0x08d9dd34 in ?? () #4 0x4046c780 in PyArray_Return () from /usr/lib/python2.3/site-packages/Numeric/_numpy.so #5 0x08d9dd44 in ?? () #6 0x40100300 in PyBaseString_Type () from /usr/lib/libpython2.3.so.1.0 #7 0x08105718 in ?? () #8 0x4007490d in PyObject_Init () from /usr/lib/libpython2.3.so.1.0 #9 0x08105ae0 in ?? () Can you make head or tail of this? /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
writing arrays to binary file
Hi everyone,
I have a short program the writes the output data to an acsii file:
import sys
import os
import string
import gisdata
from Numeric import *
def latlon(area_id, i, j):
lon_file = "lon_%s.dat" % area_id
flon= open(lon_file, 'wa')
lat_file = "lat_%s.dat" % area_id
flat = open(lat_file, 'wa')
for x in range(i):
for y in range(j):
flat.write("%f\n" % gisdata.xy2lonlat(area_id,x,y)[1])
flon.write("%f\n" % gisdata.xy2lonlat(area_id,x,y)[0])
flat.close()
flon.close()
#
if __name__ == '__main__':
tile_info ={"ibla_35n13e":[1215,1215],
"ibla_46n16e":[1215,1215],
"ibla_57n40w":[1215,1215],
}
for t in tile_info.keys():
xsize = tile_info[t][0]
ysize = tile_info[t][1]
result = latlon_(t, xsize, ysize)
Now this works but the output is in ascii form. Whenever I try to write
it binary form by creating the 2D array and then opening the file with
open("filename", 'wb'), I lose the decimals that are vital. The input
data is float with about 4 decimal places.
Does anyone have any ideas of how to improve this by writing the 2D
array in binary form?
Thanks,
Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Re: writing arrays to binary file
Thanks Travis, I don't have the new NumPy yet but this tofile() method should work fine. /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
GRIB to images
Hi, I am interesseted on reading some GRIB files using python and then converting the fields to gif images. Does anyone know if this is possible or have a program that does this? /Sheldon -- http://mail.python.org/mailman/listinfo/python-list
ftputil.py
Hi Everyone,
I recently installed a module called ftputil and I am trying to
understand how it works.
I created a simple program to download about 1500 files from my ftp and
the program looks like this:
**
#! /usr/bin/env python
#-*- coding: UTF-8 -*-
def ftp_fetch(newdir):
import sys
import ftputil
# download some files from the login directory
host = ftputil.FTPHost('ftp..xx', 'xx', 'xxx')
names = host.listdir(host.curdir)
print "There are %d files on the FTP server to be downloaded" %
len(names)
for name in names:
if host.path.isfile(name):
host.download(name, name, 'b') # remote, local, binary
mode
#make a new directory and copy a remote file into it
host.mkdir('newdir')
source = host.file('index.html', 'r') # file-like object
target = host.file('newdir/index.html', 'w') # file-like object
host.copyfileobj(source, target) # similar to shutil.copyfileobj
source.close()
target.close()
return 0
#
if __name__== "__main__":
import os
import sys
newdir = os.getcwd()
print "Downloading files from FTP into directory: %s" % newdir
#sys.exit(0)
stat = ftp_fetch(newdir)
**
Now I also need to write another function to upload files and I
haven't the faintest idea yet.
Here is the part of the program that I need explained:
host.download(name, name, 'b') # remote, local, binary mode
source = host.file('index.html', 'r') # file-like object
target = host.file('newdir/index.html', 'w') # file-like object
host.copyfileobj(source, target) # similar to shutil.copyfileobj
Any help would be greatly appreciated!
Sincerely,
Sheldon
--
http://mail.python.org/mailman/listinfo/python-list
Re: ftputil.py
Thanks! Sheldon -- http://mail.python.org/mailman/listinfo/python-list
reading binary data written in C
Hi, I have a script that I want to use to read some binary lon and lat data that was written with a C program. My script looks like this: lat = open(lat_file,'rb').read() lat = Numeric.fromstring(lat) print len(lat) print lat[0] Results: 1476225 -995001790 Or using the Float typecode: Results: 1476225 -1419.82055664 ** Now the length of the string is good, a 1215*1215 array in vector form. But the lat data is wrong. I have tried different type codes but nothing gives me a lat that is within [+90,-90] degrees. Does anyone know what is going on here? Thanks, Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Re: reading binary data written in C
Thanks, I later discovered that it was a big edian binary as well. Sheldon -- http://mail.python.org/mailman/listinfo/python-list
Printing UTF-8
I am new to unicode so please bear with my stupidity.
I am doing the following in a Python IDE called Wing with Python 23.
>>> s = "äöü"
>>> print s
äöü
>>> print s
äöü
>>> s
'\xc3\xa4\xc3\xb6\xc3\xbc'
>>> s.decode('utf-8')
u'\xe4\xf6\xfc'
>>> u = s.decode('utf-8')
>>> u
u'\xe4\xf6\xfc'
>>> print u.encode('utf-8')
äöü
>>> print u.encode('latin1')
äöü
Why can't I get äöü printed from utf-8 and I can from latin1? How
can I use utf-8 exclusivly and be able to print the characters?
I also did the same thing an the same machine in a command window...
ActivePython 2.3.2 Build 230 (ActiveState Corp.) based on
Python 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "äöü"
>>> print s
äöü
>>> s
'\x84\x94\x81'
>>> s.decode('utf-8')
Traceback (most recent call last):
File "", line 1, in ?
UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 0:
unexpected code byte
>>> u = s.decode('utf-8')
Traceback (most recent call last):
File "", line 1, in ?
UnicodeDecodeError: 'utf8' codec can't decode byte 0x84 in position 0:
unexpected code byte
>>>
Why such a difference from the IDE to the command window in what it can
do and the internal representation of the unicode?
Thanks,
Shel
--
http://mail.python.org/mailman/listinfo/python-list
urllib2 http authentication/redirection/cookie issue
I have a tricky situation here with auth/redirection/cookies and I
think I am messing something up with the handler.
I am trying to open a site http://foo.example.com which redirects
(status 302) to https://bar.example.com/ where the dashes
represent lots of subdirectories and parameters. This second site,
bar.example.com, is the site which requires authorization (status 401).
Upon authorization, the second site will give a cookie or two then
redirect (status 302) to the original site which will then load because
of the cookie.
The problem is that when I run my code, I still get an HTTP 401
exception. Any help would be much appreciated.
Code and traceback follows.
Regards,
Sheldon Neuberger
--
cj = cookielib.LWPCookieJar()
cookie_handler = urllib2.HTTPCookieProcessor(cj)
passmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmgr.add_password(None, 'https://bar.example.com',
'username', 'password')
auth_handler = urllib2.HTTPBasicAuthHandler(passmgr)
opener = urllib2.build_opener(cookie_handler, auth_handler)
urllib2.install_opener(opener)
html = urllib2.urlopen('http://foo.example.com/qux/')
-
Traceback (most recent call last):
File "dts.py", line 21, in
html = urllib2.urlopen('http://foo.example.com/qux/')
File "C:\Python25\lib\urllib2.py", line 124, in urlopen
return _opener.open(url, data)
File "C:\Python25\lib\urllib2.py", line 387, in open
response = meth(req, response)
File "C:\Python25\lib\urllib2.py", line 498, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python25\lib\urllib2.py", line 419, in error
result = self._call_chain(*args)
File "C:\Python25\lib\urllib2.py", line 360, in _call_chain
result = func(*args)
File "C:\Python25\lib\urllib2.py", line 582, in http_error_302
return self.parent.open(new)
File "C:\Python25\lib\urllib2.py", line 387, in open
response = meth(req, response)
File "C:\Python25\lib\urllib2.py", line 498, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python25\lib\urllib2.py", line 425, in error
return self._call_chain(*args)
File "C:\Python25\lib\urllib2.py", line 360, in _call_chain
result = func(*args)
File "C:\Python25\lib\urllib2.py", line 506, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
--
http://mail.python.org/mailman/listinfo/python-list
RE: Writing a program under GNU/Linux for MS Windows.
If you're just writing Python code then it will run unmodified on his Windows machine. Are you planning on using other languages too? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Levi Campbell Sent: Friday, July 18, 2008 4:27 PM To: python-list@python.org Subject: Writing a program under GNU/Linux for MS Windows. Hi, I'm trying to write a program for a friend of mine who uses windows but I use GNU/Linux. I know you can use mingw and link to the python dll, but is there a way to create a win32 service under Linux? -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
ntlm authentication
Is there any way to make urllib2 handle NTLM authentication? Sheldon Neuberger -- http://mail.python.org/mailman/listinfo/python-list
RE: Password prompt with mask
> Could I write my own if one does not exist?
Take a look at the getpass.py module. It's very short. The windows
version of the getpass function can be trivially modified to echo
something. The unix version is just as short but a bit more
complicated.
def win_getpass(prompt='Password: ', stream=None):
"""Prompt for password with echo off, using Windows getch()."""
if sys.stdin is not sys.__stdin__:
return default_getpass(prompt, stream)
import msvcrt
for c in prompt:
msvcrt.putch(c)
pw = ""
while 1:
c = msvcrt.getch()
if c == '\r' or c == '\n':
break
if c == '\003':
raise KeyboardInterrupt
if c == '\b':
pw = pw[:-1]
else:
pw = pw + c
msvcrt.putch('\r')
msvcrt.putch('\n')
return pw
--
http://mail.python.org/mailman/listinfo/python-list
