Re: How to write partial of a buffer which was returned from a C function to a file?
eryk sun at 2018/4/13 PM 12:16 wrote: On Fri, Apr 13, 2018 at 12:38 AM, Jach Fong wrote: Gregory Ewing at 2018/4/13 上午 07:25 wrote: To get around this, you may need to declare the return type as POINTER(c_char) instead: For a general character pointer that may also point to binary data, > POINTER(c_char) must be used. I had missed this statement:-( To make a quick try, I set the function's restype to ctypes.POINTER(ctypes.c_ubyte), instead of ctypes.c_char_p. It's amazing, the \x00 trap can be avoided in this way. Now I can use "mydata = bytes(buf[:n])" to extract n bytes of data and write it to file. Slicing a ctypes.POINTER(ctypes.c_char) pointer returns bytes without having to make a third copy via the bytes constructor. (Note that c_char is the fundamental C char integer type, not to be confused with c_char_p, which is a char * pointer.) However, if you're working with multi-megabyte data buffers,it's more efficient and safer to use an array view (ctypes or NumPy) on the returned buffer. After studying the example you explained in your previous post replied to Gregory Ewing, I had noticed that until today I was totally misunderstand the meaning of the c_char_p. I always think it "is" a pointer, but actually it's just a ctypes type, maybe literarily looks like a C pointer, but not a pointer from the ctypes view at all:-) from the ctypes document: "Pointer instances are created by calling the pointer() function on a ctypes type" "Fundamental data types, when returned as foreign function call results, or, for example, by retrieving structure field members or array items, are transparently converted to native Python types. In other words, if a foreign function has a restype of c_char_p, you will always receive a Python bytes object, not a c_char_p instance" That's the reason I was failed at first using c_char_p as a pointer to the returned C buffer. There is no fundamental pointer type in ctypes. If a pointer was needed, you have to create it manually. That's why the second try works. Anyway, thanks for help on cleaning my head:-) --Jach In most cases, you should free the returned pointer after you're finished processing the data buffer, else you'll have a memory leak. The library should export a function for this. --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -- https://mail.python.org/mailman/listinfo/python-list
python notifying calling script that multiprocessing tasks have finished at lower level scripts
Hi, I have a master script that executes two sequences (lists) of child scripts, i.e. script_1 to script_3, and script_4 to_script_6 (the structure is attached as a png file). The execution is sequential, e.g. running script_1, then 2 then 3. After executing the 1st sequence (script_1 to 3), master will execute the 2nd sequence (script_4 to 6). Each child script will be calling a multiprocessing function to process a task. Master script is like, for seq in seqs_to_launch: for script in seq: script().execute(data) Each child script is like, import multi_process_update class Script_n(): def execute(self, data): # some data processing multi_process_task(data_task, task_name) The multiprocessing function is like, def multi_process_task(tasks, task_name): cores_to_use = how_many_core() handler = task_handling(task_name) # task handling class task_blocks = slice_list(tasks, cores_to_use) for block in task_blocks: # spawn processes for each row block assigned to every cpu core p = multiprocessing.Process(target=handler.execute, args=(block,)) p.start() I like to know how to notify the master when the 2nd sequence is finished that all multiprocess tasks (seq 1 and 2) are completed, and then maybe close the multiprocessing. thanks -- https://mail.python.org/mailman/listinfo/python-list
Python Import Impossibility
Hello, Could you tell me how to import the installed modules ? I have successfully installed openpyxl, but When I executed ‘import openpyxl’, The following message is displayed: Traceback (most recent call last): File "", line 1, in import openpyxl ModuleNotFoundError: No module named 'openpyxl' My folder is formed as follows: [cid:image003.png@01D3D312.38C82830] Best Regards, - Naoki Morihira TEL: 01181-90-6460-6265 - -- https://mail.python.org/mailman/listinfo/python-list
Re: Filtering computer.lang.python
Mark Lawrence on Wed, 11 Apr 2018 20:07:47 +0100 typed in comp.lang.python the following: > >> for totals of 2168 fetched and 4384 killed; that is, the group >> is now 2/3 spam and the volume doesn't seem to be decreasing. >> I don't understand why other groups gatewayed to Google Groups >> aren't spammed, but from a limited sample they don't seem to be. >> >> Will >> > >I have recently given up killing all the crap directly on gg as I can't >be bothered any more. That is I would go onto gg and directly mark all >the spam as spam. Is this coincidence? There's your problem "Google Groups". Whatever it is, Google Groups doesn't seem to be "Usenet". -- pyotr filipivich Next month's Panel: Graft - Boon or blessing? -- https://mail.python.org/mailman/listinfo/python-list
Re: python notifying calling script that multiprocessing tasks have finished at lower level scripts
On 13/04/18 18:08, Daiyue Weng wrote: > (the structure > is attached as a png file). No it's not. This is a text-only list. (you know what, I'm sick of saying that) > > The execution is sequential, e.g. running script_1, then 2 then 3. > > After executing the 1st sequence (script_1 to 3), master will execute the > 2nd sequence (script_4 to 6). > > Each child script will be calling a multiprocessing function to process a > task. I could ask what motivates this convoluted-sounding structure... > > [snip] > > > I like to know how to notify the master when the 2nd sequence is finished > that all multiprocess tasks (seq 1 and 2) are completed, and then maybe > close the multiprocessing. Well, I suppose you'll have to keep all the Process objects around somewhere where they can be checked on. You might want to use a multiprocessing.Pool. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Import Impossibility
On 13/04/18 14:48, ?? ?? wrote: > Hello, > > Could you tell me how to import the installed modules ? > > I have successfully installed openpyxl, but > When I executed ‘import openpyxl’, > The following message is displayed: > Traceback (most recent call last): > File "", line 1, in > import openpyxl > ModuleNotFoundError: No module named 'openpyxl' > > My folder is formed as follows: > [cid:image003.png@01D3D312.38C82830] Is it possible that you installed the module in the wrong Python installation? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Import Impossibility
On Fri, 13 Apr 2018 12:48:55 +, ?? ?? wrote: > Hello, > > Could you tell me how to import the installed modules ? > > I have successfully installed openpyxl, How do you know it was successful? What did you do to install it? How many different Python installations do you have on your system? but When I executed ‘import > openpyxl’, The following message is displayed: > Traceback (most recent call last): > File "", line 1, in > import openpyxl > ModuleNotFoundError: No module named 'openpyxl' > > My folder is formed as follows: > [cid:image003.png@01D3D312.38C82830] There are no attachments permitted on this list. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Python regex pattern from array of hex chars
I have an array of hex chars which designate required characters.
and one happens to be \x5C or "\". What foo is required to build the
pattern to exclude all but:
regex = re.compile('[^{}]+'.format(''.join(c for c in character_class)))
I would use that in a re.sub to collapse and replace all but those
in the character_class. Obviously the escape issues break the \x5C
character.
Thanks,
jlc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python regex pattern from array of hex chars
On 2018-04-13 18:28, Joseph L. Casale wrote:
I have an array of hex chars which designate required characters.
and one happens to be \x5C or "\". What foo is required to build the
pattern to exclude all but:
regex = re.compile('[^{}]+'.format(''.join(c for c in character_class)))
I would use that in a re.sub to collapse and replace all but those
in the character_class. Obviously the escape issues break the \x5C
character.
Use re.escape:
regex = re.compile('[^{}]+'.format(re.escape(''.join(c for c in
character_class
--
https://mail.python.org/mailman/listinfo/python-list
RE: Python regex pattern from array of hex chars
-Original Message-
From: Python-list On Behalf Of MRAB
Sent: Friday, April 13, 2018 12:05 PM
To: python-list@python.org
Subject: Re: Python regex pattern from array of hex chars
> Use re.escape:
>
> regex = re.compile('[^{}]+'.format(re.escape(''.join(c for c in
> character_class
Brilliant, thanks!
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Fri, Apr 13, 2018 at 8:44 AM, Jach Fong wrote: > > After studying the example you explained in your previous post replied to > Gregory Ewing, I had noticed that until today I was totally misunderstand > the meaning of the c_char_p. I always think it "is" a pointer, but actually > it's just a ctypes type, maybe literarily looks like a C pointer, but not a > pointer from the ctypes view at all:-) Here's a list of type classes in ctypes: class metaclass = _SimpleCData PyCSimpleType _Pointer PyCPointerType _CFuncPtr PyCFuncPtrType Array PyCArrayType Structure PyCStructType Union UnionType These classes share a common _CData base class. Note that the _ctypes extension module doesn't directly expose _CData, nor any of the metaclasses. ctypes type checking primarily uses Python type checking, so we generally do not subclass these types directly, except for Structure and Union. Instead we have a set of predefined simple types that subclass _SimpleCData (e.g. c_int, c_char), and we use factory functions to create pointer types (e.g. POINTER, CFUNCTYPE), which cache the created type. For arrays, we rely on the base _CData sequence-repeat functionality (e.g. c_int * 3), which also caches the Array subclass that it creates. Type caching ensures that two expressions that create an equivalent C type return the same class. For example, if you have `c_char * 3` in two places, it should be the same type: >>> cls = ctypes.c_char * 3 >>> (ctypes.c_char * 3) is cls True The simple types c_void_p, c_char_p, and c_wchar_p are pointers. However, since they subclass _SimpleCData instead of _Pointer, they inherit the behavior of simple types. In particular they have get/set functions that implicitly convert to and from native Python types when they're used in aggregate types (arrays, structs, unions), when indexing or slicing a _Pointer instance, or as the result or argument of a function pointer (i.e. _CFuncPtr subclass). -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
eryk sun at 2018/4/14 PM 05:27 wrote: On Fri, Apr 13, 2018 at 8:44 AM, Jach Fong wrote: After studying the example you explained in your previous post replied to Gregory Ewing, I had noticed that until today I was totally misunderstand the meaning of the c_char_p. I always think it "is" a pointer, but actually it's just a ctypes type, maybe literarily looks like a C pointer, but not a pointer from the ctypes view at all:-) Here's a list of type classes in ctypes: class metaclass = _SimpleCData PyCSimpleType _Pointer PyCPointerType _CFuncPtr PyCFuncPtrType Array PyCArrayType Structure PyCStructType Union UnionType These classes share a common _CData base class. Note that the _ctypes extension module doesn't directly expose _CData, nor any of the metaclasses. ctypes type checking primarily uses Python type checking, so we generally do not subclass these types directly, except for Structure and Union. Instead we have a set of predefined simple types that subclass _SimpleCData (e.g. c_int, c_char), and we use factory functions to create pointer types (e.g. POINTER, CFUNCTYPE), which cache the created type. For arrays, we rely on the base _CData sequence-repeat functionality (e.g. c_int * 3), which also caches the Array subclass that it creates. Type caching ensures that two expressions that create an equivalent C type return the same class. For example, if you have `c_char * 3` in two places, it should be the same type: >>> cls = ctypes.c_char * 3 >>> (ctypes.c_char * 3) is cls True Thanks for your description. To digest it, I may need to dive into its source jungle:-( The simple types c_void_p, c_char_p, and c_wchar_p are pointers. However, since they subclass _SimpleCData instead of _Pointer, they inherit the behavior of simple types. The ctypes document says: "Pointer instances have a contents attribute which returns the object to which the pointer points" >>> buf0 = ctypes.create_string_buffer(b'spam') >>> pvoid = ctypes.c_void_p(ctypes.addressof(buf0)) >>> pvoid.contents Traceback (most recent call last): File "", line 1, in AttributeError: 'c_void_p' object has no attribute 'contents' >>> pvoid.value 35425816 >>> pp = ctypes.pointer(buf0) >>> pp.contents >>> pp.value Traceback (most recent call last): File "", line 1, in AttributeError: 'LP_c_char_Array_5' object has no attribute 'value' It looks like the c_void_p is not of a pointer type:-) --Jach In particular they have get/set functions that implicitly convert to and from native Python types when they're used in aggregate types (arrays, structs, unions), when indexing or slicing a _Pointer instance, or as the result or argument of a function pointer (i.e. _CFuncPtr subclass). --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Sat, Apr 14, 2018 at 1:57 AM, Jach Fong wrote: > eryk sun at 2018/4/14 PM 05:27 wrote: > >> The simple types c_void_p, c_char_p, and c_wchar_p are pointers. >> However, since they subclass _SimpleCData instead of _Pointer, they >> inherit the behavior of simple types. > > The ctypes document says: > "Pointer instances have a contents attribute which returns the object to > which the pointer points" > buf0 = ctypes.create_string_buffer(b'spam') > pvoid = ctypes.c_void_p(ctypes.addressof(buf0)) pvoid.contents > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'c_void_p' object has no attribute 'contents' pvoid.value > 35425816 > pp = ctypes.pointer(buf0) pp.contents > pp.value > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'LP_c_char_Array_5' object has no attribute 'value' > > It looks like the c_void_p is not of a pointer type:-) c_void_p is a `void *` pointer type, but it's a simple pointer type that subclasses _SimpleCData instead of _Pointer, so it doesn't have a `contents` property but instead has a `value` property. >>> hasattr(ctypes._SimpleCData, 'contents') False >>> hasattr(ctypes._SimpleCData, 'value') True >>> hasattr(ctypes._Pointer, 'contents') True >>> hasattr(ctypes._Pointer, 'value') False Don't take the Python class hierarchy so literally that you overlook what these types ultimately are in C. Just because they don't subclass _Pointer, that doesn't mean they're not pointer types. c_void_p, c_char_p, and c_wchar_p are unquestionably pointer types. This is what the "_p" suffix means in their names. It's just these particular pointer types were implemented as simple types to get the convenience of implicit conversion. That said, sometimes the implicit conversion is a problem, in which case we use, for example, POINTER(c_char) instead of c_char_p. Look in Lib/ctypes/__init__.py to review the definitions for yourself. Here they are without the __repr__ methods: class c_void_p(_SimpleCData): _type_ = "P" class c_char_p(_SimpleCData): _type_ = "z" class c_wchar_p(_SimpleCData): _type_ = "Z" This doesn't tell you much. You have to go looking for what it means to be a simple "P" type, and in particular we're concerned with how conversion to and from native Python types is implemented. You'll find the get and set C implementations for types "P", "z", and "Z" defined in Modules/_ctypes/cfield.c. For example, for type "P", it's P_get() and P_set(). For P_get(), we use PyLong_FromVoidPtr to convert the pointer value to a Python integer, except we return None for a NULL pointer. For P_set(), we require an integer value or None (NULL). Note that the function to convert a Python integer to an integral address in C depends on the size of a C `long` or `long long` compared to the size of a C `void *` pointer. In particular, this design accommodates 64-bit Windows, on which a `long` is 32-bit and thus too small for a 64-bit pointer value, so we call PyLong_AsUnsignedLongLongMask instead of PyLong_AsUnsignedLongMask. -- https://mail.python.org/mailman/listinfo/python-list
