Re: How to write partial of a buffer which was returned from a C function to a file?
Chris Angelico於 2018年4月12日星期四 UTC+8下午1時31分35秒寫道： > On Thu, Apr 12, 2018 at 2:16 PM, wrote: > > This C function returns a buffer which I declared it as a ctypes.c_char_p. > > The buffer has size 0x1 bytes long and the valid data may vary from a > > few bytes to the whole size. > > > > In every call I know how much the valid data size is, but I suppose I can't > > use slice to get it because there may be zero byte in it. What to do? > > > > You suppose? Or have you tested it? > > ChrisA Yes, I had test it once before. Now, I re-do it again to make sure. After a call which returns 3 bytes of data, I use len(buf) to check the length and get the number 24. I can see the first 24 bytes of data by using buf[:30] but buf[24] will cause an "index out of range" error. I don't know how to see what the buf[24] exactly is but I suppose it might be a zero byte. --Jach -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Thu, Apr 12, 2018 at 4:20 PM, wrote: > Chris Angelico於 2018年4月12日星期四 UTC+8下午1時31分35秒寫道： >> On Thu, Apr 12, 2018 at 2:16 PM, wrote: >> > This C function returns a buffer which I declared it as a ctypes.c_char_p. >> > The buffer has size 0x1 bytes long and the valid data may vary from a >> > few bytes to the whole size. >> > >> > In every call I know how much the valid data size is, but I suppose I >> > can't use slice to get it because there may be zero byte in it. What to do? >> > >> >> You suppose? Or have you tested it? >> >> ChrisA > > Yes, I had test it once before. Now, I re-do it again to make sure. After a > call which returns 3 bytes of data, I use len(buf) to check the length and > get the number 24. I can see the first 24 bytes of data by using buf[:30] but > buf[24] will cause an "index out of range" error. I don't know how to see > what the buf[24] exactly is but I suppose it might be a zero byte. > If you have 24 bytes, they're numbered 0 through 23. So there is no byte at 24. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On 11Apr2018 21:16, jf...@ms4.hinet.net wrote: This C function returns a buffer which I declared it as a ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid data may vary from a few bytes to the whole size. Could you show us the function? In every call I know how much the valid data size is, but I suppose I can't use slice to get it because there may be zero byte in it. What to do? Why not just return bytes? Allocate one of the correct size and copy the bytes into it, then return? Of course it is all hard to say without seeing some actual code. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: beacons and geofences
On Thu, Apr 12, 2018 at 12:33:39AM +0200, ElChino wrote: > Rafal Sikora wrote: > > > Hi! I want users’ devices to be able to monitor the maximum amount of POIs > > at > > once (geo-fences/beacons) and I need to prepare an algorithm solution for > > monitoring the POIs. How should it be implemented in Python? > > What? You'll have to describe the problem in more details > if you want any sensible answers. I just might so happen that the homework assignment did not contain much more in terms of description. Karsten -- GPG key ID E4071346 @ eu.pool.sks-keyservers.net E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346 -- https://mail.python.org/mailman/listinfo/python-list
Re: Compression of random binary data
On 4/11/18 9:29 PM, cuddlycave...@gmail.com wrote: I’m replying to your post on January 28th Nice carefully chosen non random numbers Steven D'Aprano. Was just doing what you asked, but you don’t remember 😂😂😂 Best practice is to include a quote of the thing you are replying to. It makes it much easier for people to follow the thread of the discussion, especially when there are large gaps in the timeline. --Ned. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
Chris Angelico於 2018年4月12日星期四 UTC+8下午4時05分29秒寫道： > On Thu, Apr 12, 2018 at 4:20 PM, wrote: > > Chris Angelico於 2018年4月12日星期四 UTC+8下午1時31分35秒寫道： > >> On Thu, Apr 12, 2018 at 2:16 PM, wrote: > >> > This C function returns a buffer which I declared it as a > >> > ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid > >> > data may vary from a few bytes to the whole size. > >> > > >> > In every call I know how much the valid data size is, but I suppose I > >> > can't use slice to get it because there may be zero byte in it. What to > >> > do? > >> > > >> > >> You suppose? Or have you tested it? > >> > >> ChrisA > > > > Yes, I had test it once before. Now, I re-do it again to make sure. After a > > call which returns 3 bytes of data, I use len(buf) to check the length and > > get the number 24. I can see the first 24 bytes of data by using buf[:30] > > but buf[24] will cause an "index out of range" error. I don't know how to > > see what the buf[24] exactly is but I suppose it might be a zero byte. > > > > If you have 24 bytes, they're numbered 0 through 23. So there is no byte at > 24. > > ChrisA Using a technique you mentioned in subject "how to memory dump an object?" at 16/5/21, I confirm the length of buf was decided by a \x00 byte: >>> len(buf) 24 >>> id(buf) 13553888 >>> ptr = ctypes.cast(id(buf), ctypes.POINTER(ctypes.c_ubyte)) >>> buf[:24] b'\x05ALLOTNPUT_BUFFER_SIZE\x02+' >>> bytes([ptr[i] for i in range(50)]) b'\x02\x00\x00\x00X\xa1%\x1e\x18\x00\x00\x00\xff\xff\xff\xff\x05ALLOTNPUT_BUFFER_SIZE\x02+\x00\n\x00\x00\x00\x00\x00\x00\xb0\x9b' >>> but it won't help on solving my problem. Still need someone's help:-) --Jach -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
This is the first time I am using python-list to interact with comp.lang.python forum (because there are so many spam when using browser to view it) so forgive me if something goes wrong. Python already treat the returned buffer as 'bytes'. The problem is Python don't know its size (or decides it wrong:-). --Jach Cameron Simpson at 2018/4/12 PM 02:28 wrote: On 11Apr2018 21:16, jf...@ms4.hinet.net wrote: This C function returns a buffer which I declared it as a ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid data may vary from a few bytes to the whole size. Could you show us the function? In every call I know how much the valid data size is, but I suppose I can't use slice to get it because there may be zero byte in it. What to do? Why not just return bytes? Allocate one of the correct size and copy the bytes into it, then return? Of course it is all hard to say without seeing some actual code. Cheers, Cameron Simpson --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -- https://mail.python.org/mailman/listinfo/python-list
Re: Pandas, create new column if previous column(s) are not in [None, '', np.nan]
On Wednesday, 11 April 2018 21:19:44 UTC+2, José María Mateos wrote:
> On Wed, Apr 11, 2018, at 14:48, zlj...com wrote:
> > I have a dataframe:
> > [...]
>
> This seems to work:
>
> df1 = pd.DataFrame( { 'A' : ['a', 'b', '', None, np.nan],
> 'B' : [None, np.nan, 'a',
> 'b', '']})
> df1['C'] = df1[['A', 'B']].apply(lambda x: x[0] if x[1] in [None, '', np.nan]
> else x[1], axis = 1)
>
> Two notes:
>
> - Do apply() on axis = 1, so you process every row.
> - You lambda function wasn't entirely correct, if I understood what you
> wanted to do.
>
> Cheers,
>
> --
> José María (Chema) Mateos
> https://rinzewind.org/blog-es || https://rinzewind.org/blog-en
Thanks Jose, this what I needed. Thanks also to all others.
Regards.
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Apr 12, 2018 09:39, jf...@ms4.hinet.net wrote: > > Chris Angelico於 2018年4月12日星期四 UTC+8下午1時31分35秒寫道： > > On Thu, Apr 12, 2018 at 2:16 PM, wrote: > > > This C function returns a buffer which I declared it as a > > > ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid > > > data may vary from a few bytes to the whole size. > > > > > > In every call I know how much the valid data size is, but I suppose I > > > can't use slice to get it because there may be zero byte in it. What to > > > do? > > > > > > > You suppose? Or have you tested it? > > > > ChrisA > > Yes, I had test it once before. Now, I re-do it again to make sure. After a > call which returns 3 bytes of data, I use len(buf) to check the length and > get the number 24. I can see the first 24 bytes of data by using buf[:30] but > buf[24] will cause an "index out of range" error. I don't know how to see > what the buf[24] exactly is but I suppose it might be a zero byte. Aren't you looking for the .value or the .raw property? -- https://mail.python.org/mailman/listinfo/python-list
Making matrix solver/ solver for X
Hee guys, I'm learning Python partly for school and wanted to see if I could figure some things out myself. I wanted to make like a value finder, sort of like numpy.solve (sounds more ambitious than it is) but I'm sort of stuck, since I'm a beginner. Let's say I want to find the values of matrix X with X1, X2, X3 and X4. And that X is multiplied with matrix Q and then X substracted from it. I thought about: X = np.array(['X1', 'X2', 'X3', 'X4']) Q = np.array(['Q1', 'Q2', 'Q3', 'Q4']) P = (X*Q)-X I chose array's since all the X and Q values can be different. How would I be able to find the values of X if the Q-array would be given and it would have to be solved under the condition that out of 4 P values, at least 2 would be more or equal to sum of X. So solve condition: P[0]+P[1] ,P[0]+P[2] ,P[0]+P[3] ,P[1]+P[2] ,P[1]+P[3] ,P[2]+P[3] >= np.sum(X) So summary: If Q is given here, and only X is unknown, and P has only one unknown (X-array), how can I calculate the X-matrix with the solve condition? Can you guys help me? I feel like it's an easy set of lines of code, but like I said I'm still trying to learn. And improvise to see if I understand it. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Thu, Apr 12, 2018 at 2:16 PM, wrote: This C function returns a buffer which I declared it as a ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid data may vary from a few bytes to the whole size. I think we need to see the code you're using to call this C function. The crucial issue is: are *you* allocating this 0x1 byte buffer and telling the function to read data into it, or does the function allocate the memory itself and return a pointer to it? If the function is allocating the buffer, then I don't think there's any way to make this work. The ctypes docs say this: Fundamental data types, when returned as foreign function call results ... are transparently converted to native Python types. In other words, if a foreign function has a restype of c_char_p, you will always receive a Python bytes object, not a c_char_p instance. The problem is that the only way ctypes can tell how long a bytes object to create for a c_char_p is by assuming that it points to a nul-terminated string. If it actually points to a char array that can legitimately contain zero bytes, then you're out of luck. To get around this, you may need to declare the return type as POINTER(c_char) instead: For a general character pointer that may also point to binary data, > POINTER(c_char) must be used. I'm not sure where to go from here, though, because the ctypes documentation peters out before explaining exactly what can be done with a POINTER object. Another approach would be to allocate the buffer yourself and pass it into the C function, but whether that's possible depends on the details of the C API you're using. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On 12Apr2018 16:11, Jach Fong wrote: This is the first time I am using python-list to interact with comp.lang.python forum (because there are so many spam when using browser to view it) so forgive me if something goes wrong. Python already treat the returned buffer as 'bytes'. The problem is Python don't know its size (or decides it wrong:-). I think you'll need to show us your code. It isn't clear to me your problem is. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Thu, Apr 12, 2018 at 11:25 PM, Gregory Ewing wrote: > > To get around this, you may need to declare the return type > as POINTER(c_char) instead: > >> For a general character pointer that may also point to binary data, > >> POINTER(c_char) must be used. > > I'm not sure where to go from here, though, because the > ctypes documentation peters out before explaining exactly > what can be done with a POINTER object. Pointers can be indexed and sliced. You have to be careful, however, since there's no bounds checking. Alternatively, without copying, you can create an array view on the buffer, which is bounded and thus doesn't risk an access violation (segfault). For example: Say the function returns a pointer to a buffer with the contents b"spam\x00". Let's simulate the function result using a void * pointer to initialize a char * pointer: >>> buf0 = ctypes.create_string_buffer(b'spam') >>> pvoid = ctypes.c_void_p(ctypes.addressof(buf0)) >>> result = ctypes.POINTER(ctypes.c_char).from_buffer_copy(pvoid) This pointer object has just the address of the buffer, without supporting references in _b_base_ or _objects: >>> result._b_base_ is result._objects is None True (In other words, ctypes isn't responsible for the buffer, as simulated here. Libraries that allocate their own memory for results have to provide a function to free it. Especially on Windows, you cannot rely on both Python and the DLL to use the same heap.) You can slice the pointer: >>> result[:5] b'spam\x00' Or you can access the buffer more safely as a new array view: >>> array_t = ctypes.c_char * 5 >>> pointer_t = ctypes.POINTER(array_t) >>> result.contents c_char(b's') >>> buf1 = pointer_t(result.contents)[0] >>> buf1[:] b'spam\x00' This buf1 array is a view on the buffer, not a copy. It reflects whatever changes are made to the underlying buffer: >>> buf0[:] = b'eggs\x00' >>> buf1[:] b'eggs\x00' As such, ctypes knows it doesn't have to free this memory: >>> buf1._b_needsfree_ 0 -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
Gregory Ewing at 2018/4/13 上午 07:25 wrote: On Thu, Apr 12, 2018 at 2:16 PM, wrote: This C function returns a buffer which I declared it as a ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid data may vary from a few bytes to the whole size. I think we need to see the code you're using to call this C function. The crucial issue is: are *you* allocating this 0x1 byte buffer and telling the function to read data into it, or does the function allocate the memory itself and return a pointer to it? I am working on a DLL's function. If the function is allocating the buffer, then I don't think there's any way to make this work. The ctypes docs say this: Fundamental data types, when returned as foreign function call results ... are transparently converted to native Python types. In other words, if a foreign function has a restype of c_char_p, you will always receive a Python bytes object, not a c_char_p instance. The problem is that the only way ctypes can tell how long a bytes object to create for a c_char_p is by assuming that it points to a nul-terminated string. If it actually points to a char array that can legitimately contain zero bytes, then you're out of luck. To get around this, you may need to declare the return type as POINTER(c_char) instead: For a general character pointer that may also point to binary data, > POINTER(c_char) must be used. I had missed this statement:-( To make a quick try, I set the function's restype to ctypes.POINTER(ctypes.c_ubyte), instead of ctypes.c_char_p. It's amazing, the \x00 trap can be avoided in this way. Now I can use "mydata = bytes(buf[:n])" to extract n bytes of data and write it to file. The problem was solved, and thanks for all your help. --Jach I'm not sure where to go from here, though, because the ctypes documentation peters out before explaining exactly what can be done with a POINTER object. Another approach would be to allocate the buffer yourself and pass it into the C function, but whether that's possible depends on the details of the C API you're using. --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write partial of a buffer which was returned from a C function to a file?
On Fri, Apr 13, 2018 at 12:38 AM, Jach Fong wrote: > Gregory Ewing at 2018/4/13 上午 07:25 wrote: > >> To get around this, you may need to declare the return type >> as POINTER(c_char) instead: >> >>> For a general character pointer that may also point to binary data, >> >> > POINTER(c_char) must be used. > > I had missed this statement:-( > > To make a quick try, I set the function's restype to > ctypes.POINTER(ctypes.c_ubyte), instead of ctypes.c_char_p. It's amazing, > the \x00 trap can be avoided in this way. Now I can use "mydata = > bytes(buf[:n])" to extract n bytes of data and write it to file. Slicing a ctypes.POINTER(ctypes.c_char) pointer returns bytes without having to make a third copy via the bytes constructor. (Note that c_char is the fundamental C char integer type, not to be confused with c_char_p, which is a char * pointer.) However, if you're working with multi-megabyte data buffers,it's more efficient and safer to use an array view (ctypes or NumPy) on the returned buffer. In most cases, you should free the returned pointer after you're finished processing the data buffer, else you'll have a memory leak. The library should export a function for this. -- https://mail.python.org/mailman/listinfo/python-list
Installing NETCDF4 in windows using python 3.4
Hi All, I have downloaded NETCDF4 module from https://pypi.python.org/pypi/netCDF4 e.g. netCDF4-1.3.1-cp34-cp34m-win_amd64.whl I have installed it using pip install netCDF4-1.3.1-cp34-cp34m-win_amd64.whl through the command prompt in Spyder. It has successfully installed. C:\python3>pip install netCDF4-1.3.1-cp34-cp34m-win_amd64.whl Processing c:\python3\netcdf4-1.3.1-cp34-cp34m-win_amd64.whl Requirement already satisfied: numpy>=1.7 in c:\python3\winpython-64bit-3.4.4.5qt5\python-3.4.4.amd64\lib\site-packages (from netCDF4==1.3.1) Installing collected packages: netCDF4 Found existing installation: netCDF4 1.3.2 Uninstalling netCDF4-1.3.2: Successfully uninstalled netCDF4-1.3.2 Successfully installed netCDF4-1.3.1 But when I am trying to import, it is giving an error: import netCDF4 as nc4 Traceback (most recent call last): File "", line 1, in import netCDF4 as nc4 File "C:\python3\WinPython-64bit-3.4.4.5Qt5\python-3.4.4.amd64\lib\site-packages\netCDF4__init__.py", line 3, in from ._netCDF4 import * File "netCDF4_netCDF4.pyx", line 2988, in init netCDF4._netCDF4 AttributeError: type object 'netCDF4._netCDF4.Dimension' has no attribute 'reduce_cython' How can I fix it? Suggestions would be appreciated. -- https://mail.python.org/mailman/listinfo/python-list
