Re: When creating a nested dictionary of dataframes, how can I name a dictionary based on a list name of the dataframe?
Aaron wrote:
> When creating a nested dictionary of dataframes, how can I name a
> dictionary based on a list name of the dataframe?
>
> Given the following:
>
> # START CODE
> import pandas as pd
>
> cars = {'Brand': ['Honda Civic','Toyota Corolla'],
> 'Price': [22000,25000]
> }
> df_cars = pd.DataFrame(cars, columns = ['Brand','Price'])
>
> trucks = {'Brand': ['GMC Sierra','Ford F-150'],
> 'Price': [5,48000]
> }
> df_trucks = pd.DataFrame(trucks, columns = ['Brand','Price'])
>
> list_of_dfs = [df_cars, df_trucks]
>
> # Not exactly sure how this code should be:
> dict_of_dfs = {}
> for df in list_of_dfs:
> dict_of_dfs[name_of_df] = {} # Not sure here
> dict_of_dfs[name_of_df]['results'] = df # Not sure here
> # END CODE
In the general case you can't find the name of a value, so you may start
with the names and look up the dataframes in the global namespace:
list_of_df_names = ["df_cars", "df_trucks"]
dict_of_dfs = {
name: {"results": globals()[name]}
for name in list_of_df_names
}
Personally I would probably reorganize your code a bit and avoid the df_...
names:
import pandas as pd
vehicles = dict(
cars={
'Brand': ['Honda Civic','Toyota Corolla'],
'Price': [22000,25000]
},
trucks={
'Brand': ['GMC Sierra','Ford F-150'],
'Price': [5,48000]
}
)
dict_of_dfs = {
name: {"results": pd.DataFrame(val)}
for name, val in vehicles.items()
}
--
https://mail.python.org/mailman/listinfo/python-list
ctypes & allocated memory
Hi,
Does ctypes, when using restype, frees allocated memory?
For example, will the memory allocated by "strdup" be freed after the "del"
statement? If not, how can I free it?
---
import ctypes
libc = ctypes.cdll.LoadLibrary('libc.so.6')
strdup = libc.strdup
strdup.argtypes = [ctypes.c_char_p]
strdup.restype = ctypes.c_char_p
out = strdup(b'hello').decode('utf-8')
print(out) # hello
del out
---
Thanks,
Miki
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
> Does ctypes, when using restype, frees allocated memory?
>
> For example, will the memory allocated by "strdup" be freed after the "del"
> statement? If not, how can I free it?
I've tried the following program and I'm more confused now :) Can anyone
explain the output?
---
import ctypes
import gc
import resource
def mem_usage():
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
libc = ctypes.cdll.LoadLibrary('libc.so.6')
strdup = libc.strdup
strdup.argtypes = [ctypes.c_char_p]
strdup.restype = ctypes.c_char_p
size = 1 << 20
print(f'size: {size:,}')
data = b'x' * size # 1MB
mb = mem_usage()
print(f'memory before: {mb:,}')
n = 1000
print(f'n: {n:,}')
for _ in range(n):
strdup(data)
gc.collect()
ma = mem_usage()
diff = ma - mb
print(f'memory after: {ma:,}')
print(f'diff: {diff:,}')
print(f'diff/size: {diff/size:.2f}')
---
Which prints
---
size: 1,048,576
memory before: 21,556
n: 1,000
memory after: 1,035,180
diff: 1,013,624
diff/size: 0.97
---
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
On 6/7/20 7:15 AM, Miki Tebeka wrote:
> Hi,
>
> Does ctypes, when using restype, frees allocated memory?
>
> For example, will the memory allocated by "strdup" be freed after the "del"
> statement? If not, how can I free it?
I don't think so. I did a quick google search and came up with this
discussion, which may be relevant:
https://stackoverflow.com/questions/13445568/python-ctypes-how-to-free-memory-getting-invalid-pointer-error
>
> ---
> import ctypes
>
> libc = ctypes.cdll.LoadLibrary('libc.so.6')
> strdup = libc.strdup
> strdup.argtypes = [ctypes.c_char_p]
> strdup.restype = ctypes.c_char_p
>
> out = strdup(b'hello').decode('utf-8')
> print(out) # hello
> del out
> ---
>
> Thanks,
> Miki
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Trouble with making modules 'global'
On 2020-06-04 04:18:31 -0400, Terry Reedy wrote: > On 6/3/2020 11:48 PM, pytho...@gmail.com wrote: > > (1) Main.py > > The PEP8 standard is all lower case for modules, leaving TitleCase for > classes, so, for instance, file/module 'editor' contains the definition of > class 'Editor' as its main class. And nobody swoops in to point out that he isn't required to follow PEP 8? I'm disappointed. SCNR. hp -- _ | Peter J. Holzer| Story must make more sense than reality. |_|_) || | | | h...@hjp.at |-- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" signature.asc Description: PGP signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?
On 2020-05-28 20:51:59 -0400, Terry Reedy wrote: > On 5/28/2020 5:20 PM, Peter J. Holzer wrote: [The final "security-only" releases are source-only] > > This seems a rather odd policy to me. > > Not if one considers the intended users. > Do you prefer we not make these releases? No, of couse not (also, as a Linux user I'm not terribly inconvenienced by the lack of Windows binaries). I would prefer that security fixes are distributed in the same form as the original release so that people can apply them without encountering additional hurdles. I assume (perhaps naively) that those releases are tested on the supported platforms. After building and testing them, is that much additional work to put the artifacts on the web site? (I admit that the Windows developers I know aren't very fond of automation, is that a general trait?) hp -- _ | Peter J. Holzer| Story must make more sense than reality. |_|_) || | | | h...@hjp.at |-- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" signature.asc Description: PGP signature -- https://mail.python.org/mailman/listinfo/python-list
Button press event - event handling and picking: IndexError: list index out of range
I have the following command below.
The overall aim is to allow the user to select a Y-value by pressing on the bar
graph. The colour of each bar should then change depending on what this Y-value
is.
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import matplotlib.colors
np.random.seed(12345)
df = pd.DataFrame({"values":[np.random.normal(32000,20,3650).mean(),
np.random.normal(43000,10,3650).mean(),
np.random.normal(43500,14,3650).mean(),
np.random.normal(48000,7,3650).mean()],
"index":[1992,1993,1994,1995]})
standarderrors1992 = stats.sem(np.random.normal(32000,20,3650))
standarderrors1993 = stats.sem(np.random.normal(43000,10,3650))
standarderrors1994 = stats.sem(np.random.normal(43500,14,3650))
standarderrors1995 = stats.sem(np.random.normal(48000,7,3650))
add1992 = 1.96*standarderrors1992
add1993 = 1.96*standarderrors1993
add1994 = 1.96*standarderrors1994
add1995 = 1.96*standarderrors1995
mean1992 = np.random.normal(32000,20,3650).mean()
mean1993 = np.random.normal(43000,10,3650).mean()
mean1994 = np.random.normal(43500,14,3650).mean()
mean1995 = np.random.normal(48000,7,3650).mean()
labels = [1992,1993,1994,1995]
add = [add1992,add1992,add1994,add1995]
1. This first part organises the raw data.
limits = []
def onclick(event):
plt.cla()
plt.bar(df["index"].values,df["values"].values,align='center',
alpha=0.5,yerr=add)
plt.xticks(labels)
limit = event.ydata
limits.append(limit)
if len(limits) >= 1:
plt.gcf().canvas.mpl_disconnect(plt.gcf().canvas.mpl_connect('button_press_event',
onclick))
plt.gcf().canvas.mpl_connect('button_press_event', onclick)
2. This next part allows the user to press on the graph to select a Y value.
This should be assigned to the variable 'limits'
dict = {mean1992:add1992,mean1993:add1993,mean1994:add1994,mean1995:add1995}
colourofbars = []
for key,value in dict.items():
if limits[0] > (key+(value)):
colour = 1
colourofbars.append(colour)
elif limits[0] < (key-(value)):
colour = 0
colourofbars.append(colour)
elif (limits[0] < (key+(value))) & (limits[0] > (key-(value))):
colour = ((key+(value))-limits[0])/((key+value)-(key-value))
colourofbars.append(colour)
df["colourofbars"] = colourofbars
3. Here, the list 'colourofbars' is appended based on the data above, and added
as a column to the dataframe 'df'.
cmap = plt.cm.rainbow
norm = matplotlib.colors.Normalize(vmin=1.5, vmax=4.5)
plt.bar(df["index"].values,df["values"].values,color=cmap(norm(df["colourofbars"].values)),align='center',
alpha=0.5,yerr=add)
plt.xticks(labels)
sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)
plt.gcf().colorbar(sm)
plt.show()
4. Here, a different colour is assigned to each bar in the bar chart depending
on the values in the column 'colourofbars'. I then try to plot a legend showing
this colour gradient scale.
However, I keep getting the error: IndexError: list index out of range. Could
anyone give me a helping hand as to where I am going wrong?
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
> On 7 Jun 2020, at 14:23, Miki Tebeka wrote:
>
> Hi,
>
> Does ctypes, when using restype, frees allocated memory?
>
> For example, will the memory allocated by "strdup" be freed after the "del"
> statement? If not, how can I free it?
See https://linux.die.net/man/3/strdup that tells you to use free() to delete
memory allocated by strdup.
You must remember the result of strdup and free it at an appropriate time.
Barry
>
> ---
> import ctypes
>
> libc = ctypes.cdll.LoadLibrary('libc.so.6')
> strdup = libc.strdup
> strdup.argtypes = [ctypes.c_char_p]
> strdup.restype = ctypes.c_char_p
>
> out = strdup(b'hello').decode('utf-8')
> print(out) # hello
> del out
> ---
>
> Thanks,
> Miki
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: When creating a nested dictionary of dataframes, how can I name a dictionary based on a list name of the dataframe?
Thank you for the help! Based on your response/recommendation, I am
thinking that my entire approach to solving my problem is very poor. I am
trying to pull slices from a dataframe, store them in a nested dictionary,
retrieve them, perform calculations, store the results in the same nested
dictionary (hence why I need to know the actual name of each nested
dictionary), retrieve the results, then format the results. LOL that
looks so bad when I think about it out loud. What I think I should be
doing is pull my first slice from a dataframe, perform my calculations,
then format those results in my desired final output; take my second,
third, fourth, etc. slices and then just repeat that same pattern instead
of what I initially thought I should do. Is there a standard approach for
evaluating particular slices of dataframes or does it always depend on the
situation?
On Sun, Jun 7, 2020 at 4:39 AM Peter Otten <__pete...@web.de> wrote:
> Aaron wrote:
>
> > When creating a nested dictionary of dataframes, how can I name a
> > dictionary based on a list name of the dataframe?
> >
> > Given the following:
> >
> > # START CODE
> > import pandas as pd
> >
> > cars = {'Brand': ['Honda Civic','Toyota Corolla'],
> > 'Price': [22000,25000]
> > }
> > df_cars = pd.DataFrame(cars, columns = ['Brand','Price'])
> >
> > trucks = {'Brand': ['GMC Sierra','Ford F-150'],
> > 'Price': [5,48000]
> > }
> > df_trucks = pd.DataFrame(trucks, columns = ['Brand','Price'])
> >
> > list_of_dfs = [df_cars, df_trucks]
> >
> > # Not exactly sure how this code should be:
> > dict_of_dfs = {}
> > for df in list_of_dfs:
> > dict_of_dfs[name_of_df] = {} # Not sure here
> > dict_of_dfs[name_of_df]['results'] = df # Not sure here
> > # END CODE
>
> In the general case you can't find the name of a value, so you may start
> with the names and look up the dataframes in the global namespace:
>
> list_of_df_names = ["df_cars", "df_trucks"]
> dict_of_dfs = {
>name: {"results": globals()[name]}
>for name in list_of_df_names
> }
>
> Personally I would probably reorganize your code a bit and avoid the
> df_...
> names:
>
> import pandas as pd
>
> vehicles = dict(
> cars={
> 'Brand': ['Honda Civic','Toyota Corolla'],
> 'Price': [22000,25000]
> },
> trucks={
> 'Brand': ['GMC Sierra','Ford F-150'],
> 'Price': [5,48000]
> }
> )
>
> dict_of_dfs = {
> name: {"results": pd.DataFrame(val)}
> for name, val in vehicles.items()
> }
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Button press event - event handling and picking: IndexError: list index out of range
On 8/06/20 7:06 AM, Caledonian26 wrote: ... However, I keep getting the error: IndexError: list index out of range. Could anyone give me a helping hand as to where I am going wrong? When things go wrong, Python tries to be helpful by providing a "traceback". Please copy-paste the entire traceback, to help us help you. The error-message says that (somewhere) a list is being used *but* the index is pointing to an element that does not exist, eg the fifth element of a list that contains only three members. Short answer: What you could do, is add a temporary print() to show you the exact value of the list index. Thus, giving you a pointer/where to back-track, to find the source of the problem... The longer answer: learning to use a debugger or a 'visual' IDE/web-tool. -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Button press event - event handling and picking: IndexError: list index out of range
On 2020-06-07 23:24, DL Neil via Python-list wrote: On 8/06/20 7:06 AM, Caledonian26 wrote: ... However, I keep getting the error: IndexError: list index out of range. Could anyone give me a helping hand as to where I am going wrong? When things go wrong, Python tries to be helpful by providing a "traceback". Please copy-paste the entire traceback, to help us help you. The error-message says that (somewhere) a list is being used *but* the index is pointing to an element that does not exist, eg the fifth element of a list that contains only three members. Short answer: What you could do, is add a temporary print() to show you the exact value of the list index. Thus, giving you a pointer/where to back-track, to find the source of the problem... The longer answer: learning to use a debugger or a 'visual' IDE/web-tool. FYI, that's the same good advice that ChrisA gave yesterday in reply to the same post. See the thread "Applied Data Science with Python - Assignment 2.3: clicking on chart to select Y values". -- https://mail.python.org/mailman/listinfo/python-list
Re: Button press event - event handling and picking: IndexError: list index out of range
On 8/06/20 10:38 AM, MRAB wrote: On 2020-06-07 23:24, DL Neil via Python-list wrote: On 8/06/20 7:06 AM, Caledonian26 wrote: ... However, I keep getting the error: IndexError: list index out of range. Could anyone give me a helping hand as to where I am going wrong? When things go wrong, Python tries to be helpful by providing a "traceback". Please copy-paste the entire traceback, to help us help you. The error-message says that (somewhere) a list is being used *but* the index is pointing to an element that does not exist, eg the fifth element of a list that contains only three members. Short answer: What you could do, is add a temporary print() to show you the exact value of the list index. Thus, giving you a pointer/where to back-track, to find the source of the problem... The longer answer: learning to use a debugger or a 'visual' IDE/web-tool. FYI, that's the same good advice that ChrisA gave yesterday in reply to the same post. See the thread "Applied Data Science with Python - Assignment 2.3: clicking on chart to select Y values". Apologies @MRAB, didn't realise the duplication. I think I came to that thread late, noticed @Chris (and yourself?) had responded, deleted, and moved-on. -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
On 6/7/20 2:25 PM, Barry wrote: >> Does ctypes, when using restype, frees allocated memory? >> >> For example, will the memory allocated by "strdup" be freed after the "del" >> statement? If not, how can I free it? > > See https://linux.die.net/man/3/strdup that tells you to use free() to delete > memory allocated by strdup. > > You must remember the result of strdup and free it at an appropriate time. But the problem is that by specifying the type as ctypes.c_char_p, ctypes will hide that pointer from you and return a Python object instead. I'm not sure how ctypes is doing it under the hood, but I suspect ctypes is doing it's own strdup of the string on conversion, and managing that memory, but the original pointer ends up being lost and leaking memory. The stack exchange link I posted suggests that have ctypes give you a void * pointer, and then cast that to give you a python string while still having the original pointer to free. Is this correct? -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
Hi,
> But the problem is that by specifying the type as ctypes.c_char_p,
> ctypes will hide that pointer from you and return a Python object
> instead. I'm not sure how ctypes is doing it under the hood, but I
> suspect ctypes is doing it's own strdup of the string on conversion, and
> managing that memory, but the original pointer ends up being lost and
> leaking memory.
Yes, the problem with my code I posted is that resource.getrusage report the
Python allocated memory. Once I've changed to
psutil.Process().memory_info().rss I see the memory leak. The code now looks
like:
---
...
strdup.restype = ctypes.c_void_p
free = libc.free
free.argtypes = [ctypes.c_void_p]
...
for _ in range(n):
o = strdup(data)
s = ctypes.string_at(o).decode('utf-8')
free(o)
---
This seems to fix the leak.
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
