Problem with concatenating two dataframes

2021-11-06 Thread Mahmood Naderan via Python-list
In the following code, I am trying to create some key-value pairs in a dictionary where the first element is a name and the second element is a dataframe. # Creating a dictionary data = {'Value':[0,0,0]} kernel_df = pd.DataFrame(data, index=['M1','M2','M3']) dict = {'dummy':kernel_df} # dummy  -

Re: Problem with concatenating two dataframes

2021-11-06 Thread Mahmood Naderan via Python-list
>Try this instead: > > >    dict[name] = pd.concat([dict[name], values]) OK. That fixed the problem, however, I see that they are concatenated vertically. How can I change that to horizontal? The printed dictionary in the end looks like {'dummy': Value M1  0 M2  0 M3  0, 'K1':

Re: Problem with concatenating two dataframes

2021-11-06 Thread Mahmood Naderan via Python-list
>The second argument of pd.concat is 'axis', which defaults to 0. Try >using 1 instead of 0. Unfortunately, that doesn't help... dict[name] = pd.concat( [dict[name],values], axis=1 ) {'dummy': Value M1  0 M2  0 M3  0, 'K1':Value  Value 0   10.0NaN 15.0NaN 2  

Returning the index of a row in dataframe

2021-11-14 Thread Mahmood Naderan via Python-list
Hi In the following dataframe, I want to get the index string by specifying the row number which is the same as value column. Value     global loads   0     global stores  1     local loads    2 For example, `df.iloc[1].index.name` should return "global store

Re: Returning the index of a row in dataframe

2021-11-14 Thread Mahmood Naderan via Python-list
>>> df.iloc[1].name Correct I also see that 'df.index[1]' works fine. Thanks. Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list

Using astype(int) for strings with thousand separator

2021-11-14 Thread Mahmood Naderan via Python-list
Hi While reading a csv file, some cells have values like '1,024' which I mean they contains thousand separator ','. Therefore, when I want to process them with   row = df.iloc[0].astype(int) I get the following error   ValueError: invalid literal for int() with base 10: '1,024' How can I fi

Re: Using astype(int) for strings with thousand separator

2021-11-15 Thread Mahmood Naderan via Python-list
> (see > https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) Got it. Thanks. Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list

get_axes not present?

2021-11-18 Thread Mahmood Naderan via Python-list
Hi I am using the following versions >>> import matplotlib >>> print(matplotlib. __version__) 3.3.4 >>> import pandas as pd >>> print(pd.__version__) 1.2.3 >>> import sys >>> sys.version_info sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) In my code, I use axes in

Re: get_axes not present?

2021-11-18 Thread Mahmood Naderan via Python-list
>It's not saying get_axes doesn't exist because of version skew, it's >saying that the object returned by the call to the left of it >(get_figure()) returned None, and None doesn't have methods > >Something isn't set up right, but you'll have to trace that through. Do you think the following

Re: get_axes not present?

2021-11-19 Thread Mahmood Naderan via Python-list
>And what is the result of plot()?  Is it a valid object, or is it None? Well the error happens on the plot() line. I tried to print some information like this:     print("axes=", axes)     print("axes[0]=", axes[0])     print("cnt=", cnt)     print("row=", row)     ax1 = row.plot( fontsize=fon

Re: get_axes not present?

2021-11-21 Thread Mahmood Naderan via Python-list
>The best way to get >assistance here on the list is to create a minimal, self-contained, >run-able, example program that you can post in its entirety here that >demonstrates the issue. I created a sample code with input. Since the code processes a csv file to group input rows, I also included t

Re: get_axes not present?

2021-11-21 Thread Mahmood Naderan via Python-list
>I installed the latest pandas, although on Python 3.10, and the script >worked without a problem. Yes as I wrote it works with 1.3.3 but mine is 1.2.3. I am trying to keep the current version because of the possible future consequences. In the end maybe I have to upgrade the pandas. Regards,

Re: get_axes not present?

2021-11-21 Thread Mahmood Naderan via Python-list
>Your example isn't minimal enough for me to be able to pin it down any >better than that, though. Chris, I was able to simply it even further. Please look at this: $ cat test.batch.csv Value,Value 10,2 5,2 10,2 $ cat test.py import pandas as pd import csv,sys import matplotlib import matplot

About get_axes() in Pandas 1.2.3

2021-11-22 Thread Mahmood Naderan via Python-list
Hi I asked a question some days ago, but due to the lack of minimal producing code, the topic got a bit messy. So, I have decided to ask it in a new topic with a clear minimum code. With Pandas 1.2.3 and Matplotlib 3.3.4, the following plot() functions returns error and I don't know what is wr

Re: About get_axes() in Pandas 1.2.3

2021-11-22 Thread Mahmood Naderan via Python-list
>I can help you narrow it down a bit. The problem actually occurs inside >this function call somehow. You can verify this by doing this: > > >fig,axes = plt.subplots(2,1, figsize=(20, 15)) > >print ("axes[0].get_figure()=",axes[0].get_figure()) > >You'll find that get_figure() is returning None, wh

Extracting dataframe column with multiple conditions on row values

2022-01-07 Thread Mahmood Naderan via Python-list
Hi I have a csv file like this V0,V1,V2,V3 4,1,1,1 6,4,5,2 2,3,6,7 And I want to search two rows for a match and find the column. For example, I want to search row[0] for 1 and row[1] for 5. The corresponding column is V2 (which is the third column). Then I

Writing a string with comma in one column of CSV file

2022-01-15 Thread Mahmood Naderan via Python-list
Hi, I use the following line to write some information to a CSV file which is comma delimited. f = open(output_file, 'w', newline='') wr = csv.writer(f) ... f.write(str(n) + "," + str(key) + "\n" ) Problem is that key is a string which may contain ',' and this causes the final CSV file to have

Re: Writing a string with comma in one column of CSV file

2022-01-15 Thread Mahmood Naderan via Python-list
Right. I was also able to put all columns in a string and then use writerow(). Thanks. Regards, Mahmood On Saturday, January 15, 2022, 10:33:08 PM GMT+1, alister via Python-list wrote: On Sat, 15 Jan 2022 20:56:22 + (UTC), Mahmood Naderan wrote: > Hi, > I use the following lin

Unable to install "collect" via pip3

2019-12-20 Thread Mahmood Naderan via Python-list
Hi I can install collect with pip for python2.7 $ pip install --user collect Collecting collect Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz Collecting couc

Re: Unable to install "collect" via pip3

2019-12-22 Thread Mahmood Naderan via Python-list
Yes thank you. The package is not compatible with 3.x. Regards, Mahmood On Saturday, December 21, 2019, 1:40:29 AM GMT+3:30, Barry wrote: > On 20 Dec 2019, at 15:27, Mahmood Naderan via Python-list > wrote: > > Hi > > I can install collect with pip for

Grepping words for match in a file

2019-12-28 Thread Mahmood Naderan via Python-list
Hi I have some lines in a text file like ADD R1, R2 ADD3 R4, R5, R6 ADD.MOV R1, R2, [0x10] If I grep words with this code for line in fp: if my_word in line: Then if my_word is "ADD", I get 3 matches. However, if I grep word with this code for line in fp: for word in line.split():

Install python via MS batch file

2017-05-06 Thread Mahmood Naderan via Python-list
Hello, I have downloaded python-3.6.1-amd64.exe and it is fine to install it through GUI. However, I want to write a batch file to install it via command line. Since the installation process is interactive, it seems that the auto-install batch file is difficult. What I want to do is: set python

packaging python code

2017-05-08 Thread Mahmood Naderan via Python-list
Hi, I have simple piece of code which uses two libraries (numpy and openpyxl). The script is called from another application. Currently, if someone wants to run my program, he has to first install the python completely via its installer. Is there any way to pack my .py with all required librarie

Re: packaging python code

2017-05-08 Thread Mahmood Naderan via Python-list
OK. I did that but it fails! Please see the stack D:\ThinkPad\Documents\NetBeansProjects\ExcelTest>pyinstaller exread.py 96 INFO: PyInstaller: 3.2.1 96 INFO: Python: 3.6.1 98 INFO: Platform: Windows-10-10.0.14393-SP0 103 INFO: wrote D:\ThinkPad\Documents\NetBeansProjects\ExcelTest\exread.spec

Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hello, The following code which uses openpyxl and numpy, fails to read large Excel (xlsx) files. The file si 20Mb which contains 100K rows and 50 columns. W = load_workbook(fname, read_only = True) p = W.worksheets[0] a=[] m = p.max_row n = p.max_column np.array([[i.value for i in j] for

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
, May 10, 2017 7:25 PM, Peter Otten <__pete...@web.de> wrote: Mahmood Naderan via Python-list wrote: > Hello, > > The following code which uses openpyxl and numpy, fails to read large > Excel (xlsx) files. The file si 20Mb which contains 100K rows and 50 > colu

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
n-list@python.org Date: Wednesday, May 10, 2017, 3:48 PM Mahmood Naderan via Python-list wrote: > Thanks for your reply. The openpyxl part (reading the workbook) works > fine. I printed some debug information and found that when it reaches the > np.array, after some 10 seconds, t

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
On Wed, 5/10/17, Peter Otten <__pete...@web.de> wrote: Subject: Re: Out of memory while reading excel file To: python-list@python.org Date: Wednesday, May 10, 2017, 6:30 PM Mahmood Naderan via Python-list wrote: > Well actually cells are treated as strings and not integer

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hi, I am confused with that. If you say that numpy is not suitable for my case and may have large overhead, what is the alternative then? Do you mean that numpy is a good choice here while we can reduce its overhead? Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
>a = numpy.zeros((ws.max_row, ws.max_column), dtype=float) >for y, row in enumerate(ws.rows): > a[y] = [cell.value for cell in row] Peter, As I used this code, it gave me an error that cannot convert string to float for the first cell. All cells are strings. Regards, Mahmood -- https://

Re: Out of memory while reading excel file

2017-05-10 Thread Mahmood Naderan via Python-list
Hi, I used the old fashion coding style to create a matrix and read/add the cells. W = load_workbook(fname, read_only = True) p = W.worksheets[0] m = p.max_row n = p.max_column arr = np.empty((m, n), dtype=object) for r in range(1, m): for c in range(1, n): d = p.cell(row=r, colu

Re: Out of memory while reading excel file

2017-05-11 Thread Mahmood Naderan via Python-list
I wrote this: a = np.zeros((p.max_row, p.max_column), dtype=object) for y, row in enumerate(p.rows): for cell in row: print (cell.value) a[y] = cell.value print (a[y]) For one of the cells, I see NM_198576.3 ['NM_198576.3' 'NM_198576.3' 'NM_198576.3' 'NM_1985

Re: Out of memory while reading excel file

2017-05-11 Thread Mahmood Naderan via Python-list
Thanks. That code is so simple and works. However, there are things to be considered. With the CSV format, cells in a row are separated by ',' and for some cells it writes "" around the cell content. So, if the excel looks like CHR1 11,232,445 The output file looks like CHR1,"11,232,4

Re: Out of memory while reading excel file

2017-05-11 Thread Mahmood Naderan via Python-list
Excuse me, I changed csv.writer(outstream) to csv.writer(outstream, delimiter =' ') It puts space between cells and omits "" around some content. However, between two lines there is a new empty line. In other word, the first line is the first row of excel file. The second line is empty ("\

Re: Out of memory while reading excel file

2017-05-11 Thread Mahmood Naderan via Python-list
Thanks a lot for suggestions. It is now solved. Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list

Concatenating files in order

2017-05-23 Thread Mahmood Naderan via Python-list
Hi, There are some text files ending with _chunk_i where 'i' is an integer. For example, XXX_chunk_0 XXX_chunk_1 ... I want to concatenate them in order. Thing is that the total number of files may be variable. Therefore, I can not specify the number in my python script. It has to be "for all

Re: Concatenating files in order

2017-05-23 Thread Mahmood Naderan via Python-list
>Yup. Make a list of all the file names, write a key function that >extracts the numbery bits, sort the list based on that key function, and >go to town. > >Alternatively, when you create the files in the first place, make sure >to use more leading zeros than you could possibly need. >xxx_chun

Re: Concatenating files in order

2017-05-23 Thread Mahmood Naderan via Python-list
OK guys thank you very much. It is better to sort them first. Here is what I wrote files = glob.glob('*chunk*') sorted=[[int(name.split("_")[-1]), name] for name in files] with open('final.txt', 'w') as outf: for fname in sorted: with open(fname[1]) as inf: for line in

Re: Concatenating files in order

2017-05-25 Thread Mahmood Naderan via Python-list
Hi guys, Cameron, thanks for the points. In fact the file name contains multiple '_' characters. So, I appreciate what you recommended. filenames = {} for name in glob.glob('*chunk_*'): left, right = name.rsplit('_', 1) if left.endswith('chunk') and right.isdigit(): fi

Re: Concatenating files in order

2017-05-26 Thread Mahmood Naderan via Python-list
Thank you very much. I understand that Regards, Mahmood On Friday, May 26, 2017 5:01 AM, Cameron Simpson wrote: On 25May2017 20:37, Mahmood Naderan wrote: >Cameron, thanks for the points. In fact the file name contains multiple '_' >characters. So, I appreciate what you recommended. > >

embed a package for proper fun script

2017-05-29 Thread Mahmood Naderan via Python-list
Hello, How it is possible to embed a package in my project? I mean, in my python script I have written import openpyxl So, the user may not have installed that package and doesn't understand what is pip! Please let me know the instructions or any document regarding that. Regards, Mahmood --

Re: embed a package for proper fun script

2017-05-30 Thread Mahmood Naderan via Python-list
No idea?... Regards, Mahmood On Tuesday, May 30, 2017 1:06 AM, Mahmood Naderan via Python-list wrote: Hello, How it is possible to embed a package in my project? I mean, in my python script I have written import openpyxl So, the user may not have installed that package and doesn&#

Python not able to find package but it is installed

2017-05-30 Thread Mahmood Naderan via Python-list
Hello, Although I have installed a package via pip on a centos-6.6, python interpreter still says there is no such package! Please see the output below $ python exread2.py input.xlsx tmp/output Traceback (most recent call last): File "/home/mahmood/excetest/exread2.py", line 1, in from openpyxl

Re: embed a package for proper fun script

2017-05-30 Thread Mahmood Naderan via Python-list
Thanks. I will try and come back later. Regards, Mahmood On Tuesday, May 30, 2017 2:03 PM, Paul Moore wrote: On Tuesday, 30 May 2017 08:48:34 UTC+1, Mahmood Naderan wrote: > No idea?... > > > Regards, > Mahmood > > > On Tuesday, May 30, 2017 1:06

Re: Python not able to find package but it is installed

2017-05-30 Thread Mahmood Naderan via Python-list
Well, on rocks there exist multiple pythons. But by default the active is 2.6.6 $ python -V Python 2.6.6 I have to say that the script doesn't modify sys.path. I only use sys.argv[] there I can put all dependent modules in my project folder but that will be dirty. Regards, Mahmood On Tues

Re: Python not able to find package but it is installed

2017-05-30 Thread Mahmood Naderan via Python-list
Well yes. It looks in other folders >>> import openpyxl # trying openpyxl.so # trying openpyxlmodule.so # trying openpyxl.py # trying openpyxl.pyc # trying /usr/lib64/python2.6/openpyxl.so # trying /usr/lib64/python2.6/openpyxlmodule.so # trying /usr/lib64/python2.6/openpyxl.py # trying /usr/lib64

Re: Python not able to find package but it is installed

2017-05-31 Thread Mahmood Naderan via Python-list
Consider this output [root@cluster ~]# pip --version pip 9.0.1 from /opt/rocks/lib/python2.6/site-packages/pip-9.0.1-py2.6.egg (python 2.6) [root@cluster ~]# easy_install --version distribute 0.6.10 [root@cluster ~]# find /opt -name python /opt/rocks/lib/graphviz/python /opt/rocks/bin/python /opt

openpyxl reads cell with format

2017-06-05 Thread Mahmood Naderan via Python-list
Hello guys... With openpyxl, it seems that when the content of a cell is something like "4-Feb", then it is read as "2016-02-04 00:00:00" that looks like a calendar conversion. How can I read the cell as text instead of such an automatic conversion? Regards, Mahmood -- https://mail.python.org/

Openpyxl cell format

2017-06-05 Thread Mahmood Naderan via Python-list
Hello guys, With openpyxl, it seems that when the content of a cell is something like "4-Feb", then it is read as "2016-02-04 00:00:00" that looks like a calendar conversion. How can I read the cell as text instead of such an automatic conversion? Regards, Mahmood -- https://mail.python.org/m

Re: Openpyxl cell format

2017-06-05 Thread Mahmood Naderan via Python-list
Maybe... But specifically in my case, the excel file is exported from a web page. I think there should be a way to read the content as a pure text. Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list

Re: openpyxl reads cell with format

2017-06-05 Thread Mahmood Naderan via Python-list
>if the cell is an Excel date, it IS stored as a numeric As I said, the "shape" of the cell is similar to date. The content which is "4-Feb" is not a date. It is a string which I expect from cell.value to read it as "4-Feb" and nothing else. Also, I said before that the file is downloaded from

Re: openpyxl reads cell with format

2017-06-05 Thread Mahmood Naderan via Python-list
OK thank you very much. As you said, it seems that it is too late for my python script. Regards, Mahmood On Monday, June 5, 2017 10:41 PM, Dennis Lee Bieber wrote: On Mon, 5 Jun 2017 14:46:18 + (UTC), Mahmood Naderan via Python-list declaimed the following: >>if the cell