Re: tree representation of Python data

What is the robust way to use Python to read in an XML and turn it into a JSON file? JSON dictionary is actually a tree. It is much easier to manage the tree-structured data. Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Missing global # gdal DRIVER_NAME declaration in gdal_array.py

The following warning kept coming up when running ogr2ogr. Warning 1: Missing global # gdal: DRIVER_NAME declaration in C:\Users\AppData\Local\Programs\Python\Python36\Lib\site-packages\osgeo\gdal_array.py What steps to be take to resolve this issue? Regards, David -- https://mail.python.org/m

Issues of pip install gdal and fiona

I downloaded .whl files for fiona and gdal to go with Python3.6.5. However, I am having trouble with red error messages. Though Gdal is now working, there is a warning message - Missing global ~ gdal: DRIVER_NAME declaration gdal_array,py Can anyone advise on how to resolve the issues? Regard

Re: Re: Long running process - how to speed up?

On Sat, 19 Feb 2022 at 18:51, Alan Gauld wrote: > On 19/02/2022 11:28, Shaozhong SHI wrote: > > > I have a cvs file of 932956 row > > That's not a lot in modern computing terms. > > > and have to have time.sleep in a Python > > script. > > Why? Is it

Re: Long running process - how to speed up?

On Sat, 19 Feb 2022 at 19:44, Mats Wichmann wrote: > On 2/19/22 05:09, Shaozhong SHI wrote: > > Can it be divided into several processes? > > Regards, > > David > > The answer is: "maybe". Multiprocessing doesn't happen for free, you > have to fi

Re: Long running process - how to speed up?

Can it be divided into several processes? Regards, David On Saturday, 19 February 2022, Chris Angelico wrote: > On Sat, 19 Feb 2022 at 22:59, Karsten Hilbert > wrote: > > > > > > I have a cvs file of 932956 row and have to have time.sleep in a > Python > > > > script. It takes a long time to p

Long running process - how to speed up?

I have a cvs file of 932956 row and have to have time.sleep in a Python script. It takes a long time to process. How can I speed up the processing? Can I do multi-processing? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

URLError:

The following is used in a loop to get response code for each url. print (urllib.request.urlopen(url).getcode()) However, error message says: URLError: Python 3.6.5 is being used to test whether url is live or not. Can anyone shed light on this? Regards, David -- https://mail.python.org/mai

Re: How to set environmental variables for Python

Set Operation System but not disturbing existing setting. Only to add at the command line. Regards, David On Mon, 17 Jan 2022 at 10:57, dn via Python-list wrote: > On 17/01/2022 22.31, Shaozhong SHI wrote: > > I got quite a few version of Python on my machine. > > &

How to set environmental variables for Python

I got quite a few version of Python on my machine. How do I set environmental variables for Python 3.6.1 to work? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Can Python call and use FME modules and functions such as StreamOrderCalculator?

Can we do something like import an fme.something and make use of FME modules and functions? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

ogr2ogr can not open gfs file when loading GML

My command line kept telling me that it ogr2ogr can not open gfs file. It does find it. I was trying to load GML onto PostGIS. Alternatively, how to specify XSD file to go along with reading GML files? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Installation of GeoPandas - failed at fiona

I am trying to install geopandas. I navigated to c:\programData|Anaconda3\Scripts> and typed in 'pip install geopandas'. It ran but failed at fiona. I tried import geopandas as gp, but Error Message says: No module names 'geopandas'. Can anyone help? Regards, David -- https://mail.python.org

Re: System, configuration and Python performance

On Tue, 2 Nov 2021 at 00:20, Shaozhong SHI wrote: > > > On Tue, 2 Nov 2021 at 00:09, MRAB wrote: > >> On 2021-11-01 23:02, Shaozhong SHI wrote: >> > How to configure to improve Python performance in a system like the >> > following: >> > >>

Re: System, configuration and Python performance

On Tue, 2 Nov 2021 at 00:09, MRAB wrote: > On 2021-11-01 23:02, Shaozhong SHI wrote: > > How to configure to improve Python performance in a system like the > > following: > > > > Windows 10 > > > > System > > > > Processor Intel(R) Core(TM)

System, configuration and Python performance

How to configure to improve Python performance in a system like the following: Windows 10 System Processor Intel(R) Core(TM) i7-9700 CPU @3.60GHz, 3.60 GHz Installed memory (RAM) 32.0 GB (31.8 GB usable) System type: 64-bit Operating System, x64-based processor I found that the Python script wa

Re: How to apply a self defined function in Pandas

On Sun, 31 Oct 2021 at 18:42, Shaozhong SHI wrote: > > > On Sunday, 31 October 2021, Albert-Jan Roskam > wrote: > >> >> >> > df['URL'] = df.apply(lambda x: connect(df['URL']), axis=1) >> >> >> I think you need ax

Re: How to apply a self defined function in Pandas

On Sun, 31 Oct 2021 at 19:28, MRAB wrote: > On 2021-10-31 18:48, Shaozhong SHI wrote: > > > > On Sunday, 31 October 2021, MRAB wrote: > > > > On 2021-10-31 17:25, Shaozhong SHI wrote: > > > > I defined a function and apply it to a column

Re: How to apply a self defined function in Pandas

On Sunday, 31 October 2021, MRAB wrote: > On 2021-10-31 17:25, Shaozhong SHI wrote: > >> I defined a function and apply it to a column in Pandas. But it does not >> return correct values. >> >> I am trying to test which url in a column full of url to see which one

Re: How to apply a self defined function in Pandas

On Sunday, 31 October 2021, Albert-Jan Roskam wrote: > > > > df['URL'] = df.apply(lambda x: connect(df['URL']), axis=1) > > > I think you need axis=0. Or use the Series, df['URL'] = > df.URL.apply(connect) > Any details? I will try and let you know. Regards, David -- https://mail.python.org/ma

How to apply a self defined function in Pandas

I defined a function and apply it to a column in Pandas. But it does not return correct values. I am trying to test which url in a column full of url to see which one can be connected to or not def connect(url): try: urllib.request.urlopen(url) return True except:

Re: Python script seems to stop running when handling very large dataset

On Saturday, 30 October 2021, Dieter Maurer wrote: > Shaozhong SHI wrote at 2021-10-29 23:42 +0100: > >Python script works well, but seems to stop running at a certain point > when > >handling very large dataset. > > > >Can anyone shed light on this? > > Som

Python script seems to stop running when handling very large dataset

Python script works well, but seems to stop running at a certain point when handling very large dataset. Can anyone shed light on this? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

How to store the result of df.count() as a new dataframe in Pandas?

Hello, The result of df.count() appears to be a series object. How to store the result of df.count() as a new dataframe in Pandas? That is data anyhow. Regards, David -- https://mail.python.org/mailman/listinfo/python-list

df.count() to a Pandas dataframe with column names

How to output the result of df.count() to a Pandas dataframe with column names? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Alternatives to Jupyter Notebook

Hello, Is anyone familiar with alternatives to Jupyter Notebook. My Jupyter notebook becomes unresponsive in browsers. Are there alternatives to read, edit and run Jupyter Notebook? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

SQLAlchemy fault

I read a txt file into Pandas Dataframe, and found a lot of nulls in a column. Then, I used SQLAlchemy and psycopg2. I created engine. I loaded data onto PostgreSQL. Strange thing happened. The column has no null at all. Does it mean that the data has been modified somewhere along the line?

Connecting to MS accdb and read data into Pandas

I tried the following code: import pyodbc conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=D:\my.accdb;') cursor = conn.cursor() cursor.execute('select * from table_name') for row in cursor.fetchall(): print (row) But I could not connect to .accdb. What is the

Re: Definitive guide for Regex

? Regards, David On Thu, 30 Sept 2021 at 22:02, Barry Scott wrote: > > > > On 30 Sep 2021, at 19:35, dn via Python-list > wrote: > > > > On 01/10/2021 06.16, Barry Scott wrote: > >> > >> > >>> On 30 Sep 2021, at 12:29, Shaozhong SHI >

Definitive guide for Regex

Dear All, I am trying to look for a definitive guide for Regex in Python. Can anyone help? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Automated data testing, checking, validation, reporting for data assurance

There appear to be a few options for this. Has anyone tested and got experience with automated data testing, validation and reporting? Can anyone enlighten me? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Observing long running processes of Jupyter Notebook

We have been running Jupyter Notebook processes, which take long time to run. We use nbconvert to run these in commandline. Nbconvert only writes output into a file at the end. We just wonder whether there is a way to observe the progress and printing messages when nbconvert is running. Regards

How to run Jupyter notebook in command line and get full error message?

How to run Jupyter notebook in command line and get full error messages? My VPN keeps dropping and can not run Jupyter Notebook as it is. I started to use nbconvert in command line. But, when it stops due to error, I can not see where the error occurs. In order to make life easier for debugging

How to record full error message for debugging with nbconvert?

Hi, When I use nbconvert to run Jupyter notebook, it is so difficult to see the full error message for debugging? How to save full error messages? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

ssl connection has been closed unexpectedly

Hi, I keep getting the following error when I use engine = create_engine(logging in details to postgres) df.to_sql('table_name', and etc.) OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly (Background on this error at: http://sqlalche.me/e/13/e3q8) Operat

Re: Questions about XML processing?

Hi, Hernan, Did you try to parse GML? Surely, there can be very concise and smart ways to do these things. Regards, David On Fri, 6 Nov 2020 at 20:57, Hernán De Angelis wrote: > Thank you Terry, Dan and Dieter for encouraging me to post here. I have > already solved the problem albeit with a

Dataframe to postgresql - Saving the dataframe to memory using StringIO

I found this last option is very interesting. Saving the dataframe to memory using StringIO https://naysan.ca/2020/06/21/pandas-to-postgresql-using-psycopg2-copy_from/ But, testing shows unicode argument expected, got 'str' Any working example for getting DataFrame into a PostgreSQL table direc

How to write differently to remove this type hint in Python 2.7?

Is there another way to do this? def greet(name: str) -> str: return "Hello, " + name greet File "", line 1 def greet(name: str) -> str: ^ SyntaxError: invalid syntax -- https://mail.python.org/mailman/listinfo/python-list

How to expand and flatten a nested of list of dictionaries of varied lengths?

Even worse is that, in some cases, an addition called serviceRatings as a key occur with new data unexpectedly. How to produce a robust Python/Panda script to coping with all these? Regards, David u'historicRatings': [{u'overall': {u'keyQuestionRatings': [{u'name': u'Safe', u'rating': u'Require

Are there Python ways to execute queries on PostgreSQL without getting data over?

Are there Python ways to execute queries on PostgreSQL without getting data over? Are there ways just to fire off PostgreSQL queries and not get data into Python? Regards, David -- https://mail.python.org/mailman/listinfo/python-list

Re: ValueError: arrays must all be same length

tten table? Apparently, the nested data is another table. Regards, Shao On Sun, 4 Oct 2020 at 13:55, Tim Williams wrote: > On Sun, Oct 4, 2020 at 8:39 AM Tim Williams wrote: > > > > > > > On Fri, Oct 2, 2020 at 11:00 AM Shaozhong SHI > > wrote: > >

How to handle a dictionary value that is a list

Hi, All, I was trying to handle the value of "personRoles" in a part of json dictionary. Can anyone tell me various ways to handle this? Regards, Shao "regulatedActivities": [ { "name": "Accommodation for persons who require nursing or personal care", "code": "RA2", "con

ValueError: arrays must all be same length

Hello, I got a json response from an API and tried to use pandas to put data into a dataframe. However, I kept getting this ValueError: arrays must all be same length. Can anyone help? The following is the json text. Regards, Shao { "locationId": "1-1004508435", "providerId": "1-101641521