Thanks a lot, Edmondo. Or better... Grazie mille.
On 1/17/2023 5:42 AM, Edmondo Giovannozzi wrote:
Sorry,
I was just creating an array of 400x10 elements that I fill with random
numbers:
a = np.random.randn(400,100_000)
Then I pick one element randomly, it is just a stupid sort on a
Il giorno martedì 17 gennaio 2023 alle 00:18:04 UTC+1 Dino ha scritto:
> On 1/16/2023 1:18 PM, Edmondo Giovannozzi wrote:
> >
> > As a comparison with numpy. Given the following lines:
> >
> > import numpy as np
> > a = np.random.randn(400,100_000)
> > ia = np.argsort(a[0,:])
> > a_elem = a[
On Mon, 16 Jan 2023 12:28:37 -0500, Thomas Passin wrote:
> On 1/16/2023 11:56 AM, rbowman wrote:
>> On 16 Jan 2023 15:14:06 GMT, Stefan Ram wrote:
>>
>>
>>>When none of those reasons matter, one can use dictionaries in
>>>Python as well. And then what Chandler Carruth showed us applies:
On 1/16/2023 1:18 PM, Edmondo Giovannozzi wrote:
As a comparison with numpy. Given the following lines:
import numpy as np
a = np.random.randn(400,100_000)
ia = np.argsort(a[0,:])
a_elem = a[56, ia[0]]
I have just taken an element randomly in a numeric table of 400x10 elements
To find it w
On Jan 15, 2023 05:26, Dino wrote:
Hello, I have built a PoC service in Python Flask for my work, and - now
that the point is made - I need to make it a little more performant (to
be honest, chances are that someone else will pick up from where I left
off, and implement the
On 1/16/2023 11:56 AM, rbowman wrote:
On 16 Jan 2023 15:14:06 GMT, Stefan Ram wrote:
When none of those reasons matter, one can use dictionaries in Python
as well. And then what Chandler Carruth showed us applies:
I am missing something. Where is the data in your dictionary coming from
On 2023-01-15 18:06:36 -0500, Thomas Passin wrote:
> You especially want to avoid letting the database engine do full-table
> scans over and over. And you never want to send a lot of rows to
> Python and do post-filtering on them if you can avoid it.
Another thing to avoid: Lots of small queries.
On 2023-01-16 09:12:30 +1300, dn via Python-list wrote:
> On 16/01/2023 08.36, Weatherby,Gerard wrote:
> > I think any peformance improvements would have to come from a language
> > change or better indexing of the data.
> Expanding on @Peter's post: databases (relational or not) are best organise
Il giorno domenica 15 gennaio 2023 alle 05:26:50 UTC+1 Dino ha scritto:
> Hello, I have built a PoC service in Python Flask for my work, and - now
> that the point is made - I need to make it a little more performant (to
> be honest, chances are that someone else will pick up from where I left
>
On 1/16/2023 10:14 AM, Stefan Ram wrote:
However, operating systems and databases also try to cache
information in main memory that is estimated to be accessed
often.
Yes, and you can only know by testing, when that's possible. Also, if
you know that you have the same queries repeated over
On 16 Jan 2023 15:14:06 GMT, Stefan Ram wrote:
> When none of those reasons matter, one can use dictionaries in Python
> as well. And then what Chandler Carruth showed us applies:
I am missing something. Where is the data in your dictionary coming from?
--
https://mail.python.org/mailman/li
On 1/16/2023 2:53 AM, David wrote:
See here:
https://docs.python.org/3/reference/expressions.html#assignment-expressions
https://realpython.com/python-walrus-operator/
Thank you, brother.
--
https://mail.python.org/mailman/listinfo/python-list
Just wanted to take a moment to express my gratitude to everyone who
responded here. You have all been so incredibly helpful. Thank you
Dino
On 1/14/2023 11:26 PM, Dino wrote:
Hello, I have built a PoC service in Python Flask for my work, and - now
that the point is made - I need to make
On Mon, 16 Jan 2023 at 16:15, Dino wrote:
> BTW, can you tell me what is going on here? what's := ?
>
> while (increase := add_some(conn,adding)) == 0:
See here:
https://docs.python.org/3/reference/expressions.html#assignment-expressions
https://realpython.com/python-walrus-operator/
--
On 1/15/2023 2:23 PM, Weatherby,Gerard wrote:
That’s about what I got using a Python dictionary on random data on a high
memory machine.
https://github.com/Gerardwx/database_testing.git
It’s not obvious to me how to get it much faster than that.
Gerard, you are a rockstar. This is going to b
On 16/01/23 2:27 am, Dino wrote:
Do you have any idea about the speed of a SELECT query against a 100k
rows / 300 Mb Sqlite db?
That depends entirely on the nature of the query and how the
data is indexed. If it's indexed in a way that allows sqlite to
home in directly on the wanted data, it wi
On 1/15/2023 4:49 PM, Stefan Ram wrote:
dn writes:
Some programmers don't realise that SQL can also be used for
calculations, eg the eponymous COUNT(), which saves (CPU-time and
coding-effort) over post-processing in Python.
Yes, I second that! Sometimes, people only re-invent things
in
data that has to be sent back to the client.
From: Python-list on
behalf of Stefan Ram
Date: Sunday, January 15, 2023 at 5:03 PM
To: python-list@python.org
Subject: Re: Fast lookup of bulky "table"
*** Attention: This is an external email. Use caution responding, opening
atta
On Sun, 15 Jan 2023 08:27:29 -0500, Dino wrote:
> Do you have any idea about the speed of a SELECT query against a 100k
> rows / 300 Mb Sqlite db?
https://www.sqlite.org/speed.html
The site is old but has a number of comparisons. I have not used SQLite
with Python yet but with both C and C# I'
On 16/01/2023 08.36, Weatherby,Gerard wrote:
I think any peformance improvements would have to come from a language change
or better indexing of the data.
Exactly!
Expanding on @Peter's post: databases (relational or not) are best
organised according to use. Some must accept rapid insert/upd
On 1/15/2023 2:39 PM, Peter J. Holzer wrote:
On 2023-01-15 10:38:22 -0500, Thomas Passin wrote:
On 1/15/2023 6:14 AM, Peter J. Holzer wrote:
On 2023-01-14 23:26:27 -0500, Dino wrote:
Anyway, my Flask service initializes by loading a big "table" of 100k rows
and 40 columns or so (memory footpri
I think any peformance improvements would have to come from a language change
or better indexing of the data.
From: Python-list on
behalf of Weatherby,Gerard
Date: Sunday, January 15, 2023 at 2:25 PM
To: Dino , python-list@python.org
Subject: Re: Fast lookup of bulky "table"
Th
On 2023-01-15 10:38:22 -0500, Thomas Passin wrote:
> On 1/15/2023 6:14 AM, Peter J. Holzer wrote:
> > On 2023-01-14 23:26:27 -0500, Dino wrote:
> > > Anyway, my Flask service initializes by loading a big "table" of 100k rows
> > > and 40 columns or so (memory footprint: order of 300 Mb)
> >
> > 30
-list@python.org
Subject: Re: Fast lookup of bulky "table"
*** Attention: This is an external email. Use caution responding, opening
attachments or clicking on links. ***
Thank you for your answer, Lars. Just a clarification: I am already
doing a rough measuring of my queries.
A fresh que
Thank you, Peter. Yes, setting up my own indexes is more or less the
idea of the modular cache that I was considering. Seeing others think in
the same direction makes it look more viable.
About Scalene, thank you for the pointer. I'll do some research.
Do you have any idea about the speed o
Thank you for your answer, Lars. Just a clarification: I am already
doing a rough measuring of my queries.
A fresh query without any caching: < 4s.
Cached full query: < 5 micro-s (i.e. 6 orders of magnitude faster)
Desired speed for my POC: 10 Also, I didn't want to ask a question with way t
On 1/15/2023 6:14 AM, Peter J. Holzer wrote:
On 2023-01-14 23:26:27 -0500, Dino wrote:
Hello, I have built a PoC service in Python Flask for my work, and - now
that the point is made - I need to make it a little more performant (to be
honest, chances are that someone else will pick up from where
On 2023-01-14 23:26:27 -0500, Dino wrote:
> Hello, I have built a PoC service in Python Flask for my work, and - now
> that the point is made - I need to make it a little more performant (to be
> honest, chances are that someone else will pick up from where I left off,
> and implement the same serv
Hey,
before you start optimizing. I would suggest, that you measure response times and query
times, data search times and so on. In order to save time, you have to know where you
"loose" time.
Does your service really have to load the whole table at once? Yes that might
lead to quicker respon
Hello, I have built a PoC service in Python Flask for my work, and - now
that the point is made - I need to make it a little more performant (to
be honest, chances are that someone else will pick up from where I left
off, and implement the same service from scratch in a different language
(G
30 matches
Mail list logo
