Re: Python 2.7 range Function provokes a Memory Error

2023-03-06 Thread Chris Angelico
On Tue, 7 Mar 2023 at 16:53, Stephen Tucker wrote: > > Hi again, > > I tried xrange, but I got an error telling me that my integer was too big > for a C long. > > Clearly, xrange in Py2 is not capable of dealing with Python (that is, > possibly very long) integers. That's because Py2 has two diff

Re: Python 2.7 range Function provokes a Memory Error

2023-03-06 Thread Stephen Tucker
Hi again, I tried xrange, but I got an error telling me that my integer was too big for a C long. Clearly, xrange in Py2 is not capable of dealing with Python (that is, possibly very long) integers. I am raising this because, (a) IF xrange in Py3 is a simple "port" from Py2, then it won't handl

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread rbowman
On Mon, 6 Mar 2023 21:55:37 -0500, Dino wrote: > ne issue that was also correctly foreseen by some is that there's going > to be a new request at every user key stroke. Known problem. JavaScript > programmers use a trick called "debounceing" to be reasonably sure that > the user is done typing bef

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Dino
On 3/4/2023 10:43 PM, Dino wrote: I need fast text-search on a large (not huge, let's say 30k records totally) list of items. Here's a sample of my raw data (a list of US cars: model and make) Gentlemen, thanks a ton to everyone who offered to help (and did help!). I loved the part where so

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Greg Ewing via Python-list
On 7/03/23 6:49 am, avi.e.gr...@gmail.com wrote: But the example given wanted to match something like "V6" in middle of the text and I do not see how that would work as you would now need to search 26 dictionaries completely. It might even make things worse, as there is likely to be a lot of

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Greg Ewing via Python-list
On 7/03/23 4:35 am, Weatherby,Gerard wrote: If mailing space is a consideration, we could all help by keeping our replies short and to the point. Indeed. A thread or two of untrimmed quoted messages is probably more data than Dino posted! -- Greg -- https://mail.python.org/mailman/listinfo/p

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Thomas Passin
On 3/6/2023 12:49 PM, avi.e.gr...@gmail.com wrote: Thomas, I may have missed any discussion where the OP explained more about proposed usage. If the program is designed to load the full data once, never get updates except by re-reading some file, and then handles multiple requests, then some

RE: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread avi.e.gross
Ah, thanks Dino. Autocomplete within a web page can be an interesting scenario but also a daunting one. Now, do you mean you have a web page with a text field, initially I suppose empty, and the user types a single character and rapidly a drop-down list or something is created and shown? And as th

Re: Bug 3.11.x behavioral, open file buffers not flushed til file closed.

2023-03-06 Thread Dieter Maurer
aapost wrote at 2023-3-5 09:35 -0500: > ... >If a file is still open, even if all the operations on the file have >ceased for a time, the tail of the written operation data does not get >flushed to the file until close is issued and the file closes cleanly. This is normal: the buffer is flushed if

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Thomas Passin
On 3/6/2023 7:28 AM, Dino wrote: On 3/5/2023 9:05 PM, Thomas Passin wrote: I would probably ingest the data at startup into a dictionary - or perhaps several depending on your access patterns - and then you will only need to to a fast lookup in one or more dictionaries. If your access patte

RE: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread avi.e.gross
Thomas, I may have missed any discussion where the OP explained more about proposed usage. If the program is designed to load the full data once, never get updates except by re-reading some file, and then handles multiple requests, then some things may be worth doing. It looked to me, and I ma

RE: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread avi.e.gross
Gerard, I was politely pointing out how it was more than the minimum necessary and might gets repeated multiple times as people replied. The storage space is a resource someone else provides and I prefer not abusing it. However, since the OP seems to be asking a question focused on how long it ta

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Dino
On 3/5/2023 9:05 PM, Thomas Passin wrote: I would probably ingest the data at startup into a dictionary - or perhaps several depending on your access patterns - and then you will only need to to a fast lookup in one or more dictionaries. If your access pattern would be easier with SQL querie

Re: Cutting slices

2023-03-06 Thread Christian Gollwitzer
Am 05.03.23 um 23:43 schrieb Stefan Ram: The following behaviour of Python strikes me as being a bit "irregular". A user tries to chop of sections from a string, but does not use "split" because the separator might become more complicated so that a regular expression will be required

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread rbowman
On Mon, 6 Mar 2023 07:40:29 -0500, Dino wrote: > The idea that someone types into an input field and matches start > dancing in the browser made me think that this was exactly what I > needed, and hence I figured that asking here about Whoosh would be a > good idea. I know realize that Whoosh woul

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread rbowman
On Mon, 6 Mar 2023 15:32:09 +, Weatherby,Gerard wrote: > Increased performance may be achieved by building dictionaries “aa”,”ab” > ... “zz. And so on. Or a trie. There have been several implementations but I believe this is the most active: https://pypi.org/project/PyTrie/ -- https://mai

Re: Bug 3.11.x behavioral, open file buffers not flushed til file closed.

2023-03-06 Thread aapost
On 3/5/23 19:02, Cameron Simpson wrote: On 05Mar2023 10:38, aapost wrote: Additionally (not sure if this still applies): flush() does not necessarily write the file’s data to disk. Use flush() followed by os.fsync() to ensure this behavior. Yes. You almost _never_ need or want this behaviour

Re: RE: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Dino
Thank you for taking the time to write such a detailed answer, Avi. And apologies for not providing more info from the get go. What I am trying to achieve here is supporting autocomplete (no pun intended) in a web form field, hence the -i case insensitive example in my initial question. Your

Re: Bug 3.11.x behavioral, open file buffers not flushed til file closed.

2023-03-06 Thread aapost
On 3/5/23 09:35, aapost wrote: I have run in to this a few times and finally reproduced it. Whether it is as expected I am not sure since it is slightly on the user, but I can think of scenarios where this would be undesirable behavior.. This occurs on 3.11.1 and 3.11.2 using debian 12 testing,

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Dino
On 3/5/2023 1:19 AM, Greg Ewing wrote: I just did a similar test with your actual data and got about the same result. If that's fast enough for you, then you don't need to do anything fancy. thank you, Greg. That's what I am going to do in fact. -- https://mail.python.org/mailman/listinfo/pyth

Re: Bug 3.11.x behavioral, open file buffers not flushed til file closed.

2023-03-06 Thread Weatherby,Gerard
Add f.reconfigure it you want line buffering in your example: f = open("abc", "w") f.reconfigure(line_buffering=True) for i in range(5): f.write(str(i) + "\n") More Pythonic would be: with open("abc", "w") as f: for i in range(5000): print(i,file=f) From: Python-list on behalf

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Thomas Passin
On 3/6/2023 10:32 AM, Weatherby,Gerard wrote: Not sure if this is what Thomas meant, but I was also thinking dictionaries. Dino could build a set of dictionaries with keys “a” through “z” that contain data with those letters in them. (I’m assuming case insensitive search) and then just search

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Weatherby,Gerard
“Dino, Sending lots of data to an archived forum is not a great idea. I snipped most of it out below as not to replicate it.” Surely in 2023, storage is affordable enough there’s no need to criticize Dino for posting complete information. If mailing space is a consideration, we could all help by

Re: Fast full-text searching in Python (job for Whoosh?)

2023-03-06 Thread Weatherby,Gerard
Not sure if this is what Thomas meant, but I was also thinking dictionaries. Dino could build a set of dictionaries with keys “a” through “z” that contain data with those letters in them. (I’m assuming case insensitive search) and then just search “v” if that’s what the user starts with. Increa

Re: Bug 3.11.x behavioral, open file buffers not flushed til file closed.

2023-03-06 Thread Barry
> On 6 Mar 2023, at 01:42, Greg Ewing via Python-list > wrote: > > On 6/03/23 1:02 pm, Cameron Simpson wrote: >> Also, fsync() need not expedite the data getting to disc. It is equally >> valid that it just blocks your programme _until_ the data have gone to disc. > > Or until it *thinks* t