Re: How to perform a long running dry run transaction without blocking

>> Anyway, thanks so much for your help. This discussion has been very useful, >> and I think I will proceed at first, exactly how you suggested, by queuing >> every validation job (using celery). Then I will explore whether or not I >> can apply the "on timeout" strategy in a small patch. >>

Re: How to perform a long running dry run transaction without blocking

On 2/7/25 10:20, Robert Leach wrote: Anyway, thanks so much for your help.  This discussion has been very useful, and I think I will proceed at first, exactly how you suggested, by queuing every validation job (using celery).  Then I will explore whether or not I can apply the "on timeout"

Re: How to perform a long running dry run transaction without blocking

>> I'd run across a stackexchange answer >> that suggested setting a >> statement timeout (specific to postgres) in a django view. If I did that >> for the validation view, it wouldn't necessarily 100% correspond to a >> blocking issue, but since

Re: How to perform a long running dry run transaction without blocking

On 2/7/25 05:49, Robert Leach wrote: Ccing list Alright I am trying to reconcile this with from below, 'The largest studies take just under a minute'. The context of the 'The largest studies take just under a minute' statement is that it's not loading the hefty/time-consuming raw data. It's

Re: How to perform a long running dry run transaction without blocking

On 2/6/25 11:25, Robert Leach wrote: Have you considered a validation app? Have it read the inputs and look in db for conflicts, rather than attempt the insert. Zero transactions necessary I did consider that about a year or two ago when I first conceived the data validation interface. Doin

Re: How to perform a long running dry run transaction without blocking

On 2/6/25 12:08, Robert Leach wrote: Alright I am trying to reconcile this with from below, 'The largest studies take just under a minute'. The context of the 'The largest studies take just under a minute' statement is that it's not loading the hefty/time-consuming raw data.  It's only val

Re: How to perform a long running dry run transaction without blocking

>>> The load to the development server does no validation? >>> >>> If so what is the purpose? >>> >>> The background processes are other validation runs? >> It's the same code that executes in both cases (with or without the >> `--validate` flag). All that that flag does is it (effectively) rai

Re: How to perform a long running dry run transaction without blocking

> Have you considered a validation app? Have it read the inputs and look in db > for conflicts, rather than attempt the insert. Zero transactions necessary I did consider that about a year or two ago when I first conceived the data validation interface. Doing that now would solve the problem o

Re: How to perform a long running dry run transaction without blocking

On 2/6/25 09:58, Robert Leach wrote: They enter the data in an excel spreadsheet containing about a dozen inter-related worksheets, named, for example: Study, Animals, Samples, Tissues, Treatments, Sequences Long term is there a thought to have them enter directly into database where validat

Re: How to perform a long running dry run transaction without blocking

> Have you considered a validation app? Have it read the inputs and look in db for conflicts, rather than attempt the insert. Zero transactions necessary

Re: How to perform a long running dry run transaction without blocking

> Please reply to list also. > Ccing list 👍🏻 >> They enter the data in an excel spreadsheet containing about a dozen >> inter-related worksheets, named, for example: Study, Animals, Samples, >> Tissues, Treatments, Sequences > > Long term is there a thought to have them enter directly into dat

Re: How to perform a long running dry run transaction without blocking

On 2/6/25 08:53, Robert Leach wrote: Great questions! Responses inline... Please reply to list also. Ccing list Load to where existing table or temporary table? Existing tables. Note that (and this is a point of contention in our team), we have some fields that must be universally unique

Re: How to perform a long running dry run transaction without blocking

On 2/6/25 07:40, Robert Leach wrote: Comments inline. Let me try and distill all the back story down to an oversimplified explanation: I created a scientific data submission validation interface that helps researchers compile their data submissions to our database. To do this, I decided to a

How to perform a long running dry run transaction without blocking

I've been trying to solve this problem in Django and I've finally decided after over a year going down this path that there's no way to solve it in Django (in a database agnostic fashion). So I'm thinking that I need to explore a Postgres-specific solution. FYI, I'm not a database expert. I just