Re: Insert records in the tavke only if they are not exist
Hi, Adrian, On Sat, Dec 7, 2024 at 5:07 PM Adrian Klaver wrote: > > On 12/7/24 12:17, Igor Korot wrote: > > Hi, Adrian, > > > > On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver > > wrote: > >> > >> On 12/7/24 09:59, Igor Korot wrote: > >>> Hi, Adrian, > >>> > >>> On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver > >>> wrote: > > On 12/6/24 19:36, Igor Korot wrote: > > Hi, All, > > > > When my application starts up, ot creates some tables and insert records > > in them. > > > > When the app starts for the second time it should check if the tables > > and the records in them are exist and skip the process. > > Does this also apply to starts after the second time? > >>> > >>> Yes, it does. > >>> > >>> However, if the new table is created in the meantime - the new record > >>> should be created, because it is not there yet. > >> > >> What determines that a new table needs to be created and populated? > > > > Consider the following scenario: > > I did and it made me start twitching. Good.. > > > > > 1. Program is installed. > > 2. Program is started for the first time. > > 3. My tables are created and populated > > What distinguishes your tables from other users' tables? I am making the tables and naming them with a specific pattern. Now I'm curious - if I start psql and will want to create a table named pg_am, what will happen? I presume psql will produce error saying that the system table with that name exists and the user can't create ome. Am I right? > > > 4. Program is executed. > > 5. User closes the program. > > 6.. Later on the user decides that there is a need > > for another table (inside psql or any other client) > > The above is where I started twitching. > > How do you keep them out of your tables? My tables are named with the specific pattern. Also - see above. > > How do you get these changes to play nice with the existing structure? Again - not sure what you mean here > > > 7. Then the program starts for the second time. > > > > At this time all my tables that were created will stay (courtesy > > of CREATE TABLE IF NOT EXIST). > > You are depending on folks not knowing about DROP TABLE and/or you > having thought out the permissions for access thoroughly. If one of my tables will be dropped - it will be re-created. I'm using CREATE TABLE IF NOT EXIST. > > > > All records that were there are staying unchanged. > > INSERT/UPDATE against your tables is not a possibility? It is.. All I;m saying that when the app starts-up, this is done automatically And if the user decides to insert some data - that's on him > > > However, for the table that was made in between the runs > > will be added > > > > Now if the program is installed on 2 different machines > > and started simultaneously on both - I want to ensure that > > only 1 set of tables is made and only 1 set of records in them > > is available > > I assume this means they are both pointing at the same instance of a > database? Correct. > > This is the part that confuses me. > > If you are going to allow ad hoc and at will changes how do you know > what is actually the correct change? I don't. Think about MS ACCESS-like applications. ACCESS creates the internal tables to keep track of some internals. Is there a possibility of a clash with the user table? Ofc there is. But MS is still doing it nevertheless, because chances of this occurring are really slim. And if a user acquire access to those internal tables and start modify them and in the process screw something up, well he uses MS product as a developer and so should know better as a developer And if MS can do it - why can't I? > > > > > Now, the creation/population is done inside a transaction. > > I'm not sure that a transaction is going to solve the issue I raised > above, it will just make one thing happen with no guarantee that it is > the correct outcome. What is considered correct outcome is strongly on user if we are talking about my app. > > > >> Are you talking about the front end that the user launches or the > >> backend that runs the database or something else? > > > > Front-end. > > This is a C++ app. > > To me this is the tail wagging the dog. The thought of allowing users to > change the database structure and you dealing with it after the fact is > just disturbing to me. But not to me. My application targets developers, not end users. Thank you. > > > > >> > >> > >>> > >>> Thank you. > > -- > Adrian Klaver > adrian.kla...@aklaver.com >
Re: Insert records in the tavke only if they are not exist
But not to me. My application targets developers, not end users. Thank you. What does your app enable developers to do?
Re: Insert records in the tavke only if they are not exist
On 12/7/24 09:59, Igor Korot wrote: Hi, Adrian, On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver wrote: On 12/6/24 19:36, Igor Korot wrote: Hi, All, When my application starts up, ot creates some tables and insert records in them. When the app starts for the second time it should check if the tables and the records in them are exist and skip the process. Does this also apply to starts after the second time? Yes, it does. However, if the new table is created in the meantime - the new record should be created, because it is not there yet. What determines that a new table needs to be created and populated? Is the data expected to change over time? Data change is possible. This is one of the tables: "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30) NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\" integer" ));"; "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON \"sys.abcatfmt\"(\"abf_name\" ASC);"; My understanding is that "INSERT OR IGNORE" will check the unique It is INSERT ... ON CONFLICT DO NOTHING | UPDATE. index and will not do anything if the recrd with such data on the index already exists. So you have the option of either skipping the insert or updating selected fields in the row. Everything is good, except what if I have a connection from 2 different users? Is it possible to have the app start up without external users connecting? Not sure what you mean here. Are you asking if the computer can run the app? Then the answer is NO. App is always started by the user. Please clarify. This is going to need a more detailed explanation of what 'app' means. Are you talking about the front end that the user launches or the backend that runs the database or something else? Thank you. I can run this inside transaction, but will this be enough? Will stating transaction lock the DB and the second user will wait for transaction to complete? Thank you. -- Adrian Klaver adrian.kla...@aklaver.com -- Adrian Klaver adrian.kla...@aklaver.com
Re: Insert records in the tavke only if they are not exist
On 12/7/24 15:58, Igor Korot wrote: Hi, Adrian, What distinguishes your tables from other users' tables? I am making the tables and naming them with a specific pattern. Now I'm curious - if I start psql and will want to create a table named pg_am, what will happen? I presume psql will produce error saying that the system table with that name exists and the user can't create ome. Am I right? When you refer to psql are you talking about the CLI program or the Postgres server in general? As to table names it depends. A schema is a namespace so if you put the table in it's own schema then the server will not complain: create table test_sch.pg_am(id integer); CREATE TABLE \d *.pg_am Table "pg_catalog.pg_am" Column | Type | Collation | Nullable | Default ---+-+---+--+- oid | oid | | not null | amname| name| | not null | amhandler | regproc | | not null | amtype| "char" | | not null | Indexes: "pg_am_oid_index" PRIMARY KEY, btree (oid) "pg_am_name_index" UNIQUE CONSTRAINT, btree (amname) Table "test_sch.pg_am" Column | Type | Collation | Nullable | Default +-+---+--+- id | integer | | | I would strongly advise against this naming schema for the above reason. 4. Program is executed. 5. User closes the program. 6.. Later on the user decides that there is a need for another table (inside psql or any other client) The above is where I started twitching. How do you keep them out of your tables? My tables are named with the specific pattern. Also - see above. Yes, but if you allow users into the database: a) They can figure out what that pattern is. b) Mess with the tables. How do you get these changes to play nice with the existing structure? Again - not sure what you mean here Examples: 1) A user creates table that creates a FK relationship to one of 'your' tables and you then change 'your' table and invalidate that relationship. 2) A user changes the structure of 'your' tables. If one of my tables will be dropped - it will be re-created. I'm using CREATE TABLE IF NOT EXIST. This means every time the app is run it could potentially run one or more schema/data migrations. How are you going to track the state of the database in order to get it back to what it was when the change or changes occurred? All records that were there are staying unchanged. INSERT/UPDATE against your tables is not a possibility? It is.. All I;m saying that when the app starts-up, this is done automatically Again, using what point in time state? And if the user decides to insert some data - that's on him No the point of this thread is it's on you or you would not be asking how to restore to a known state. This is the part that confuses me. If you are going to allow ad hoc and at will changes how do you know what is actually the correct change? I don't. Think about MS ACCESS-like applications. ACCESS creates the internal tables to keep track of some internals. Is there a possibility of a clash with the user table? Ofc there is. But MS is still doing it nevertheless, because chances of this occurring are really slim. And if a user acquire access to those internal tables and start modify them and in the process screw something up, well he uses MS product as a developer and so should know better as a developer And if MS can do it - why can't I? Because MS has the lawyers to keep at bay anyone complaining that Access blew up their application. Now, the creation/population is done inside a transaction. I'm not sure that a transaction is going to solve the issue I raised above, it will just make one thing happen with no guarantee that it is the correct outcome. What is considered correct outcome is strongly on user if we are talking about my app. Except you are asking how to deal with changes and so you are taking ownership of them. And in the case you describe, two different instances of the app making changes, how do you decide which one wins? To me this is the tail wagging the dog. The thought of allowing users to change the database structure and you dealing with it after the fact is just disturbing to me. But not to me. My application targets developers, not end users. Who are users in your application. Thank you. Thank you. -- Adrian Klaver adrian.kla...@aklaver.com -- Adrian Klaver adrian.kla...@aklaver.com
Re: Insert records in the tavke only if they are not exist
On Sat, Dec 7, 2024 at 4:59 PM Igor Korot wrote: > > I am making the tables and naming them with a specific pattern. > > Now I'm curious - if I start psql and will want to create a table > named pg_am, what will happen? > I presume psql will produce error saying that the system table with > that name exists and the user can't create ome. > Am I right? > > You are wasting people's time asking a question like this when it takes but seconds to experiment. Most of what you are doing should be considered something to test out and observe the behavior yourself. Formalize it as a test if possible if you rely heavily on it so you can be aware if a bug were to be introduced that broke such a behavior, or even just to get lots of exposure to concurrency if you are designing a system especially reliant on specific concurrent behavior. As it stands the methodology you describe is quite non-traditional for a relational client-server database system. David J.
Re: Insert records in the tavke only if they are not exist
Hi, Adrian, On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver wrote: > > On 12/6/24 19:36, Igor Korot wrote: > > Hi, All, > > > > When my application starts up, ot creates some tables and insert records > > in them. > > > > When the app starts for the second time it should check if the tables > > and the records in them are exist and skip the process. > > Does this also apply to starts after the second time? Yes, it does. However, if the new table is created in the meantime - the new record should be created, because it is not there yet. > > Is the data expected to change over time? Data change is possible. This is one of the tables: "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30) NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\" integer" ));"; "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON \"sys.abcatfmt\"(\"abf_name\" ASC);"; My understanding is that "INSERT OR IGNORE" will check the unique index and will not do anything if the recrd with such data on the index already exists. > > > > > Everything is good, except what if I have a connection from 2 different > > users? > Is it possible to have the app start up without external users connecting? Not sure what you mean here. Are you asking if the computer can run the app? Then the answer is NO. App is always started by the user. Please clarify. Thank you. > > > > > I can run this inside transaction, but will this be enough? Will stating > > transaction lock the DB and the second user will wait for transaction to > > complete? > > > > Thank you. > > > > -- > Adrian Klaver > adrian.kla...@aklaver.com >
Re: Insert records in the tavke only if they are not exist
Hi, Adrian, On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver wrote: > > On 12/7/24 09:59, Igor Korot wrote: > > Hi, Adrian, > > > > On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver > > wrote: > >> > >> On 12/6/24 19:36, Igor Korot wrote: > >>> Hi, All, > >>> > >>> When my application starts up, ot creates some tables and insert records > >>> in them. > >>> > >>> When the app starts for the second time it should check if the tables > >>> and the records in them are exist and skip the process. > >> > >> Does this also apply to starts after the second time? > > > > Yes, it does. > > > > However, if the new table is created in the meantime - the new record > > should be created, because it is not there yet. > > What determines that a new table needs to be created and populated? Consider the following scenario: 1. Program is installed. 2. Program is started for the first time. 3. My tables are created and populated 4. Program is executed. 5. User closes the program. 6.. Later on the user decides that there is a need for another table (inside psql or any other client) 7. Then the program starts for the second time. At this time all my tables that were created will stay (courtesy of CREATE TABLE IF NOT EXIST). All records that were there are staying unchanged. However, for the table that was made in between the runs will be added Now if the program is installed on 2 different machines and started simultaneously on both - I want to ensure that only 1 set of tables is made and only 1 set of records in them is available Now, the creation/population is done inside a transaction. > > > > >> > >> Is the data expected to change over time? > > > > Data change is possible. > > > > This is one of the tables: > > > > "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30) > > NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\" > > integer" ));"; > > "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON > > \"sys.abcatfmt\"(\"abf_name\" ASC);"; > > > > My understanding is that "INSERT OR IGNORE" will check the unique > > It is INSERT ... ON CONFLICT DO NOTHING | UPDATE. > > > index and will not do anything if the recrd > > with such data on the index already exists. Good. I'm using INSERT ON CONFLICT DO NOTHING. > > So you have the option of either skipping the insert or updating > selected fields in the row. > > > > >> > >>> > >>> Everything is good, except what if I have a connection from 2 different > >>> users? > >> Is it possible to have the app start up without external users connecting? > > > > Not sure what you mean here. > > > > Are you asking if the computer can run the app? > > Then the answer is NO. App is always started by the user. > > Please clarify. > > This is going to need a more detailed explanation of what 'app' means. > > Are you talking about the front end that the user launches or the > backend that runs the database or something else? Front-end. This is a C++ app. > > > > > > Thank you. > > > >> > >>> > >>> I can run this inside transaction, but will this be enough? Will stating > >>> transaction lock the DB and the second user will wait for transaction to > >>> complete? > >>> > >>> Thank you. > >>> > >> > >> -- > >> Adrian Klaver > >> adrian.kla...@aklaver.com > >> > > -- > Adrian Klaver > adrian.kla...@aklaver.com >
Re: Errors when restoring backup created by pg_dumpall
On Sat, Dec 7, 2024 at 12:25 PM PopeRigby wrote: > > It actually looks like setting those all to have public fixed all the > errors, including the one with lldap. So, how can I get it to not put > public there automatically for next time? > > I assume you mean "get it to put public there" (i.e., the "not" is a typo) You cannot. The security team has decided to not permit an opt-in bypass of the lock-downs implemented to fix CVE-2018-1058. Your only real choice at the moment is to replace the function call in the generated expression with a custom function and in that custom function's create function command attach a "set search_path to public" clause. That will prevent inlining and also ensure the public schema is in the search_path when executing the public.ll_to_earth function call. With that in place the empty search_path in the dump file will no longer matter. David J.
Re: Insert records in the tavke only if they are not exist
On 12/6/24 19:36, Igor Korot wrote: Hi, All, When my application starts up, ot creates some tables and insert records in them. When the app starts for the second time it should check if the tables and the records in them are exist and skip the process. Does this also apply to starts after the second time? Is the data expected to change over time? Everything is good, except what if I have a connection from 2 different users? Is it possible to have the app start up without external users connecting? I can run this inside transaction, but will this be enough? Will stating transaction lock the DB and the second user will wait for transaction to complete? Thank you. -- Adrian Klaver adrian.kla...@aklaver.com
Re: Insert records in the tavke only if they are not exist
On 12/7/24 12:17, Igor Korot wrote: Hi, Adrian, On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver wrote: On 12/7/24 09:59, Igor Korot wrote: Hi, Adrian, On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver wrote: On 12/6/24 19:36, Igor Korot wrote: Hi, All, When my application starts up, ot creates some tables and insert records in them. When the app starts for the second time it should check if the tables and the records in them are exist and skip the process. Does this also apply to starts after the second time? Yes, it does. However, if the new table is created in the meantime - the new record should be created, because it is not there yet. What determines that a new table needs to be created and populated? Consider the following scenario: I did and it made me start twitching. 1. Program is installed. 2. Program is started for the first time. 3. My tables are created and populated What distinguishes your tables from other users tables? 4. Program is executed. 5. User closes the program. 6.. Later on the user decides that there is a need for another table (inside psql or any other client) The above is where I started twitching. How do you keep them out of your tables? How do you get these changes to play nice with the existing structure? 7. Then the program starts for the second time. At this time all my tables that were created will stay (courtesy of CREATE TABLE IF NOT EXIST). You are depending on folks not knowing about DROP TABLE and/or you having thought out the permissions for access thoroughly. All records that were there are staying unchanged. INSERT/UPDATE against your tables is not a possibility? However, for the table that was made in between the runs will be added Now if the program is installed on 2 different machines and started simultaneously on both - I want to ensure that only 1 set of tables is made and only 1 set of records in them is available I assume this means they are both pointing at the same instance of a database? This is the part that confuses me. If you are going to allow ad hoc and at will changes how do you know what is actually the correct change? Now, the creation/population is done inside a transaction. I'm not sure that a transaction is going to solve the issue I raised above, it will just make one thing happen with no guarantee that it is the correct outcome. Are you talking about the front end that the user launches or the backend that runs the database or something else? Front-end. This is a C++ app. To me this is the tail wagging the dog. The thought of allowing users to change the database structure and you dealing with it after the fact is just disturbing to me. Thank you. -- Adrian Klaver adrian.kla...@aklaver.com
Re: Errors when restoring backup created by pg_dumpall
On 12/5/24 14:48, Adrian Klaver wrote:
On 12/5/24 14:32, PopeRigby wrote:
On 12/1/24 13:55, Tom Lane wrote:
Adrian Klaver writes:
On 12/1/24 13:14, Tom Lane wrote:
It would be useful to know what is the command at line 4102
of all.sql.
It is here:
https://gist.github.com/poperigby/fcb59eb6c22c6051800e06a0ec482b49
CREATE TABLE public.geodata_places (
id integer NOT NULL,
name character varying(200) NOT NULL,
longitude double precision NOT NULL,
latitude double precision NOT NULL,
"countryCode" character(2) NOT NULL,
"admin1Code" character varying(20),
"admin2Code" character varying(80),
"modificationDate" date NOT NULL,
"earthCoord" public.earth GENERATED ALWAYS AS
(public.ll_to_earth(latitude, longitude)) STORED,
"admin1Name" character varying,
"admin2Name" character varying,
"alternateNames" character varying
);
Ah! Then the failure occurs because we do a planning pass on the
GENERATED expression (I don't remember exactly why that's needed
during CREATE TABLE). So maybe messing with the dump script's
search_path setting *would* be enough to get you past that.
Having said that, the CREATE should have been seeing the new-style
definition of ll_to_earth() if the 1.2 version of earthdistance
was correctly installed.
regards, tom lane
So, is there anything I can do to fix this particular backup? I'm
kind of stuck here. There's also the issue with it for some reason
failing to connect to the lldap database after it literally just
created it. Some weird things going on.
In the pg_dumpall sql script did you change:
SELECT pg_catalog.set_config('search_path', '', false);
to
SELECT pg_catalog.set_config('search_path', 'public', false);
?
Show us the connection error you got for the lldap database.
It actually looks like setting those all to have public fixed all the
errors, including the one with lldap. So, how can I get it to not put
public there automatically for next time?
