Best Tool for PostgreSQL Auditing and Storing Audit Logs Separately

[vijay patil]

Hi All,

We are exploring auditing solutions for our PostgreSQL database and are
considering using pgaudit for this purpose. However, we have a few
questions:

   1.

   *What is the best tool for auditing PostgreSQL databases?*
   -

  We are specifically looking for a solution that offers detailed
  auditing capabilities and is compatible with our setup.
  2.

   *Can we store the audit information separately from PostgreSQL logs if
   we decide to use pgaudit?*
   -

  We would prefer to keep the audit logs in a separate file or location
  for easier management and analysis.


We appreciate any help or suggestions!


Thanks

Vijay

Re: Help with PhD Dissertation

[Greg Sabino Mullane]

On Tue, Apr 15, 2025 at 7:20 AM Karsten Hilbert 
wrote:

> > The survey [...] is completely anonymous.
>
> Nope.
>

Can you elaborate on this, please, for my sake and others?

Identify system databases

[Igor Korot]

Hi, ALL,
Is there a field in the pg_databases table which indicates that particular
DB is a system one?

Or it can be determined ony by name?

Thank you.

Re: Identify system databases

[David G. Johnston]

On Tuesday, April 15, 2025, Igor Korot  wrote:

> Hi, ALL,
> Is there a field in the pg_databases table which indicates that particular
> DB is a system one?
>

What is a system database?

David J.

Re: Identify system databases

[Adrian Klaver]

On 4/15/25 07:48, Igor Korot wrote:

Hi, ALL,
Is there a field in the pg_databases table which indicates that 
particular DB is a system one?


No there is not. As a practical matter they are no different then any 
user created database, other then template0 is created with datallowconn 
 set to false.




Or it can be determined ony by name?

Thank you.



--
Adrian Klaver
adrian.kla...@aklaver.com

Re: Help with PhD Dissertation

[Karsten Hilbert]

Am Tue, Apr 15, 2025 at 11:26:35AM -0400 schrieb Osmel Brito-Bigott:

> I'm not collecting emails in the survey

Don't get me wrong, I am not questioning your good faith or intent.

Karsten
-- 
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

Re: Best Tool for PostgreSQL Auditing and Storing Audit Logs Separately

[vijay patil]

Hi Kenan,

Thanks for the quick response.

Thanks
Vijay

On Tue, Apr 15, 2025 at 4:44 PM KENAN ÇİFTÇİ  wrote:

> Hi,
>
> You can use pgaudit and pgauditlogtofile extension (
> https://github.com/fmbiete/pgauditlogtofile) together to write audit logs
> in a separate file.
>
> yours,
>
> Kenan Çiftçi
>
> On Tue, Apr 15, 2025 at 1:44 PM vijay patil 
> wrote:
>
>> Hi All,
>>
>> We are exploring auditing solutions for our PostgreSQL database and are
>> considering using pgaudit for this purpose. However, we have a few
>> questions:
>>
>>1.
>>
>>*What is the best tool for auditing PostgreSQL databases?*
>>-
>>
>>   We are specifically looking for a solution that offers detailed
>>   auditing capabilities and is compatible with our setup.
>>   2.
>>
>>*Can we store the audit information separately from PostgreSQL logs
>>if we decide to use pgaudit?*
>>-
>>
>>   We would prefer to keep the audit logs in a separate file or
>>   location for easier management and analysis.
>>
>>
>> We appreciate any help or suggestions!
>>
>>
>> Thanks
>>
>> Vijay
>>
>

Re: Fwd: Identify system databases

[Tom Lane]

"David G. Johnston"  writes:
> On Tue, Apr 15, 2025 at 11:20 AM Adrian Klaver 
> wrote:
>> If what you say is true why does initdb lack an option to not create
>> them on creating a cluster?

> By creating the initial three databases the system is more usable due to
> having established conventions.  They are conventional databases, not
> system ones.

Precisely.  For example, the only reason for the "postgres" database
to exist is so that there is a reasonable default database for clients
to connect to.  If we didn't have it we'd need some other convention.
(Indeed, we used to not have it, and back then the default client
behavior was usually to connect to template1.  That led to people
creating random junk in template1 and then being surprised when
CREATE DATABASE copied it into new databases.)

In the end this discussion is about terminology.  These three
databases are part of the ecosystem and clients generally expect them
to be there.  But the server doesn't depend on them to function.
Does that make them "system" databases?  All depends on what you
think that means.

regards, tom lane

Re: Fwd: Identify system databases

[Adrian Klaver]

On 4/15/25 09:48, David G. Johnston wrote:
On Tue, Apr 15, 2025 at 9:31 AM Adrian Klaver > wrote:






Agreed.

The fact that initdb creates the template0, template1 and postgres
databases and you can't change that makes them system not user
databases.


Based on that definition there is a boundary in the system where OIDs 
are considered bootstrap/system OIDs versus user OIDs.  Key off of 
that.  Though since the names never change, and there are always/only 
three, it seems pointless to use the OID aspect of initdb as a basis.


My definition of a "system database" would be a database that, if it 
didn't exist, would cause the system to break.  i.e., is a database 
whose presence is integral to the operations of the system.  None of 
these qualify under that definition.  Which is why there is no column in 
pg_database identifying system databases - there are none.


1) Try connecting(not using single user mode) to a cluster without them.

2) Trying creating a database or databases to replace them without them 
existing in the first place.


If what you say is true why does initdb lack an option to not create 
them on creating a cluster?




David J.



--
Adrian Klaver
adrian.kla...@aklaver.com

Re: Fwd: Identify system databases

[David G. Johnston]

On Tue, Apr 15, 2025 at 11:20 AM Adrian Klaver 
wrote:

> On 4/15/25 09:48, David G. Johnston wrote:
> > On Tue, Apr 15, 2025 at 9:31 AM Adrian Klaver  > > wrote:
> >
>
> >
> > Agreed.
> >
> > The fact that initdb creates the template0, template1 and postgres
> > databases and you can't change that makes them system not user
> > databases.
> >
> >
> > Based on that definition there is a boundary in the system where OIDs
> > are considered bootstrap/system OIDs versus user OIDs.  Key off of
> > that.  Though since the names never change, and there are always/only
> > three, it seems pointless to use the OID aspect of initdb as a basis.
> >
> > My definition of a "system database" would be a database that, if it
> > didn't exist, would cause the system to break.  i.e., is a database
> > whose presence is integral to the operations of the system.  None of
> > these qualify under that definition.  Which is why there is no column in
> > pg_database identifying system databases - there are none.
>
> 1) Try connecting(not using single user mode) to a cluster without them.
>

I did.  Here, the full test.
-- freshly initdb cluster exists and is running

❯ cat ~/Downloads/test-drops.bash
createdb mydb
psql -c 'alter database template0 is_template false;' mydb
dropdb template0
psql -c 'alter database template1 is_template false;' mydb
dropdb template1
dropdb --maintenance-db mydb postgres
psql -c '\l' mydb

❯ bash ~/Downloads/test-drops.bash
ALTER DATABASE
ALTER DATABASE
   List of databases
 Name | Owner  | Encoding | Locale Provider |  Collate   |   Ctype|
Locale | ICU Rules | Access privileges
--++--+-++++---+---
 mydb | davidj | UTF8 | libc| en_US.utf8 | en_US.utf8 |
   |   |
(1 row)



> 2) Trying creating a database or databases to replace them without them
> existing in the first place.
>

Why?


> If what you say is true why does initdb lack an option to not create
> them on creating a cluster?
>

Well, it has to create one database because it is not possible to connect
to the cluster without naming a specific database.

initdb could allow the user to specify the initial database name and only
create that single database.  It would work.

By creating the initial three databases the system is more usable due to
having established conventions.  They are conventional databases, not
system ones.

Plus, it leaves the cluster in a state that is generally much more
user-friendly.  And, as shown, people who dislike the convention can
rearrange things into whatever form they desire.

David J.

Re: Help with PhD Dissertation

[Karsten Hilbert]

> > > The survey [...] is completely anonymous.
> >
> > Nope.
> >
> 
> Can you elaborate on this, please, for my sake and others?

I was, certainly somewhat tersely, trying to point out that
using Google Forms nearly precludes anonymity unless one
accesses them via Tor or some such.

Best regards,
Karsten
-- 
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

Fwd: Identify system databases

[Igor Korot]

-- Forwarded message -
From: Igor Korot 
Date: Tue, Apr 15, 2025 at 11:16 AM
Subject: Re: Identify system databases
To: Adrian Klaver 


Hi, Adrian,

On Tue, Apr 15, 2025 at 10:07 AM Adrian Klaver 
wrote:

> On 4/15/25 07:48, Igor Korot wrote:
> > Hi, ALL,
> > Is there a field in the pg_databases table which indicates that
> > particular DB is a system one?
>
> No there is not. As a practical matter they are no different then any
> user created database, other then template0 is created with datallowconn
>   set to false.


Understood. Will filter by name…

Thank you.


>
> >
> > Or it can be determined ony by name?
> >
> > Thank you.
> >
>
> --
> Adrian Klaver
> adrian.kla...@aklaver.com
>
>

Fwd: Identify system databases

[Igor Korot]

-- Forwarded message -
From: Igor Korot 
Date: Tue, Apr 15, 2025 at 11:15 AM
Subject: Re: Identify system databases
To: David G. Johnston 


Hi, David,

On Tue, Apr 15, 2025 at 9:56 AM David G. Johnston <
david.g.johns...@gmail.com> wrote:

> On Tuesday, April 15, 2025, Igor Korot  wrote:
>
>> Hi, ALL,
>> Is there a field in the pg_databases table which indicates that
>> particular DB is a system one?
>>
>
> What is a system database?
>

I consider system database a database that is created by default when the
server is run for the first time.

Kind of a predefined set of databases…

As an example - there is a database called Postgres. It is created when the
server runs for the first time, whether I as a user like it or not.

Thank you.


> David J.
>
>

Re: Fwd: Identify system databases

[Adrian Klaver]

On 4/15/25 09:21, Igor Korot wrote:






Hi, David,

On Tue, Apr 15, 2025 at 9:56 AM David G. Johnston 
mailto:david.g.johns...@gmail.com>> wrote:


On Tuesday, April 15, 2025, Igor Korot mailto:ikoro...@gmail.com>> wrote:

Hi, ALL,
Is there a field in the pg_databases table which indicates that
particular DB is a system one?


What is a system database?


I consider system database a database that is created by default when 
the server is run for the first time.


Agreed.

The fact that initdb creates the template0, template1 and postgres 
databases and you can't change that makes them system not user databases.


Further from here:

https://www.postgresql.org/docs/devel/app-initdb.html

"The postgres database is a default database meant for use by users, 
utilities and third party applications"


indicates that postgres is expected to be in a cluster(system).

Lastly the CREATE DATABASE depends on template1 to be there to create at 
least the first user database. Also template0 is a "clean" template 
database that allows for using a new encoding in new database. It is 
also used by pg_dump/restore:


CREATE DATABASE test WITH TEMPLATE = template0 ...



Kind of a predefined set of databases…

As an example - there is a database called Postgres. It is created when 
the server runs for the first time, whether I as a user like it or not.


Thank you.


David J.



--
Adrian Klaver
adrian.kla...@aklaver.com

Re: ICU Collations and Collation Updates

[Paul Foerster]

Hi Laurenz,

> On 14 Apr 2025, at 19:36, Laurenz Albe  wrote:
> 
> You cannot "bake in into" PostgreSQL, but you can grab the ICU source,
> install it in /usr/local or similar and build PostgreSQL against that.
> You will have to fiddle with CFLAGS and LDFLAGS so that the build process
> uses the C headers and libraries from your build of the ICU library.

I think I'm going to look into this. I'm not a developer, so I'll probably 
fail. Building PostgreSQL from source is easy and well documented. I have 
scripted this. But other than that, I'll probably fail. 🤣 Still worth a try, 
though.

Thanks,
Paul

Re: Fwd: Identify system databases

[David G. Johnston]

On Tue, Apr 15, 2025 at 9:31 AM Adrian Klaver 
wrote:

> On 4/15/25 09:21, Igor Korot wrote:
> >
> >
>
> > Hi, David,
> >
> > On Tue, Apr 15, 2025 at 9:56 AM David G. Johnston
> > mailto:david.g.johns...@gmail.com>> wrote:
> >
> > On Tuesday, April 15, 2025, Igor Korot  > > wrote:
> >
> > Hi, ALL,
> > Is there a field in the pg_databases table which indicates that
> > particular DB is a system one?
> >
> >
> > What is a system database?
> >
> >
> > I consider system database a database that is created by default when
> > the server is run for the first time.
>
> Agreed.
>
> The fact that initdb creates the template0, template1 and postgres
> databases and you can't change that makes them system not user databases.
>

Based on that definition there is a boundary in the system where OIDs are
considered bootstrap/system OIDs versus user OIDs.  Key off of that.
Though since the names never change, and there are always/only three, it
seems pointless to use the OID aspect of initdb as a basis.

My definition of a "system database" would be a database that, if it didn't
exist, would cause the system to break.  i.e., is a database whose presence
is integral to the operations of the system.  None of these qualify under
that definition.  Which is why there is no column in pg_database
identifying system databases - there are none.

The system will continue to operate if you do:

initdb
createdb newdb
psql -c 'alter database template0 is_template false;'
dropdb template0
psql -c 'alter database template1 is_template false;'
dropdb template1
dropdb --maintenance-db newdb postgres


An operational definition worth considering, though, is that any database
owned by the bootstrap superuser is a system database.  After all, the
system owner created/owns them?  If you want non-system databases for your
application, assign their ownership to a non-system role.

David J.

Re: Help with PhD Dissertation

[Osmel Brito-Bigott]

I'm not collecting emails in the survey

El mar, 15 abr 2025 a la(s) 11:25 a.m., Karsten Hilbert (
karsten.hilb...@gmx.net) escribió:

> > > > The survey [...] is completely anonymous.
> > >
> > > Nope.
> > >
> >
> > Can you elaborate on this, please, for my sake and others?
>
> I was, certainly somewhat tersely, trying to point out that
> using Google Forms nearly precludes anonymity unless one
> accesses them via Tor or some such.
>
> Best regards,
> Karsten
> --
> GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B
>
>
>

Re: Best Tool for PostgreSQL Auditing and Storing Audit Logs Separately

[KENAN ÇİFTÇİ]

Hi,

You can use pgaudit and pgauditlogtofile extension (
https://github.com/fmbiete/pgauditlogtofile) together to write audit logs
in a separate file.

yours,

Kenan Çiftçi

On Tue, Apr 15, 2025 at 1:44 PM vijay patil 
wrote:

> Hi All,
>
> We are exploring auditing solutions for our PostgreSQL database and are
> considering using pgaudit for this purpose. However, we have a few
> questions:
>
>1.
>
>*What is the best tool for auditing PostgreSQL databases?*
>-
>
>   We are specifically looking for a solution that offers detailed
>   auditing capabilities and is compatible with our setup.
>   2.
>
>*Can we store the audit information separately from PostgreSQL logs if
>we decide to use pgaudit?*
>-
>
>   We would prefer to keep the audit logs in a separate file or
>   location for easier management and analysis.
>
>
> We appreciate any help or suggestions!
>
>
> Thanks
>
> Vijay
>

Re: Help with PhD Dissertation

[Karsten Hilbert]

Am Mon, Apr 14, 2025 at 09:27:30PM -0400 schrieb Osmel Brito-Bigott:

> forms.gle
>
> The survey [...] is completely anonymous.

Nope.

Karsten Hilbert
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

Re: Fwd: Identify system databases

[Ron Johnson]

On Tue, Apr 15, 2025 at 3:11 PM Tom Lane  wrote:

> "David G. Johnston"  writes:
> > On Tue, Apr 15, 2025 at 11:20 AM Adrian Klaver <
> adrian.kla...@aklaver.com>
> > wrote:
> >> If what you say is true why does initdb lack an option to not create
> >> them on creating a cluster?
>
> > By creating the initial three databases the system is more usable due to
> > having established conventions.  They are conventional databases, not
> > system ones.
>
> Precisely.  For example, the only reason for the "postgres" database
> to exist is so that there is a reasonable default database for clients
> to connect to.  If we didn't have it we'd need some other convention.
> (Indeed, we used to not have it, and back then the default client
> behavior was usually to connect to template1.  That led to people
> creating random junk in template1 and then being surprised when
> CREATE DATABASE copied it into new databases.)
>

Interesting history.  I've always interpreted the "postgres" database as
being the system database, but now I know it's not.

-- 
Death to , and butter sauce.
Don't boil me, I'm still alive.
 lobster!

Re: Fwd: Identify system databases

[David G. Johnston]

On Tue, Apr 15, 2025 at 5:24 PM Adrian Klaver 
wrote:

> >
> > By creating the initial three databases the system is more usable due to
> > having established conventions.  They are conventional databases, not
> > system ones.
>
> The bottom line is the Postgres project has built an infrastructure
> around those names that the community is plugged into.
>
>
So, the answer is no, there is no dedicated pg_database column to identify
the system/conventional databases created by initdb.  They are identified
by name.  You must know from documentation or experience that they are
"postgres", "template1" and "template0".

Pick whichever of "system" or "convention" or some other word makes you
happy to categorize those three databases.  Though I suggest "the initdb"
databases since neither of the other terms is used anywhere in the
documentation.

David J.

Re: Identify system databases

[Tom Lane]

Adrian Klaver  writes:
> On 4/15/25 07:48, Igor Korot wrote:
>> Is there a field in the pg_databases table which indicates that 
>> particular DB is a system one?

> No there is not. As a practical matter they are no different then any 
> user created database, other then template0 is created with datallowconn 
>   set to false.

Yeah; in a very real sense there isn't such a thing as a "system
database" in Postgres.  There are pre-created databases that have
specific uses (such as being the default template for CREATE
DATABASE), but those uses are identified by name not by OID.
You can drop template1, and the only thing that will be unhappy
is CREATE DATABASE, and if you make a new DB that is named
template1 then CREATE DATABASE will be happy again.

regards, tom lane

Re: Fwd: Identify system databases

[Adrian Klaver]

On 4/15/25 11:35, David G. Johnston wrote:
On Tue, Apr 15, 2025 at 11:20 AM Adrian Klaver 
mailto:adrian.kla...@aklaver.com>> wrote:




1) Try connecting(not using single user mode) to a cluster without them.




First I have had this discussion before and was not successful and I 
expect the same outcome this time, but I am half Dutch so that is not a 
concern.




I did.  Here, the full test.
-- freshly initdb cluster exists and is running

❯ cat ~/Downloads/test-drops.bash
createdb mydb


From Postgres log:

CREATE DATABASE mydb;

From here:

https://www.postgresql.org/docs/current/sql-createdatabase.html

"template

The name of the template from which to create the new database, or 
DEFAULT to use the default template (template1).

"

I know I can use --template=template with createdb, but that still means 
there has to be a database existing in the initial cluster to use as a 
template. You can't get around that fact.




2) Trying creating a database or databases to replace them without them
existing in the first place.


Why?


See above. It boils down to, there has to be 'seed'(system) databases 
extant in the cluster created by initdb for a user to move forward.





If what you say is true why does initdb lack an option to not create
them on creating a cluster?


Well, it has to create one database because it is not possible to 
connect to the cluster without naming a specific database.


Exactly, the cluster needs a database created by the system to function.



initdb could allow the user to specify the initial database name and 
only create that single database.  It would work.


From an earlier post of mine in this thread:

"Further from here:

https://www.postgresql.org/docs/devel/app-initdb.html

"The postgres database is a default database meant for use by users, 
utilities and third party applications"


indicates that postgres is expected to be in a cluster(system).

Lastly the CREATE DATABASE depends on template1 to be there to create at 
least the first user database. Also template0 is a "clean" template 
database that allows for using a new encoding in new database. It is 
also used by pg_dump/restore:


CREATE DATABASE test WITH TEMPLATE = template0 ...
"

It would be possible to do as you say, but then you would need to make 
the system nature of the process explicit by updating pg_database to 
include information on what the default template database, the 'clean' 
template and the 'default connection database' are. As it it stands now 
it is implicit in the naming and the use of those names by the server 
code and outside code.




By creating the initial three databases the system is more usable due to 
having established conventions.  They are conventional databases, not 
system ones.


Then remove them or rename them and see how long it takes the mail 
server to start glowing with posts to --general and --bugs.


The bottom line is the Postgres project has built an infrastructure 
around those names that the community is plugged into.




Plus, it leaves the cluster in a state that is generally much more 
user-friendly.  And, as shown, people who dislike the convention can 
rearrange things into whatever form they desire.


David J.



--
Adrian Klaver
adrian.kla...@aklaver.com