On Sat, May 8, 2021 at 2:08 PM Michael Paquier <mich...@paquier.xyz> wrote:
>
> On Thu, May 06, 2021 at 09:33:53PM +0530, Dilip Kumar wrote:
> > On Thu, May 6, 2021 at 5:42 PM Michael Paquier <mich...@paquier.xyz> wrote:
> > >
> > > On Thu, May 06, 2021 at 10:45:53AM +0530, Dilip Kumar wrote:
> > > > Thanks, Robert and Michael for your input. I will try to understand
> > > > how it is done in the example shared by you and come up with the test
> > > > once I get time. I assume this is not something urgent.
> > >
> > > Thanks. FWIW, I'd rather see this gap closed asap, as features should
> > > come with proper tests IMO.
> >
> > I have done this please find the attached patch.
>
> No objections to take the approach to mark the lz4-related test with a
> special flag and skip them. I have three comments:
> - It would be good to document this new flag. See the comment block
> on top of %dump_test_schema_runs.
> - There should be a test for --no-toast-compression. You can add a
> new command in %pgdump_runs, then unmatch the expected output with the
> option.
> - I would add one test case with COMPRESSION pglz somewhere to check
> after the case of ALTER TABLE COMPRESSION commands not generated as
> this depends on default_toast_compression. A second test I'd add is a
> materialized view with a column switched to use lz4 as compression
> method with an extra ALTER command in create_sql.
I have fixed some of them, I could not write the test cases where we
have to ensure that 'ALTER TABLE COMPRESSION' is not appearing in the
dump. Basically, I don't have knowledge of the perl language so even
after trying for some time I could not write those 2 test cases. I
have fixed the remaining comments.
> > I don't have much idea about the MSVC script, but I will try to see
> > some other parameters and fix this.
>
> Thanks! I can dive into that if that's an issue. Let's make things
> compatible with what upstream provides, meaning that we should have
> some documentation pointing to the location of their deliverables,
> equally to what we do for the Perl and OpenSSL dependencies for
> example.
I have changed the documentation and also updated the Solution.pm. I
could not verify the windows build yet as I am not having windows
environment.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
From c69ab6dd0fde8e39b7d5aeeb4d74119621c31f58 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Thu, 6 May 2021 21:32:12 +0530
Subject: [PATCH v2 1/2] Add pg_dump test case for TOAST compression
---
src/bin/pg_dump/t/002_pg_dump.pl | 64 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 86113df..3ea6f10 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -374,6 +374,10 @@ my %pgdump_runs = (
# of the pg_dump runs happening. This is what "seeds" the
# system with objects to be dumped out.
#
+# There is also a flag 'lz4' which can be set if the test case
+# depends upon lz4 library. And if this flag is set then the
+# test will not be executed if HAVE_LIBLZ4 is not set.
+#
# Building of this hash takes a bit of time as all of the regexps
# included in it are compiled. This greatly improves performance
# as the regexps are used for each run the test applies to.
@@ -2068,6 +2072,27 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE MATERIALIZED VIEW matview_compression' => {
+ create_order => 20,
+ create_sql => 'CREATE MATERIALIZED VIEW
+ dump_test.matview_compression (col2) AS
+ SELECT col2 FROM dump_test.test_table;
+ ALTER MATERIALIZED VIEW dump_test.matview_compression
+ ALTER COLUMN col2 SET COMPRESSION lz4',
+ regexp => qr/^
+ \QCREATE MATERIALIZED VIEW dump_test.matview_compression AS\E
+ \n\s+\QSELECT test_table.col2\E
+ \n\s+\QFROM dump_test.test_table\E
+ \n\s+\QWITH NO DATA;\E
+ .*
+ \QALTER TABLE ONLY dump_test.matview_compression ALTER COLUMN col2 SET COMPRESSION lz4;\E\n
+ /xms,
+ lz4 => 1,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE POLICY p1 ON test_table' => {
create_order => 22,
create_sql => 'CREATE POLICY p1 ON dump_test.test_table
@@ -2334,6 +2359,26 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_compression' => {
+ create_order => 3,
+ create_sql => 'CREATE TABLE dump_test.test_compression (
+ col1 int,
+ col2 text COMPRESSION lz4
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_compression (\E\n
+ \s+\Qcol1 integer,\E\n
+ \s+\Qcol2 text\E\n
+ \);\n
+ .*
+ \QALTER TABLE ONLY dump_test.test_compression ALTER COLUMN col2 SET COMPRESSION lz4;\E\n
+ /xms,
+ lz4 => 1,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE measurement PARTITIONED BY' => {
create_order => 90,
create_sql => 'CREATE TABLE dump_test.measurement (
@@ -3377,6 +3422,7 @@ my $port = $node->port;
# If it doesn't then we will skip all the COLLATION-related tests.
my $collation_support = 0;
my $collation_check_stderr;
+my $supports_lz4 = check_pg_config("#define HAVE_LIBLZ4 1");
$node->psql(
'postgres',
"CREATE COLLATION testing FROM \"C\"; DROP COLLATION testing;",
@@ -3445,6 +3491,12 @@ foreach my $run (sort keys %pgdump_runs)
next;
}
+ # Skip any lz4 compression related test
+ if (!$supports_lz4 && defined($tests{$test}->{lz4}))
+ {
+ next;
+ }
+
# If there is a like entry, but no unlike entry, then we will test the like case
if ($tests{$test}->{like}->{$test_key}
&& !defined($tests{$test}->{unlike}->{$test_key}))
@@ -3502,6 +3554,12 @@ foreach my $test (
next;
}
+ # Skip any lz4 compression related test
+ if (!$supports_lz4 && defined($tests{$test}->{lz4}))
+ {
+ next;
+ }
+
# Add terminating semicolon
$create_sql{$test_db} .= $tests{$test}->{create_sql} . ";";
}
@@ -3600,6 +3658,12 @@ foreach my $run (sort keys %pgdump_runs)
next;
}
+ # Skip any lz4 compression related test
+ if (!$supports_lz4 && defined($tests{$test}->{lz4}))
+ {
+ next;
+ }
+
if ($run_db ne $test_db)
{
next;
--
1.8.3.1
From 7c71f46cfe09bcf0e1c6c4e6f093b0178d60f5cd Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Sat, 8 May 2021 20:12:56 +0530
Subject: [PATCH v2 2/2] LZ4 option for windows build and documentation update
---
doc/src/sgml/install-windows.sgml | 9 +++++++++
src/tools/msvc/Solution.pm | 10 ++++++++++
2 files changed, 19 insertions(+)
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index 64687b1..bad8431 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -339,6 +339,15 @@ $ENV{MSBFLAGS}="/m";
</para></listitem>
</varlistentry>
+ <varlistentry>
+ <term><productname>lz4</productname></term>
+ <listitem><para>
+ Required for supporting lz4 compression method for compressing the
+ table data. Binaries can be downloaded from
+ <ulink url="https://github.com/lz4/lz4/releases";></ulink>.
+ </para></listitem>
+ </varlistentry>
+
</variablelist>
</para>
</sect2>
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index d2bc7ab..4658637 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -532,6 +532,10 @@ sub GenerateFiles
$define{HAVE_LIBXSLT} = 1;
$define{USE_LIBXSLT} = 1;
}
+ if ($self->{options}->{lz4})
+ {
+ $define{HAVE_LIBLZ4} = 1;
+ }
if ($self->{options}->{openssl})
{
$define{USE_OPENSSL} = 1;
@@ -1051,6 +1055,11 @@ sub AddProject
$proj->AddIncludeDir($self->{options}->{xslt} . '\include');
$proj->AddLibrary($self->{options}->{xslt} . '\lib\libxslt.lib');
}
+ if ($self->{options}->{lz4})
+ {
+ $proj->AddIncludeDir($self->{options}->{lz4} . '\include');
+ $proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4_static.lib');
+ }
if ($self->{options}->{uuid})
{
$proj->AddIncludeDir($self->{options}->{uuid} . '\include');
@@ -1162,6 +1171,7 @@ sub GetFakeConfigure
$cfg .= ' --with-uuid' if ($self->{options}->{uuid});
$cfg .= ' --with-libxml' if ($self->{options}->{xml});
$cfg .= ' --with-libxslt' if ($self->{options}->{xslt});
+ $cfg .= ' --with-lz4' if ($self->{options}->{lz4});
$cfg .= ' --with-gssapi' if ($self->{options}->{gss});
$cfg .= ' --with-icu' if ($self->{options}->{icu});
$cfg .= ' --with-tcl' if ($self->{options}->{tcl});
--
1.8.3.1
