[fpc-pascal] code example where AnsiString used in FCL (SqlDB) causes data loss

[Graeme Geldenhuys]

Hi,

Here is an example [proof if you will] of the problem. I wrote a small
test program that reads data from a Firebird database where the database
and field charset is set to UTF8.

I compile the program, then run it. No recompiles between the two runs.
The first run my system is set to have a UTF-8 locale. The second run is
where I set my system to have a ISO8859-1 (Latin-1) locale. The program
outputs the DefaultSystemCodePage to the console.

Because the locale changes the behaviour of String (aka AnsiString) in
the RTL and FCL, the first run works, but the second run corrupts my data.

Console output:

[unicode_test]$ export LANG=en_US.UTF-8
[unicode_test]$ ./unicodetest
65001

[unicode_test]$ export LANG=en_US.ISO8859-1
[unicode_test]$ ./unicodetest
28591


In my test program I write the data read from the database to a file
using TFileStream, thus console and file encoding settings will not
affect the data being written to file. TFileStream is simply writing bytes.

The “locale_utf8.png” screenshots shows the actual data in the database
on the left, and the read (and saved to a file “output.data”) data on
the right

Compiled with 64-bit FPC 3.0.1 (updated yesterday) on my FreeBSD 10.3
system. Firebird v2.5.4 is being used. I can supply a backup of the test
Firebird database too if needed - it is small.


I'm honestly trying very hard to understand the string changes
implemented in FPC 3.x, and the best way to use it going forward. In
this example I tried everything I learned from the recent mailing list
discussions. My concern with the usage of String/AnsiString still
stands, as this test program shows.

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp

program project1;

{$mode objfpc}
{$H+}
{$modeswitch unicodestrings} // this makes String = UnicodeString

 // use UTF8String instead of String
{$DEFINE u8}

uses
  cwstring,
  classes,
  sysutils,
  db,
  sqldb,
  IBConnection;

const
  cBOM = #$EF#$BB#$BF;

var
  FDatabase: TIBConnection;
  FTransaction: TSQLTransaction;
  FQuery: TSQLQuery;
  f: TFileStream;
  u: {$IFDEF u8} UTF8String {$ELSE} String {$ENDIF};
begin
  writeln(DefaultSystemCodePage);
  FDatabase := TIBConnection.Create(nil);
  FDatabase.Dialect := 3;
  FDatabase.LoginPrompt := False;
  FDatabase.CharSet := 'UTF8';
  FDatabase.DatabaseName := '192.168.0.2:/data/devel/data/unicode_test.fdb';
  FDatabase.UserName := 'sysdba';
  FDatabase.Password := 'masterkey';

  FTransaction := TSQLTransaction.Create(nil);
  FDatabase.Transaction := FTransaction;

  FQuery := TSQLQuery.Create(nil);
  FQuery.DataBase := FDatabase;
  FQuery.SQL.Text := 'SELECT DESCRIPTION, UNIVALUE FROM UNICODE'; // where Description = ''Ligatures''';

  FDatabase.Connected := True;
  FQuery.Open;

  f := TFileStream.Create('output.data', fmCreate);
  f.Write(cBOM[1], 3);

  FQuery.First;
  while not FQuery.EOF do
  begin
    // field one
    u := FQuery.FieldByName('DESCRIPTION').AsString;
    f.write(u[1], Length(u));
    f.WriteByte(10); // new line
    // field two
    u := FQuery.FieldByName('UNIVALUE').AsString;
    f.write(u[1], Length(u));
    f.WriteByte(10); // new line
    f.WriteByte(10); // new line to separate records
    FQuery.Next;
  end;
  f.Free;

  FDatabase.Connected := False;

  FQuery.Free;
  FTransaction.Free;
  FDatabase.Free;
end.


_______________________________________________
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal