[fpc-pascal] what is the FPC equivalent of Delphi's zlib unit?

[Dennis Poon]

I need to use CompressBuf and DecompressBuf of zlib in fpc.

Thanks,

Dennis
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] what is the FPC equivalent of Delphi's zlib unit?

[Michael Van Canneyt]

There are 3 options:

 paszlib.pas pascal implementation of libz.

 zstream.pp  stream interface, uses paszlib.

 zlib.pp  native interface to C libz library.

Michael.

On Mon, 29 Jul 2013, Dennis Poon wrote:


I need to use CompressBuf and DecompressBuf of zlib in fpc.

Thanks,

Dennis



___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] what is the FPC equivalent of Delphi's zlib unit?

[Mark Morgan Lloyd]

Michael Van Canneyt wrote:

There are 3 options:

 paszlib.pas pascal implementation of libz.

 zstream.pp  stream interface, uses paszlib.

 zlib.pp  native interface to C libz library.


Is any one of those more "natural" than the rest, because e.g. it's 
already used for GIF handling?


--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] what is the FPC equivalent of Delphi's zlib unit?

[Dmitry Boyarintsev]

GIF or PNG?
PNG is using deflate algorithm.
GIF is using its own.


On Sun, Jul 28, 2013 at 4:53 PM, Mark Morgan Lloyd <
markmll.fpc-pas...@telemetry.co.uk> wrote:

> Michael Van Canneyt wrote:
>
>> There are 3 options:
>>
>>  paszlib.pas pascal implementation of libz.
>>
>>  zstream.pp  stream interface, uses paszlib.
>>
>>  zlib.pp  native interface to C libz library.
>>
>
> Is any one of those more "natural" than the rest, because e.g. it's
> already used for GIF handling?
>
> --
> Mark Morgan Lloyd
> markMLl .AT. telemetry.co .DOT. uk
>
> [Opinions above are the author's, not those of his employers or colleagues]
>
> __**_
> fpc-pascal maillist  -  
> fpc-pascal@lists.freepascal.**org
> http://lists.freepascal.org/**mailman/listinfo/fpc-pascal
>
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] what is the FPC equivalent of Delphi's zlib unit?

[Dennis Poon]

thanks a lot.

Dennis

Michael Van Canneyt wrote:


There are 3 options:

 paszlib.pas pascal implementation of libz.

 zstream.pp  stream interface, uses paszlib.

 zlib.pp  native interface to C libz library.

Michael.

On Mon, 29 Jul 2013, Dennis Poon wrote:


I need to use CompressBuf and DecompressBuf of zlib in fpc.

Thanks,

Dennis



___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal


-
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2013.0.3349 / Virus Database: 3209/6527 - Release Date: 07/28/13


___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Console Encoding in Windows (Local VS. UTF8)

[Noah Silva]

Hi,

This answer is a bit late, but...


2013/7/9 Dennis Poon 

>
> I have followed steps in the Wiki, etc., but to little avail, so I have
>> some questions for anyone who knows more than me:
>> ...
>
> ...
>
> 4. I write the output to a file instead of the console, and it's ok.
>>
>
> Please state the windows version you are using. XP or Windows 7?


Windows 7, but there isn't any difference in this case.


>  I deal with chinese in my programs so I know your problems. The same
> delphi 5 program works differently on XP and Windows 7.  Looks like Windows
> 7 has removed support for non unicode (I am not sure whether the Unicode it
> uses is UTF8, UTF16 or UTF32).
>
> When you say "removed support", what specifically are you talking about ?
 They both use the NT kernel and so use UTF16 most everywhere internally,
but the assume that most programs use local encodings.


> Seems that all filenames in XP are treated as unicode code. If you type a
> non unicode file name in Explorer, it will be auto converted to unicode.
>

Filenames as accessed from where?  Filenames in NTFS are always stored in
Unicode, so far as I know.  When you enter a file name in Explorer, how do
you know if it is being entered in Unicode or not?  Basically, explorer is
probably a Unicode-aware program (so far as I know), so I would assume any
text you enter starts life as Unicode via the IME.


> Also, in XP, when text is copied to MS Office from other programs and vice
> versa, XP seems to do an automatic conversion to UTF-8 and vice versa.
>

MS Office is also of course Unicode Aware, and the newer XML file formats
they use save everything as as Unicode to the best of my knowledge.  When
importing/exporting text files, clipboard, etc., they will use the local
encoding for most things.

That is if you copy some text in your program which is encoding in SJIS and
> paste it to Word, the text seems to be auto converted to unicode.
>
> Seems reasonable.


> As for file handling, it seems some programs will read the first 2 bytes
> of the text file to determine the the encoding of the file. Google about it.
>

This is well known, though many programs use much more than 2 bytes in
order to guess if there is no BOM.  Though Microsoft's guessing algorithm
is rather entertaining at times.


> Sorry, I don't have exact answers to your questions. Just share some of my
> experience.
>
>
Well certainly the more the merrier, other people who search this list may
benefit from your comments even if they aren't relevant to my case.

My problem is this:
I want to output text to the console and have it not be corrupted.  I have
the text in my program as UTF8, but when I write it to the console, it
shows corrupted garbage.  This happens whether I write it out as UTF8 or
ANSI (SJIS).  Since the console basically works with local encoding, it
should work when I use UTF8ToAnsi to convert the text before writing it,
but it doesn't.

Since it works with other programs and things like the "type" command, it
occurred to be that it might be a FPC issue, or that I might be interfering
with some built-in auto-conversion feature (which is why I tried to output
as UTF8 as well).

I'll figure it out eventually, though.

Dennis
>
>
Thank you,
Noah Silva

> __**_
> fpc-pascal maillist  -  
> fpc-pascal@lists.freepascal.**org
> http://lists.freepascal.org/**mailman/listinfo/fpc-pascal
>
>
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Console Encoding in Windows (Local VS. UTF8)

[Noah Silva]

Hello everyone,

OK I have checked this out:
2013/7/9 Tomas Hajny 

>
> There is code page 65001 for UTF-8, however I don't know since which
> version of MS Windows it is supported (and to which extent - e.g.
> including the console output). You may change the output code page for
> console using the Windows API call SetConsoleOutputCP, but e.g. our unit
> Crt currently resets the code page used for output to console to the
> "main" process code page (as returned by GetACP) on each write and thus
> ignores the user selection - this will probably change in trunk in the
> future as part of the Unicode support activities, but the current state is
> like this.
>
>
According to MSDN, GetACP returns ANSI code page (i.e. never Unicode pages
CP_UTF8)
On my system, the codepage it returns is 932 (SJIS), so writing SJIS to the
console should be no issue, even is writeln resets the code page every
time.

http://msdn.microsoft.com/ja-JP/goglobal/cc305152.aspx

Yet, doing the following yields garbage:
  Writeln(UTF8ToANSI(Object.Text));

(and I have verified that the Text field content is indeed in UTF8 format).

As for my FPC version:
Lazarus 1.1
2012-11-06
FPC 2.6.1
SVN 37432
x86_64-win64-win32/win64

(I have various other versions installed, but all on OS X machines...)

Tomas
>
>  thank you,
 Noah Silva

>
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Console Encoding in Windows (Local VS. UTF8)

[Noah Silva]

Hi,

2013/7/9 Jonas Maebe 

>
> On 09 Jul 2013, at 11:02, Noah Silva wrote:
>
>  1. What encoding "should" I be writing to the terminal?
>>
>
> The console code page. You can get it using the following function:
>   function GetConsoleOutputCP : UINT; stdcall; external 'kernel32' name
> 'GetConsoleOutputCP';
>
>
>
This is 932 on my system, as expected.

http://msdn.microsoft.com/ja-JP/goglobal/cc305152.aspx

>  from experimenting
>> with text files using the cat command in powershell, it seems that local
>> ("ANSI") encoding should be used.  This makes sense since older versions
>> of
>> windows only supported local encodings.
>>
>
> The ansi code page is (or at least can be) different, that's the the
> result of
>   function GetACP:UINT; stdcall; external 'kernel32' name 'GetACP';
>
> This is also 932, as expected.
(I assume they should always be the same though, I don't think most
programs check the console output page before writing).


>  2. Is there any reason why writing out data in the local encoding (with
>> write statements, etc.) should get corrupted?  For example is some level
>> of
>> the RTL assuming something about the encoding? (I don't think so, but...)
>>
>
> Not in 2.6.x. In 2.7.x, every ansistring is tagged with a code page (the
> ansi code page by default) and the RTL will convert it to the console code
> page before writing it.
>
> And I am using 2.6, so it should be clean, yet somehow it's not working
out that way.  I think I will have to debut writeln...


> Jonas
>
> Thank you,
   Noah Silva
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Console Encoding in Windows (Local VS. UTF8)

[Noah Silva]

Hi,

2013/7/9 Michael Schnell 

> On 07/09/2013 11:02 AM, Noah Silva wrote:
>
>>
>> I convert it to UTF8 before displaying it
>>
>>  Not a good idea.
>
> Well if the console is UTF8


> The FPC developers are right now busy implementing the new Delphi Strings.
> This _could_ mean that the application programmer can  use any encoding
> (such as multiple different ANSI byte-codes, UTF-8, UTF-16, ...), but in
> fact to be 100% Delphi compatible ("nothing less, nothing more"), it seems
> that only UTF-16 will gain full decent support (e.g. class inheritance, in
> TStringList, the Lazarus user API etc.)
>
> Using UTF16 for internal string handling is a sensible option.  That's
what the Windows API does and what f.e. SAP's ABAP does.  On the other
hand, UTF8 is very common in files and transferring string data via things
like HTTP/XML, so it has to be fully supported any way around it.  OS X
uses UTF8 as the "local" encoding (so you never have to worry there, except
in Java), and apparently so does GTK2.  To make it simple, UTF8 can be used
everywhere that ANSI encodings were used, because it is ASCII compatible
when only ASCII is used.  UTF16 and UTF32 can't easily be substituted
because they contain "padding" bytes for normal ASCII.

For things like WideString, this is fine.  The reason UTF16/UTF32 are
popular for in-memory variables is that it's easy to achieve higher
performance.  For example, with UTF32, there is no need to "decode" the
string to find out what character you are on, how many bytes a certain
character takes up, etc.  If you want the 4th character of a string, you
simply go to the 4th 4 byte array element and retrieve the value.  UTF16
works the same way if you are dealing with the 99% of characters in use
that take only two bytes (but this leads to bugs because people usually
don't handle the remaining 1% properly).  So you gain processing speed and
code simplicity by using UTF16 or UTF32.  You lose out on memory if you are
dealing with ASCII data only - which is no big deal in most cases.  UTF8
saves memory and is more ASCII compatible, but requires more
decoding/encoding.  Since they represent the same character set, it doesn't
really matter in the end - there is a trade-off either way.  If you are
doing mainly I/O, UTF8 is convenient, if you are doing heavy duty string
processing, UTF32 is convenient.  Either way, not supporting one or the
other is simply not an option if you want to be able to write Unicode
compliant programs.  One can be the "main" way used by internal routines,
and this is UTF16 in more operating systems than not.

If FPC used UTF8 for everything and automatically converted it, then calls
to the Win32 API would be slowed down by this, so it makes sense to use
UTF16 on Windows, but... then again if GTK2 requires UTF8 then you have the
same (but opposite) problem there.  Lazarus also doesn't support only
Windows, so we have to think a little wider than Delphi.

Another interesting point is that I have heard no end of complaints about
Delphi's Unicode strategy, so while we want to be compatible, perhaps we
should consider how to do that while possibly avoiding some of the same
pitfalls.

To address your specific points:
1.Lazarus User API already supports UTF8 so far as I know.
2. TStringList could easily support both, but as long as the conversion
to/from other code pages (especially UTF8) is automatic, I wouldn't mind.
3. Not sure what class inheritance has to do with this...

-Michael

Thank you,
Noah Silva

p.s.: Unicode is an area that I know a lot about, so if anyone working on
the RTL needs help testing, let me know...


> __**_
> fpc-pascal maillist  -  
> fpc-pascal@lists.freepascal.**org
> http://lists.freepascal.org/**mailman/listinfo/fpc-pascal
>
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Console Encoding in Windows (Local VS. UTF8)

[Noah Silva]

Hi,

2013/7/9 Jonas Maebe 

>
> The console code page. You can get it using the following function:
>   function GetConsoleOutputCP : UINT; stdcall; external 'kernel32' name
> 'GetConsoleOutputCP';
>
>  2. Is there any reason why writing out data in the local encoding (with
>> write statements, etc.) should get corrupted?  For example is some level
>> of
>> the RTL assuming something about the encoding? (I don't think so, but...)
>>
>
> Not in 2.6.x. In 2.7.x, every ansistring is tagged with a code page (the
> ansi code page by default) and the RTL will convert it to the console code
> page before writing it.
>
>
> Oddly enough, I can get my (2.6.1) version to change the garbage it shows
but but saving the file as UTF8 with BOM.  (Until now, I had been using
UTF8 w/o BOM).

http://wiki.freepascal.org/LCL_Unicode_Support/ja

And the garbage it shows is the same garbage I see if I convert from UTF8
to ANSI manually using UTF8toAnsi or UTF8ToSys.

Detecting the Unicode BOM or not seems to be a strange way to switch the
behavior of the compiler, I wonder why a $define wasn't used instead.
 (Looking at the difference in handling, the BOM way seems much more
convenient, so I will convert things over to that...)

Jonas
>

Thank you,
Noah Silva

> __**_
> fpc-pascal maillist  -  
> fpc-pascal@lists.freepascal.**org
> http://lists.freepascal.org/**mailman/listinfo/fpc-pascal
>
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

[fpc-pascal] Re: Console Encoding in Windows (Local VS. UTF8)

[Noah Silva]

Hello everyone!

Success!  Writeln was indeed the main culprit.

The following works:
  WritelnUTF8(('あいうえお秋葉原'));

(Where:
1. WritelnUTF8 is a procedure that uses the Win32 API WriteConsole
directly, bypassing whatever writeln is doing.
2. Where the source code is saves as UTF8 with BOM
3. Where I don't set the console output page to UTF8)

But I'm outputting UTF8 to an ANSI console and it works?!
Preseumably, somewhere the RTL sees that my source code is UTF8 and decodes
to convert it to ANSI and that gets output and it works since the codepage
is ANSI.

A cleaner path would be to set the output codepage to ANSI and tell the
compiler not to touch my UTF8, but that doesn't seem to work.

My guess is that Writeln is somehow/somewhere trying to convert the already
converted text again, which would very certainly result in garbage!

Once I got the idea to use WriteConsole, I searched and found this:
http://forum.lazarus.freepascal.org/index.php?topic=17548.0

(svn: 37432)

thank you,
Noah Silva


2013/7/9 Noah Silva 

> Hi,
>
> I deal with in Japanese (and sometimes other languages) in a lot of my
> programs, and nothing I do seems to work consistently on Windows systems.
>  (OS X is no problem).
>
> I have followed steps in the Wiki, etc., but to little avail, so I have
> some questions for anyone who knows more than me:
> 1. What encoding "should" I be writing to the terminal?  from
> experimenting with text files using the cat command in powershell, it seems
> that local ("ANSI") encoding should be used.  This makes sense since older
> versions of windows only supported local encodings.
> 2. Is there any reason why writing out data in the local encoding (with
> write statements, etc.) should get corrupted?  For example is some level of
> the RTL assuming something about the encoding? (I don't think so, but...)
> 3. Is there a way to set the output to UTF8 so I can just write out UTF8
> and be done with it?
>
> Just to give an example:
> 1. I read in an SJIS CSV file, and display it on the screen, and it's
> corrupted.
> 2. I convert it to UTF8 before displaying it, and it's still corrupted.
> 3.I cat the file to the screen and it's ok.
> 4. I write the output to a file instead of the console, and it's ok.
>
> Something seems odd.
>
> Does anyone else have these issues?
>
> Thank you,
> Noah Silva
>
> p.s.: I know that the actual data in my programs isn't broken, because if
> I write it to a file or database, there is no problem with corruption.
>
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal