Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Vincent Snijders]

2011/12/7 Graeme Geldenhuys :
> Hi,
>
> I did a simple GetTickCount() timing around this loop. Delphi executes
> the loop in 20 ticks. FPC 2.6.0-rc2 takes 10585 ticks The outer
> loop runs 200400 iterations. The types for BitValue, ByteValue and
> RandSeed is of type Byte.
>
> 01  for Index := 1 to Length(Source) do
> 02  begin
> 03    OrdValue := Ord(Source[Index]);
> 04    for BitCount := 0 to 7 do
> 05    begin
> 06      BitValue := Byte(OrdValue and (1 shl BitCount) = 1 shl BitCount);
> 07      RandSeed := ByteValue;
> 08      ByteValue := (((Random(255) + 1) div 2) * 2) + BitValue;
> 09      Result[(Index - 1) * 8 + BitCount + 1] := AnsiChar(ByteValue);
> 10    end;
> 11  end;
>

I have one question about this code, why is RandSeed set inside the
loop and not outside the loop or even at the program start?
If you change the randseed, then the random number generator has to be
initialized and that is more costly for a stateful RNG as the mersenne
twister.

Vincent
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Re: Why is Random(255) some 529x slower compared to Delphi7?

[Graeme Geldenhuys]

On 9 December 2011 09:55, Vincent Snijders  wrote:
> Can you compile the fpc version for 32 bits (may make some difference)
> and run it on windows (probably make no difference) to reduce the
> differences in platform when you compare with Delphi?


---FPC 2.4.5 (r17953 from fixes_2_4 branch) ---
Y:\tests\random_test>C:\FPC\2.4.5\bin\ppc386.exe speedtest.pas
Free Pascal Compiler version 2.4.5 [2011/08/15] for i386
Copyright (c) 1993-2010 by Florian Klaempfl

Y:\tests\random_test>speedtest
Length(psData) = 200401
ByteValue: 180
Length(Source) = 200401
First Loop
elapsed time: 7421
-


So that's 7421 (FPC 32bit) divided by 11 (Delphi 7) = 674 times slower


I'm compiling FPC 2.5.1 for 32bit windows now, to see if there is any
difference between FPC versions, but I doubt there would be a big
difference.

What is weird is that the 32bit FPC is so much faster than the 64bit
FPC. Strange...

-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Florian Klaempfl]

Am 09.12.2011 08:59, schrieb Graeme Geldenhuys:
> On 9 December 2011 09:47, Florian Klaempfl wrote:
>>
>> According to measurements of me and other peoples, random is only 7-10
>> times slower (depending on the CPU).
> 
> What do you feed your computer, 

Nothing, but I don't mess with things I don't understand :) Changing
randseed (and makes absolutely no change except if your random generator
might be bad ;)) during random number generation is really a bad idea:

c:\>test
Length(psData) = 200401
ByteValue: 187
Length(Source) = 200401
First Loop
elapsed time: 6751

c:\>test
Length(psData) = 200401
ByteValue: 174
Length(Source) = 200401
First Loop
elapsed time: 60

> because mine differs vastly from yours.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Florian Klaempfl]

Am 09.12.2011 09:02, schrieb Vincent Snijders:
> 2011/12/7 Graeme Geldenhuys :
>> Hi,
>>
>> I did a simple GetTickCount() timing around this loop. Delphi executes
>> the loop in 20 ticks. FPC 2.6.0-rc2 takes 10585 ticks The outer
>> loop runs 200400 iterations. The types for BitValue, ByteValue and
>> RandSeed is of type Byte.
>>
>> 01  for Index := 1 to Length(Source) do
>> 02  begin
>> 03OrdValue := Ord(Source[Index]);
>> 04for BitCount := 0 to 7 do
>> 05begin
>> 06  BitValue := Byte(OrdValue and (1 shl BitCount) = 1 shl BitCount);
>> 07  RandSeed := ByteValue;
>> 08  ByteValue := (((Random(255) + 1) div 2) * 2) + BitValue;
>> 09  Result[(Index - 1) * 8 + BitCount + 1] := AnsiChar(ByteValue);
>> 10end;
>> 11  end;
>>
> 
> I have one question about this code, why is RandSeed set inside the
> loop and not outside the loop or even at the program start?
> If you change the randseed, then the random number generator has to be
> initialized and that is more costly for a stateful RNG as the mersenne
> twister.

Oops, mails crossed. The assignment to randseed is indeed the problem.
Why is it done? Bad random generator of delphi :)?
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 10:02, Vincent Snijders wrote:
>
> I have one question about this code, why is RandSeed set inside the
> loop and not outside the loop or even at the program start?


For the full code as used by tiOPF, see the following URL.

 
http://tiopf.svn.sourceforge.net/viewvc/tiopf/tiOPF2/Trunk/Options/tiEncryptSimple.pas?revision=2143&view=markup

I didn't write this encryption code, I merely debugged why the unit
tests for this unit took so long to complete, compared to under
Delphi.

Looking at the code again, I have no idea how it will affect the
encryption algorithm if I move the assignment to RandSeed outside the
loop (just after the first assignment to ByteValue). As a test, made
your suggested change and ran the unit tests. The unit tests still
pass, and the performance under FPC has drastically improved. FPC
(64-bit) is now roughly 4 times slower that Delphi. Much better
indeed, and an acceptable result.

I'll see if I can get hold of the original author of this code and
what they think about the suggested change - and if it will affect the
encryption algorithm at all.

Thanks Vincent for mentioning this. I totally overlooked that.


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Felipe Monteiro de Carvalho]

On Fri, Dec 9, 2011 at 9:39 AM, Graeme Geldenhuys
 wrote:
> I didn't write this encryption code, I merely debugged why the unit
> tests for this unit took so long to complete, compared to under
> Delphi.

It is specifically written in the Delphi documentation that Random
should not be utilized for encryption...

-- 
Felipe Monteiro de Carvalho
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 10:42, Felipe Monteiro de Carvalho  wrote:
>
> It is specifically written in the Delphi documentation that Random
> should not be utilized for encryption...

Delphi documentation mentions a lot of things you mustn't do... Does
that stop anybody. ;-)
Like I said, I didn't write that code, and I don't specialise in
encryption algorithms. But Florian might be right, the RandSeed
assignment might have been added due to the bad random generator of
Delphi - hopefully the original author of the code can shed some
light.


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Dimitri Smits]

- "Felipe Monteiro de Carvalho"  schreef:

> On Fri, Dec 9, 2011 at 9:39 AM, Graeme Geldenhuys
>  wrote:
> > I didn't write this encryption code, I merely debugged why the unit
> > tests for this unit took so long to complete, compared to under
> > Delphi.
> 
> It is specifically written in the Delphi documentation that Random
> should not be utilized for encryption...
> 

true, (but) looking at the code again, it seems that you always have a 
predictable sequence when using the same algorithm. Not sure if that is a good 
thing or a bad one in cryptology :-). After all, when you do not randomize() 
first, randseed has a default startupvalue (and otherwise it is typically 
seeded with a timestamp of somesorts). 

I don't remember where I read it (ages ago), and the comment in Delphi seems to 
negate that this is the used algorithm, but this 'predictable sequence from the 
same seed'-property is especially true when using a LCG 
pseudo-random-number-generator.

Just to be sure, the wikipedia article DOES mention that Delphi (and every 
other HL language that matters :-)) supplies a Random functionality that is 
based on a LCG.

http://en.wikipedia.org/wiki/Linear_congruential_generator

And in the java realm there are numerous other algorithms available, but the 
default implementation with the language libraries is a LCG, as does the C(++).

Reading the article again, I find a few paragraphs corresponding to the Delphi 
help. Excerpt from the 'advantages and disadvantages' part of the page:

--
LCGs should not be used for applications where high-quality randomness is 
critical. For example, it is not suitable for a Monte Carlo simulation because 
of the serial correlation (among other things). They should also not be used 
for cryptographic applications; see cryptographically secure pseudo-random 
number generator for more suitable generators. If a linear congruential 
generator is seeded with a character and then iterated once, the result is a 
simple classical cipher called an affine cipher; this cipher is easily broken 
by standard frequency analysis.
--

kind regards,
Dimitri Smits
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Jonas Maebe]

On 09 Dec 2011, at 09:39, Graeme Geldenhuys wrote:


Looking at the code again, I have no idea how it will affect the
encryption algorithm if I move the assignment to RandSeed outside the
loop


It will improve the randomness of the generated numbers. Changing the  
random seed all the time removes any properties the random generator  
normally has. In other words, the original code is wrong.



Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Dimitri Smits]

- "Graeme Geldenhuys"  schreef:

> On 9 December 2011 10:42, Felipe Monteiro de Carvalho  wrote:
> >
> > It is specifically written in the Delphi documentation that Random
> > should not be utilized for encryption...
> 
> Delphi documentation mentions a lot of things you mustn't do... Does
> that stop anybody. ;-)

only those that take their jobs seriously and read help/manuals iso assuming? 
:-)

> Like I said, I didn't write that code, and I don't specialise in
> encryption algorithms. But Florian might be right, the RandSeed
> assignment might have been added due to the bad random generator of
> Delphi - hopefully the original author of the code can shed some
> light.
> 

I actually doubt that that codesnippet does any real encryption. From what I 
understand from it (with the provided code and assuming types), the 
source-message is bitstreamed. In the resulting message, the LSB of every 
octet/byte/8-bitvalue is the one you need to focus on, the rest is random data. 
Also, as a result, the result-message is 8 times as large. And that LSB is also 
easily decrypted => 
parse the LSB from every byte in your data and reconstruct using OR the (index 
mod 8)-th bit on the (index div 8) octet.

hardly encrypted ;-)

kind regards,
Dimitri Smits
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

[fpc-pascal] Re: Why is Random(255) some 529x slower compared to Delphi 7?

[Virgo Pärna]

On Fri, 09 Dec 2011 09:19:53 +0100, Florian Klaempfl  
wrote:
>
> Oops, mails crossed. The assignment to randseed is indeed the problem.
> Why is it done? Bad random generator of delphi :)?
>

I don't know, how bad Delphis random generator is, but I once years ago did 
make a mistake
of  calling randomize before random. It was a school assignment for using Turbo 
Pascal graphics.
So I made program, that was supposed to fill screen with random dots (colour 
was also set with 
random). Anyway, when I called randomize before random, then as a result dots 
didn't fill the 
screen, but were mostly along the diagonal of the screen. So I'd guess, that 
setting randseed 
multiple times could be actually bad for randomness of results. Or maybe modern 
algorithms are
better at situations like this?

-- 
Virgo Pärna 
virgo.pa...@mail.ee

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Re: Why is Random(255) some 529x slower compared to Delphi 7?

[Dimitri Smits]

- "Virgo Pärna"  schreef:

> On Fri, 09 Dec 2011 09:19:53 +0100, Florian Klaempfl
>  wrote:
> >
> > Oops, mails crossed. The assignment to randseed is indeed the
> problem.
> > Why is it done? Bad random generator of delphi :)?
> >
> 
> I don't know, how bad Delphis random generator is, but I once
> years ago did make a mistake
> of  calling randomize before random. It was a school assignment for
> using Turbo Pascal graphics.
> So I made program, that was supposed to fill screen with random dots
> (colour was also set with 
> random). Anyway, when I called randomize before random, then as a
> result dots didn't fill the 
> screen, but were mostly along the diagonal of the screen. So I'd
> guess, that setting randseed 
> multiple times could be actually bad for randomness of results. Or
> maybe modern algorithms are
> better at situations like this?
> 

Randomize() is supposed to be called only once to seed the generator with an 
initial value. 

If you made it something like so:

begin
  for i := 0 to 1000 do
  begin
randomize();
pixel[random(screenwidth),random(screenheight)]:= clSomeColor;
  end;
end;

then you seed with a timestamp before randoming and your distribution does not 
change much with a lcg. Not the fault of the algorithm itself, just your 
mistake of doing the randomize in the loop itself (like you mentioned yourself).


kind regards,
Dimitri Smits
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Re: Why is Random(255) some 529x slower compared to Delphi 7?

[Marco van de Voort]

In our previous episode, Dimitri Smits said:
> 
> Randomize() is supposed to be called only once to seed the generator with an 
> initial value. 
> 
> If you made it something like so:
> 
> begin
>   for i := 0 to 1000 do
>   begin
> randomize();
> pixel[random(screenwidth),random(screenheight)]:= clSomeColor;
>   end;
> end;
> 
> then you seed with a timestamp before randoming and your distribution does
> not change much with a lcg.  Not the fault of the algorithm itself, just
> your mistake of doing the randomize in the loop itself (like you mentioned
> yourself).

(specially since high quality rng's might call OS dependent sources for
initial entropy in that first call)
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Reimar Grabowski]

On Fri, 09 Dec 2011 07:27:46 +0100
Jürgen Hestermann  wrote:

> 
> 
> Reimar Grabowski schrieb:
> > The parameter should default to FALSE to not break existing code relying on 
> > FPCs random function 
> And what about existing code coming from Delphi/Turbo Pascal? This was a 
> strong argument in the past for doing even crap coding.

Not by me, but the FPC developers. I am not interested in Delphi/TP 
compatability and never was, so I don't care if Delphi/TP code works or not, 
but I care if code written exclusively in and for FPC continues to work as it 
did. I think that Delphi is a burden for FPC and Lazarus, but the developers 
have other opinions, so be it.
I don't need a change at all. It may be irritating for a Delphi developer that 
his code runs slower, but profiling is no secret art and changing a random 
implementations to a less 'good' but faster one is a matter of minutes.
Like people already said, lots of talk about a 'problem' that 10 years noone 
has seen as one.

Another 2 cents
R.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Marco van de Voort]

In our previous episode, Reimar Grabowski said:
> Like people already said, lots of talk about a 'problem' that 10 years noone 
> has seen as one.

(it's afaik not the first. Has been noticed once or twice before. But those
people used it in unittests, and simply changed without much ado when the
problem was identified.  Contrary to the fact that before the twister
change, people complaining about random being "not as random as Delphi"
several times per year)
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Reimar Grabowski]

On Fri, 9 Dec 2011 10:47:15 +0200
Graeme Geldenhuys  wrote:

> Like I said, I didn't write that code, and I don't specialise in
> encryption algorithms.
That's not your fault. Your fault was to not identify the problem correctly but 
blame FPCs implementation and later rant about "double standards" and "unusable 
code", without the knowledge to back up your statements. Next time please take 
the time to identify the problem correctly before jumping to conclusions.
No offense ment.

R.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

[fpc-pascal] How to poll for a byte in Input?

[tcoq]

Hello,
I'm trying to poll the input stream to see whether there is a new character 
available, without blocking my software.
I've tried "Read" and "THandleStream.ReadByte" but with no success: when the 
input is empty, the program just waits.

Is there a way to ask whether the input stream is not empty, without waiting 
and without using threads/processes?
And, in addition, is there an OS-independent way (linux, windows)?

something like:
var 
  inputStream: THandleStream;
  aByte: Byte;
begin
  InputStream := THandleStream.Create(StdInputHandle);
  try
while true do
begin
  {$I-}
  aByte := InputStream.ReadByte;
  DoSomethingWithByte(aByte);
  result := true;
end;
  except
On E: EStreamError do result := false;
  end;
  //This code here is never executed
  ThereIsNothingInInputStream_DoDefault;
end;
  

(PS. In windows, I traced the "readByte" procedure down to a Windows API 
function: ReadFile in Kernel32. This seems to be a blocking function).
Best regards,
Thierry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Michael Van Canneyt]

On Fri, 9 Dec 2011, tcoq wrote:


Hello,
I'm trying to poll the input stream to see whether there is a new character 
available, without blocking my software.
I've tried "Read" and "THandleStream.ReadByte" but with no success: when the 
input is empty, the program just waits.

Is there a way to ask whether the input stream is not empty, without waiting 
and without using threads/processes?
And, in addition, is there an OS-independent way (linux, windows)?

something like:
var
 inputStream: THandleStream;
 aByte: Byte;
begin
 InputStream := THandleStream.Create(StdInputHandle);
 try
   while true do
   begin
 {$I-}
 aByte := InputStream.ReadByte;
 DoSomethingWithByte(aByte);
 result := true;
   end;
 except
   On E: EStreamError do result := false;
 end;
 //This code here is never executed
 ThereIsNothingInInputStream_DoDefault;
end;



You must use fpSelect() or Poll on the handle on linux/unix.
On windows, you'll need to WaitForObject() or so on the file handle.

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[tcoq]

Thanks Michael,
I'll try that!
Thierry
- Mail Original -
De: "Michael Van Canneyt" 
À: "FPC-Pascal users discussions" 
Envoyé: Vendredi 9 Décembre 2011 16h57:47 GMT +01:00 Amsterdam / Berlin / Berne 
/ Rome / Stockholm / Vienne
Objet: Re: [fpc-pascal] How to poll for a byte in Input?
...

You must use fpSelect() or Poll on the handle on linux/unix.
On windows, you'll need to WaitForObject() or so on the file handle.

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Marco van de Voort]

In our previous episode, Michael Van Canneyt said:
> > Is there a way to ask whether the input stream is not empty, without 
> > waiting and without using threads/processes?
> > And, in addition, is there an OS-independent way (linux, windows)?
> >
> > something like:
> > var
> >  inputStream: THandleStream;
> >  aByte: Byte;
> > begin
> >  InputStream := THandleStream.Create(StdInputHandle);
> >  try
> >while true do
> >begin
> >  {$I-}
> >  aByte := InputStream.ReadByte;
> >  DoSomethingWithByte(aByte);
> >  result := true;
> >end;
> >  except
> >On E: EStreamError do result := false;
> >  end;
> >  //This code here is never executed
> >  ThereIsNothingInInputStream_DoDefault;
> > end;
> 
> You must use fpSelect() or Poll on the handle on linux/unix.
> On windows, you'll need to WaitForObject() or so on the file handle.

Be careful on streams that might have buffers though. Then you first need to
check if there are still bytes in the buffer.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 15:51, Reimar Grabowski wrote:
>knowledge to back up your statements. Next time please take the time to 
>identify the problem >correctly before jumping to conclusions.
> No offense ment.

No offense take. Two unknown (to most) facts came out of this
discussion.  1) the FPC Random() function IS slower than Delphi's. 2)
FPC implements a different random generator than Delphi. Is this good
or bad? Considering the "must be Delphi compatible" term thrown around
so often. Even with the original (slightly non-optimal usage of
Random) code, Delphi ran it fine and fast, FPC didn't. So from a
porting point of view (Delphi to FPC), there was a problem which
others might get caught in as well.

Either way, thanks to Vincent's sharp eye, which everybody but Vincent
overlooked, the speed problem in the tiOPF code can be greatly reduced
to acceptable levels with FPC - even though FPC has a different random
generator.

Thanks to all that participated and helped solve my problem. It was an
interesting discussion as always.


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 12:42, Jonas Maebe > wrote:
>
> It will improve the randomness of the generated numbers.

Thanks Jonas.



-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 12:50, Dimitri Smits  wrote:
>
> I actually doubt that that codesnippet does any real encryption.

It isn't. The sample code / test program I posted is just a snippet of
the actual unit. No point in posting the whole unit here, just to
point out that a single section of code in one method is the location
of the slow-running code under FPC. I only supplied the slow-running
code to demonstrate the problem.


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[tcoq]

On windows, I did not find WaitForSingleObject, which is actually going to wait 
for the file.
However I found ReadFileEx in the Windows API, which is an asynchronous read on 
the file. I'll try using that.

- Mail Original -
De: "Michael Van Canneyt" 
À: "FPC-Pascal users discussions" 
Envoyé: Vendredi 9 Décembre 2011 16h57:47 GMT +01:00 Amsterdam / Berlin / Berne 
/ Rome / Stockholm / Vienne
Objet: Re: [fpc-pascal] How to poll for a byte in Input?
>...
You must use fpSelect() or Poll on the handle on linux/unix.
On windows, you'll need to WaitForObject() or so on the file handle.

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Jorge Aldo G. de F. Junior]

Thats an old problem on the way TStream was implemented.

I for one needed to know the result size of a uncompress stream (IE.:
i have a compressed stream and want to know in advance how many bytes
the uncompress method would yield).

The stream system is currently a mess, sometimes you can know in
advance how many bytes you can read, sometimes you cant, depends on
the class you are dealing with.

2011/12/9 tcoq :
> On windows, I did not find WaitForSingleObject, which is actually going to 
> wait for the file.
> However I found ReadFileEx in the Windows API, which is an asynchronous read 
> on the file. I'll try using that.
>
> - Mail Original -
> De: "Michael Van Canneyt" 
> À: "FPC-Pascal users discussions" 
> Envoyé: Vendredi 9 Décembre 2011 16h57:47 GMT +01:00 Amsterdam / Berlin / 
> Berne / Rome / Stockholm / Vienne
> Objet: Re: [fpc-pascal] How to poll for a byte in Input?
>>...
> You must use fpSelect() or Poll on the handle on linux/unix.
> On windows, you'll need to WaitForObject() or so on the file handle.
>
> Michael.
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Jorge Aldo G. de F. Junior]

Well, lets go to theory :

One way to build a cypher is to XOR the stream that must be encrypted
against a fixed value.

But, this is easy to break using statistical methods.

So the next logical way to do this is to XOR the stream against
another stream of numbers, kind of one time password.

But, the stream of numbers must be shared by both ends.

How ? Simple, generate the stream on-the-fly using a PRNG.

Now you only need to share the seed of the PRNG and the encrypted stream.

But for this to work the same PRNG must be used on both ends, and the
PRNG must be strong enough with a period much longer than the size of
the data being encrypted.

Whats wrong with calling RANDOMIZE during the loop ? The problem is
that you keep seeding the PRNG with the Now() (current time stamp),
defeating the whole idea. Now, instead of XORing your plain text
against a pseudo-random sequence, you are XORing your plain text
against the current time/date, something quite easily predictable.

Even in your case, seeding a Mersenne Twister in every loop
interaction will put that algorithm exactly in its worst case : the
startup sequence.

The mersenne Twister is slow to start generating good random numbers.

So your program is bug to the guts in this case.

To solve this all you have to do is to guarantee that both
communication ends use the same random number generation algorithm and
seed. This will work. (IE.: the fix will break compatibility with old
binaries). And dont seed against the last result, thats not a good
idea (This will keep the MT restarting).

A good alternative to a simple encryption algorithm is to use
cryptographic hash functions like SHA.

hash your key using SHA.

Split your file in blocks the same size of the SHA key.

XOR the first block against the first key,

output the bytes.

Hash the SHA key itself, generating a second key,

XOR the second block against that second key.

repeat until end of data. PAD the last block to match the Hash size.


2011/12/9 Graeme Geldenhuys :
> On 9 December 2011 12:50, Dimitri Smits  wrote:
>>
>> I actually doubt that that codesnippet does any real encryption.
>
> It isn't. The sample code / test program I posted is just a snippet of
> the actual unit. No point in posting the whole unit here, just to
> point out that a single section of code in one method is the location
> of the slow-running code under FPC. I only supplied the slow-running
> code to demonstrate the problem.
>
>
> --
> Regards,
>   - Graeme -
>
>
> ___
> fpGUI - a cross-platform Free Pascal GUI toolkit
> http://fpgui.sourceforge.net
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Jorge Aldo G. de F. Junior]

even if FPC implemented a ultra high tech PRNG it would be compatible
with DELPHI :

1 - Delphi asserts that you should not use the Random function to
encryption porpuses.
2 - Delphi asserts no speed guarantees.
3 - Delphi asserts no randomness quality guarantees.

IE : to be compatible you only need the function to have the same
calling parameters and result type.

Formally, in that case, even this would be "compatible" :

Function Random(): Real;
Begin
Result := 4; { i throwed a dice to reach that result ! }
End;

2011/12/9 Graeme Geldenhuys :
> On 9 December 2011 12:42, Jonas Maebe > wrote:
>>
>> It will improve the randomness of the generated numbers.
>
> Thanks Jonas.
>
>
>
> --
> Regards,
>   - Graeme -
>
>
> ___
> fpGUI - a cross-platform Free Pascal GUI toolkit
> http://fpgui.sourceforge.net
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Michael Van Canneyt]

On Fri, 9 Dec 2011, Jorge Aldo G. de F. Junior wrote:


Thats an old problem on the way TStream was implemented.

I for one needed to know the result size of a uncompress stream (IE.:
i have a compressed stream and want to know in advance how many bytes
the uncompress method would yield).


This information is not stored in the compressed data. 
To be able to give you this number, the compression stream

would need to actually decompress the whole stream.



The stream system is currently a mess, sometimes you can know in
advance how many bytes you can read, sometimes you cant, depends on
the class you are dealing with.


There are simply streams which have no size.

In each case, I am working on the 'StreamCapabilities' property, 
which should be able to tell you what is available.


Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Readline substitute

[noreply]

> 2011/5/1 Johann Glaser :
>
>> If you find any improvements or comments don't hesitate to send me an
>> EMail.
>
> I just used it. I only needed readline() and add_history(), these two
> functions are already enough to make a ReadLn() substitute that really
> works and It works like a charm! Thank you!
>
> I think this code really deserves to be included in FPC.

Be very careful with ReadLine GPL library. It is not LGPL as far as i
know, and therefore it is viral stallmanware... requires we relicense
anything else as gpl if you use the library...

See what happened to common lisp clisp when they used it. Had to change
license to GPL even though they didn't want to.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Jorge Aldo G. de F. Junior]

I think the main problem is that TStream tries to abstract two kinds
of resources :

Character based resources and block based resources. The end result is
that it is no good for neither of the two cases.

2011/12/9 Michael Van Canneyt :
>
>
> On Fri, 9 Dec 2011, Jorge Aldo G. de F. Junior wrote:
>
>> Thats an old problem on the way TStream was implemented.
>>
>> I for one needed to know the result size of a uncompress stream (IE.:
>> i have a compressed stream and want to know in advance how many bytes
>> the uncompress method would yield).
>
>
> This information is not stored in the compressed data. To be able to give
> you this number, the compression stream
> would need to actually decompress the whole stream.
>
>
>>
>> The stream system is currently a mess, sometimes you can know in
>> advance how many bytes you can read, sometimes you cant, depends on
>> the class you are dealing with.
>
>
> There are simply streams which have no size.
>
> In each case, I am working on the 'StreamCapabilities' property, which
> should be able to tell you what is available.
>
>
> Michael.
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] CloseThread needed? still unclear

[noreply]

> On 28.11.2011 21:25, nore...@z505.com wrote:
>>> Am 14.11.2011 02:32, schrieb nore...@z505.com:
 First I thought I would post this to the mailing list to ask you what
 the
 proper way to program with threads is. If we must call closethread on
 MS
 Win machines but not unix machines, then something needs to change to
 make
 the threading in the RTL more cross platform (and all the docs would
 need
 to mention things about closethread if it is needed).
>>>
>>> The bug was already mentioned here
>>> http://mantis.freepascal.org/view.php?id=13160 and fixed for 2.2.2.
>>> "CloseThread" was implemented in a cross platform way.
>>>
>>> Regards,
>>> Sven
>>
>> Hmm, but the compiler I'm using right now is a 2.4 and I think there is
>> a
>> leak if I don't call closethread.
>
> That is exactly my point: CloseThread was implemented because of that
> leak (see the comments in the bug issue I linked to).
>

I see. I think I found some documentation that still doesn't include
CloseThread in the examples.. I will send patches or recommendations to
the documentation soon.  For example closethread is now documented as a
function, but some of the other pages that show threading examples just
use endthread. I thought maybe closethread somehow was tied into endthread
but this not the case.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[noreply]

> On 9 December 2011 09:47, Florian Klaempfl wrote:
>>
>> According to measurements of me and other peoples, random is only 7-10
>> times slower (depending on the CPU).
>
> What do you feed your computer, because mine differs vastly from yours.
>

In europe electricity is sometimes 220 volts. Twice as fast as 110 volts
in Canada, but I'm not sure about africa ;-)

Can the compiler send a warning message WARNING: random is faster if you
use the other unit in soandso.pas.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[tcoq]

I have found and tried WaitForSingleObject, using the following syntax:

ThereIsACharacterToRead :=   WaitForSingleObject(StdInputHandle,0) = 0;
While ThereIsACharacterToRead do
begin
  aByte := InputStream.ReadByte;
  ThereIsACharacterToRead :=   WaitForSingleObject(StdInputHandle,0) = 0;
end;

However, WaitForSingleObject always gives back 0, meaning there should be 
information. My logging tells me that information arrives at least 500ms 
afterwards and that ReadByte waits for it. Something is not working.
Thierry

- Mail Original -
De: "tcoq" 
À: "FPC-Pascal users discussions" 
Envoyé: Vendredi 9 Décembre 2011 17h56:13 GMT +01:00 Amsterdam / Berlin / Berne 
/ Rome / Stockholm / Vienne
Objet: Re: [fpc-pascal] How to poll for a byte in Input?

On windows, I did not find WaitForSingleObject, which is actually going to wait 
for the file.
However I found ReadFileEx in the Windows API, which is an asynchronous read on 
the file. I'll try using that.

- Mail Original -
De: "Michael Van Canneyt" 
À: "FPC-Pascal users discussions" 
Envoyé: Vendredi 9 Décembre 2011 16h57:47 GMT +01:00 Amsterdam / Berlin / Berne 
/ Rome / Stockholm / Vienne
Objet: Re: [fpc-pascal] How to poll for a byte in Input?
>...
You must use fpSelect() or Poll on the handle on linux/unix.
On windows, you'll need to WaitForObject() or so on the file handle.

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Readline substitute

[Jonas Maebe]

On 09 Dec 2011, at 19:37, nore...@z505.com wrote:

> See what happened to common lisp clisp when they used it. Had to change
> license to GPL even though they didn't want to.

They changed the license because they didn't want to stop using GNU 
libreadline. If they had switched to a BSD-licensed libreadline, they would not 
have had to start distributing their code also under a GPL-compatible license.

You may know this, but I think it is important not to spread the false 
impression that the GPL is like some kind of "virus" that "sticks" to your code 
once you have been "touched" by it, because that is not true at all. If that 
were true, then large parts of the FPC RTL would have been GPL by now since for 
a while it contained GPL'd code from the FreeCLX.


Jonas___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Michael Van Canneyt]

On Fri, 9 Dec 2011, Jorge Aldo G. de F. Junior wrote:


I think the main problem is that TStream tries to abstract two kinds
of resources :

Character based resources and block based resources. The end result is
that it is no good for neither of the two cases.


You are not forced to use it :-)

I've been using it for many years, never had any problems...

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Michael Van Canneyt]

On Fri, 9 Dec 2011, tcoq wrote:


I have found and tried WaitForSingleObject, using the following syntax:

ThereIsACharacterToRead :=   WaitForSingleObject(StdInputHandle,0) = 0;
While ThereIsACharacterToRead do
begin
 aByte := InputStream.ReadByte;
 ThereIsACharacterToRead :=   WaitForSingleObject(StdInputHandle,0) = 0;
end;

However, WaitForSingleObject always gives back 0, meaning there should be
information.  My logging tells me that information arrives at least 500ms
afterwards and that ReadByte waits for it.  Something is not working.


I do not think this is correct if the timeout is zero. 
You should specify a timeout.


Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Vinzent Höfler]

On Thu, 08 Dec 2011 10:52:01 +0100, Graeme Geldenhuys  
 wrote:



On 8 December 2011 11:33, Henry Vermaak  wrote:


I agree, quality first.


I would normally agree with that. But such huge magnitudes slower
(20ms vs 10585ms) on a new Quad-Core type system? That just seems a
bit excessive, and considering most use cases are not even for
statistical type applications.


Well, considering that the MT is a very bad choice as PRNG for a stream-
cipher (even if it's never used in "production code", it still gives me the
creeps), I'd say, your use case isn't the typical one, either. ;)

In my case I needed a thread-safe version, so may I request that
System.Random() shall be protected against concurrent access - making it
even slower?

I mean, using your own random generator for whatever reason (statistical
properties, cryptographically strong, lightning fast) isn't exactly rocket
science. ;)



Especially because you could speed-up your code alone by using the 64
random bits you are actually getting from the Random() call instead of
generating a stream of 512 bits for each single character. ;)

So, to come to a conclusion, I do not see any reason why the current
implementation of System.Random() shall be changed.


Vinzent.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 19:55, Jorge Aldo G. de F. Junior  wrote:
> Well, lets go to theory :

In case you didn't notice the unit name this code comes from
tiEncryptSimple.pas
The name should be a good enough hint that it wasn't meant for
real-world apps. ;-)  For real-world apps, tiOPF has other encryption
units based on DES and Blowfish.

But thanks for the theory. I'll keep it for a rainy day.


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Why is Random(255) some 529x slower compared to Delphi 7?

[Graeme Geldenhuys]

On 9 December 2011 20:54,  wrote:
> In europe electricity is sometimes 220 volts. Twice as fast as 110 volts
> in Canada, but I'm not sure about africa ;-)

:-)  South Africa uses 220 volts too.


-- 
Regards,
  - Graeme -


___
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] How to poll for a byte in Input?

[Chad Berchek]

On 12/9/2011 9:44 AM, tcoq wrote:

Hello,
I'm trying to poll the input stream to see whether there is a new character 
available, without blocking my software.


Maybe TInputPipeStream.NumBytesAvailable can help. You can find it in 
the documentation of the Pipes unit.


I tried an experiment to make sure it would work. I included the code 
for the experiment below. It works when stdin is from another process. 
It does not seem to work if stdin is from the console and I'm not sure 
why. I tried adding the Windows API call SetConsoleMode(StdInputHandle, 
1) to disable line-input mode but it didn't seem to make any difference.


Regarding WaitForSingleObject, there are some comments on the MSDN 
documentation page regarding the use of this function for console input. 
In particular, one observation was that if the console is in line-input 
mode, WaitForSingleObject can return but ReadConsole will still block 
until a carriage return is entered. Perhaps the same is true for 
ReadFile. I have not tested this. See:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms687032%28v=vs.85%29.aspx

Code for the test: There are two programs. You can run WriteToPipe to 
see writing to the stdin stream of one process from another process. You 
can also test piped input using TestPipes by typing a command like:


type TestPipes.pas | TestPipes

However this does not seem to work:

TestPipes < TestPipes.pas

I don't know why exactly it doesn't appear to work with redirected input 
from a file or from the console. By the way, I'm using Windows XP Pro 
64-bit. Here is the test code.


- WriteToPipe.pas -

program WriteToPipe;

{$MODE ObjFpc}

uses
  SysUtils,
  Process,
  Windows;

var
  AProc: TProcess;
  AString: string;
  ACount: Integer;
begin
  WriteLn('This will take about 20 seconds.');
  AProc := TProcess.Create(nil);
  try
AProc.CommandLine := 'TestPipes.exe';
AProc.Options := [poUsePipes];
AProc.Execute;
try
  for ACount := 0 to 5 do
  begin
AString := IntToStr(ACount);
AProc.Input.WriteBuffer(AString[1], Length(AString));
Sleep (3000);
  end;
finally
  AProc.Terminate(0);
end;
  finally
AProc.Free;
  end;
end.

- End -

- TestPipes.pas -

program TestPipes;

{$MODE ObjFpc}

uses
  Classes,
  Pipes,
  SysUtils,
  Windows;

var
  AStream: TInputPipeStream;
  AString: string;
  AFile: TextFile;
begin
  Assign(AFile, 'output.txt');
  Rewrite(AFile);
  try
AStream := TInputPipeStream.Create(StdInputHandle);
while True do
begin
  if AStream.NumBytesAvailable > 0 then
  begin
SetLength(AString, AStream.NumBytesAvailable);
AStream.ReadBuffer(AString[1], Length(AString));
WriteLn(AFile, 'I just got this: ', AString);
  end else
  begin
WriteLn(AFile, 'Waiting...');
Sleep(1000);
  end;
end;
  finally
Close(AFile);
  end;
end.

- End -
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Readline substitute

[Krishna]

On Fri, Apr 29, 2011 at 10:18 PM, Johann Glaser  wrote:
> Hi!
>
> I want to equip a program with a command line and therefore want a
> powerful and user-friendly input prompt. Currently I use
>  ReadLn(CmdLine);
> which is not as user-friendly as desired. :-)
>
> The usual answer for this question is libreadline. I already worked with
> it (in plain C) but before I do a header translation for FreePascal I
> wanted to ask you for your opinion if there is a better alternative or
> anybody already has a header translation or even programmer-friendly
> wrapper for the history and auto-completion stuff.

linenoise[1] and editline[2] are alternatives. Don't know if there are
pascal bindings to those.

[1]: https://github.com/antirez/linenoise
[2]: http://www.thrysoee.dk/editline/


-- 
Programming is difficult business.
It should never be undertaken in ignorance.
--Douglas Crockford, "Javascript: The Good Parts"
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal