Re: [PHP-DEV] [PATCH] LFS support for PHP 5.5.1

[X Ryl]

On Fri, Aug 30, 2013 at 4:52 PM, Johannes Schlüter
 wrote:
>
> On Fri, 2013-08-30 at 16:43 +0200, X Ryl wrote:
> > Hi,
> >
> >
> >   Please find attached a patch for adding large file size support to
> > PHP 5.5.1.
>
> The patch didn't make it. Please send as text/plain (i.e. using .txt
> extension)


Done.

>
>
> > It does so by, from the PHP's side, getting double instead of int for
> > the file size/ offset functions, when the size is larger than 2^31.
>
> This has some problems - for further handling onemight need the exact
> file size (i.e. content length headers, checking structures, reading
> specific positions)
>
> > This means that files with size:
> > - up to 2^32 bytes works as previously (integer returned / used)
> > - up to 2^52 bytes can be handled correctly (double's mantissa is 52
> > bits, no loss in precision here)
>
> This might work for the initial operation, but as soon as the user does
> a calculation ("give me the last ten bytes") this will cause issues.

No it won't. double mantissa are integers, works like integer.
So you have the complete 52 bits of precision here, not a single
rounding error can occur.
The exponent in that case will be 0, so "2^0 * integer_with_52_bits"
is still an integer with 52 bits of precision.

So the "filesize - 10" is still exact as long as the filesize is less
than 2^52 (see below when it's larger)



>
> > - from 2^52 up to 2^64 will have their size rounded, yet, reading and
> > writing will work as expected since it's done in the PHP's binary.

I'll try to improve this, since it's causing misunderstanding.
Typically, when you start storing a value that does not fit in 52
bits, it's shifted (understand: round to the closed multiple of two)
until it fits the range.
So for example, a file size that's "2^53 + 203" will be stored as
9007199254741196 instead of 9007199254741195 (the error is 1 here)

Yet, the reported value will be wrong, but if you seek close to the
end position and read it you'll still be able to read it completely
(since the  double value will be converted back to 
64-bit integer in the C code). As long as you're dealing with
positions/seek in a multiple of 2^52, you'll be fine.
So if you need 100% correct value for such a large file, you can still
do it by looping, AT WORST, 4096 times ( = 2^64 / 2^52) a seek with
SEEK_CUR.
Not a real showstopper when the current version does not even allow
you to know the actual size of the file, not even speaking of reading
it!)

If you either have to handle such large file NOW, then chance are high
you're already using a 64 bits system.
This patch if for us, poor souls, struck with 32 bits system, yet
wanting to report correct size for our movies in our web-based file
manager, wanting to stream correctly such files and so on.

> >
> >
> > The changes are:
> > - Some size_t are changed to off_t wherever required.
>
> This disqualifies from 5.5 and allows use in 5.6 only.

Ok
Cyril
diff -ur php-5.5.1/ext/phar/phar_internal.h 
php-5.5.1.new/ext/phar/phar_internal.h
--- php-5.5.1/ext/phar/phar_internal.h  2013-07-18 16:37:33.0 +0200
+++ php-5.5.1.new/ext/phar/phar_internal.h  2013-08-15 10:42:39.332887601 
+0200
@@ -534,8 +534,15 @@
return FAILURE;
 }
 #else
-# define phar_stream_copy_to_stream(src, dest, maxlen, len)
_php_stream_copy_to_stream_ex((src), (dest), (maxlen), (len) STREAMS_CC 
TSRMLS_CC)
-
+static inline size_t phar_stream_copy_to_stream(php_stream *src, php_stream 
*dest, size_t maxlen, size_t *len)
+{
+   off_t _maxlen = maxlen == (size_t)PHP_STREAM_COPY_ALL ? 
PHP_STREAM_COPY_ALL : maxlen, _len = 0;
+   size_t ret = php_stream_copy_to_stream_ex(src, dest, _maxlen, &_len);
+   if (ret == SUCCESS) {
+   if (len) *len = (size_t)_len;
+   } else if (len) *len = 0;
+   return ret;
+}
 #endif
 
 #if PHP_VERSION_ID >= 6
diff -ur php-5.5.1/ext/standard/file.c php-5.5.1.new/ext/standard/file.c
--- php-5.5.1/ext/standard/file.c   2013-07-18 16:37:33.0 +0200
+++ php-5.5.1.new/ext/standard/file.c   2013-08-15 10:42:39.432888125 +0200
@@ -570,7 +570,7 @@
char *filename;
int filename_len;
zval *data;
-   int numbytes = 0;
+   off_t numbytes = 0;
long flags = 0;
zval *zcontext = NULL;
php_stream_context *context = NULL;
@@ -618,7 +618,7 @@
 
switch (Z_TYPE_P(data)) {
case IS_RESOURCE: {
-   size_t len;
+   off_t len;
if (php_stream_copy_to_stream_ex(srcstream, stream, 
PHP_STREAM_COPY_ALL, &len) != SUCCESS) {
numbytes = -1;
  

[PHP-DEV] [PATCH] LFS support for PHP 5.5.1

[X Ryl]

Hi,

  Please find attached a patch for adding large file size support to PHP
5.5.1.
Basically, it allows 32 bits machine to address file larger than 4GB, get
correct results when asking for their size, allows to read and write them,
etc...

It does so by, from the PHP's side, getting double instead of int for the
file size/ offset functions, when the size is larger than 2^31.

This means that files with size:
- up to 2^32 bytes works as previously (integer returned / used)
- up to 2^52 bytes can be handled correctly (double's mantissa is 52 bits,
no loss in precision here)
- from 2^52 up to 2^64 will have their size rounded, yet, reading and
writing will work as expected since it's done in the PHP's binary.

The changes are:
- Some size_t are changed to off_t wherever required.
- The code with unique mmap now loops the mmap until the complete
file/stream is done processing
- Fix for the mmap of a popen's pipe that can't work (unrelated, but easy
to fix while working on the mmap code)
- Change the return type based on the actual range of the manipulated
number (so if the value fit in a integer, a integer is used, and the code
that used to work still works, but if it does not fit, a double is used,
and the code that used to fails now works)

Please notice that I'm not a PHP developer, I don't have any time left for
maintaining this patch, but I'm sure this patch has a value. So, I deny any
right on it and put it in public domain for whoever wants to improve it,
integrate it.


Let me know your remarks.
Cyril
-- 
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP-DEV] A fix for limited 32 bits supports

[X Ryl]

Hi,

  I was a bit disappointed to see that filesize, stat et al don't work on
32bits systems, because they are using 32 bits integer.
I've found a patch written by Wes, but unfortunately it didn't worked.

Well, now, I've found this:

http://bugs.php.net/bug.php?id=48886

This is done right, it works for all streams, not just file.
I think it just the best solution now without breaking the ABI with modules.
With this patch you get 52 bits filesize (which is more than enough with a
32 bits system), 52 bits offset, and so on.

With unmodified PHP code, it's working because the type is used as a double
when it overcome the 32 bit limit.
So instead of this ugly fix : "sprintf("%u", $value)" for 2GB up to 4GB, you
have this one "sprintf("%.0f", $value)", which works with file size < 4503TB
on 32 bits machines.

Please advise.

Re: [PHP-DEV] Functions that fails on large files

[X Ryl]

The patch in 27792 doesn't work, but the one in 48886 does (from my own
tests)


2009/9/13 Laurent Léonard 

> Hi,
>
> What about that bug ? http://bugs.php.net/bug.php?id=27792
>
> I see there is a patch here on this related bug report :
> http://bugs.php.net/bug.php?id=48886
>
> Do you plan to fix the issue ?
>
> Thank you,
> --
> Laurent Léonard
>

Re: [PHP-DEV] 5.3.1 process

[X Ryl]

Hi, I would like to know if PHP 5.3.1 will fix the remanent big file (> 4GB)
issue on 32bits system.

There was 2 patch posted to fix this, the first one was in bug #48886 and
another one is there:
http://www.php.net/~wez/lfs.diff

I don't know what is the expected protocol to get a patch examined, but I
hope you dev will at least try them.
I personnally have tried the one from bug #48886 and it's working as
described (it's using php's double to return/handle filesize when it can't
be stored in an 32bits int.
As the integral part of a double is 52bits wide, this give way larger space
for the filesize).
I haven't tried the new one from wez, but the previous one didn't work.

Hope it will get considered.
Regards,
X





2009/9/19 Pierre Joye 

> hi Johannes,
>
> It is a really great improvement that we don't have to worry anymore
> about being in a release phase for a given branch. However one thing I
> would like to improve in the merge frequency. Having a large merge the
> day (or the day before) a RC does not sound too god to me. It would be
> much safer to merge more often so we can valid them before the release
> or point you to some missing bits if necessary.
>
> Thoughts&comments welcome,
>
> Cheers,
>
> 2009/9/4 Johannes Schlüter :
> > Hi,
> >
> > a new tool provides new opportunities and challenges, with the move to
> > svn I took some of the new possibilities and after many discussions
> > created a new release process which I'll try for 5.3.1 for which RC1 is
> > about to be packed. (having trouble accessing the snaps box to build
> > it...)
> >
> > The very short story for most devs is to go on and work on the branches
> > as before (meaning trunk, PHP_5_3, PHP_5_2, commit at once etc.)
> >
> > The longer story is this:
> >
> > I've created a new branch PHP_5_3_1. This release branch receives
> > selective merges by the RM (me) only!
> >
> > This branch will then be used to create RCs as often as needed
> > containing these fixes, as we won't have snaps for this branch only svn
> > checkouts and RCs will be available for testing. Therefore the aim is to
> > provide stable RCs and the only change between the last RC and the
> > stable release shall be the version number change.
> >
> > Once 5.3.1 is to be published the PHP_5_3_1 branch will be converted
> > into a php_5_3_1 tag. (svn mv) And the 5.3.1 NEWS will be merged back
> > into PHP_5_3.
> >
> > When committing to PHP_5_3 please be conservative till 5.3.1 is out to
> > ease merging and testing (we have no snaps for PHP_5_3_1) and try to
> > indicate whether you think the change should be merged or not and tell
> > me if you think I missed to merge a commit.
> >
> > Once 5.3.1 is released this process will be reviewed and either be
> > documented or modified.
> >
> > Hoping this works out,
> > johannes
> >
> >
> >
> > --
> > PHP Internals - PHP Runtime Development Mailing List
> > To unsubscribe, visit: http://www.php.net/unsub.php
> >
> >
>
>
>
> --
> Pierre
>
> http://blog.thepimp.net | http://www.libgd.org
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

[PHP-DEV] [PATCH] Fix for 32 bits limit on file size (see bug 48886)

[X Ryl]

Hi,

 I'm not a C developer, so I can't really help with the attached patch.
However, I've tried it with PHP 5.3.0 version I have and it does what it
claims, that is, it gives correct behaviour for the following functions:
filesize, fstat, stat when used on file > 4GB on a 32 bits machine.
Currently, php 5.x doesn't give the correct output for this code (on my
32bits server):
echo filesize("path/to/4.3GBfile");  // expecting 4617089843, got 322122547

With the patch applied, the good result is returned.
>From the patch author, it improves the tests results.
>From what I've understood, the patch switch to php's double type when the
size overflow the 32bits limit of int.
It doesn't seem to break any existing behaviour (I'm using this on my
multiple project since july, and no bad experience so far).

The explaination by the patch author: http://bugs.php.net/bug.php?id=48886
-- 
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php