[issue14455] plistlib unable to read json and binary plist files

Roundup Robot added the comment: New changeset 728f626ee337 by R David Murray in branch 'default': whatsnew: plistlib new api and deprecations (#14455) http://hg.python.org/cpython/rev/728f626ee337 -- ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Serhiy: the issue should now be fixed. I finally understand why I was so sure that Apple's code serialised large positive numbers as negative numbers: due to a bug in PyObjC large positive numbers end up as NSNumber values that are interpreted as negative val

[issue14455] plistlib unable to read json and binary plist files

Roundup Robot added the comment: New changeset 0121c2b7dcce by Ronald Oussoren in branch 'default': Issue #14455: fix handling of unsigned long long values for binary plist files http://hg.python.org/cpython/rev/0121c2b7dcce -- ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: > I'm going to do some more spelunking to find out what's going on here, and > will adjust the plistlib code to fully represent all values of unsigned > 64-bit integers (likely based on your code for supporting 128-bit integers) My last patch supports only val

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Reopening because Cocoa behaves differently that I had noticed earlier... The (Objective-C) code below serialises an NSDictionary with an unsigned long of value ULLONG_MAX and then reads it back. I had expected that restored value contained a negative number,

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: You have at least one way to create a 128 bit CFNumber. Read plist file (and you can create plist in XML format with big integers in any text editor). In any case it is not good to produce incorrect plist for big integers. If you don't want to support intege

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: kCFNumberSInt128Type is not public API, see the list of number types in . I agree that CFBinaryPlist.c contains support for those, and for writ

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: > However, I have no idea how to write that file using Apple's APIs. Look in CFBinaryPList.c. It have a code for creating 128-bit integers: CFSInt128Struct val; val.high = 0; val.low = bigint; *plist = CFNumberC

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: However, I have no idea how to write that file using Apple's APIs. I'd prefer to either be compatible with Apple's API (current behavior), or just outright reject values that cannot be represented as a 64-bit signed integer. The file you generated happens

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: I see that plistlib incorrectly writes large ints from 2**63 to 2**64-1 as negative values. >>> d = plistlib.dumps({'a': 18446744073709551615}, fmt=plistlib.FMT_BINARY) >>> plistlib.loads(d) {'a': -1} My patch did this correct (as 128-bit integer), and as yo

[issue14455] plistlib unable to read json and binary plist files

Changes by Ronald Oussoren : -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue14455] plistlib unable to read json and binary plist files

Roundup Robot added the comment: New changeset 1a8149ba3000 by Ronald Oussoren in branch 'default': Issue #14455: Fix some issues with plistlib http://hg.python.org/cpython/rev/1a8149ba3000 -- ___ Python tracker __

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Conversion to XML results in: $ plutil -convert xml1 -o - 18446744073709551615.plist http://www.apple.com/DTDs/PropertyList-1.0.dtd";> a 18446744073709551615 This is the same as what I get with my latest patch: >>> import plistlib >>> pl

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: I can't test on OSX, but I see that Apple's code can write any 128-bit integers and read signed and unsigned 64-bit integers. Can Apple's utilities read this file? What is a result? -- Added file: http://bugs.python.org/file33213/18446744073709551615

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Updated patch. -- Added file: http://bugs.python.org/file33207/negative_int_support-2.txt ___ Python tracker ___ __

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Attached a script (using PyObjC) that demonstrates the behavior of Apple's Foundation framework with large integers. The same behavior should occur when the script is rewritten in Objective-C. -- Added file: http://bugs.python.org/file33206/apple-beh

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: > * I don't like supporting 128 bit integers because Apple's public APIs > don't support those values. That is, the value 'kCFNumberSInt128Type' > is not in a public header for the OSX 10.9 SDK. At least we should support integers from -2**63 to 2**64-1 (s

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: The attached patch should fix the open issues: * Negative integers are supported (based on Serhiy's patch), but without support for 128-bit integer (as per my previous comment) * Test updates for this * Updated version tags in the documentation * Documente

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I'm working on an update for your patch that addresses these comments: * I don't like supporting 128 bit integers because Apple's public APIs don't support those values. That is, the value 'kCFNumberSInt128Type' is not in a public header for the OSX 10.9 SD

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: Yet one nitpick. Perhaps _write_object() should raise TypeError instead of InvalidFileException. -- ___ Python tracker ___ __

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: According to [1] Apple's libraries write any signed 128-bit integers, but read only integers from -2**63 to 2**64-1 (e.g. signed and unsigned 64-bit integers). [1] http://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c -- _

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: oops... thanks for the patch. I'll review later this week, in particular the 128 bit integer support because I don't know if Apple's libraries support those. -- ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: Currently negative integers are not supported in binary format. Here is a patch which adds support for negative integers and large integers up to 128 bit. -- Added file: http://bugs.python.org/file32937/plistlib_int.patch

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: These changes are worth to mention in What's News. "versionchanged" below writePlistToBytes() is wrong. Perhaps below dump() too. "versionadded" is needed for new functions: dump(), dumps(), load(), loads(). -- __

[issue14455] plistlib unable to read json and binary plist files

Roundup Robot added the comment: New changeset 602e0a0ec67e by Ned Deily in branch 'default': Issue #14455: Fix maybe_open typo in Plist.fromFile(). http://hg.python.org/cpython/rev/602e0a0ec67e -- ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: I'm not sure about docstrings text ("return" vs "returns", I don't remember what is better), but we can bikeshed it after beta 1. -- ___ Python tracker _

[issue14455] plistlib unable to read json and binary plist files

Roundup Robot added the comment: New changeset 673ca119dbd0 by Ronald Oussoren in branch 'default': Issue #14455: plistlib now supports binary plists and has an updated API. http://hg.python.org/cpython/rev/673ca119dbd0 -- nosy: +python-dev ___ Python

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Updated patch after next round of reviews. -- Added file: http://bugs.python.org/file32754/issue-14455-v10.txt ___ Python tracker ___ _

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: I have added a few comments on Rietveld. Besides formatting nitpicks your have forgot third argument in new warns and missed some details in tests. As for the rest the patch LGTM. If you have no time I will fixed this minor issues and will commited the patch

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I've attached an updated version of the patch that should fix most of the issues found during review. I've also changed the two FMT_ constants to an enum.Enum (but still expose the constants themselves as module global names because that's IMHO more convenie

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: It's too large and complicated patch. I would like to have a chance to quick review it before committing. You will have time to commit. -- ___ Python tracker ___

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I for the most part agree with the comments and will provide an updated patch on thursday. Would you mind if I committed that without further review (due to cutting it awfully close to the deadline for beta 1)? Some comments I want to reply to specifically:

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: I added a lot of comments on Rietveld. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I'd really like to include this patch in 3.4, but haven't managed to do any opensource work in the previous period and don't know when I'll be able to actually commit this (and more importantly, be available when issues crop up) :-( -- _

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Actually attach the latest version of the patch. -- Added file: http://bugs.python.org/file30874/issue-14455-v8.txt ___ Python tracker ___

[issue14455] plistlib unable to read json and binary plist files

Ned Deily added the comment: Ronald, I think v8 of the patch is missing (and plistlib_generate_testdata.py was uploaded twice). -- ___ Python tracker ___ ___

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: v8 of the patch contains 1 change from v7: the test data is encoded in base64. This was primarily done to ensure that the file has usable line lengths. A nice side effect is that it is now harder than ever to manually change the test data, as the comment menti

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Updated test-data generator: it now encodes the data using base64, to make it easier to generate a file with limited line lengths. -- Added file: http://bugs.python.org/file30790/plistlib_generate_testdata.py ___ Pyt

[issue14455] plistlib unable to read json and binary plist files

Changes by Ronald Oussoren : Removed file: http://bugs.python.org/file30526/plistlib_generate_testdata.py ___ Python tracker ___ ___ Python-bu

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: This version should be better: * There should be no lines longer than 80 characters * Changed coding style of the (private) XML plist writer classes to PEP8 * Public API is now dump/dumps and load/loads, the old API is still available and deprecated ->

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I've attached a slightly updated version of the patch. The documentation now lists dump and load as the primary API, with the old API as a deprecated alternative. The code hasn't been changed to relect this yet, but does contain a number of tweaks and bugfix

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: I have added comments on Rietveld. I have to apologize for unwitting misleading of d9pouces. Functional version of the patch is enough Pythonic and it looks more clear to me than object-oriented one. -- ___ Pytho

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: Any review would be greatly appreciated. One thing I'm not too happy about is the use of magic numbers in the binary plist support code, but I think that using constants or a dispatch table would not make the code any clearer. --

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: Let me review your patch. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I intend to commit my latest version of the patch during the europython sprints, with a minor change: don't include dump(s) and load(s), that change (and the other items on "open issues" in my last post) can be addressed later. -- ___

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: The fifth version of the patch should be much cleaner. Changes: * Coding style cleanup, the new code uses PEP8 conformant names for methods and variables. * Explicitly make private classes private by prefixing their name with an underscore (including the

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: v4 passes the included tests. The testsuite isn't finished yet. -- Added file: http://bugs.python.org/file30530/issue14455-v4.txt ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: The test failure I'm getting is caused by a difference in the order in which items are written to the archive. I'm working on a fix. -- ___ Python tracker ___

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: See also: #18168: request for the sort_keys option #11101: request for an option to ignore 'None' values when writing #9256: datetime.datetime objects created by plistlib don't include timezone information (and looking at the code I'd say that timezones are ig

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: v3 is still a work in progress, and still fails some tests * Replaced test data by data generated by a helper script (to make it easier to update) * Use 'subtest' feature of unittest library in 3.4 * Small tweaks to plist library (the dump/load/dumps/loads

[issue14455] plistlib unable to read json and binary plist files

Changes by Ronald Oussoren : Added file: http://bugs.python.org/file30525/issue14455-v3.txt ___ Python tracker ___ ___ Python-bugs-list mailin

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I've attached issue14455-v2.txt with an updated patch. The patch is still very much a work in progress, I haven't had as much time to work on this as I'd like. This version: * Should apply cleanly to the tip of the default branch * Move around some code. * D

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I've started work on integrating the latest patch. Some notes (primarily for my own use): * I'll drop support for the JSON format, that format is not used by Apple's libraries (although the plutil tool can convert plists to JSON) * The patch (plistlib_wit

[issue14455] plistlib unable to read json and binary plist files

d9pouces added the comment: I just signed this agreement. Thanks for accepting this patch! -- ___ Python tracker ___ ___ Python-bugs-l

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: d9pouces: are you willing to sign a contributor agreement? The agreement is needed before we can add these changes to the stdlib, and I'd like to that for the 3.4 release. More information on the contributor agreement: http://www.python.org/psf/contrib/contr

[issue14455] plistlib unable to read json and binary plist files

Mark Grandi added the comment: are any more changes needed to the code that is already posted as a patch in this bug report? or are the changes you wanted to see happen in msg157669 not happen yet? -- ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: plutil(1) supports writing json format. That written, the opensource parts of CoreFoundation on opensource.apple.com don't support reading or writing json files. I'm therefore -1 w.r.t. adding support for json formatted plist files, support for json can be

[issue14455] plistlib unable to read json and binary plist files

Mark Grandi added the comment: Where are you even seeing these json property lists? I just checked the most recent documentation for NSPropertyListSerialization, and they have not updated the enum for NSPropertyListFormat. It seems that if even Apple doesn't support writing json property lists

[issue14455] plistlib unable to read json and binary plist files

d9pouces added the comment: The plutil (Apple's command-line tool to convert plist files from a format to another) returns an error if you try to convert a XML plist with dates to JSON. -- ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

Mark Grandi added the comment: Hi, I noticed in the latest message that d9pounces posted that "JSON format does not allow dates and data, so XML is used by default to write files.". Rthe XML version of plists also do not really 'support' those types, and they are converted as follows: NSDat

[issue14455] plistlib unable to read json and binary plist files

d9pouces added the comment: Here is the new patch, allowing read and write binary, json and xml plist files. It includes both the plistlib.py and test/test_plistlib.py patches. JSON format does not allow dates and data, so XML is used by default to write files. I use the json library to write

[issue14455] plistlib unable to read json and binary plist files

d9pouces added the comment: I'm working on a class, BinaryPlistParser, which allow to both read and write binary files. I've also added a parameter fmt to writePlist and readPlist, to specify the format ('json', 'xml1' or 'binary1', using XML by default). These constants are used by Apple fo

[issue14455] plistlib unable to read json and binary plist files

Ronald Oussoren added the comment: I (as one of the Mac maintainers) like the new functionality, but would like to see some changes: 1) as others have noted it is odd that binary and json plists can be read but not written 2) there need to be tests, and I'd add two or even three set of tests

[issue14455] plistlib unable to read json and binary plist files

Éric Araujo added the comment: Keep it simple: if a few functions work, there is no need at all to add classes. Before doing more work though I suggest you wait for the feedback of the Mac maintainers. -- nosy: +eric.araujo ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

d9pouces added the comment: storchaka > I'm trying to take care of your remarks. So, I'm working on a more object-oriented code, with both write and read functions. I just need to write some test cases. IMHO, we should add a new parameter to the writePlist function, to allow the use of the bin

[issue14455] plistlib unable to read json and binary plist files

Changes by Georges Martin : -- nosy: +jrjsmrtn ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue14455] plistlib unable to read json and binary plist files

Serhiy Storchaka added the comment: This patch is for Python 2. New features are accepted only for Python 3.3+. I ported the patch, but since I have no Mac, I can't check. To date code was specified incorrectly. The length of integers was calculated incorrectly. To convert integers, you can

[issue14455] plistlib unable to read json and binary plist files

Changes by Ned Deily : -- nosy: +ned.deily ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue14455] plistlib unable to read json and binary plist files

R. David Murray added the comment: Hmm. Apparently what I meant was -u instead of -c (unified diff). I just use the 'hg diff' command myself, which does the right thing :) Of course, to do that you need to have a checkout. (We can probably use the context diff.) -- __

[issue14455] plistlib unable to read json and binary plist files

d9pouces added the comment: Here is the new patch. I assumed that you meant to use diff -c instead of the raw diff command. -- keywords: +patch Added file: http://bugs.python.org/file25076/context.diff ___ Python tracker

[issue14455] plistlib unable to read json and binary plist files

R. David Murray added the comment: Thanks for the patch. Could you upload it as a context diff? -- nosy: +r.david.murray stage: -> patch review versions: +Python 3.3 -Python 2.7 ___ Python tracker __

[issue14455] plistlib unable to read json and binary plist files

New submission from d9pouces : Hi, Plist files have actually three flavors : XML ones, binary ones, and now (starting from Mac OS X 10.7 Lion) json one. The plistlib.readPlist function can only read XML plist files and thus cannot read binary and json ones. The binary format is open and descr