Re: [compress] not reading archive stream completely

2013-01-17 Thread Bear Giles
I think a number of applications use a concatenation of a standard archive
format and custom data. The most well known probably is .rpm which is/was a
cpio stream immediately followed by additional information (iirc - it might
go the other way). In any case a developer might expect to have the input
stream placed at the end of the archive, not the end of the input stream.

On the zip 'central directory' - one of the big 'wins' for zip format is
that it allows you to seek directly to a file instead of having to scan it
sequentially. For various reasons (e.g., the need to support streaming
modes) it has to go at the end of the archive. The unix backup format has
the directory at the top of the archive but it was optimized for backups
that spanned multiple tapes so the cost of precomputing the values was
worth it.

Bear


On Thu, Jan 17, 2013 at 7:42 AM, Torsten Curdt  wrote:

> > For tar it would be one block (usualy 512 bytes), for zip the full
> > central directory has to be read which could be quite a bit.
> >
>
> Urgh. Because that's at the end for zips? That's not so good then.
>
>
> >
> > I currently plan to implement it inside getNextEntry as it is cleaner.
> > In the tar case I vaguely recall some implementation only write one EOF
> > marker so a more careful aproach is needed in order to not read beyond
> > the end of the archive (likely mark and reset if the stream supports
> > mark).
> >
>
> Hm. That suddenly makes (2) much interesting again.
> I can see the back and forth on this :)
>


[COMPRESS] pkware header id for crypto extensions

2015-09-02 Thread Bear Giles
Hi, I know that the implementation of the PKWARE AES crypto is subject to
license restrictions but is it possible to recognize the extension fields
so anyone scanning an unfamiliar file will at least know what the extra
field headers contain?

I don't know if code to parse the contents (solely using the standard JCE)
would trigger export restrictions. This would NOT be decrypting the data,
just adding a thin layer of code to parse a standardized ASN.1 object.

(BTW there are some other header types documented but except for a few I
don't know what they are.)

The header IDs (per
https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) are:

  0x0014PKCS#7 Store for X.509 Certificates
  0x0015X.509 Certificate ID and Signature for
individual file
  0x0016X.509 Certificate ID for Central Directory
  0x0017Strong Encryption Header
  0x0019PKCS#7 Encryption Recipient Certificate List

The definitions are:

   4.5.9 -PKCS#7 Store for X.509 Certificates (0x0014):

This field MUST contain information about each of the certificates
files may be signed with. When the Central Directory Encryption
feature is enabled for a ZIP file, this record will appear in
the Archive Extra Data Record, otherwise it will appear in the
first central directory record and will be ignored in any
other record.


Note: all fields stored in Intel low-byte/high-byte order.

Value Size Description
-  ---
(Store) 0x00142 bytes  Tag for this "extra" block type
TSize 2 bytes  Size of the store data
TData TSizeData about the store


   4.5.10 -X.509 Certificate ID and Signature for individual file (0x0015):

This field contains the information about which certificate in
the PKCS#7 store was used to sign a particular file. It also
contains the signature data. This field can appear multiple
times, but can only appear once per certificate.

Note: all fields stored in Intel low-byte/high-byte order.

Value Size Description
-  ---
(CID)   0x00152 bytes  Tag for this "extra" block type
TSize 2 bytes  Size of data that follows
TData TSizeSignature Data

   4.5.11 -X.509 Certificate ID and Signature for central directory (0x0016):

This field contains the information about which certificate in
the PKCS#7 store was used to sign the central directory structure.
When the Central Directory Encryption feature is enabled for a
ZIP file, this record will appear in the Archive Extra Data Record,
otherwise it will appear in the first central directory record.

Note: all fields stored in Intel low-byte/high-byte order.

Value Size Description
-  ---
(CDID)  0x00162 bytes  Tag for this "extra" block type
TSize 2 bytes  Size of data that follows
TData TSizeData

   4.5.12 -Strong Encryption Header (0x0017):

Value Size Description
-  ---
0x00172 bytes  Tag for this "extra" block type
TSize 2 bytes  Size of data that follows
Format2 bytes  Format definition for this record
AlgID 2 bytes  Encryption algorithm identifier
Bitlen2 bytes  Bit length of encryption key
Flags 2 bytes  Processing flags
CertData  TSize-8  Certificate decryption extra field data
   (refer to the explanation for CertData
in the section describing the
Certificate Processing Method under
the Strong Encryption Specification)

See the section describing the Strong Encryption Specification
for details.  Refer to the section in this document entitled
"Incorporating PKWARE Proprietary Technology into Your Product"
for more information.

   4.5.14 -PKCS#7 Encryption Recipient Certificate List (0x0019):

This field MAY contain information about each of the certificates
used in encryption processing and it can be used to identify who is
allowed to decrypt encrypted files.  This field should only appear
in the archive extra data record. This field is not required and
serves only to aid archive modifications by preserving public
encryption key data. Individual security requirements may dictate
that this data be omitted to deter information exposure.

Note: all fields stored in Intel low-byte/high-byte order.

 Value Size Description
 -  ---
(CStore) 0x00192 bytes  Tag for this "extra" block type
 TSize 2 bytes  Size of the store data
 TData TSize

Re: [COMPRESS] pkware header id for crypto extensions

2015-09-03 Thread Bear Giles
I've forked at github.com/beargiles/commons-compress and committed some
initial work - five new ExtraHeader classes, their registration, two test
files and four unit tests.

One test file uses certificate encryption and signature but does not
encrypt filenames. The second test file uses certificate encryption and
signature and encrypts filenames. I am using my Symantec certificate for
bgi...@coyotesong.com and can verify that information is present with
'strings'.

The results of the tests are a little confusing. The good:

- the use of strong encryption is recognized
- the substitution of an encrypted filenames is recognized. (recorded
filename is 1, not LICENSE.txt).

The bad:

- I'm only seeing the headers loaded in one unit test. I don't know if the
rest should be reading the headers or not so I don't know if I'm properly
handling them or if they're quietly handled by the default handler.

- one unit test blows up due to missing central directory (it's encrypted).

I guess the next step is either finding a way to have the unit tests
trigger reading the extra headers or confirm that it's not possible.

Bear

On Wed, Sep 2, 2015 at 3:58 PM, Bear Giles  wrote:

> Hi, I know that the implementation of the PKWARE AES crypto is subject to
> license restrictions but is it possible to recognize the extension fields
> so anyone scanning an unfamiliar file will at least know what the extra
> field headers contain?
>
> I don't know if code to parse the contents (solely using the standard JCE)
> would trigger export restrictions. This would NOT be decrypting the data,
> just adding a thin layer of code to parse a standardized ASN.1 object.
>
> (BTW there are some other header types documented but except for a few I
> don't know what they are.)
>
> The header IDs (per
> https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) are:
>
>   0x0014PKCS#7 Store for X.509 Certificates
>   0x0015X.509 Certificate ID and Signature for
> individual file
>   0x0016X.509 Certificate ID for Central Directory
>   0x0017Strong Encryption Header
>   0x0019PKCS#7 Encryption Recipient Certificate List
>
> The definitions are:
>
>4.5.9 -PKCS#7 Store for X.509 Certificates (0x0014):
>
> This field MUST contain information about each of the certificates
> files may be signed with. When the Central Directory Encryption
> feature is enabled for a ZIP file, this record will appear in
> the Archive Extra Data Record, otherwise it will appear in the
> first central directory record and will be ignored in any
> other record.
>
>
> Note: all fields stored in Intel low-byte/high-byte order.
>
> Value Size Description
> -  ---
> (Store) 0x00142 bytes  Tag for this "extra" block type
> TSize 2 bytes  Size of the store data
> TData TSizeData about the store
>
>
>4.5.10 -X.509 Certificate ID and Signature for individual file (0x0015):
>
> This field contains the information about which certificate in
> the PKCS#7 store was used to sign a particular file. It also
> contains the signature data. This field can appear multiple
> times, but can only appear once per certificate.
>
> Note: all fields stored in Intel low-byte/high-byte order.
>
> Value Size Description
> -  ---
> (CID)   0x00152 bytes  Tag for this "extra" block type
> TSize 2 bytes  Size of data that follows
> TData TSizeSignature Data
>
>4.5.11 -X.509 Certificate ID and Signature for central directory (0x0016):
>
> This field contains the information about which certificate in
> the PKCS#7 store was used to sign the central directory structure.
> When the Central Directory Encryption feature is enabled for a
> ZIP file, this record will appear in the Archive Extra Data Record,
> otherwise it will appear in the first central directory record.
>
> Note: all fields stored in Intel low-byte/high-byte order.
>
> Value Size Description
> -  ---
> (CDID)  0x00162 bytes  Tag for this "extra" block type
> TSize 2 bytes  Size of data that follows
> TData TSizeData
>
>4.5.12 -Strong Encryption Header (0x0017):
>
> Value Size Description
> -  ---
> 0x00172 bytes  Tag for this "extra" block type
> T

Re: [COMPRESS] pkware header id for crypto extensions

2015-09-04 Thread Bear Giles
I just came across a press release that suggests that it is possible to
support AES encryption for zip files:

http://www.prnewswire.com/news-releases/pkware-announces-free-licensing-program-for-secure-ziptm-and-pkzip-reader-technologies-72382192.html

(We might have discussed this several years ago...)

Bear

On Thu, Sep 3, 2015 at 10:12 PM, Bear Giles  wrote:

> I've forked at github.com/beargiles/commons-compress and committed some
> initial work - five new ExtraHeader classes, their registration, two test
> files and four unit tests.
>
> One test file uses certificate encryption and signature but does not
> encrypt filenames. The second test file uses certificate encryption and
> signature and encrypts filenames. I am using my Symantec certificate for
> bgi...@coyotesong.com and can verify that information is present with
> 'strings'.
>
> The results of the tests are a little confusing. The good:
>
> - the use of strong encryption is recognized
> - the substitution of an encrypted filenames is recognized. (recorded
> filename is 1, not LICENSE.txt).
>
> The bad:
>
> - I'm only seeing the headers loaded in one unit test. I don't know if the
> rest should be reading the headers or not so I don't know if I'm properly
> handling them or if they're quietly handled by the default handler.
>
> - one unit test blows up due to missing central directory (it's encrypted).
>
> I guess the next step is either finding a way to have the unit tests
> trigger reading the extra headers or confirm that it's not possible.
>
> Bear
>
> On Wed, Sep 2, 2015 at 3:58 PM, Bear Giles  wrote:
>
>> Hi, I know that the implementation of the PKWARE AES crypto is subject to
>> license restrictions but is it possible to recognize the extension fields
>> so anyone scanning an unfamiliar file will at least know what the extra
>> field headers contain?
>>
>> I don't know if code to parse the contents (solely using the standard
>> JCE) would trigger export restrictions. This would NOT be decrypting the
>> data, just adding a thin layer of code to parse a standardized ASN.1 object.
>>
>> (BTW there are some other header types documented but except for a few I
>> don't know what they are.)
>>
>> The header IDs (per
>> https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) are:
>>
>>   0x0014PKCS#7 Store for X.509 Certificates
>>   0x0015X.509 Certificate ID and Signature for
>> individual file
>>   0x0016X.509 Certificate ID for Central Directory
>>   0x0017Strong Encryption Header
>>   0x0019PKCS#7 Encryption Recipient Certificate List
>>
>> The definitions are:
>>
>>4.5.9 -PKCS#7 Store for X.509 Certificates (0x0014):
>>
>> This field MUST contain information about each of the certificates
>> files may be signed with. When the Central Directory Encryption
>> feature is enabled for a ZIP file, this record will appear in
>> the Archive Extra Data Record, otherwise it will appear in the
>> first central directory record and will be ignored in any
>> other record.
>>
>>
>> Note: all fields stored in Intel low-byte/high-byte order.
>>
>> Value Size Description
>> -  ---
>> (Store) 0x00142 bytes  Tag for this "extra" block type
>> TSize 2 bytes  Size of the store data
>> TData TSizeData about the store
>>
>>
>>4.5.10 -X.509 Certificate ID and Signature for individual file (0x0015):
>>
>> This field contains the information about which certificate in
>> the PKCS#7 store was used to sign a particular file. It also
>> contains the signature data. This field can appear multiple
>> times, but can only appear once per certificate.
>>
>> Note: all fields stored in Intel low-byte/high-byte order.
>>
>> Value Size Description
>> -  ---
>> (CID)   0x00152 bytes  Tag for this "extra" block type
>> TSize 2 bytes  Size of data that follows
>> TData TSizeSignature Data
>>
>>4.5.11 -X.509 Certificate ID and Signature for central directory (0x0016):
>>
>> This field contains the information about which certificate in
>> the PKCS#7 store was used to sign the central directory structure.
>> When the Central Directory Encryption feature is enabled fo

Re: [COMPRESS] pkware header id for crypto extensions

2015-09-05 Thread Bear Giles
I was just reminded of the risks of making assumptions. There's two
separate implementations of AES strong encryption - WinZip AES and PKWare.
I've been focused on the latter but we should recognize both types of
headers.

I've pushed a bit more code that partially parses the PKWare headers. It's
enough to identify the encryption and hash algorithms used but a lot is
left unparsed.

Bear

On Fri, Sep 4, 2015 at 12:17 PM, Bear Giles  wrote:

> I just came across a press release that suggests that it is possible to
> support AES encryption for zip files:
>
>
> http://www.prnewswire.com/news-releases/pkware-announces-free-licensing-program-for-secure-ziptm-and-pkzip-reader-technologies-72382192.html
>
> (We might have discussed this several years ago...)
>
> Bear
>
> On Thu, Sep 3, 2015 at 10:12 PM, Bear Giles  wrote:
>
>> I've forked at github.com/beargiles/commons-compress and committed some
>> initial work - five new ExtraHeader classes, their registration, two test
>> files and four unit tests.
>>
>> One test file uses certificate encryption and signature but does not
>> encrypt filenames. The second test file uses certificate encryption and
>> signature and encrypts filenames. I am using my Symantec certificate for
>> bgi...@coyotesong.com and can verify that information is present with
>> 'strings'.
>>
>> The results of the tests are a little confusing. The good:
>>
>> - the use of strong encryption is recognized
>> - the substitution of an encrypted filenames is recognized. (recorded
>> filename is 1, not LICENSE.txt).
>>
>> The bad:
>>
>> - I'm only seeing the headers loaded in one unit test. I don't know if
>> the rest should be reading the headers or not so I don't know if I'm
>> properly handling them or if they're quietly handled by the default handler.
>>
>> - one unit test blows up due to missing central directory (it's
>> encrypted).
>>
>> I guess the next step is either finding a way to have the unit tests
>> trigger reading the extra headers or confirm that it's not possible.
>>
>> Bear
>>
>> On Wed, Sep 2, 2015 at 3:58 PM, Bear Giles  wrote:
>>
>>> Hi, I know that the implementation of the PKWARE AES crypto is subject
>>> to license restrictions but is it possible to recognize the extension
>>> fields so anyone scanning an unfamiliar file will at least know what the
>>> extra field headers contain?
>>>
>>> I don't know if code to parse the contents (solely using the standard
>>> JCE) would trigger export restrictions. This would NOT be decrypting the
>>> data, just adding a thin layer of code to parse a standardized ASN.1 object.
>>>
>>> (BTW there are some other header types documented but except for a few I
>>> don't know what they are.)
>>>
>>> The header IDs (per
>>> https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) are:
>>>
>>>   0x0014PKCS#7 Store for X.509 Certificates
>>>   0x0015X.509 Certificate ID and Signature for
>>> individual file
>>>   0x0016X.509 Certificate ID for Central Directory
>>>   0x0017Strong Encryption Header
>>>   0x0019PKCS#7 Encryption Recipient Certificate List
>>>
>>> The definitions are:
>>>
>>>4.5.9 -PKCS#7 Store for X.509 Certificates (0x0014):
>>>
>>> This field MUST contain information about each of the certificates
>>> files may be signed with. When the Central Directory Encryption
>>> feature is enabled for a ZIP file, this record will appear in
>>> the Archive Extra Data Record, otherwise it will appear in the
>>> first central directory record and will be ignored in any
>>> other record.
>>>
>>>
>>> Note: all fields stored in Intel low-byte/high-byte order.
>>>
>>> Value Size Description
>>> -  ---
>>> (Store) 0x00142 bytes  Tag for this "extra" block type
>>> TSize 2 bytes  Size of the store data
>>> TData TSizeData about the store
>>>
>>>
>>>4.5.10 -X.509 Certificate ID and Signature for individual file (0x0015):
>>>
>>> This field contains the information about which certificate in
>>> the PKCS#7 store was used to sign a particular file. It also
>>> contains

Re: [COMPRESS] pkware header id for crypto extensions

2015-09-14 Thread Bear Giles
No problem, I've also been preempted.

I want to capture as much information as possible but the documentation is
a weird mixture of details and hand-waving. Some is detailed, some is
"compatible with the format used by the Microsoft CRYPTOAPI".  How is
anyone supposed to use that if all they have is the APPNOTE.txt file?
Anyway I can demonstrate that the headers are recognized in at least one
test case so I'll just remove the debugging code and add a few getters.

I also found documentation on WinZip so I can also capture that information
later. It uses a different compression method ID to indicate encryption
instead of new extra fields.

FWIW my initial impression from what's in the fields is that WinZip is
competently implemented but PKWare is a LOT more solid. I can give details
if anyone wants to be bored. :-)

Bear

On Mon, Sep 14, 2015 at 1:34 AM, Stefan Bodewig  wrote:

> Hi Bear
>
> sorry for the late reply.  My spare time is currently severly limited
> but this should be getting better by the end of the month.
>
> On 2015-09-05, Bear Giles wrote:
>
> > I was just reminded of the risks of making assumptions. There's two
> > separate implementations of AES strong encryption - WinZip AES and
> PKWare.
> > I've been focused on the latter but we should recognize both types of
> > headers.
>
> IIRC WinZip AES works rather differently by using a separate
> "compression method".  I haven't looked into the details so far, though.
> It should be fine to start with just one implementation IMHO.
>
> > I've pushed a bit more code that partially parses the PKWare headers.
> It's
> > enough to identify the encryption and hash algorithms used but a lot is
> > left unparsed.
>
> Unless we really wanted to implement the crypto part this sounds
> like a lot already.
>
> Thanks
>
> Stefan
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


Re: [COMPRESS] pkware header id for crypto extensions

2015-09-20 Thread Bear Giles
I took out some debugging scaffolding so you can check it out. One of the
unit tests fails but it should since it's trying to read an archive with an
encrypted CD. (I haven't gotten around to wrapping it with an exception
handler that fails if an exception isn't thrown.)

Theoretically you can do a pull request. I think.

Bear

On Tue, Sep 15, 2015 at 8:03 AM, Stefan Bodewig  wrote:

> On 2015-09-14, Bear Giles wrote:
>
> > I want to capture as much information as possible but the documentation
> is
> > a weird mixture of details and hand-waving. Some is detailed, some is
> > "compatible with the format used by the Microsoft CRYPTOAPI".  How is
> > anyone supposed to use that if all they have is the APPNOTE.txt file?
>
> Compared to some other "specs" we implement, this is brilliant ;-)
>
> > Anyway I can demonstrate that the headers are recognized in at least one
> > test case so I'll just remove the debugging code and add a few getters.
>
> Sounds great.
>
> > FWIW my initial impression from what's in the fields is that WinZip is
> > competently implemented but PKWare is a LOT more solid. I can give
> details
> > if anyone wants to be bored. :-)
>
> Expect me to ask you for the boring details once I start breathing
> again.  Probably better off-list :-)
>
> Cheers
>
> Stefan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


Re: [COMPRESS] pkware header id for crypto extensions

2015-09-21 Thread Bear Giles
I have a moment to explain why the PKWARE crypto is stronger than the
WinZip crypto.

1. The key has validation data, the VData and VCRC32 values.

There are a couple different ways to compute a message integrity check that
allows you to verify the encryption key and that the message contents have
not been modified. This is only available after encrypting all of the data
- in streaming mode this information would have to be written to a footer.
That's not difficult but it means that something has been written to the
disk before it's verified and that might cause problems.

Message integrity checks also require maintenance of a second key that's
independent of the first. Some ciphers have an encryption mode that handles
message integrity at the same time it does decryption but many ciphers do
not and you can't assume all readers supports those ciphers and modes.

Key validation won't detect message corruption but the file CRC gives us a
modest level of protection. It's only modest though since it's easy to set
the message CRC to whatever value you want if you can modify 4 bytes.
That's impractical with a true HMAC with strong algorithms.

2. Encrypted random data (ERD)

You can often predict the initial contents of a file from the filename
extension. For instance a .xml file probably starts with  or the compressed version of the same. This means you have
a "known plaintext" problem and it is a powerful tool used to break the
legacy ZIP encryption.

ERD handles that by prepending random data to the file. The reader knows to
ignore those bytes. The ERD is encrypted in the header with the same key
(but all-zero IV?) to avoid the same "known plaintext" problem.

One of the WW-II (or WW-I?) German ciphers was broken this way. Messages
were supposed to start with a unique value but one clerk repeated one by
accident. It was enough to break the cipher since all messages started the
same way.

Bear

On Sun, Sep 20, 2015 at 8:36 PM, Bear Giles  wrote:

> I took out some debugging scaffolding so you can check it out. One of the
> unit tests fails but it should since it's trying to read an archive with an
> encrypted CD. (I haven't gotten around to wrapping it with an exception
> handler that fails if an exception isn't thrown.)
>
> Theoretically you can do a pull request. I think.
>
> Bear
>
> On Tue, Sep 15, 2015 at 8:03 AM, Stefan Bodewig 
> wrote:
>
>> On 2015-09-14, Bear Giles wrote:
>>
>> > I want to capture as much information as possible but the documentation
>> is
>> > a weird mixture of details and hand-waving. Some is detailed, some is
>> > "compatible with the format used by the Microsoft CRYPTOAPI".  How is
>> > anyone supposed to use that if all they have is the APPNOTE.txt file?
>>
>> Compared to some other "specs" we implement, this is brilliant ;-)
>>
>> > Anyway I can demonstrate that the headers are recognized in at least one
>> > test case so I'll just remove the debugging code and add a few getters.
>>
>> Sounds great.
>>
>> > FWIW my initial impression from what's in the fields is that WinZip is
>> > competently implemented but PKWare is a LOT more solid. I can give
>> details
>> > if anyone wants to be bored. :-)
>>
>> Expect me to ask you for the boring details once I start breathing
>> again.  Probably better off-list :-)
>>
>> Cheers
>>
>> Stefan
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
>


[COMPRESS] pull request for pkware crypto headers

2015-10-24 Thread Bear Giles
​I'm still unclear on the final mile of a submission. I think I've properly
merged everything upstream and submitted a pull request for my work but I
wouldn't put much money on that bet.  Let me know if there's problems - it
should only be a half-dozen files or so.

Also I can't build locally due to an 'animal' plugin warning. I have no
idea what the problem is since I've tried several different JDKs.

Bear​


[COMPRESS] PKWare header questions

2015-12-13 Thread Bear Giles
1. I created the PKWareExtraHeader as a marker - it should make it clear
that the headers will only be present in PKWare-created files and everyone
else can ignore them. This is in contract to other headers that are much
more standard. It was an oversight if this got lost when I had to refork
the project.

2. ExtraFieldUtils and Tests - I'm not that familiar with the code since
I'm more focused on the crypto than the general functionality. The former
is an oversight. The latter is something that stumped me. During
development I had System.out statements that verified I was reading the
expected values but we obviously don't want to leave that in the final
code. Is there a way to read a file and then inspect all of the headers
within the test framework?

Bear


Re: [compress] Where to Place Big Test Archives?

2011-07-27 Thread Bear Giles
In case you're talking about my test data the uncompressed test case could
be externally compressed and the test code adjusted accordingly. It never
occurred to me to do that since you never compress dump files like that in
the wild - you use the native compression instead - but the file is almost
entirely \0. I guess I also assumed the test data would be a separate
download.

You definitely want to have both compressed and uncompressed test data
though, and large enough files and directories to force multiple segments. I
think I managed to fail on nearly every case during development.

Bear


On Wed, Jul 27, 2011 at 10:30 AM, Gary Gregory wrote:

> On Wed, Jul 27, 2011 at 9:45 AM, Luc Maisonobe 
> wrote:
> > Le 27/07/2011 14:54, Stefan Bodewig a écrit :
> >>
> >> On 2011-07-27, Torsten Curdt wrote:
> >>
> >>> But to be clear: this is quite some work just to safe some disk space
> >>> for people who check out the whole of commons but do not care about
> >>> compress.
> >>
> >> Actually my main concern was/is the size of the source distribution and
> >> network bandwidth rather than disk space.
> >
> > I second that. I know people who even in occidental countries still have
> no
> > access to ADSL and can use only 56k modems, with unreliable connexion.
> For
> > them, 15MB is huge.
> >
> > We should take care of the digital divide.
>
> I do not see how. If I want to be a developer on [compress], I need to
> be able to run the full test suite.
>
> The best workaround I can see is to create a separate download for the
> test data. This allows developers who need sources for debugging to
> still get them without downloading GBs of test data.
>
> Gary
>
>
> >
> > Luc
> >
> >> If I'm the only one who
> >> thinks a 15MB file in svn trunk is excessive, then I'll be happy to shut
> >> up - I'm going to need the files anyway. ;-)
> >>
> >> Stefan
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> >> For additional commands, e-mail: dev-h...@commons.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> > For additional commands, e-mail: dev-h...@commons.apache.org
> >
> >
>
>
>
> --
> Thank you,
> Gary
>
> http://garygregory.wordpress.com/
> http://garygregory.com/
> http://people.apache.org/~ggregory/
> http://twitter.com/GaryGregory
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


Re: [VOTE] Release Compress 1.3 based on RC2

2011-10-31 Thread Bear Giles
I found a few issues (1330) with Fortify. As anyone who's used it knows the
vast majority of those are of the "but that's what I intended" variety, but
there were 180 cases of unclosed resource streams and 354 cases of
potential denial of service.

In the first case we all write

  InputStream is = get some stream;
  is.read();
  is.close(); // maybe, or maybe we just let it go out of scope.

but Fortify wants:

  InputStream is = null;
  try {
  is = get some stream;
  is.read();
  } finally {
 if (is != null) {
   is.close();  // could possibly throw a second exception here
 }
  }

This ensures that the input stream will always be closed. Going out of
scope SHOULD call the finalize method but there's no guarantee that this
will happen and it's possible that the stream was left open.


In the second case we write something like:

  public int read(byte[] b, int from, int length) throws IOException {
 int read = in.read(b, from, length);
 this.count(read);
 return read;
  }

but Fortify wants something like

  public int read(byte[] b, int from, int length) throws IOException {
 if (b != null || b.length < length) {
   throw new IllegalArgumentException("buffer must be at least " +
length + " bytes long.");
 }
 // and probably also
 if (from < 0 || length < 1) {
   throw new IllegalArgumentException("...");
 }

 int read = in.read(b, from, length);
 this.count(read);
 return read;
  }

which is hard to argue with even though it will take a little longer to
execute.

On the one hand these seem extremely picky. On the other hand this is a
library used by others, not our own code where we have a lot more
flexibility in changing it if things go wrong. (I know, people can get the
source and compile it themselves but it will take time to track down the
problem, identify a solution, etc.)

Is this something worth addressing now or should it wait until 2.0?

Bear Giles


On Sun, Oct 30, 2011 at 7:38 PM, Gary Gregory wrote:

> +1
>
> Looks good on Oracle Java 1.6.0_29 and 1.7.0_01, Maven 3.0.3 on Windows 7
> 64 bit.
>
> Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500)
> Maven home: C:\Java\apache-maven-3.0.3\bin\..
> Java version: 1.7.0_01, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_01\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
>
> Gary
>
> On Sun, Oct 30, 2011 at 2:41 PM, Jörg Schaible  >wrote:
>
> > Stefan Bodewig wrote:
> >
> > > Hi all,
> > >
> > > compared to RC1 Michael Kuss' name has been fixed and he's been added
> as
> > > contributor.  A bunch of additional tests increased coverage (still
> some
> > > areas are not covered but coverage is better than it has been for any
> > > prior release).  A few plugins have been upgraded.
> > >
> > >   Compress 1.3 RC2 is available for review here:
> > > http://people.apache.org/~bodewig/compress-1.3-RC2/
> > >
> > >   Maven artifacts are here:
> > >
> >
> >
> https://repository.apache.org/content/repositories/orgapachecommons-111/org/apache/commons/commons-
> > compress/1.3/
> > >
> > >   Details of changes since 1.1 are in the release notes:
> > >
> http://people.apache.org/~bodewig/compress-1.3-RC2/RELEASE-NOTES.txt
> > > http://people.apache.org/~bodewig/compress-1.3-RC2/site/changes-
> > report.html
> > >
> > >   I have tested this with Oracle JDK 1.5 and OpenJDK 1.6 using maven2.
> > >
> > >   The tag is here (svn revision 1190156):
> > >
> >
> http://svn.apache.org/viewvc/commons/proper/compress/tags/COMPRESS_1.3_RC2/
> > >
> > >   Site:
> > > http://people.apache.org/~bodewig/compress-1.3-RC2/site/
> > >
> > >   Clirr Report (compared to 1.2):
> > > http://people.apache.org/~bodewig/compress-1.3-RC2/site/clirr-
> > report.html
> > >
> > >   RAT Report:
> > > http://people.apache.org/~bodewig/compress-1.3-RC2/site/rat-
> > report.html
> > >
> > >   Votes, please.  This vote will close in 96 hours, 0500 GMT
> 01-November
> > >   2011
> > >
> > >   [ ] +1 Release these artifacts
> > >   [ ] +0 OK, but...
> > >   [ ] -0 OK, but really should fix...
> > >   [ ] -1 I oppose this release because...
> >
> > +1
> >
> > Build from source with my complete compiler zoo.
> >
> > - Jörg
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> > For additional commands, e-mail: dev-h...@commons.apache.org
> >
> >
>
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> JUnit in Action, 2nd Ed: <http://goog_1249600977>http://bit.ly/ECvg0
> Spring Batch in Action: <http://s.apache.org/HOq>http://bit.ly/bqpbCK
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>


Re: [VOTE] Release Compress 1.3 based on RC2

2011-10-31 Thread Bear Giles
Quick followup - I know there's also a school of thought that bad
parameters should be passed to the InputStream and let it throw the
exception since it's the method that actually cares about the values.
 Maybe a negative offset will have meaning at some point in the future. So
you can argue that the code would be cleaner if we (explicitly) let the
InputStream method handle bad parameters or it will be cleaner if we
explicitly check for null or short buffers or if we do a mixture.


On Mon, Oct 31, 2011 at 2:59 PM, Bear Giles  wrote:

> I found a few issues (1330) with Fortify. As anyone who's used it knows
> the vast majority of those are of the "but that's what I intended" variety,
> but there were 180 cases of unclosed resource streams and 354 cases of
> potential denial of service.
>
> In the first case we all write
>
>   InputStream is = get some stream;
>   is.read();
>   is.close(); // maybe, or maybe we just let it go out of scope.
>
> but Fortify wants:
>
>   InputStream is = null;
>   try {
>   is = get some stream;
>   is.read();
>   } finally {
>  if (is != null) {
>is.close();  // could possibly throw a second exception here
>  }
>   }
>
> This ensures that the input stream will always be closed. Going out of
> scope SHOULD call the finalize method but there's no guarantee that this
> will happen and it's possible that the stream was left open.
>
>
> In the second case we write something like:
>
>   public int read(byte[] b, int from, int length) throws IOException {
>  int read = in.read(b, from, length);
>  this.count(read);
>  return read;
>   }
>
> but Fortify wants something like
>
>   public int read(byte[] b, int from, int length) throws IOException {
>  if (b != null || b.length < length) {
>throw new IllegalArgumentException("buffer must be at least " +
> length + " bytes long.");
>  }
>  // and probably also
>  if (from < 0 || length < 1) {
>throw new IllegalArgumentException("...");
>  }
>
>  int read = in.read(b, from, length);
>  this.count(read);
>  return read;
>   }
>
> which is hard to argue with even though it will take a little longer to
> execute.
>
> On the one hand these seem extremely picky. On the other hand this is a
> library used by others, not our own code where we have a lot more
> flexibility in changing it if things go wrong. (I know, people can get the
> source and compile it themselves but it will take time to track down the
> problem, identify a solution, etc.)
>
> Is this something worth addressing now or should it wait until 2.0?
>
> Bear Giles
>
>
> On Sun, Oct 30, 2011 at 7:38 PM, Gary Gregory wrote:
>
>> +1
>>
>> Looks good on Oracle Java 1.6.0_29 and 1.7.0_01, Maven 3.0.3 on Windows 7
>> 64 bit.
>>
>> Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500)
>> Maven home: C:\Java\apache-maven-3.0.3\bin\..
>> Java version: 1.7.0_01, vendor: Oracle Corporation
>> Java home: C:\Program Files\Java\jdk1.7.0_01\jre
>> Default locale: en_US, platform encoding: Cp1252
>> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
>>
>> Gary
>>
>> On Sun, Oct 30, 2011 at 2:41 PM, Jörg Schaible > >wrote:
>>
>> > Stefan Bodewig wrote:
>> >
>> > > Hi all,
>> > >
>> > > compared to RC1 Michael Kuss' name has been fixed and he's been added
>> as
>> > > contributor.  A bunch of additional tests increased coverage (still
>> some
>> > > areas are not covered but coverage is better than it has been for any
>> > > prior release).  A few plugins have been upgraded.
>> > >
>> > >   Compress 1.3 RC2 is available for review here:
>> > > http://people.apache.org/~bodewig/compress-1.3-RC2/
>> > >
>> > >   Maven artifacts are here:
>> > >
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachecommons-111/org/apache/commons/commons-
>> > compress/1.3/
>> > >
>> > >   Details of changes since 1.1 are in the release notes:
>> > >
>> http://people.apache.org/~bodewig/compress-1.3-RC2/RELEASE-NOTES.txt
>> > > http://people.apache.org/~bodewig/compress-1.3-RC2/site/changes-
>> > report.html
>> > >
>> > >   I have tested this with Oracle JDK 1.5 and OpenJDK 1.6 using maven2.
>> > >
>> > >   The tag is here (svn revision 119

[compress] Support for strong encryption in zip files?

2013-06-11 Thread Bear Giles
What is the status on strong encryption in zip files? That is - I know we
don't have it, but is there a reason why it hasn't been added or is it just
an itch that hasn't been scratched?

I can see several levels of values

1. add ZipExtraField handlers for the five extra fields of interest (0x14,
0x15, 0x16, 0x17, 0x19) even if encryption/decryption isn't supported. It
could still provide useful information, e.g., identifying users able to
decrypt the file.
2. add read-only support.
3. add write support.

The last two items can be further split by password and X.509 certificate.
The latter is particularly interesting since it means you can have an
unprivileged user create an encrypted archive without having to reveal the
password.

The two potential issues I've found so far are 1) export/import
restrictions that may limit functionality and 2) language in the PKWARE
documentation at http://www.pkware.com/documents/casestudies/APPNOTE.TXT.

I think the first point is easily handled since it can and should be punted
to the user's configuration. The default configuration is designed to be
legal everywhere and the user (even US domestic users) must take explicit
action to enable stronger encryption.

On the second point the language is:

7.0 Strong Encryption Specification
---

   7.0.1 Portions of the Strong Encryption technology defined in this
   specification are covered under patents and pending patent applications.
   Refer to the section in this document entitled "Incorporating
   PKWARE Proprietary Technology into Your Product" for more information.


10.0 Incorporating PKWARE Proprietary Technology into Your Product
--

   10.1 The Use or Implementation in a product of APPNOTE technological
   components pertaining to either strong encryption or patching requires
   a separate, executed license agreement from PKWARE. Please contact
   PKWARE at zipfor...@pkware.com or +1-414-289-9788 with regard to
   acquiring such a license.

   10.2 Additional information regarding PKWARE proprietray technology is
   available at http://www.pkware.com/appnote.


I don't know if this is outdated language, if it's a pro forma
request, if Apache commons already has a license agreement, if Apache
commons won't use it as a matter of principle, etc.

--- Legacy Encryption ---

A related question involves support for the legacy encryption. It's
easily cracked but people might need to read old encrypted files. It's
even possible that some people may still be forced to use it because
of contractual obligations.

I will resist the temptation to offer a flag to make the decryption
key optional.

Bear


Re: [compress] Support for strong encryption in zip files?

2013-06-12 Thread Bear Giles
This morning I realized that the license requirement could be a real
problem if it applies to the end developer instead of or in addition to any
libraries. On the other hand it might just be a completely reasonable
requirement that the code be tested for compatibility with PKWARE and the
license requirement would stop with the library. I have no idea.

Fortunately I think it would be easy to design this around a injected
interface, something like

   byte[] encrypt(byte[] plaintext, List fields)

and likewise for decryption. The fields are immutable for decryption but
not encryption. (It looks like streaming isn't always possible since the
headers contain checksum information but I need to re-check that.) This
could be generalized to allow arbitrary transformations - compression,
encryption, etc., albeit at the loss of interoperability if developers
create their own. In this case the encryption could be provided in a
separate library.

On the encryption itself I can understand the concerns about the false
sense of security with the legacy encryption but the newer stuff looks
strong - it supports AES, has a unique encryption key for each file,
prepends randomized data to block known plaintext attacks, etc. You can
encrypt the metadata (filenames, etc.) I definitely think embedded
encryption is better than running it through PGP since a damaged file is
much more recoverable - just scan through the file for the next
(unencrypted) header. PGP also chunks data but there's no correlation
between the PGP chunks and the zip entries.

A clean room implementation based on the APPNOTE is looking like it might
be a challenge. It's a good overview but it doesn't document key things
like how the per-file session key is generated from the master key. I don't
know how much I can peek into other open-source implementations without
running afoul of "original work" requirements - obviously I would never
cut-and-paste but some of this has only one possible implementation modulo
window-dressing. (E.g., how do you call the method?) Maybe that's provided
as part of a developer pack after you contact them about a license.

Bear


Re: [compress] Support for strong encryption in zip files?

2013-06-12 Thread Bear Giles
Those regulations are why the default JCE policy is so restricted. I have
to assume that there were a lot of lawyers involved in making sure that the
Java approach to extending the JCE capabilities to more than 56/512 bits
compiled with the law.

There encryption in two places in a zip file: the files and the central
directory. The former could be easily handled via a plugin architecture,
the latter would require much tighter integration but it's a fairly recent
addition - you can provide file encryption without necessarily providing
metadata encryption.

As to the five extra fields themselves I don't have all of my notes with me
but they are:

0x0014, 0x0019 - list of recipients as a collection of X.509 certificates
in PKCS7 structure.
0x0015, 0x0016 - certificate ID + "signature data". I'm not sure yet what
they mean by that.
0x0017 - identification of encryption algorithm, key length, processing
flag (decrypt using password, certificate or both)

You can't use certs for decryption, just authentication, so I don't know
what their status would be. Strictly speaking all commons really needs is
the ability to compare a certificate ID provided by the user with the
certificate ID in the extra field to give a yes/no answer on whether to
bother trying to decrypt the data or to just skip the entry.

The keying material is kept in a "decryption header" preceding the
compressed file data. See "Single Password Symmetric Encryption Method"
section 3. It looks like it should be straight JCE with simple lookups - no
dependency on anything other than the standard JCE API.

Bear


On Wed, Jun 12, 2013 at 2:09 PM, Gary Gregory wrote:

> Why not start with encryption as provided out of the box by Java (6?).
>
> It's up to a user to install the string enc policy files or BouncyCastle or
> whatnot.
>
> It's all plugable IIRC.
>
> Gary
>
>
> On Wed, Jun 12, 2013 at 4:04 PM, Siegfried Goeschl 
> wrote:
>
> > Hi folks,
> >
> > don't know if this was already mentioned but providing strong encryption
> > causes administrative head-ache
> >
> > http://www.apache.org/dev/**crypto.html<
> http://www.apache.org/dev/crypto.html>
> >
> > Cheers,
> >
> > Siegfried Goeschl
> >
> > On 12.06.13 16:29, Bear Giles wrote:
> >
> >> This morning I realized that the license requirement could be a real
> >> problem if it applies to the end developer instead of or in addition to
> >> any
> >> libraries. On the other hand it might just be a completely reasonable
> >> requirement that the code be tested for compatibility with PKWARE and
> the
> >> license requirement would stop with the library. I have no idea.
> >>
> >> Fortunately I think it would be easy to design this around a injected
> >> interface, something like
> >>
> >> byte[] encrypt(byte[] plaintext, List fields)
> >>
> >> and likewise for decryption. The fields are immutable for decryption but
> >> not encryption. (It looks like streaming isn't always possible since the
> >> headers contain checksum information but I need to re-check that.) This
> >> could be generalized to allow arbitrary transformations - compression,
> >> encryption, etc., albeit at the loss of interoperability if developers
> >> create their own. In this case the encryption could be provided in a
> >> separate library.
> >>
> >> On the encryption itself I can understand the concerns about the false
> >> sense of security with the legacy encryption but the newer stuff looks
> >> strong - it supports AES, has a unique encryption key for each file,
> >> prepends randomized data to block known plaintext attacks, etc. You can
> >> encrypt the metadata (filenames, etc.) I definitely think embedded
> >> encryption is better than running it through PGP since a damaged file is
> >> much more recoverable - just scan through the file for the next
> >> (unencrypted) header. PGP also chunks data but there's no correlation
> >> between the PGP chunks and the zip entries.
> >>
> >> A clean room implementation based on the APPNOTE is looking like it
> might
> >> be a challenge. It's a good overview but it doesn't document key things
> >> like how the per-file session key is generated from the master key. I
> >> don't
> >> know how much I can peek into other open-source implementations without
> >> running afoul of "original work" requirements - obviously I would never
> >> cut-and-paste but some of this has only one possible implementation
> modulo
> >

Re: [compress] Support for strong encryption in zip files?

2013-06-16 Thread Bear Giles
What would be the best way to start pushing code? It will be more tightly
integrated with the existing code than my earlier 'dump' contribution so I
can't just dump a source ball. I also don't want to post code snippets here
if it will cause people problems.

Thanks,

Bear


On Thu, Jun 13, 2013 at 1:46 AM, Stefan Bodewig  wrote:

> On 2013-06-12, Siegfried Goeschl wrote:
>
> > Hi folks,
>
> > don't know if this was already mentioned but providing strong
> > encryption causes administrative head-ache
>
> Actually, it doesn't.  All that has to be done, is editing a page in the
> CMS, adding a notice to the README and sending a single email.  Been
> there, have got the T-shirt.
>
> I even think we don't need to do anything as long as we stick to plain
> JCE.
>
> Hmm, we already have AES support in the new 7z package.  I'll take care
> of the crypto policy stuff.
>
> Stefan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


Re: [compress] Support for strong encryption in zip files?

2013-06-17 Thread Bear Giles
Would that trigger the 'export' clause?

The main deliverable I hope to have in the next few days is a method for
password-based decryption that will be called at the end of getNextEntry().
It looks like it will take a Cipher initialized by the master password and
provide either a Cipher initialized to the per-file session key or null if
the header didn't validate.

Certificate-based decryption is a little more work but not a lot.

At that point password-based decryption would just need a way to get the
initial password and cipher, and a way to run the data through the Cipher
prior to decompression. Both would probably be better handled by someone
familiar with the code base.

Bear


Re: [compress] Strong Crypto in Tests

2013-10-23 Thread Bear Giles
Should that be PKCS7Padding? Or would that be worse - I don't recall if
it's one of the "must have" paddings in the spec.

http://crypto.stackexchange.com/questions/9043/what-is-the-difference-between-pkcs5-padding-and-pkcs7-padding

In practice it's unlikely that a software-based crypto provider will be so
pedantic that it throws an exception if you try to use PKCS5 with a 256-bit
cipher but I'm not sure if that's true if it's a hardware-based crypto
provider. Anyone who springs for hardware will probably want something that
strictly complies with the spec and PKCS5 is only defined for 64-bit
ciphers.

Bear


On Wed, Oct 23, 2013 at 5:03 AM, Stefan Bodewig  wrote:

> On 2013-10-23, Jörg Schaible wrote:
>
> >  boolean supportedKeyLength(int keyLen) throws NoSuchAlgorithmException
> >  {
> >if (Cipher.getMaxAllowedKeyLength("AES/ECB/PKCS5Padding") < keyLen) {
> >  System.err.println("WARNING: " + getName()
> >+ " not executed, environment does not support " + keyLen
> >+ "-bit keys for AES");
> >  return false;
> >}
> >return true;
>
> Looks a lot nicer than my "catch Exception" approach, thanks.
>
> Stefan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


[commons-compress] traditional zip encryption patch to come

2016-02-14 Thread Bear Giles
We discussed adding support for the traditional zip encryption algorithm a
while back. IIRC the concensus was loosely against it since it's so weak
and we don't want to mislead people.

Since then I can add two points.

First, I heard from someone who said that she has to use the traditional
encryption algorithm for legal reasons. To be more precise she don't have
to use the traditional zip encryption algorithm specifically - she has to
use some form of encryption and for various reasons (e.g., ancient embedded
systems that can't be updated) this is the best she can do. It's enough to
satisfy the legal requirement that the data not be sent in plaintext and
her exposure is limited since it's only used during the brief window when
the embedded systems make a dialup connection to the central server.

In any case it's someone with a good use case for why she needs to continue
using the traditional algorithm.

Second, I've been looking at the strong encryption algorithms (WinZip and
the un-distributable PKWare) and they'll need some way to insert encryption
and decryption into the ZIP streams. The traditional approach may
illustrate problems we can expect with WinZip strong encryption.

My approach is to create two Filter classes -
TraditionalZipEncryptionInputStream and
TraditionalZipEncryptionOutputStream - and inject them at the lowest level.

There are two problems that prevent this from being completely transparent.

First, the ciphertext is precisely 12 bytes larger than the plaintext due
to the presence of an encrypted salt. I know that ZipEntry has a
'compressed size' field that's used with reading and skipping the actual
stream but I don't know if the decompression logic uses an internal EOF
marker or if it uses the number of bytes. This might cause problems and I
don't want to make any assumptions.

Second, the algorithm requires the CRC of the original data. The last byte
of the encrypted salt should be the same as the low byte of the CRC. If not
it's assumed that the provided password is incorrect. (If it matches
there's still a remote chance that it's incorrect but overall we can save a
lot of wasted effort.) This isn't an issue in streaming mode - just use 0 -
but it requires a way to get that value to the constructor when the header
value is not 0.

Thoughts?

Bear


Re: [commons-compress] traditional zip encryption patch to come

2016-02-16 Thread Bear Giles
I can create test files. (I still need to create some PKWare test files as
well...). I also need to figure out a way to extract the encrypted content
to I can verify my implementation of the crypto logic itself. I think that
will just require disabling any check for the 'encrypted' bit.

​Source at
https://github.com/beargiles/commons-compress/blob/traditional-encryption/src/main/java/org/apache/commons/compress/archivers/zip/TraditionalZipEncryption.java
.

Test at
https://github.com/beargiles/commons-compress/blob/traditional-encryption/src/test/java/org/apache/commons/compress/archivers/zip/TraditionalZipEncryptionTest.java
​

​Bear​



On Tue, Feb 16, 2016 at 9:08 AM, Stefan Bodewig  wrote:

> On 2016-02-15, Bear Giles wrote:
>
> > We discussed adding support for the traditional zip encryption algorithm
> a
> > while back. IIRC the concensus was loosely against it since it's so weak
> > and we don't want to mislead people.
>
> I'm pretty sure we should ad a big warning about the quality of
> encryption to expect, but there are good reasons to implement decryption
> - and not quite so good reasons for encryption, too :-)
>
> > Second, I've been looking at the strong encryption algorithms (WinZip and
> > the un-distributable PKWare) and they'll need some way to insert
> encryption
> > and decryption into the ZIP streams. The traditional approach may
> > illustrate problems we can expect with WinZip strong encryption.
>
> True. Even though WinZip strong encryption will need a different
> approach as it is a "method" of its own rather than used in addition to
> one of the other methods.
>
> > My approach is to create two Filter classes -
> > TraditionalZipEncryptionInputStream and
> > TraditionalZipEncryptionOutputStream - and inject them at the lowest
> level.
>
> > There are two problems that prevent this from being completely
> transparent.
>
> > First, the ciphertext is precisely 12 bytes larger than the plaintext due
> > to the presence of an encrypted salt. I know that ZipEntry has a
> > 'compressed size' field that's used with reading and skipping the actual
> > stream but I don't know if the decompression logic uses an internal EOF
> > marker or if it uses the number of bytes. This might cause problems and I
> > don't want to make any assumptions.
>
> Without encryption you get both flavors. DEFLATE uses an internal
> marker, STORED uses the number of bytes, for example. From looking
> through appnote I don't see a definitive answer, we should probably look
> at real world archives.
>
> > Second, the algorithm requires the CRC of the original data. The last
> byte
> > of the encrypted salt should be the same as the low byte of the CRC. If
> not
> > it's assumed that the provided password is incorrect. (If it matches
> > there's still a remote chance that it's incorrect but overall we can
> save a
> > lot of wasted effort.) This isn't an issue in streaming mode - just use
> 0 -
> > but it requires a way to get that value to the constructor when the
> header
> > value is not 0.
>
> We may even run into a situation - when writing to a RandomAccessFile -
> where we move back to write the CRC into the header after we've written
> the file's contents. This would likely require us to calculate the CRC
> before actually starting to encrypt the data.
>
> Stefan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


Re: [Compress] Hiring Coder

2013-11-29 Thread Bear Giles
I've been looking at the strong crypto side of it. I have a test file but the 
published spec is vague at times (unless you are in windows and can use their 
crypto library), not a huge hurdle but I've been busy. I should at least submit 
a patch that recognizes the four addl block types even if we don't do anything 
with them. ( I do NOT want to peek at other implementations. Cleanroom all the 
way.)

What are the license issues? I had looked at the legal stuff briefly but could 
have easily overlooked it since I was only looking at the tech note that 
documents their numerous versions.

Bear


Sent from my iPad

> On Nov 29, 2013, at 12:57 AM, Stefan Bodewig  wrote:
> 
>> On 2013-11-29, dam6923 . wrote:
>> 
>> Is there any policy in place or method already laid out for putting
>> bounties on certain task items?  In particular, I could chip something
>> in for the following two tickets:
> 
> What Phil said.
> 
>> 1) https://issues.apache.org/jira/browse/COMPRESS-88
>> 2) https://issues.apache.org/jira/browse/COMPRESS-238
> 
> Note that COMPRESS-88 is extremely unlikely to end up in an official ASF
> distribution anyway - the real problem is PKWARE's licensing of the
> encryption parts of the spec.  This is unless you mean the WinZIP AES
> implementation or the "old encryption" rather than the strong crypto
> protocol described by PKWARE.
> 
> What Compress can do is provide hooks that an implementation of
> COMPRESS-88 can be maintained outside of Compress.  I plan to seriously
> get the ball rolling for Compress 2.x (which should provide said hooks
> in the ZIP package) after christmas but have too much on my plate to
> drive this right now.
> 
> Stefan
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



Re: svn commit: r1584206 - /commons/proper/compress/trunk/src/main/java/org/apache/commons/compress/archivers/sevenz/Coders.java

2014-04-03 Thread Bear Giles
I'm bummed. One of my interview torture questions was whether this compiles.

class _ {
  static _ _ ;
  _() {
  }
  _(_ _) {
  }
}


On Thu, Apr 3, 2014 at 6:44 AM, Stefan Bodewig  wrote:

> On 2014-04-03, Emmanuel Bourg wrote:
>
> >
> http://openjdk.5641.n7.nabble.com/Warning-about-single-underscore-identifier-td145974.html
>
> They are deprecating the use of "_" for method argument names so that
> one day I may be able to use it for the exact same purpose I've done so
> for :-)
>
> Stefan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>