Re: [crypto][chimera] Next steps

[Gary Gregory]

On Feb 21, 2016 11:59 PM, "Benedikt Ritter"  wrote:
>
> Hi again,
>
> 2016-02-20 12:15 GMT+01:00 Benedikt Ritter :
>
> > Hi,
> >
> > I'd like to discuss the next steps for moving the Chimera component to
> > Apache Commons. So far, none of the other PMC members has expressed his
or
> > her thoughts about this. If nobody brings up objections about moving the
> > component to Apache Commons, I'm assuming lazy consensus about this.
> >
> > So the next steps would be:
> > - decide on a name for the new component (my proposal was Apache Commons
> > Crypto)
> > - move code to an Apache repo (probably git?!)
> > - request a Jira project
> > - setup maven build
> > - setup project website
> > - work on an initial release under Apache Commons coordinates
> >
> > Anything missing?
> >
>
> Given the recent discussion, I'd like to update to above list:
>
> - Define the scope of the new Component (in our wiki? in the repo on
> github?)
> - Define the group of initial committers
> - Decide on a name
> - Think about IP clearance - The code seems to belong to Intel. What do we
> have to do to move the code the the ASF? Go through the Incubator?
> - Move code to Apache Commons
> - request Jira project
> - setup maven build
> - setup project website
> - work on an initial release
>
> Thoughts?

Set up CI build.

Gary

> Benedikt
>
>
> >
> > Regards,
> > Benedikt
> >
> > --
> > http://home.apache.org/~britter/
> > http://twitter.com/BenediktRitter
> > http://github.com/britter
> >
>
>
>
> --
> http://home.apache.org/~britter/
> http://twitter.com/BenediktRitter
> http://github.com/britter

RE: [crypto][chimera] Next steps

[Xu, Cheng A]

Hi Gary,
We use JNI to get to Openssl.

Ferd

-Original Message-
From: Gary Gregory [mailto:garydgreg...@gmail.com] 
Sent: Monday, February 22, 2016 2:57 PM
To: Commons Developers List
Subject: Re: [crypto][chimera] Next steps

Curious: How to you get to OpenSSL, JNI? JNA?

Gary

On Sun, Feb 21, 2016 at 10:51 PM, Chen, Haifeng 
wrote:

> Thanks Gary.
>
> >> Would Commons Crypto focus only on AES? If so, Commons Crypto AES or
> Commons AES would be a better name.
> Currently, this module supports only AES modes. To help folks with
> information for making decision, a little further clarification from me may
> help.
>
> The project doesn't implement directly the cryptographic algorithms. It
> provides:
> 1.  It provides a thin layer of Cipher to abstract the under-layer Cipher
> implementations. (currently support JCE Cipher or OpenSSL Cipher
> implementations). This is for optimization purposes, for example OpenSSL
> Cipher implementation provides 17+x performance for AES/CTR comparing with
> JDK 6 and 5x comparing JDK 7/8.
> 2.  It provides a layer of stream and channel implementations abstracting
> Input source and Output target utilizing the Cipher layer. The layer can be
> used easily by applications that needs to encrypt/decrypt data streams or
> channels.
> 3.  Additionally, it provides a secure random utility classes to help
> generate TRUE random numbers for key generation.
>
> While there is no technical barrier for it to support other algorithms
> such as RC4 through JCE or OpenSSL. Just depends how widely this is
> required.
>
> Thanks,
> Haifeng
>
> -Original Message-
> From: Gary Gregory [mailto:garydgreg...@gmail.com]
> Sent: Monday, February 22, 2016 11:36 AM
> To: Commons Developers List 
> Subject: Re: [crypto][chimera] Next steps
>
> Would Commons Crypto focus only on AES? If so, Commons Crypto AES or
> Commons AES would be a better name.
>
> Gary
>
> On Sun, Feb 21, 2016 at 6:28 PM, Chen, Haifeng 
> wrote:
>
> > Thanks Benedikt, Uma, Gary, Ole, and Emmanuel Bourg for your support!
> > It's great to have Chimera to be part of Apache Commons.
> >
> > >>[ Emmanuel Bourg] Define the scope of the project? Do we go after
> > >>Bouncy
> > Castle and aim for an equivalent feature set?
> > Agree to make a clear scope of the project. The original Chimera scope
> > is not go after a Bouncy Castle style of library. It targets the gap
> > between the application and the under cipher implementations. For
> > example, applications uses a lot of InputStream/OutputStream or
> > Channel concepts to read / write a stream of data. Application can
> > share these Crypto streams/channel implementations abstracting various
> input and output types.
> > Chimera also targets to very important performance optimizations on
> > AES which is used worldwide. I suggest this module to be still
> > lightweight and clear in involving, which is the same ideology of Apache
> Commons.
> >
> > >> [Uma] Here I would like to introduce Haifeng, who lead the efforts
> > >> in
> > Chimera github project.
> > Thanks Uma for introduction.
> >
> > >> [Uma] Me and Haifeng had some discussion yesterday for the list to
> > >> get
> > commit prevs. May be he could probably get list. Then I think Commons
> > PMC can make a decision on it.
> > Chimera has 5 long standing contributors on github. We can also invite
> > those who contributed the HDFS encryption feature (HDFS-6134 and
> > HADOOP-10150) to continue participate the involution of this project
> > if they want.
> >
> > Thanks,
> > Haifeng
> >
> > -Original Message-
> > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
> > Sent: Sunday, February 21, 2016 3:07 AM
> > To: Commons Developers List 
> > Subject: Re: [crypto][chimera] Next steps
> >
> > Hi Benedikt,
> >
> >  Thank you for the Next steps discussion. I thought of pinging you on
> > this
> > :-)
> >
> >  Here I would like to introduce Haifeng, who lead the efforts in
> > Chimera github project.
> >
> > I think Apache Commons Crypto looks good and self descriptive IMO.
> >  I am +1
> >
> > Me and Haifeng had some discussion yesterday for the list to get
> > commit prevs. May be he could probably get list. Then I think Commons
> > PMC can make a decision on it.
> >
> >
> > >move code to an Apache repo (probably git?!)
> > +1 for git
> >
> > >- setup maven build
> > If this point is just about maven build alone, then we should set up
> > Jenkins CI build boat as well right, may be we can add this point?
> >
> > Regards,
> > Uma
> >
> > On 2/20/16, 8:29 AM, "Gary Gregory"  wrote:
> >
> > >Who are the committers comming along for this component?
> > >
> > >Do we have enough of them?
> > >
> > >I like Apache Commons Crypto.
> > >
> > >Gary
> > >On Feb 20, 2016 3:15 AM, "Benedikt Ritter"  wrote:
> > >
> > >> Hi,
> > >>
> > >> I'd like to discuss the next steps for moving the Chimera component
> > >>to  Apache Commons. So far, none of the other PMC members has
> > >>expressed his or  her thoughts about

Re: [crypto][chimera] Next steps

[Nick Burch]

On Sun, 21 Feb 2016, Gary Gregory wrote:

Curious: How to you get to OpenSSL, JNI? JNA?


I know that Tomcat has done quite a bit of work around pulling in OpenSSL 
in order to do SNI and ALPN. Mark Thomas gave a good talk on it at 
ApacheCon last year, slides are at:

http://events.linuxfoundation.org/sites/events/files/slides/2015-09-24-HTTP2%20and%20Java.pdf

No idea if that approach would be of use for this, but though I'd flag it 
up in case people didn't know about the work Tomcat have been doing in the 
same sort of area!


Nick


On Sun, Feb 21, 2016 at 10:51 PM, Chen, Haifeng 
wrote:


Thanks Gary.


Would Commons Crypto focus only on AES? If so, Commons Crypto AES or

Commons AES would be a better name.
Currently, this module supports only AES modes. To help folks with
information for making decision, a little further clarification from me may
help.

The project doesn't implement directly the cryptographic algorithms. It
provides:
1.  It provides a thin layer of Cipher to abstract the under-layer Cipher
implementations. (currently support JCE Cipher or OpenSSL Cipher
implementations). This is for optimization purposes, for example OpenSSL
Cipher implementation provides 17+x performance for AES/CTR comparing with
JDK 6 and 5x comparing JDK 7/8.
2.  It provides a layer of stream and channel implementations abstracting
Input source and Output target utilizing the Cipher layer. The layer can be
used easily by applications that needs to encrypt/decrypt data streams or
channels.
3.  Additionally, it provides a secure random utility classes to help
generate TRUE random numbers for key generation.

While there is no technical barrier for it to support other algorithms
such as RC4 through JCE or OpenSSL. Just depends how widely this is
required.

Thanks,
Haifeng

-Original Message-
From: Gary Gregory [mailto:garydgreg...@gmail.com]
Sent: Monday, February 22, 2016 11:36 AM
To: Commons Developers List 
Subject: Re: [crypto][chimera] Next steps

Would Commons Crypto focus only on AES? If so, Commons Crypto AES or
Commons AES would be a better name.

Gary

On Sun, Feb 21, 2016 at 6:28 PM, Chen, Haifeng 
wrote:


Thanks Benedikt, Uma, Gary, Ole, and Emmanuel Bourg for your support!
It's great to have Chimera to be part of Apache Commons.


[ Emmanuel Bourg] Define the scope of the project? Do we go after
Bouncy

Castle and aim for an equivalent feature set?
Agree to make a clear scope of the project. The original Chimera scope
is not go after a Bouncy Castle style of library. It targets the gap
between the application and the under cipher implementations. For
example, applications uses a lot of InputStream/OutputStream or
Channel concepts to read / write a stream of data. Application can
share these Crypto streams/channel implementations abstracting various

input and output types.

Chimera also targets to very important performance optimizations on
AES which is used worldwide. I suggest this module to be still
lightweight and clear in involving, which is the same ideology of Apache

Commons.



[Uma] Here I would like to introduce Haifeng, who lead the efforts
in

Chimera github project.
Thanks Uma for introduction.


[Uma] Me and Haifeng had some discussion yesterday for the list to
get

commit prevs. May be he could probably get list. Then I think Commons
PMC can make a decision on it.
Chimera has 5 long standing contributors on github. We can also invite
those who contributed the HDFS encryption feature (HDFS-6134 and
HADOOP-10150) to continue participate the involution of this project
if they want.

Thanks,
Haifeng

-Original Message-
From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com]
Sent: Sunday, February 21, 2016 3:07 AM
To: Commons Developers List 
Subject: Re: [crypto][chimera] Next steps

Hi Benedikt,

 Thank you for the Next steps discussion. I thought of pinging you on
this
:-)

 Here I would like to introduce Haifeng, who lead the efforts in
Chimera github project.

I think Apache Commons Crypto looks good and self descriptive IMO.
 I am +1

Me and Haifeng had some discussion yesterday for the list to get
commit prevs. May be he could probably get list. Then I think Commons
PMC can make a decision on it.



move code to an Apache repo (probably git?!)

+1 for git


- setup maven build

If this point is just about maven build alone, then we should set up
Jenkins CI build boat as well right, may be we can add this point?

Regards,
Uma

On 2/20/16, 8:29 AM, "Gary Gregory"  wrote:


Who are the committers comming along for this component?

Do we have enough of them?

I like Apache Commons Crypto.

Gary
On Feb 20, 2016 3:15 AM, "Benedikt Ritter"  wrote:


Hi,

I'd like to discuss the next steps for moving the Chimera component
to  Apache Commons. So far, none of the other PMC members has
expressed his or  her thoughts about this. If nobody brings up
objections about moving the  component to Apache Commons, I'm
assuming lazy consensus about this.

So the next steps would be:
- dec

Re: [crypto][chimera] Next steps

[Colin P. McCabe]

I would highly recommend shading this library when it is used in
Hadoop and/or Spark, to prevent version skew problems between Hadoop
and Spark like we have had in the past.

What is the strategy for handling JNI components?  I think at a
minimum, we should include the version number in the native library
name to avoid problems when deploying multiple versions of Chimera.
This is something that has been problematic in Hadoop with
libhadoop.so.

Is this library going to have Scala interfaces as well as Java ones,
or just Java?

cheers,
Colin

On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter  wrote:
> Hi,
>
> I'd like to discuss the next steps for moving the Chimera component to
> Apache Commons. So far, none of the other PMC members has expressed his or
> her thoughts about this. If nobody brings up objections about moving the
> component to Apache Commons, I'm assuming lazy consensus about this.
>
> So the next steps would be:
> - decide on a name for the new component (my proposal was Apache Commons
> Crypto)
> - move code to an Apache repo (probably git?!)
> - request a Jira project
> - setup maven build
> - setup project website
> - work on an initial release under Apache Commons coordinates
>
> Anything missing?
>
> Regards,
> Benedikt
>
> --
> http://home.apache.org/~britter/
> http://twitter.com/BenediktRitter
> http://github.com/britter

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Jochen Wiedmann]

On Mon, Feb 22, 2016 at 6:28 PM, Colin P. McCabe  wrote:

> What is the strategy for handling JNI components?

Wrong question, IMO. Should better be: What are the reasons for using
JNI components? Couldn't they be replaced? If so, that would very much
enhance the long term prospects of crypto|chimera|whatever.

Jochen

-- 
The next time you hear: "Don't reinvent the wheel!"

http://www.keystonedevelopment.co.uk/wp-content/uploads/2014/10/evolution-of-the-wheel-300x85.jpg

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Gangumalla, Uma]

>I would highly recommend shading this library when it is used in
Hadoop and/or Spark, to prevent version skew problems between Hadoop
and Spark like we have had in the past.
[uma]Ha. This avoids multiple jars versions issues. Agreed IMO.

>I think at a
minimum, we should include the version number in the native library
name to avoid problems when deploying multiple versions of Chimera.
This is something that has been problematic in Hadoop with
libhadoop.so.
[uma]I think this is very valid suggestion. We can maintain version number
with native lib. Also here target is to bundle libchimera.so along with
jars. Ideally it should be less confusion, but its good idea to have
version number along.

>Is this library going to have Scala interfaces as well as Java ones,
or just Java?
[uma] Currently it is focussing on java. If there is a demand for Scala
specifically may be we can think on that.

Regards,
Uma

On 2/22/16, 9:28 AM, "Colin P. McCabe"  wrote:

>I would highly recommend shading this library when it is used in
>Hadoop and/or Spark, to prevent version skew problems between Hadoop
>and Spark like we have had in the past.
>
>What is the strategy for handling JNI components?  I think at a
>minimum, we should include the version number in the native library
>name to avoid problems when deploying multiple versions of Chimera.
>This is something that has been problematic in Hadoop with
>libhadoop.so.
>
>Is this library going to have Scala interfaces as well as Java ones,
>or just Java?
>
>cheers,
>Colin
>
>On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter 
>wrote:
>> Hi,
>>
>> I'd like to discuss the next steps for moving the Chimera component to
>> Apache Commons. So far, none of the other PMC members has expressed his
>>or
>> her thoughts about this. If nobody brings up objections about moving the
>> component to Apache Commons, I'm assuming lazy consensus about this.
>>
>> So the next steps would be:
>> - decide on a name for the new component (my proposal was Apache Commons
>> Crypto)
>> - move code to an Apache repo (probably git?!)
>> - request a Jira project
>> - setup maven build
>> - setup project website
>> - work on an initial release under Apache Commons coordinates
>>
>> Anything missing?
>>
>> Regards,
>> Benedikt
>>
>> --
>> http://home.apache.org/~britter/
>> http://twitter.com/BenediktRitter
>> http://github.com/britter


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Gary Gregory]

All files should follow the Commons Maven naming scheme to make it easy to
reach from Maven, Ivy and so on.

This will be commons-crypto-1.0.jar for example.

Gary

On Mon, Feb 22, 2016 at 1:06 PM, Gangumalla, Uma 
wrote:

> >I would highly recommend shading this library when it is used in
> Hadoop and/or Spark, to prevent version skew problems between Hadoop
> and Spark like we have had in the past.
> [uma]Ha. This avoids multiple jars versions issues. Agreed IMO.
>
> >I think at a
> minimum, we should include the version number in the native library
> name to avoid problems when deploying multiple versions of Chimera.
> This is something that has been problematic in Hadoop with
> libhadoop.so.
> [uma]I think this is very valid suggestion. We can maintain version number
> with native lib. Also here target is to bundle libchimera.so along with
> jars. Ideally it should be less confusion, but its good idea to have
> version number along.
>
> >Is this library going to have Scala interfaces as well as Java ones,
> or just Java?
> [uma] Currently it is focussing on java. If there is a demand for Scala
> specifically may be we can think on that.
>
> Regards,
> Uma
>
> On 2/22/16, 9:28 AM, "Colin P. McCabe"  wrote:
>
> >I would highly recommend shading this library when it is used in
> >Hadoop and/or Spark, to prevent version skew problems between Hadoop
> >and Spark like we have had in the past.
> >
> >What is the strategy for handling JNI components?  I think at a
> >minimum, we should include the version number in the native library
> >name to avoid problems when deploying multiple versions of Chimera.
> >This is something that has been problematic in Hadoop with
> >libhadoop.so.
> >
> >Is this library going to have Scala interfaces as well as Java ones,
> >or just Java?
> >
> >cheers,
> >Colin
> >
> >On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter 
> >wrote:
> >> Hi,
> >>
> >> I'd like to discuss the next steps for moving the Chimera component to
> >> Apache Commons. So far, none of the other PMC members has expressed his
> >>or
> >> her thoughts about this. If nobody brings up objections about moving the
> >> component to Apache Commons, I'm assuming lazy consensus about this.
> >>
> >> So the next steps would be:
> >> - decide on a name for the new component (my proposal was Apache Commons
> >> Crypto)
> >> - move code to an Apache repo (probably git?!)
> >> - request a Jira project
> >> - setup maven build
> >> - setup project website
> >> - work on an initial release under Apache Commons coordinates
> >>
> >> Anything missing?
> >>
> >> Regards,
> >> Benedikt
> >>
> >> --
> >> http://home.apache.org/~britter/
> >> http://twitter.com/BenediktRitter
> >> http://github.com/britter
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: [crypto][chimera] Next steps

[Gangumalla, Uma]

>All files should follow the Commons Maven naming scheme to make it easy to
>reach from Maven, Ivy and so on.
>This will be commons-crypto-1.0.jar for example.
Sure. Thanks Gary. We will follow the naming convention here from Commons.

Regards,
Uma

On 2/22/16, 1:20 PM, "Gary Gregory"  wrote:

>All files should follow the Commons Maven naming scheme to make it easy to
>reach from Maven, Ivy and so on.
>
>This will be commons-crypto-1.0.jar for example.
>
>Gary
>
>On Mon, Feb 22, 2016 at 1:06 PM, Gangumalla, Uma
>
>wrote:
>
>> >I would highly recommend shading this library when it is used in
>> Hadoop and/or Spark, to prevent version skew problems between Hadoop
>> and Spark like we have had in the past.
>> [uma]Ha. This avoids multiple jars versions issues. Agreed IMO.
>>
>> >I think at a
>> minimum, we should include the version number in the native library
>> name to avoid problems when deploying multiple versions of Chimera.
>> This is something that has been problematic in Hadoop with
>> libhadoop.so.
>> [uma]I think this is very valid suggestion. We can maintain version
>>number
>> with native lib. Also here target is to bundle libchimera.so along with
>> jars. Ideally it should be less confusion, but its good idea to have
>> version number along.
>>
>> >Is this library going to have Scala interfaces as well as Java ones,
>> or just Java?
>> [uma] Currently it is focussing on java. If there is a demand for Scala
>> specifically may be we can think on that.
>>
>> Regards,
>> Uma
>>
>> On 2/22/16, 9:28 AM, "Colin P. McCabe"  wrote:
>>
>> >I would highly recommend shading this library when it is used in
>> >Hadoop and/or Spark, to prevent version skew problems between Hadoop
>> >and Spark like we have had in the past.
>> >
>> >What is the strategy for handling JNI components?  I think at a
>> >minimum, we should include the version number in the native library
>> >name to avoid problems when deploying multiple versions of Chimera.
>> >This is something that has been problematic in Hadoop with
>> >libhadoop.so.
>> >
>> >Is this library going to have Scala interfaces as well as Java ones,
>> >or just Java?
>> >
>> >cheers,
>> >Colin
>> >
>> >On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter 
>> >wrote:
>> >> Hi,
>> >>
>> >> I'd like to discuss the next steps for moving the Chimera component
>>to
>> >> Apache Commons. So far, none of the other PMC members has expressed
>>his
>> >>or
>> >> her thoughts about this. If nobody brings up objections about moving
>>the
>> >> component to Apache Commons, I'm assuming lazy consensus about this.
>> >>
>> >> So the next steps would be:
>> >> - decide on a name for the new component (my proposal was Apache
>>Commons
>> >> Crypto)
>> >> - move code to an Apache repo (probably git?!)
>> >> - request a Jira project
>> >> - setup maven build
>> >> - setup project website
>> >> - work on an initial release under Apache Commons coordinates
>> >>
>> >> Anything missing?
>> >>
>> >> Regards,
>> >> Benedikt
>> >>
>> >> --
>> >> http://home.apache.org/~britter/
>> >> http://twitter.com/BenediktRitter
>> >> http://github.com/britter
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
>
>
>-- 
>E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>Java Persistence with Hibernate, Second Edition
>
>JUnit in Action, Second Edition 
>Spring Batch in Action 
>Blog: http://garygregory.wordpress.com
>Home: http://garygregory.com/
>Tweet! http://twitter.com/GaryGregory


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Gary Gregory]

On Mon, Feb 22, 2016 at 1:26 PM, Gangumalla, Uma 
wrote:

>
> >All files should follow the Commons Maven naming scheme to make it easy to
> >reach from Maven, Ivy and so on.
> >This will be commons-crypto-1.0.jar for example.
> Sure. Thanks Gary. We will follow the naming convention here from Commons.
>

For jar files, this will happen automatically if you follow the Commons
Maven conventions. For .dll and .so files, you/we'll have to adjust the
make files, unless there is a smarter way.

Gary


> Regards,
> Uma
>
> On 2/22/16, 1:20 PM, "Gary Gregory"  wrote:
>
> >All files should follow the Commons Maven naming scheme to make it easy to
> >reach from Maven, Ivy and so on.
> >
> >This will be commons-crypto-1.0.jar for example.
> >
> >Gary
> >
> >On Mon, Feb 22, 2016 at 1:06 PM, Gangumalla, Uma
> >
> >wrote:
> >
> >> >I would highly recommend shading this library when it is used in
> >> Hadoop and/or Spark, to prevent version skew problems between Hadoop
> >> and Spark like we have had in the past.
> >> [uma]Ha. This avoids multiple jars versions issues. Agreed IMO.
> >>
> >> >I think at a
> >> minimum, we should include the version number in the native library
> >> name to avoid problems when deploying multiple versions of Chimera.
> >> This is something that has been problematic in Hadoop with
> >> libhadoop.so.
> >> [uma]I think this is very valid suggestion. We can maintain version
> >>number
> >> with native lib. Also here target is to bundle libchimera.so along with
> >> jars. Ideally it should be less confusion, but its good idea to have
> >> version number along.
> >>
> >> >Is this library going to have Scala interfaces as well as Java ones,
> >> or just Java?
> >> [uma] Currently it is focussing on java. If there is a demand for Scala
> >> specifically may be we can think on that.
> >>
> >> Regards,
> >> Uma
> >>
> >> On 2/22/16, 9:28 AM, "Colin P. McCabe"  wrote:
> >>
> >> >I would highly recommend shading this library when it is used in
> >> >Hadoop and/or Spark, to prevent version skew problems between Hadoop
> >> >and Spark like we have had in the past.
> >> >
> >> >What is the strategy for handling JNI components?  I think at a
> >> >minimum, we should include the version number in the native library
> >> >name to avoid problems when deploying multiple versions of Chimera.
> >> >This is something that has been problematic in Hadoop with
> >> >libhadoop.so.
> >> >
> >> >Is this library going to have Scala interfaces as well as Java ones,
> >> >or just Java?
> >> >
> >> >cheers,
> >> >Colin
> >> >
> >> >On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter 
> >> >wrote:
> >> >> Hi,
> >> >>
> >> >> I'd like to discuss the next steps for moving the Chimera component
> >>to
> >> >> Apache Commons. So far, none of the other PMC members has expressed
> >>his
> >> >>or
> >> >> her thoughts about this. If nobody brings up objections about moving
> >>the
> >> >> component to Apache Commons, I'm assuming lazy consensus about this.
> >> >>
> >> >> So the next steps would be:
> >> >> - decide on a name for the new component (my proposal was Apache
> >>Commons
> >> >> Crypto)
> >> >> - move code to an Apache repo (probably git?!)
> >> >> - request a Jira project
> >> >> - setup maven build
> >> >> - setup project website
> >> >> - work on an initial release under Apache Commons coordinates
> >> >>
> >> >> Anything missing?
> >> >>
> >> >> Regards,
> >> >> Benedikt
> >> >>
> >> >> --
> >> >> http://home.apache.org/~britter/
> >> >> http://twitter.com/BenediktRitter
> >> >> http://github.com/britter
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> >> For additional commands, e-mail: dev-h...@commons.apache.org
> >>
> >>
> >
> >
> >--
> >E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> >Java Persistence with Hibernate, Second Edition
> >
> >JUnit in Action, Second Edition 
> >Spring Batch in Action 
> >Blog: http://garygregory.wordpress.com
> >Home: http://garygregory.com/
> >Tweet! http://twitter.com/GaryGregory
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

commons-beanutils deserialization gadget

[Chris Frohoff]

All,
I already sent something similar to the private security list 
(secur...@apache.org) earlier this month and it was suggested that I post it to 
the dev list for discussion.
There is a Java deserialization "gadget" in the commons-beanutils library that 
can be used along with others in the JRE to achieve code/command execution in 
applications deserializing untrusted data, similar to the ones recently 
discussed in the commons-collections library. Specifically, an instance of the 
commons-beanutils class `BeanComparator` can be used during deserialization of 
a containing collection to invoke the `getOutputProperties()` getter method on 
a TemplatesImpl instance, defining arbitrary and potentially malicious classes 
(very similar to the CommonsCollections2 gadget chain). A proof-of-concept 
payload generator is available in this gist: 
https://gist.github.com/frohoff/9eb8811761ff989b3ac0 
To be clear, I don't believe that this should necessarily be a concern of 
library developers and that protections should really be implemented by code 
doing the unsafe deserialization. This is in line with the previous Apache 
Commons statement regarding the vunlerabilities discourse from last year 
(https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread).
 That being said, some people seem to disagree, and there seems to be a desire 
for libraries to provide mitigations, so a new release with a patch may be 
warranted.
I discovered this last year, but for better or worse, wanted to wait until I 
could publicly disclose a similar issue with only JRE (<7u21) classes 
(https://gist.github.com/frohoff/24af7913611f8406eaf3) to further emphasize 
that this is a more pervasive problem that should be addressed in a more robust 
way than playing "whack-a-mole" with gadget classes 
(seehttps://gist.github.com/frohoff/24af7913611f8406eaf3#mitigation for 
recommendations).
I'm happy answer questions, review code/patches, and otherwise help in any way 
I can.
Regards,
-Chris Frohoff

Further references:
Beanutils gadget chain: https://gist.github.com/frohoff/9eb8811761ff989b3ac0 
AppSecCali Marshalling Pickles Talk: 
http://www.slideshare.net/frohoff1/appseccali-2015-marshalling-picklesysoserial 
payload generator: https://github.com/frohoff/ysoserialJRE 7u21 Advisory: 
https://gist.github.com/frohoff/24af7913611f8406eaf3Apache Commons statement: 
https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread

Re: commons-beanutils deserialization gadget

[Chris Frohoff]

Re-sending the references since the formatting and links seemed to have gotten 
a bit messed up.

Further references:

Beanutils gadget chain: 
https://gist.github.com/frohoff/9eb8811761ff989b3ac0 
 
AppSecCali Marshalling Pickles Talk: 
http://www.slideshare.net/frohoff1/appseccali-2015-marshalling-pickles 

ysoserial payload generator: 
https://github.com/frohoff/ysoserial 

JRE 7u21 Advisory: 
https://gist.github.com/frohoff/24af7913611f8406eaf3 

Apache Commons statement: 
https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread


On Monday, February 22, 2016 1:59 PM, Chris Frohoff  wrote:



All,

I already sent something similar to the private security list 
(secur...@apache.org) earlier this month and it was suggested that I post it to 
the dev list for discussion.

There is a Java deserialization "gadget" in the commons-beanutils library that 
can be used along with others in the JRE to achieve code/command execution in 
applications deserializing untrusted data, similar to the ones recently 
discussed in the commons-collections library. Specifically, an instance of the 
commons-beanutils class `BeanComparator` can be used during deserialization of 
a containing collection to invoke the `getOutputProperties()` getter method on 
a TemplatesImpl instance, defining arbitrary and potentially malicious classes 
(very similar to the CommonsCollections2 gadget chain). A proof-of-concept 
payload generator is available in this gist: https://gist.github.com/ 
frohoff/9eb8811761ff989b3ac0 

To be clear, I don't believe that this should necessarily be a concern of 
library developers and that protections should really be implemented by code 
doing the unsafe deserialization. This is in line with the previous Apache 
Commons statement regarding the vunlerabilities discourse from last year 
(https://blogs.apache.org/ foundation/entry/apache_ commons_statement_to_ 
widespread). That being said, some people seem to disagree, and there seems to 
be a desire for libraries to provide mitigations, so a new release with a patch 
may be warranted.

I discovered this last year, but for better or worse, wanted to wait until I 
could publicly disclose a similar issue with only JRE (<7u21) classes 
(https://gist.github.com/ frohoff/24af7913611f8406eaf3) to further emphasize 
that this is a more pervasive problem that should be addressed in a more robust 
way than playing "whack-a-mole" with gadget classes 
(seehttps://gist.github.com/ frohoff/24af7913611f8406eaf3# mitigation for 
recommendations).

I'm happy answer questions, review code/patches, and otherwise help in any way 
I can.

Regards,

-Chris Frohoff


Further references:

Beanutils gadget chain: https://gist.github. com/frohoff/ 9eb8811761ff989b3ac0 
AppSecCali Marshalling Pickles Talk: http://www.slideshare.net/ 
frohoff1/appseccali-2015- marshalling-pickles
ysoserial payload generator: https://github.com/ frohoff/ysoserial
JRE 7u21 Advisory: https://gist.github.com/ frohoff/24af7913611f8406eaf3
Apache Commons statement: https://blogs. apache.org/foundation/entry/ 
apache_commons_statement_to_ widespread

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Colin P. McCabe]

Hi Jochen,

Many CPUs come with built-in support for certain cryptographic and/or
hash/checksum-related primitives.  For example, modern x86 CPUs have
CRC32C implemented in hardware.  Currently, this must be accessed via
inline assembly expressed in JNI.  It is worth it... at least in the
case of checksumming, you often see 5x or 10x reductions in the amount
of CPU used.  The gains for moving from pure Java to using the openSSL
AES functions are similar.  Perhaps someday Java will gain native
support for these features.  Until that point, though, JNI will be
necessary to get reasonable performance on modern hardware.

best,
Colin

On Mon, Feb 22, 2016 at 10:02 AM, Jochen Wiedmann
 wrote:
> On Mon, Feb 22, 2016 at 6:28 PM, Colin P. McCabe  wrote:
>
>> What is the strategy for handling JNI components?
>
> Wrong question, IMO. Should better be: What are the reasons for using
> JNI components? Couldn't they be replaced? If so, that would very much
> enhance the long term prospects of crypto|chimera|whatever.
>
> Jochen
>
> --
> The next time you hear: "Don't reinvent the wheel!"
>
> http://www.keystonedevelopment.co.uk/wp-content/uploads/2014/10/evolution-of-the-wheel-300x85.jpg

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Gary Gregory]

Checksum via JNI should be done in the commons-codec project.

Gary
On Feb 22, 2016 3:14 PM, "Colin P. McCabe"  wrote:

> Hi Jochen,
>
> Many CPUs come with built-in support for certain cryptographic and/or
> hash/checksum-related primitives.  For example, modern x86 CPUs have
> CRC32C implemented in hardware.  Currently, this must be accessed via
> inline assembly expressed in JNI.  It is worth it... at least in the
> case of checksumming, you often see 5x or 10x reductions in the amount
> of CPU used.  The gains for moving from pure Java to using the openSSL
> AES functions are similar.  Perhaps someday Java will gain native
> support for these features.  Until that point, though, JNI will be
> necessary to get reasonable performance on modern hardware.
>
> best,
> Colin
>
> On Mon, Feb 22, 2016 at 10:02 AM, Jochen Wiedmann
>  wrote:
> > On Mon, Feb 22, 2016 at 6:28 PM, Colin P. McCabe 
> wrote:
> >
> >> What is the strategy for handling JNI components?
> >
> > Wrong question, IMO. Should better be: What are the reasons for using
> > JNI components? Couldn't they be replaced? If so, that would very much
> > enhance the long term prospects of crypto|chimera|whatever.
> >
> > Jochen
> >
> > --
> > The next time you hear: "Don't reinvent the wheel!"
> >
> >
> http://www.keystonedevelopment.co.uk/wp-content/uploads/2014/10/evolution-of-the-wheel-300x85.jpg
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [crypto][chimera] Next steps

[James Carman]

Not a big fan of introducing JNI-based library to Commons. I'm -0

On Sat, Feb 20, 2016 at 6:15 AM Benedikt Ritter  wrote:

> Hi,
>
> I'd like to discuss the next steps for moving the Chimera component to
> Apache Commons. So far, none of the other PMC members has expressed his or
> her thoughts about this. If nobody brings up objections about moving the
> component to Apache Commons, I'm assuming lazy consensus about this.
>
> So the next steps would be:
> - decide on a name for the new component (my proposal was Apache Commons
> Crypto)
> - move code to an Apache repo (probably git?!)
> - request a Jira project
> - setup maven build
> - setup project website
> - work on an initial release under Apache Commons coordinates
>
> Anything missing?
>
> Regards,
> Benedikt
>
> --
> http://home.apache.org/~britter/
> http://twitter.com/BenediktRitter
> http://github.com/britter
>

Re: [crypto][chimera] Next steps

[Gary Gregory]

We already have commons-daemon that has C bits.

Gary
On Feb 22, 2016 3:27 PM, "James Carman"  wrote:

> Not a big fan of introducing JNI-based library to Commons. I'm -0
>
> On Sat, Feb 20, 2016 at 6:15 AM Benedikt Ritter 
> wrote:
>
> > Hi,
> >
> > I'd like to discuss the next steps for moving the Chimera component to
> > Apache Commons. So far, none of the other PMC members has expressed his
> or
> > her thoughts about this. If nobody brings up objections about moving the
> > component to Apache Commons, I'm assuming lazy consensus about this.
> >
> > So the next steps would be:
> > - decide on a name for the new component (my proposal was Apache Commons
> > Crypto)
> > - move code to an Apache repo (probably git?!)
> > - request a Jira project
> > - setup maven build
> > - setup project website
> > - work on an initial release under Apache Commons coordinates
> >
> > Anything missing?
> >
> > Regards,
> > Benedikt
> >
> > --
> > http://home.apache.org/~britter/
> > http://twitter.com/BenediktRitter
> > http://github.com/britter
> >
>

Re: [crypto][chimera] Next steps

[James Carman]

Still not a fan. My vote stands
On Mon, Feb 22, 2016 at 6:37 PM Gary Gregory  wrote:

> We already have commons-daemon that has C bits.
>
> Gary
> On Feb 22, 2016 3:27 PM, "James Carman" 
> wrote:
>
> > Not a big fan of introducing JNI-based library to Commons. I'm -0
> >
> > On Sat, Feb 20, 2016 at 6:15 AM Benedikt Ritter 
> > wrote:
> >
> > > Hi,
> > >
> > > I'd like to discuss the next steps for moving the Chimera component to
> > > Apache Commons. So far, none of the other PMC members has expressed his
> > or
> > > her thoughts about this. If nobody brings up objections about moving
> the
> > > component to Apache Commons, I'm assuming lazy consensus about this.
> > >
> > > So the next steps would be:
> > > - decide on a name for the new component (my proposal was Apache
> Commons
> > > Crypto)
> > > - move code to an Apache repo (probably git?!)
> > > - request a Jira project
> > > - setup maven build
> > > - setup project website
> > > - work on an initial release under Apache Commons coordinates
> > >
> > > Anything missing?
> > >
> > > Regards,
> > > Benedikt
> > >
> > > --
> > > http://home.apache.org/~britter/
> > > http://twitter.com/BenediktRitter
> > > http://github.com/britter
> > >
> >
>

commons-beanutils deserialization gadget

[Chris Frohoff]

All,

I already sent something similar to the private security list (
secur...@apache.org) earlier this month and it was suggested that I post it
to the dev list for discussion.

There is a Java deserialization "gadget" in the commons-beanutils library
that can be used along with others in the JRE to achieve code/command
execution in applications deserializing untrusted data, similar to the ones
recently discussed in the commons-collections library. Specifically, an
instance of the commons-beanutils class `BeanComparator` can be used during
deserialization of a containing collection to invoke the
`getOutputProperties()` getter method on a TemplatesImpl instance, defining
arbitrary and potentially malicious classes (very similar to the
CommonsCollections2 gadget chain). A proof-of-concept payload generator is
available in this gist: https://gist.github.com/frohoff/9eb8811761ff989b3ac0


To be clear, I don't believe that this should necessarily be a concern of
library developers and that protections should really be implemented by
code doing the unsafe deserialization. This is in line with the previous
Apache Commons statement regarding the vunlerabilities discourse from last
year (
https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread).
That being said, some people seem to disagree, and there seems to be a
desire for libraries to provide mitigations, so a new release with a patch
may be warranted.

I discovered this last year, but for better or worse, wanted to wait until
I could publicly disclose a similar issue with only JRE (<7u21) classes (
https://gist.github.com/frohoff/24af7913611f8406eaf3) to further emphasize
that this is a more pervasive problem that should be addressed in a more
robust way than playing "whack-a-mole" with gadget classes (see
https://gist.github.com/frohoff/24af7913611f8406eaf3#mitigation for
recommendations).

I'm happy answer questions, review code/patches, and otherwise help in any
way I can.

Regards,

-Chris Frohoff


Further references:

Beanutils gadget chain: https://gist.github.com/frohoff/9eb8811761ff989b3ac0

AppSecCali Marshalling Pickles Talk:
http://www.slideshare.net/frohoff1/appseccali-2015-marshalling-pickles
ysoserial payload generator: https://github.com/frohoff/ysoserial
JRE 7u21 Advisory: https://gist.github.com/frohoff/24af7913611f8406eaf3
Apache Commons statement:
https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread

Re: [crypto][chimera] Next steps

[sebb]

On 22 February 2016 at 23:43, James Carman  wrote:
> Still not a fan. My vote stands

I'm inclined to agree with James here.

> On Mon, Feb 22, 2016 at 6:37 PM Gary Gregory  wrote:
>
>> We already have commons-daemon that has C bits.

Have you tried building it?

>>
>> Gary
>> On Feb 22, 2016 3:27 PM, "James Carman" 
>> wrote:
>>
>> > Not a big fan of introducing JNI-based library to Commons. I'm -0
>> >
>> > On Sat, Feb 20, 2016 at 6:15 AM Benedikt Ritter 
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I'd like to discuss the next steps for moving the Chimera component to
>> > > Apache Commons. So far, none of the other PMC members has expressed his
>> > or
>> > > her thoughts about this. If nobody brings up objections about moving
>> the
>> > > component to Apache Commons, I'm assuming lazy consensus about this.
>> > >
>> > > So the next steps would be:
>> > > - decide on a name for the new component (my proposal was Apache
>> Commons
>> > > Crypto)
>> > > - move code to an Apache repo (probably git?!)
>> > > - request a Jira project
>> > > - setup maven build
>> > > - setup project website
>> > > - work on an initial release under Apache Commons coordinates
>> > >
>> > > Anything missing?
>> > >
>> > > Regards,
>> > > Benedikt
>> > >
>> > > --
>> > > http://home.apache.org/~britter/
>> > > http://twitter.com/BenediktRitter
>> > > http://github.com/britter
>> > >
>> >
>>

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

RE: [crypto][chimera] Next steps

[Xu, Cheng A]

At this point, it has just Java interfaces only.

-Original Message-
From: Colin P. McCabe [mailto:cmcc...@apache.org] 
Sent: Tuesday, February 23, 2016 1:29 AM
To: Hadoop Common
Cc: Commons Developers List
Subject: Re: [crypto][chimera] Next steps

I would highly recommend shading this library when it is used in
Hadoop and/or Spark, to prevent version skew problems between Hadoop
and Spark like we have had in the past.

What is the strategy for handling JNI components?  I think at a
minimum, we should include the version number in the native library
name to avoid problems when deploying multiple versions of Chimera.
This is something that has been problematic in Hadoop with
libhadoop.so.

Is this library going to have Scala interfaces as well as Java ones,
or just Java?

cheers,
Colin

On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter  wrote:
> Hi,
>
> I'd like to discuss the next steps for moving the Chimera component to
> Apache Commons. So far, none of the other PMC members has expressed his or
> her thoughts about this. If nobody brings up objections about moving the
> component to Apache Commons, I'm assuming lazy consensus about this.
>
> So the next steps would be:
> - decide on a name for the new component (my proposal was Apache Commons
> Crypto)
> - move code to an Apache repo (probably git?!)
> - request a Jira project
> - setup maven build
> - setup project website
> - work on an initial release under Apache Commons coordinates
>
> Anything missing?
>
> Regards,
> Benedikt
>
> --
> http://home.apache.org/~britter/
> http://twitter.com/BenediktRitter
> http://github.com/britter

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [crypto][chimera] Next steps

[Benedikt Ritter]

I'm confused. None of the other PMC members has expressed whether he or she
want's the see Chimera/crypto joining Apache Commons, yet we're already
discussing how JNI bindings should be handled.

I'd like to see:
1) a clear statement whether Chimera/crypto should become part of Apache
Commons. Do we need a vote for that?
2) Discuss a plan on how to do that (I've described a possible plan [1])
3) After that is clear: discuss design details regarding the component.

Thanks! :-)
Benedikt

[1] http://markmail.org/message/74j4el6bpfpt4evs

2016-02-23 3:03 GMT+01:00 Xu, Cheng A :

> At this point, it has just Java interfaces only.
>
> -Original Message-
> From: Colin P. McCabe [mailto:cmcc...@apache.org]
> Sent: Tuesday, February 23, 2016 1:29 AM
> To: Hadoop Common
> Cc: Commons Developers List
> Subject: Re: [crypto][chimera] Next steps
>
> I would highly recommend shading this library when it is used in
> Hadoop and/or Spark, to prevent version skew problems between Hadoop
> and Spark like we have had in the past.
>
> What is the strategy for handling JNI components?  I think at a
> minimum, we should include the version number in the native library
> name to avoid problems when deploying multiple versions of Chimera.
> This is something that has been problematic in Hadoop with
> libhadoop.so.
>
> Is this library going to have Scala interfaces as well as Java ones,
> or just Java?
>
> cheers,
> Colin
>
> On Sat, Feb 20, 2016 at 3:15 AM, Benedikt Ritter 
> wrote:
> > Hi,
> >
> > I'd like to discuss the next steps for moving the Chimera component to
> > Apache Commons. So far, none of the other PMC members has expressed his
> or
> > her thoughts about this. If nobody brings up objections about moving the
> > component to Apache Commons, I'm assuming lazy consensus about this.
> >
> > So the next steps would be:
> > - decide on a name for the new component (my proposal was Apache Commons
> > Crypto)
> > - move code to an Apache repo (probably git?!)
> > - request a Jira project
> > - setup maven build
> > - setup project website
> > - work on an initial release under Apache Commons coordinates
> >
> > Anything missing?
> >
> > Regards,
> > Benedikt
> >
> > --
> > http://home.apache.org/~britter/
> > http://twitter.com/BenediktRitter
> > http://github.com/britter
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>


-- 
http://home.apache.org/~britter/
http://twitter.com/BenediktRitter
http://github.com/britter