Dave,
Il Mer 23 Dic 2020, 19:58 Dave Fisher <w...@apache.org> ha scritto:
>
>
> > On Dec 23, 2020, at 10:55 AM, Matteo Merli <matteo.me...@gmail.com>
> wrote:
> >
> > Hi Dave,
> >
> > these are not dependencies that are needed (or would be pulled) from
> > the Pulsar build.
> >
> > The JMH is only used if someone wants to run a benchmark on the
> > LightProto code. Pulsar does not depend on that.
>
> Excellent. You’ll need to note that for developers if benchmarking is
> exposed in Pulsar.
>
JMH is only for micro benchmarks, this is something users don't use, only
Pulsar (Splunk) developers
Matteo,
Very nice work!
Looking forward to see this land into community code
I guess we cannot leverage it on BK as we are with Protobuf 3.x
Enrico
> >
> > From Pulsar build we would need:
> > 1. At build time -> The LightProto Maven Plugin with the generator module
> > 2. At runtime (and in Apache release) -> No dependencies
> >
> >
> > Matteo
> >
> >
> >
> > --
> > Matteo Merli
> > <matteo.me...@gmail.com>
> >
> > On Wed, Dec 23, 2020 at 10:20 AM Dave Fisher <w...@apache.org> wrote:
> >>
> >> Hi Matteo,
> >>
> >> I looked at the dependencies in Splunk LightPro and the following are
> not acceptable in an Apache Binary:
> >>
> >> <dependency>
> >> <groupId>org.openjdk.jmh</groupId>
> >> <artifactId>jmh-core</artifactId>
> >> <version>${jmh.version}</version>
> >> </dependency>
> >> <dependency>
> >> <groupId>org.openjdk.jmh</groupId>
> >> <artifactId>jmh-generator-annprocess</artifactId>
> >> <version>${jmh.version}</version>
> >> <scope>provided</scope>
> >> </dependency>
> >>
> >> These are GPL according to mvnrepository.com.
> >>
> >> The other dependencies are fine and all class A or B.
> >>
> >> Have a look at https://www.apache.org/legal/resolved.html
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Dec 23, 2020, at 9:35 AM, Matteo Merli <mme...@apache.org> wrote:
> >>>
> >>> ## Motivation
> >>>
> >>> In the Pulsar wire protocol, we are using Google Protobuf in order to
> perform
> >>> serialization/deserialization of the commands that are exchanged
> between
> >>> clients and brokers.
> >>>
> >>> Because of the overhead involved with the regular Protobuf
> implementation, since
> >>> very early on, we have been using a modified version of Protobuf 2.4.1.
> >>> The modifications were done to ensure a more efficient serialization
> code that
> >>> used thread local caches for the objects used in the process.
> >>>
> >>> There are few issues with the current approach:
> >>> 1. The patch to the Protobuf code generator is only based on version
> 2.4.1 and
> >>> cannot be upgraded to newer Protobuf versions
> >>> 2. The new Protobuf version, 3.xx, do have the same performance issues
> as the
> >>> 2.x versions.
> >>> 3. The thread-local approach for reusing objects is not ideal.
> Thread-local
> >>> access is not free and it would be better to instead cache the root
> objects
> >>> only.
> >>>
> >>> ## Goal
> >>>
> >>> Have an efficient and maintainable way to perform
> serialization/deserialization
> >>> of Pulsar protocol.
> >>>
> >>> The current proposal is to switch from the patched Protobuf 2.4.1 and
> use a
> >>> different code generator, Splunk LightProto:
> >>> https://github.com/splunk/lightproto.
> >>>
> >>> This code generator has the following features/goals:
> >>>
> >>> 1. Generate the fastest possible Java code for Protobuf SerDe
> >>> 2. 100% Compatible with proto2 definition and wire protocol
> >>> 3. Zero-copy deserialization using Netty ByteBuf
> >>> 4. Deserialize from direct memory
> >>> 5. Zero heap allocations in serialization / deserialization
> >>> 6. Lazy deserialization of strings and bytes
> >>> 7. Reusable mutable objects
> >>> 8. No runtime dependency library
> >>> 9. Java based code generator with Maven plugin
> >>>
> >>> There is extensive testing to ensure the generated code serializes and
> parses
> >>> the same bytes in the same way as the Google Protobuf does.
> >>>
> >>>
> >>> --
> >>> Matteo Merli
> >>> <mme...@apache.org>
> >>
>
>
