Jason, that's an interesting point regarding the Java client. Do we know
what clients in other languages do in these cases?
Ismael
On Tue, 21 Aug 2018, 17:30 Jason Gustafson, <ja...@confluent.io> wrote:
> Hi Dongjin,
>
> One of the complications is that old versions of the API will not expect a
> new error code. However, since we expect this to be a fatal error anyway
> for old clients, it may still be more useful to return the correct error
> code. For example, the Kafka clients use the following code to convert the
> error code:
>
> public static Errors forCode(short code) {
> Errors error = codeToError.get(code);
> if (error != null) {
> return error;
> } else {
> log.warn("Unexpected error code: {}.", code);
> return UNKNOWN_SERVER_ERROR;
> }
> }
>
> If we return an unsupported error code, it will be converted to an UNKNOWN
> error, but at least we will get the message in the log with the correct
> code. That seems preferable to returning a misleading error code. So I
> wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even for
> older versions.
>
> Also, one question just to check my understanding. I think we would only
> use this error code when we /know/ that zstd was in use and the client
> doesn't support it? This is true if either 1) the message needs
> down-conversion and we encounter a zstd compressed message, or 2) if the
> topic is explicitly configured to use zstd. However, if the compression
> type is set to "producer," then the fetched data may or may not be
> compressed with zstd. In this case, we return the data to the client and
> expect it to fail parsing. Is that correct?
>
> Thanks,
> Jason
>
>
>
> On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee <dong...@apache.org> wrote:
>
> > Ismael, Jason and all,
> >
> > I rewrote the backward compatibility strategy & its alternatives like
> > following, based on Ismael & Jason's comments. Since it is not updated to
> > the wiki yet, don't hesitate to give me a message if you have any opinion
> > on it.
> >
> > ```
> > *Backward Compatibility*
> >
> > We need to establish some backward-compatibility strategy for the case an
> > old client subscribes a topic using ZStandard implicitly (i.e.,
> > 'compression.type' configuration of given topic is 'producer' and the
> > producer compressed the records with ZStandard). We have the following
> > options for this situation:
> >
> > *A. Support ZStandard to the old clients which can understand v0, v1
> > messages only.*
> >
> > This strategy necessarily requires the down-conversion of v2 message
> > compressed with Zstandard into v0 or v1 messages, which means a
> > considerable performance degradation. So we rejected this strategy.
> >
> > *B. Bump the API version and support only v2-available clients*
> >
> > With this approach, we can message the old clients that they are old and
> > should be upgraded. However, there are still several options for the
> Error
> > code.
> >
> > *B.1. INVALID_REQUEST (42)*
> >
> > This option gives the client so little information; the user can be
> > confused about why the client worked correctly in the past suddenly
> > encounters a problem. So we rejected this strategy.
> >
> > *B.2. CORRUPT_MESSAGE (2)*
> >
> > This option gives inaccurate information; the user can be surprised and
> > misunderstand that the log files are broken in some way. So we rejected
> > this strategy.
> >
> > *B.3 UNSUPPORTED_FOR_MESSAGE_FORMAT (43)*
> >
> > The advantage of this approach is we don't need to define a new error
> code;
> > we can reuse it and that's all.
> >
> > The disadvantage of this approach is that it is also a little bit vague;
> > This error code is defined as a work for KIP-98[^1] and now returned in
> the
> > transaction error.
> >
> > *B.4. UNSUPPORTED_COMPRESSION_TYPE (new)*
> >
> > The advantage of this approach is that it is clear and provides an exact
> > description. The disadvantage is we need to add a new error code.
> > ```
> >
> > *It seems like what we need to choose is now so clear:
> > UNSUPPORTED_FOR_MESSAGE_FORMAT (B.3) or UNSUPPORTED_COMPRESSION_TYPE
> > (B.4).*
> > The first one doesn't need a new error message but the latter is more
> > explicit. Which one do you prefer? Since all of you have much more
> > experience and knowledge than me, I will follow your decision. The wiki
> > page will be updated following the decision also.
> >
> > Best,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/KAFKA-4990
> >
> > On Sun, Aug 19, 2018 at 4:58 AM Ismael Juma <isma...@gmail.com> wrote:
> >
> > > Sounds reasonable to me.
> > >
> > > Ismael
> > >
> > > On Sat, 18 Aug 2018, 12:20 Jason Gustafson, <ja...@confluent.io>
> wrote:
> > >
> > > > Hey Ismael,
> > > >
> > > > Your summary looks good to me. I think it might also be a good idea
> to
> > > add
> > > > a new UNSUPPORTED_COMPRESSION_TYPE error code to go along with the
> > > version
> > > > bumps. We won't be able to use it for old api versions since the
> > clients
> > > > will not understand it, but we can use it going forward so that we're
> > not
> > > > stuck in a similar situation with a new message format and a new
> codec
> > to
> > > > support. Another option is to use UNSUPPORTED_FOR_MESSAGE_FORMAT, but
> > it
> > > is
> > > > not as explicit.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Aug 17, 2018 at 5:19 PM, Ismael Juma <ism...@juma.me.uk>
> > wrote:
> > > >
> > > > > Hi Dongjin and Jason,
> > > > >
> > > > > I would agree. My summary:
> > > > >
> > > > > 1. Support zstd with message format 2 only.
> > > > > 2. Bump produce and fetch request versions.
> > > > > 3. Provide broker errors whenever possible based on the request
> > version
> > > > and
> > > > > rely on clients for the cases where the broker can't validate
> > > efficiently
> > > > > (example message format 2 consumer that supports the latest fetch
> > > version
> > > > > but doesn't support zstd).
> > > > >
> > > > > If there's general agreement on this, I suggest we update the KIP
> to
> > > > state
> > > > > the proposal and to move the rejected options to its own section.
> And
> > > > then
> > > > > start a vote!
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Aug 17, 2018 at 4:00 PM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hi Dongjin,
> > > > > >
> > > > > > Yes, that's a good summary. For clients which support v2, the
> > client
> > > > can
> > > > > > parse the message format and hopefully raise a useful error
> message
> > > > > > indicating the unsupported compression type. For older clients,
> our
> > > > > options
> > > > > > are probably (1) to down-convert to the old format using no
> > > compression
> > > > > > type, or (2) to return an error code. I'm leaning toward the
> latter
> > > as
> > > > > the
> > > > > > simpler solution, but the challenge is finding a good error code.
> > Two
> > > > > > possibilities might be INVALID_REQUEST or CORRUPT_MESSAGE. The
> > > downside
> > > > > is
> > > > > > that old clients probably won't get a helpful message. However,
> at
> > > > least
> > > > > > the behavior will be consistent in the sense that all clients
> will
> > > fail
> > > > > if
> > > > > > they do not support zstandard.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > > On Fri, Aug 17, 2018 at 8:08 AM, Dongjin Lee <dong...@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Thanks Jason, I reviewed the down-converting logic following
> your
> > > > > > > explanation.[^1] You mean the following routines, right?
> > > > > > >
> > > > > > > -
> > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/
> > > > > > > main/scala/kafka/server/KafkaApis.scala#L534
> > > > > > > -
> > > > > > > https://github.com/apache/kafka/blob/trunk/clients/src/
> > > > > > > main/java/org/apache/kafka/common/record/
> > LazyDownConversionRecords.
> > > > > > > java#L165
> > > > > > > -
> > > > > > > https://github.com/apache/kafka/blob/trunk/clients/src/
> > > > > > > main/java/org/apache/kafka/common/record/RecordsUtil.java#L40
> > > > > > >
> > > > > > > It seems like your stance is like following:
> > > > > > >
> > > > > > > 1. In principle, Kafka does not change the compression codec
> when
> > > > > > > down-converting, since it requires inspecting the fetched data,
> > > which
> > > > > is
> > > > > > > expensive.
> > > > > > > 2. However, there are some cases the fetched data is inspected
> > > > anyway.
> > > > > In
> > > > > > > this case, we can provide compression conversion from Zstandard
> > to
> > > > > > > classical ones[^2].
> > > > > > >
> > > > > > > And from what I understand, the cases where the client without
> > > > > ZStandard
> > > > > > > support receives ZStandard compressed records can be organized
> > into
> > > > two
> > > > > > > cases:
> > > > > > >
> > > > > > > a. The 'compression.type' configuration of given topic is
> > > 'producer'
> > > > > and
> > > > > > > the producer compressed the records with ZStandard. (that is,
> > using
> > > > > > > ZStandard implicitly.)
> > > > > > > b. The 'compression.type' configuration of given topic is
> > 'zstd';
> > > > that
> > > > > > is,
> > > > > > > using ZStandard explicitly.
> > > > > > >
> > > > > > > As you stated, we don't have to handle the case b specially.
> So,
> > It
> > > > > seems
> > > > > > > like we can narrow the focus of the problem by joining case 1
> and
> > > > case
> > > > > b
> > > > > > > like the following:
> > > > > > >
> > > > > > > > Given the topic with 'producer' as its 'compression.type'
> > > > > > configuration,
> > > > > > > ZStandard compressed records and old client without ZStandard,
> is
> > > > there
> > > > > > any
> > > > > > > case we need to inspect the records and can change the
> > compression
> > > > > type?
> > > > > > If
> > > > > > > so, can we provide compression type converting?
> > > > > > >
> > > > > > > Do I understand correctly?
> > > > > > >
> > > > > > > Best,
> > > > > > > Dongjin
> > > > > > >
> > > > > > > [^1]: I'm sorry, I found that I was a little bit
> misunderstanding
> > > how
> > > > > API
> > > > > > > version works, after reviewing the downconvert logic & the
> > protocol
> > > > > > > documentation <https://kafka.apache.org/protocol>.
> > > > > > > [^2]: None, Gzip, Snappy, Lz4.
> > > > > > >
> > > > > > > On Tue, Aug 14, 2018 at 2:16 AM Jason Gustafson <
> > > ja...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > > But in my opinion, since the client will fail with the API
> > > > version,
> > > > > > so
> > > > > > > we
> > > > > > > > > don't need to down-convert the messages anyway. Isn't it?
> > So, I
> > > > > think
> > > > > > > we
> > > > > > > > > don't care about this case. (I'm sorry, I am not familiar
> > with
> > > > > > > > down-convert
> > > > > > > > > logic.)
> > > > > > > >
> > > > > > > >
> > > > > > > > Currently the broker down-converts automatically when it
> > receives
> > > > an
> > > > > > old
> > > > > > > > version of the fetch request (a version which is known to
> > predate
> > > > the
> > > > > > > > message format in use). Typically when down-converting the
> > > message
> > > > > > > format,
> > > > > > > > we use the same compression type, but there is not much point
> > in
> > > > > doing
> > > > > > so
> > > > > > > > when we know the client doesn't support it. So if zstandard
> is
> > in
> > > > > use,
> > > > > > > and
> > > > > > > > we have to down-convert anyway, then we can choose to use a
> > > > different
> > > > > > > > compression type or no compression type.
> > > > > > > >
> > > > > > > > From my perspective, there is no significant downside to
> > bumping
> > > > the
> > > > > > > > protocol version and it has several potential benefits.
> Version
> > > > bumps
> > > > > > are
> > > > > > > > cheap. The main question mark in my mind is about
> > > down-conversion.
> > > > > > > Figuring
> > > > > > > > out whether down-conversion is needed is hard generally
> without
> > > > > > > inspecting
> > > > > > > > the fetched data, which is expensive. I think we agree in
> > > principle
> > > > > > that
> > > > > > > we
> > > > > > > > do not want to have to pay this cost generally and prefer the
> > > > clients
> > > > > > to
> > > > > > > > fail when they see an unhandled compression type. The point I
> > was
> > > > > > making
> > > > > > > is
> > > > > > > > that there are some cases where we are either inspecting the
> > data
> > > > > > anyway
> > > > > > > > (because we have to down-convert the message format), or we
> > have
> > > an
> > > > > > easy
> > > > > > > > way to tell whether zstandard is in use (the topic has it
> > > > configured
> > > > > > > > explicitly). In the latter case, we don't have to handle it
> > > > > specially.
> > > > > > > But
> > > > > > > > we do have to decide how we will handle down-conversion to
> > older
> > > > > > formats.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Sun, Aug 12, 2018 at 5:15 PM, Dongjin Lee <
> > dong...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Colin and Jason,
> > > > > > > > >
> > > > > > > > > Thanks for your opinions. In summarizing, the Pros and Cons
> > of
> > > > > > bumping
> > > > > > > > > fetch API version are:
> > > > > > > > >
> > > > > > > > > Cons:
> > > > > > > > >
> > > > > > > > > - The Broker can't know whether a given message batch is
> > > > compressed
> > > > > > > with
> > > > > > > > > zstd or not.
> > > > > > > > > - Need some additional logic for the topic explicitly
> > > configured
> > > > to
> > > > > > use
> > > > > > > > > zstd.
> > > > > > > > >
> > > > > > > > > Pros:
> > > > > > > > >
> > > > > > > > > - The broker doesn't need to conduct expensive
> > down-conversion.
> > > > > > > > > - Can message the users to update their client.
> > > > > > > > >
> > > > > > > > > So, opinions for the backward-compatibility policy by far:
> > > > > > > > >
> > > > > > > > > - A: bump the API version - +2 (Colin, Jason)
> > > > > > > > > - B: leave unchanged - +1 (Viktor)
> > > > > > > > >
> > > > > > > > > Here are my additional comments:
> > > > > > > > >
> > > > > > > > > @Colin
> > > > > > > > >
> > > > > > > > > I greatly appreciate your response. In the case of the
> > > dictionary
> > > > > > > > support,
> > > > > > > > > of course, this issue should be addressed later so we don't
> > > need
> > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > first version. You are right - it is not late to try it
> after
> > > > some
> > > > > > > > > benchmarks. What I mean is, we should keep in mind on that
> > > > > potential
> > > > > > > > > feature.
> > > > > > > > >
> > > > > > > > > @Jason
> > > > > > > > >
> > > > > > > > > You wrote,
> > > > > > > > >
> > > > > > > > > > Similarly, if we have to down-convert anyway because the
> > > client
> > > > > > does
> > > > > > > > not
> > > > > > > > > understand the message format, then we could also use a
> > > different
> > > > > > > > > compression type.
> > > > > > > > >
> > > > > > > > > But in my opinion, since the client will fail with the API
> > > > version,
> > > > > > so
> > > > > > > we
> > > > > > > > > don't need to down-convert the messages anyway. Isn't it?
> > So, I
> > > > > think
> > > > > > > we
> > > > > > > > > don't care about this case. (I'm sorry, I am not familiar
> > with
> > > > > > > > down-convert
> > > > > > > > > logic.)
> > > > > > > > >
> > > > > > > > > Please give more opinions. Thanks!
> > > > > > > > >
> > > > > > > > > - Dongjin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 8, 2018 at 6:41 AM Jason Gustafson <
> > > > ja...@confluent.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Colin,
> > > > > > > > > >
> > > > > > > > > > The problem for the fetch API is that the broker does not
> > > > > generally
> > > > > > > > know
> > > > > > > > > if
> > > > > > > > > > a batch was compressed with zstd unless it parses it. I
> > think
> > > > the
> > > > > > > goal
> > > > > > > > > here
> > > > > > > > > > is to avoid the expensive down-conversion that is needed
> to
> > > > > ensure
> > > > > > > > > > compatibility because it is only necessary if zstd is
> > > actually
> > > > in
> > > > > > > use.
> > > > > > > > > But
> > > > > > > > > > as long as old clients can parse the message format, they
> > > > should
> > > > > > get
> > > > > > > a
> > > > > > > > > > reasonable error if they see an unsupported compression
> > type
> > > in
> > > > > the
> > > > > > > > > > attributes. Basically the onus is on users to ensure that
> > > their
> > > > > > > > consumers
> > > > > > > > > > have been updated prior to using zstd. It seems like a
> > > > reasonable
> > > > > > > > > tradeoff
> > > > > > > > > > to me. There are a couple cases that might be worth
> > thinking
> > > > > > through:
> > > > > > > > > >
> > > > > > > > > > 1. If a topic is explicitly configured to use zstd, then
> we
> > > > don't
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > check the fetched data for the compression type to know
> if
> > we
> > > > > need
> > > > > > > > > > down-conversion. If we did bump the Fetch API version,
> then
> > > we
> > > > > > could
> > > > > > > > > handle
> > > > > > > > > > this case by either down-converting using a different
> > > > compression
> > > > > > > type
> > > > > > > > or
> > > > > > > > > > returning an error.
> > > > > > > > > > 2. Similarly, if we have to down-convert anyway because
> the
> > > > > client
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > understand the message format, then we could also use a
> > > > different
> > > > > > > > > > compression type.
> > > > > > > > > >
> > > > > > > > > > For the produce API, I think it's reasonable to bump the
> > api
> > > > > > version.
> > > > > > > > > This
> > > > > > > > > > can be used by clients to check whether a broker supports
> > > zstd.
> > > > > For
> > > > > > > > > > example, we might support a list of preferred compression
> > > types
> > > > > in
> > > > > > > the
> > > > > > > > > > producer and we could use the broker to detect which
> > version
> > > to
> > > > > > use.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 7, 2018 at 1:32 PM, Colin McCabe <
> > > > cmcc...@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for bumping this, Dongjin. ZStd is a good
> > > compression
> > > > > > codec
> > > > > > > > > and I
> > > > > > > > > > > hope we can get this support in soon!
> > > > > > > > > > >
> > > > > > > > > > > I would say we can just bump the API version to
> indicate
> > > that
> > > > > > ZStd
> > > > > > > > > > support
> > > > > > > > > > > is expected in new clients. We probably need some way
> of
> > > > > > > indicating
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > older clients that they can't consume the partitions,
> as
> > > > well.
> > > > > > > > Perhaps
> > > > > > > > > > we
> > > > > > > > > > > can use the UNSUPPORTED_FOR_MESSAGE_FORMAT error?
> > > > > > > > > > >
> > > > > > > > > > > The license thing seems straightforward -- it's just a
> > > matter
> > > > > of
> > > > > > > > adding
> > > > > > > > > > > the text to the right files as per ASF guidelines.
> > > > > > > > > > >
> > > > > > > > > > > With regard to the dictionary support, do we really
> need
> > > that
> > > > > in
> > > > > > > the
> > > > > > > > > > first
> > > > > > > > > > > version? Hopefully message batches are big enough that
> > > this
> > > > > > isn't
> > > > > > > > > > needed.
> > > > > > > > > > > Some benchmarks might help here.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Aug 7, 2018, at 08:02, Dongjin Lee wrote:
> > > > > > > > > > > > As Kafka 2.0.0 was released, let's reboot this issue,
> > > > KIP-110
> > > > > > > > > > > > <https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> > > > > > > > > > > 110%3A+Add+Codec+for+ZStandard+Compression>
> > > > > > > > > > > > .
> > > > > > > > > > > >
> > > > > > > > > > > > For newcomers, Here is some summary of the history:
> > > KIP-110
> > > > > was
> > > > > > > > > > > originally
> > > > > > > > > > > > worked for the issue KAFKA-4514 but, it lacked
> > benchmark
> > > > > > results
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > the
> > > > > > > > > > > > agreement of the community. Later, Ivan Babrou and
> some
> > > > other
> > > > > > > users
> > > > > > > > > who
> > > > > > > > > > > > adopted the patch provided their excellent
> performance
> > > > report
> > > > > > > which
> > > > > > > > > is
> > > > > > > > > > > now
> > > > > > > > > > > > included in the KIP, but it postponed again because
> of
> > > the
> > > > > > > > community
> > > > > > > > > > was
> > > > > > > > > > > > busy for 2.0.0 release. It is why I now reboot this
> > > issue.
> > > > > > > > > > > >
> > > > > > > > > > > > The following is the current status of the feature:
> You
> > > can
> > > > > > check
> > > > > > > > the
> > > > > > > > > > > > current draft implementation here
> > > > > > > > > > > > <https://github.com/apache/kafka/pull/2267>. It is
> > based
> > > > on
> > > > > > zstd
> > > > > > > > > 1.3.5
> > > > > > > > > > > and
> > > > > > > > > > > > periodically rebased onto the latest trunk[^1].
> > > > > > > > > > > >
> > > > > > > > > > > > The issues that should be addressed is like
> following:
> > > > > > > > > > > >
> > > > > > > > > > > > *1. Backward Compatibility*
> > > > > > > > > > > >
> > > > > > > > > > > > To support old consumers, we need to take a strategy
> to
> > > > > handle
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > > > consumers. Current candidates are:
> > > > > > > > > > > >
> > > > > > > > > > > > - Bump API version
> > > > > > > > > > > > - Leave unchanged: let the old clients fail.
> > > > > > > > > > > > - Improve the error messages:
> > > > > > > > > > > >
> > > > > > > > > > > > *2. Dictionary Support*
> > > > > > > > > > > >
> > > > > > > > > > > > To support zstd's dictionary feature in the future
> (if
> > > > > needed),
> > > > > > > we
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > sketch how it should be and leave some room for it.
> As
> > of
> > > > > now,
> > > > > > > > there
> > > > > > > > > > has
> > > > > > > > > > > > been no discussion on this topic yet.
> > > > > > > > > > > >
> > > > > > > > > > > > *3. License*
> > > > > > > > > > > >
> > > > > > > > > > > > To use this feature, we need to add license of zstd
> and
> > > > > > zstd-jni
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > project. (Thanks to Viktor Somogyi for raising this
> > > issue!)
> > > > > It
> > > > > > > > seems
> > > > > > > > > > like
> > > > > > > > > > > > what Apache Spark did would be a good example but
> there
> > > has
> > > > > > been
> > > > > > > no
> > > > > > > > > > > > discussion yet.
> > > > > > > > > > > >
> > > > > > > > > > > > You can find the details of the above issues in the
> KIP
> > > > > > document.
> > > > > > > > > > Please
> > > > > > > > > > > > have a look when you are free, and give me feedback.
> > All
> > > > > kinds
> > > > > > of
> > > > > > > > > > > > participating are welcome.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Dongjin
> > > > > > > > > > > >
> > > > > > > > > > > > [^1]: At the time of writing, commit 6b4fb8152.
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Jul 14, 2018 at 10:45 PM Dongjin Lee <
> > > > > > dong...@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for the late reply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In short, I could not submit the updated KIP by the
> > > > feature
> > > > > > > > freeze
> > > > > > > > > > > > > deadline of 2.0.0. For this reason, it will not be
> > > > included
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > 2.0.0
> > > > > > > > > > > > > release and all discussion for this issue were
> > > postponed
> > > > > > after
> > > > > > > > the
> > > > > > > > > > > release
> > > > > > > > > > > > > of 2.0.0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have been updating the PR following recent
> updates.
> > > > Just
> > > > > > > now, I
> > > > > > > > > > > rebased
> > > > > > > > > > > > > it against the latest trunk and updated the zstd
> > > version
> > > > > into
> > > > > > > > > 1.3.5.
> > > > > > > > > > > If you
> > > > > > > > > > > > > need some request, don't hesitate to notify me.
> (But
> > > not
> > > > > this
> > > > > > > > > thread
> > > > > > > > > > -
> > > > > > > > > > > just
> > > > > > > > > > > > > send me the message directly.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Dongjin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jul 10, 2018 at 11:57 PM Bobby Evans <
> > > > > > bo...@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> I there any update on this. The performance
> > > > improvements
> > > > > > are
> > > > > > > > > quite
> > > > > > > > > > > > >> impressive and I really would like to stop forking
> > > kafka
> > > > > > just
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > this
> > > > > > > > > > > > >> in.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Bobby
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Wed, Jun 13, 2018 at 8:56 PM Dongjin Lee <
> > > > > > > dong...@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Ismael,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Oh, I forgot all of you are on working frenzy
> for
> > > 2.0!
> > > > > No
> > > > > > > > > problem,
> > > > > > > > > > > take
> > > > > > > > > > > > >> > your time. I am also working at another issue
> now.
> > > > Thank
> > > > > > you
> > > > > > > > for
> > > > > > > > > > > > >> letting me
> > > > > > > > > > > > >> > know.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Best,
> > > > > > > > > > > > >> > Dongjin
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Wed, Jun 13, 2018, 11:44 PM Ismael Juma <
> > > > > > > isma...@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Sorry for the delay Dongjin. Everyone is busy
> > > > > finalising
> > > > > > > > > 2.0.0.
> > > > > > > > > > > This
> > > > > > > > > > > > >> KIP
> > > > > > > > > > > > >> > > seems like a great candidate for 2.1.0 and
> > > hopefully
> > > > > > there
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > >> more
> > > > > > > > > > > > >> > of
> > > > > > > > > > > > >> > > a discussion next week. :)
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Ismael
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > On Wed, 13 Jun 2018, 05:17 Dongjin Lee, <
> > > > > > > dong...@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > > Hello. I just updated my draft
> implementation:
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > 1. Rebased to latest trunk (commit 5145d6b)
> > > > > > > > > > > > >> > > > 2. Apply ZStd 1.3.4
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > You can check out the implementation from
> here
> > > > > > > > > > > > >> > > > <https://github.com/apache/kafka/pull/2267
> >.
> > If
> > > > you
> > > > > > > > > > experience
> > > > > > > > > > > any
> > > > > > > > > > > > >> > > problem
> > > > > > > > > > > > >> > > > running it, don't hesitate to give me a
> > mention.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Best,
> > > > > > > > > > > > >> > > > Dongjin
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > On Tue, Jun 12, 2018 at 6:50 PM Dongjin Lee
> <
> > > > > > > > > > dong...@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > > Here is the short conclusion about the
> > license
> > > > > > > problem:
> > > > > > > > > *We
> > > > > > > > > > > can
> > > > > > > > > > > > >> use
> > > > > > > > > > > > >> > > zstd
> > > > > > > > > > > > >> > > > > and zstd-jni without any problem, but we
> > need
> > > to
> > > > > > > include
> > > > > > > > > > their
> > > > > > > > > > > > >> > license,
> > > > > > > > > > > > >> > > > > e.g., BSD license.*
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Both of BSD 2 Clause License & 3 Clause
> > > License
> > > > > > > requires
> > > > > > > > > to
> > > > > > > > > > > > >> include
> > > > > > > > > > > > >> > the
> > > > > > > > > > > > >> > > > > license used, and BSD 3 Clause License
> > > requires
> > > > > that
> > > > > > > the
> > > > > > > > > > name
> > > > > > > > > > > of
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > > > > contributor can't be used to endorse or
> > > promote
> > > > > the
> > > > > > > > > product.
> > > > > > > > > > > > >> That's
> > > > > > > > > > > > >> > it
> > > > > > > > > > > > >> > > > > <
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> http://www.mikestratton.net/2011/12/is-bsd-license-
> > > > > > > > > > > compatible-with-apache-2-0-license/
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > - They are not listed in the list of
> > > prohibited
> > > > > > > licenses
> > > > > > > > > > > > >> > > > > <https://www.apache.org/legal/
> > > > > > > resolved.html#category-x>
> > > > > > > > > > also.
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Here is how Spark did for it
> > > > > > > > > > > > >> > > > > <https://issues.apache.org/
> > > > > jira/browse/SPARK-19112
> > > > > > >:
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > - They made a directory dedicated to the
> > > > > dependency
> > > > > > > > > license
> > > > > > > > > > > files
> > > > > > > > > > > > >> > > > > <
> > > > > > https://github.com/apache/spark/tree/master/licenses
> > > > > > > >
> > > > > > > > > and
> > > > > > > > > > > added
> > > > > > > > > > > > >> > > > licenses
> > > > > > > > > > > > >> > > > > for Zstd
> > > > > > > > > > > > >> > > > > <
> > > > > > > > > > > > >> >
> > > https://github.com/apache/spark/blob/master/licenses/
> > > > > > > > > > > LICENSE-zstd.txt
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > &
> > > > > > > > > > > > >> > > > > Zstd-jni
> > > > > > > > > > > > >> > > > > <
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> https://github.com/apache/
> > spark/blob/master/licenses/
> > > > > > > > > > > LICENSE-zstd-jni.txt
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > > > .
> > > > > > > > > > > > >> > > > > - Added a link to the original license
> files
> > > in
> > > > > > > LICENSE.
> > > > > > > > > > > > >> > > > > <
> > > > https://github.com/apache/spark/pull/18805/files
> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > If needed, I can make a similar update.
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Thanks for pointing out this problem,
> > Viktor!
> > > > Nice
> > > > > > > > catch!
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Best,
> > > > > > > > > > > > >> > > > > Dongjin
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > On Mon, Jun 11, 2018 at 11:50 PM Dongjin
> > Lee <
> > > > > > > > > > > dong...@apache.org>
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > >> I greatly appreciate your comprehensive
> > > > > reasoning.
> > > > > > > so:
> > > > > > > > +1
> > > > > > > > > > > for b
> > > > > > > > > > > > >> > until
> > > > > > > > > > > > >> > > > now.
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >> For the license issues, I will have a
> check
> > > on
> > > > > how
> > > > > > > the
> > > > > > > > > over
> > > > > > > > > > > > >> projects
> > > > > > > > > > > > >> > > are
> > > > > > > > > > > > >> > > > >> doing and share the results.
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >> Best,
> > > > > > > > > > > > >> > > > >> Dongjin
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >> On Mon, Jun 11, 2018 at 10:08 PM Viktor
> > > > Somogyi <
> > > > > > > > > > > > >> > > > viktorsomo...@gmail.com>
> > > > > > > > > > > > >> > > > >> wrote:
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >>> Hi Dongjin,
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >>> A couple of comments:
> > > > > > > > > > > > >> > > > >>> I would vote for option b. in the
> > "backward
> > > > > > > > > compatibility"
> > > > > > > > > > > > >> section.
> > > > > > > > > > > > >> > > My
> > > > > > > > > > > > >> > > > >>> reasoning for this is that users
> upgrading
> > > to
> > > > a
> > > > > > zstd
> > > > > > > > > > > compatible
> > > > > > > > > > > > >> > > version
> > > > > > > > > > > > >> > > > >>> won't start to use it automatically, so
> > > manual
> > > > > > > > > > > reconfiguration
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> > > > >>> required.
> > > > > > > > > > > > >> > > > >>> Therefore an upgrade won't mess up the
> > > > cluster.
> > > > > If
> > > > > > > not
> > > > > > > > > all
> > > > > > > > > > > the
> > > > > > > > > > > > >> > > clients
> > > > > > > > > > > > >> > > > >>> are
> > > > > > > > > > > > >> > > > >>> upgraded but just some of them and
> they'd
> > > > start
> > > > > to
> > > > > > > use
> > > > > > > > > > zstd
> > > > > > > > > > > > >> then it
> > > > > > > > > > > > >> > > > would
> > > > > > > > > > > > >> > > > >>> cause errors in the cluster. I'd like to
> > > > presume
> > > > > > > > though
> > > > > > > > > > that
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> > is
> > > > > > > > > > > > >> > > a
> > > > > > > > > > > > >> > > > >>> very
> > > > > > > > > > > > >> > > > >>> obvious failure case and nobody should
> be
> > > > > > surprised
> > > > > > > if
> > > > > > > > > it
> > > > > > > > > > > didn't
> > > > > > > > > > > > >> > > work.
> > > > > > > > > > > > >> > > > >>> I wouldn't choose a. as I think we
> should
> > > bump
> > > > > the
> > > > > > > > fetch
> > > > > > > > > > and
> > > > > > > > > > > > >> > produce
> > > > > > > > > > > > >> > > > >>> requests if it's a change in the message
> > > > format.
> > > > > > > > > Moreover
> > > > > > > > > > if
> > > > > > > > > > > > >> some
> > > > > > > > > > > > >> > of
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > >>> producers and the brokers are upgraded
> but
> > > > some
> > > > > of
> > > > > > > the
> > > > > > > > > > > consumers
> > > > > > > > > > > > >> > are
> > > > > > > > > > > > >> > > > not,
> > > > > > > > > > > > >> > > > >>> then we wouldn't prevent the error when
> > the
> > > > old
> > > > > > > > consumer
> > > > > > > > > > > tries
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > > > consume
> > > > > > > > > > > > >> > > > >>> the zstd compressed messages.
> > > > > > > > > > > > >> > > > >>> I wouldn't choose c. either as I think
> > > binding
> > > > > the
> > > > > > > > > > > compression
> > > > > > > > > > > > >> type
> > > > > > > > > > > > >> > > to
> > > > > > > > > > > > >> > > > an
> > > > > > > > > > > > >> > > > >>> API is not so obvious from the
> developer's
> > > > > > > > perspective.
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >>> I would also prefer to use the existing
> > > > binding,
> > > > > > > > however
> > > > > > > > > > we
> > > > > > > > > > > must
> > > > > > > > > > > > >> > > > respect
> > > > > > > > > > > > >> > > > >>> the licenses:
> > > > > > > > > > > > >> > > > >>> "The code for these JNI bindings is
> > licenced
> > > > > under
> > > > > > > > > > 2-clause
> > > > > > > > > > > BSD
> > > > > > > > > > > > >> > > > license.
> > > > > > > > > > > > >> > > > >>> The native Zstd library is licensed
> under
> > > > > 3-clause
> > > > > > > BSD
> > > > > > > > > > > license
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > > > GPL2"
> > > > > > > > > > > > >> > > > >>> Based on the FAQ page
> > > > > > > > > > > > >> > > > >>> https://www.apache.org/legal/
> > > > > > > resolved.html#category-a
> > > > > > > > > > > > >> > > > >>> we may use 2- and 3-clause BSD licenses
> > but
> > > > the
> > > > > > > Apache
> > > > > > > > > > > license
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> > not
> > > > > > > > > > > > >> > > > >>> compatible with GPL2. I'm hoping that
> the
> > > > > > "3-clause
> > > > > > > > BSD
> > > > > > > > > > > license
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > > > GPL2"
> > > > > > > > > > > > >> > > > >>> is really not an AND but an OR in this
> > case,
> > > > but
> > > > > > I'm
> > > > > > > > no
> > > > > > > > > > > lawyer,
> > > > > > > > > > > > >> > just
> > > > > > > > > > > > >> > > > >>> wanted
> > > > > > > > > > > > >> > > > >>> to make the point that we should watch
> out
> > > for
> > > > > > > > licenses.
> > > > > > > > > > :)
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >>> Regards,
> > > > > > > > > > > > >> > > > >>> Viktor
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >>> On Sun, Jun 10, 2018 at 3:02 AM Ivan
> > Babrou
> > > <
> > > > > > > > > > > ibob...@gmail.com>
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >>> > Hello,
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>> > This is Ivan and I still very much
> > support
> > > > the
> > > > > > > fact
> > > > > > > > > that
> > > > > > > > > > > zstd
> > > > > > > > > > > > >> > > > >>> compression
> > > > > > > > > > > > >> > > > >>> > should be included out of the box.
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>> > Please think about the environment,
> you
> > > can
> > > > > save
> > > > > > > > > quite a
> > > > > > > > > > > lot
> > > > > > > > > > > > >> of
> > > > > > > > > > > > >> > > > >>> hardware
> > > > > > > > > > > > >> > > > >>> > with it.
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>> > Thank you.
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>> > On Sat, Jun 9, 2018 at 14:14 Dongjin
> > Lee <
> > > > > > > > > > > dong...@apache.org>
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>> > > Since there are no responses for a
> > > week, I
> > > > > > > decided
> > > > > > > > > to
> > > > > > > > > > > > >> > reinitiate
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > >>> > > discussion thread.
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > > > 110%3A+Add+Codec+for+ZStandard+Compression
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > > This KIP is about to introduce
> > ZStandard
> > > > > > > > Compression
> > > > > > > > > > > into
> > > > > > > > > > > > >> > Apache
> > > > > > > > > > > > >> > > > >>> Kafka.
> > > > > > > > > > > > >> > > > >>> > > The reason why it is posted again
> has
> > a
> > > > > story:
> > > > > > > It
> > > > > > > > > was
> > > > > > > > > > > > >> > originally
> > > > > > > > > > > > >> > > > >>> posted
> > > > > > > > > > > > >> > > > >>> > to
> > > > > > > > > > > > >> > > > >>> > > the dev mailing list more than one
> > year
> > > > ago
> > > > > > but
> > > > > > > > > since
> > > > > > > > > > it
> > > > > > > > > > > > >> has no
> > > > > > > > > > > > >> > > > >>> > performance
> > > > > > > > > > > > >> > > > >>> > > report included, it was postponed
> > later.
> > > > But
> > > > > > > Some
> > > > > > > > > > people
> > > > > > > > > > > > >> > > (including
> > > > > > > > > > > > >> > > > >>> Ivan)
> > > > > > > > > > > > >> > > > >>> > > reported excellent performance
> report
> > > with
> > > > > the
> > > > > > > > draft
> > > > > > > > > > PR,
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> > > work
> > > > > > > > > > > > >> > > > >>> is now
> > > > > > > > > > > > >> > > > >>> > > reactivated.
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > > The updated KIP document includes
> some
> > > > > > expected
> > > > > > > > > > > problems and
> > > > > > > > > > > > >> > > their
> > > > > > > > > > > > >> > > > >>> > > candidate alternatives. Please have
> a
> > > look
> > > > > > when
> > > > > > > > you
> > > > > > > > > > are
> > > > > > > > > > > > >> free,
> > > > > > > > > > > > >> > and
> > > > > > > > > > > > >> > > > >>> give
> > > > > > > > > > > > >> > > > >>> > me a
> > > > > > > > > > > > >> > > > >>> > > feedback. All kinds of participating
> > are
> > > > > > > welcome.
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > > Best,
> > > > > > > > > > > > >> > > > >>> > > Dongjin
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > > --
> > > > > > > > > > > > >> > > > >>> > > *Dongjin Lee*
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > > *A hitchhiker in the mathematical
> > > world.*
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> > > *github: <http://goog_969573159/
> > > >github
> > > > > > > > > > > .com/dongjinleekr
> > > > > > > > > > > > >> > > > >>> > > <http://github.com/dongjinleekr
> > > >linkedin:
> > > > > > > > > > > > >> > > > >>> > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > >> > > > >>> > > <http://kr.linkedin.com/in/
> > dongjinleekr
> > > > > > > > >slideshare:
> > > > > > > > > > > > >> > > > >>> > www.slideshare.net/dongjinleekr
> > > > > > > > > > > > >> > > > >>> > > <http://www.slideshare.net/
> > dongjinleekr
> > > >*
> > > > > > > > > > > > >> > > > >>> > >
> > > > > > > > > > > > >> > > > >>> >
> > > > > > > > > > > > >> > > > >>>
> > > > > > > > > > > > >> > > > >> --
> > > > > > > > > > > > >> > > > >> *Dongjin Lee*
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >> *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >> *github: <http://goog_969573159/>github
> > > > > > > > > .com/dongjinleekr
> > > > > > > > > > > > >> > > > >> <http://github.com/dongjinleekr
> >linkedin:
> > > > > > > > > > > > >> > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > >> > > > >> <http://kr.linkedin.com/in/dongjinleekr
> > > > > > >slideshare:
> > > > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr
> > > > > > > > > > > > >> > > > >> <http://www.slideshare.net/dongjinleekr
> >*
> > > > > > > > > > > > >> > > > >>
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > --
> > > > > > > > > > > > >> > > > > *Dongjin Lee*
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > *github: <http://goog_969573159/>
> > > > > > > > github.com/dongjinleekr
> > > > > > > > > > > > >> > > > > <http://github.com/dongjinleekr>linkedin:
> > > > > > > > > > > > >> > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > >> > > > > <http://kr.linkedin.com/in/dongjinleekr
> > > > > >slideshare:
> > > > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr
> > > > > > > > > > > > >> > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > --
> > > > > > > > > > > > >> > > > *Dongjin Lee*
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > *github: <http://goog_969573159/>github
> > > > > > > .com/dongjinleekr
> > > > > > > > > > > > >> > > > <http://github.com/dongjinleekr>linkedin:
> > > > > > > > > > > > >> > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > >> > > > <http://kr.linkedin.com/in/dongjinleekr
> > > > >slideshare:
> > > > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr
> > > > > > > > > > > > >> > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> *Dongjin Lee*
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *github: <http://goog_969573159/>
> > > > github.com/dongjinleekr
> > > > > > > > > > > > >> <http://github.com/dongjinleekr>linkedin:
> > > > > > kr.linkedin.com/in/
> > > > > > > > > > > dongjinleekr
> > > > > > > > > > > > >> <http://kr.linkedin.com/in/dongjinleekr
> >slideshare:
> > > > > > > > > > > www.slideshare.net/dongjinleekr
> > > > > > > > > > > > >> <http://www.slideshare.net/dongjinleekr>*
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > > >
> > > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > >
> > > > > > > > > > > > *github: <http://goog_969573159/>
> > > github.com/dongjinleekr
> > > > > > > > > > > > <http://github.com/dongjinleekr>linkedin:
> > > > > kr.linkedin.com/in/
> > > > > > > > > > > dongjinleekr
> > > > > > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > > > > > > > > www.slideshare.net/dongjinleekr
> > > > > > > > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Dongjin Lee*
> > > > > > > > > >
> > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > >
> > > > > > > > > > *github: <http://goog_969573159/>
> github.com/dongjinleekr
> > > > > > > > > > <http://github.com/dongjinleekr>linkedin:
> > > kr.linkedin.com/in/
> > > > > > > > > dongjinleekr
> > > > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > > > > www.slideshare.net/
> > > > > > > > > dongjinleekr
> > > > > > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Dongjin Lee*
> > > > > > > >
> > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > >
> > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > > <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/
> > > > > > > dongjinleekr
> > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > > www.slideshare.net/
> > > > > > > dongjinleekr
> > > > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/
> > dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> www.slideshare.net/
> > dongjinleekr
> > > <http://www.slideshare.net/dongjinleekr>*
> > >
> >
>
