Regarding the NULLs not being deleted - I saw this
https://issues.apache.org/jira/browse/KAFKA-8522 which might explain this
case
On Sun, Dec 6, 2020 at 3:02 PM Nitay Kufert <nita...@ironsrc.com> wrote:
> Hey,
> First of all I want to apologize for thinking our own implementation
> of StateRestoreListener was a default log (got confused there).
> This is our implementation (which shows its not really the # of object,
> just the diff in offsets)
>
>>
>
> override def onRestoreStart(topicPartition: TopicPartition,
>
> storeName: String,
>
> startingOffset: Long,
>
> endingOffset: Long): Unit = {
>
> logger.info(
>
> "Started restoration of partition",
>
> "storeName" -> storeName,
>
> "partition" -> topicPartition.partition.toString,
>
> "totalRecordsToBeRestored" -> (endingOffset -
>> startingOffset).toString
>
> )
>
>
> Another thing i've noticed when checking the "problematic" topic was that
> is has many many NULL values which i assumed need to be deleted (because of
> *delete.retention.ms
> <http://delete.retention.ms>* )
>
> Sorry again for the confusion :)
>
> On Wed, Dec 2, 2020 at 12:53 AM Guozhang Wang <wangg...@gmail.com> wrote:
>
>> Hello Nitay,
>>
>> Sorry for the late reply.
>>
>> Would you be able to share the actual log entries from where you infers
>> the
>> elapsed time and the total number of records restored?
>>
>>
>> Guozhang
>>
>>
>> On Tue, Nov 24, 2020 at 3:44 AM Nitay Kufert <nita...@ironsrc.com> wrote:
>>
>> > Hey,
>> > I get the log *after* the restart was triggered for my app (and my app
>> > actually restarted, meaning i get it as part of my app bootstrap
>> logging)
>> >
>> > On Tue, Nov 24, 2020 at 12:03 AM Guozhang Wang <wangg...@gmail.com>
>> wrote:
>> >
>> > > Hello Nitay,
>> > >
>> > > Thanks for letting us know about your observations. When you restart
>> the
>> > > application from empty local state Kafka Streams will try to restore
>> all
>> > > the records up to the current log end for those global KTables, and
>> > during
>> > > this period of time there should be no processing.
>> > >
>> > > Do you mind sharing where you get the "totalRecordsToBeRestored", is
>> it
>> > > before the restarting was triggered?
>> > >
>> > > Guozhang
>> > >
>> > > On Mon, Nov 23, 2020 at 4:15 AM Nitay Kufert <nita...@ironsrc.com>
>> > wrote:
>> > >
>> > > > Hey all,
>> > > > We have been running a kafka-stream based service in production for
>> the
>> > > > last couple of years (we have 4 brokers on this specific cluster).
>> > > > Inside this service, which does a lot of stuff - there are 2
>> > > GlobalKTables
>> > > > (which are based on 2 compacted topics).
>> > > >
>> > > > When I restart my app and clean the local state - the restoration of
>> > > those
>> > > > topics begins, and the weird thing I am noticing is that for *much
>> > *fewer
>> > > > messages, 1 topic takes *a lot* more time to complete.
>> > > >
>> > > > Let's call the compacted topics A and B, both are compacted and
>> mainly
>> > > > configured the same:
>> > > > Replication 3
>> > > > Number of Partitions 36
>> > > >
>> > > > max.message.bytes 25000000
>> > > > segment.index.bytes 2097152
>> > > > min.cleanable.dirty.ratio 0.1
>> > > > cleanup.policy compact
>> > > > delete.retention.ms 900000
>> > > > segment.bytes 52428800
>> > > > segment.ms 3600000
>> > > > *WIth the exception, that for topic B, we use *
>> > > > cleanup.policy compact,delete
>> > > > retention.ms 604800000
>> > > > *The behaviour i've noticed for a single partition:*
>> > > > Topic A has 62,552 totalRecordsToBeRestored - and it takes around
>> 20s
>> > > > Topic B has 24,730,506 totalRecordsToBeRestored - and it takes
>> around
>> > 1s
>> > > >
>> > > > It is worth mentioning that the data that B holds for each record
>> *is
>> > > much
>> > > > much bigger *(A holds an integer while B holds a big object)*.*
>> > > >
>> > > > Now, I get the feeling that the reason is because the data in B is
>> > always
>> > > > relatively "fresh", while the data in topic A is mostly stale (the
>> > > business
>> > > > logic behind it suggests that topic A updates at a very low rate -
>> and
>> > a
>> > > > lot of keys would never be updated)
>> > > > So, for example, it's probably holding keys that haven't been
>> updated
>> > > since
>> > > > 2018.
>> > > > Topic B keeps getting updated every couple of milliseconds.
>> > > >
>> > > > Another difference that I just realized i might share is that topic
>> A
>> > is
>> > > > being "joined" by 6 other streams, while topic B is only being
>> joined
>> > by
>> > > 2.
>> > > >
>> > > > I find it hard to explain the relation between keeping "old" records
>> > and
>> > > > the huge difference in number of records and their size.
>> > > >
>> > > > So I guess I am missing some basic concept when it comes to
>> > understanding
>> > > > compacted topics and the way the broker saves and fetches the data
>> OR
>> > > maybe
>> > > > we have some underlying problem which can explain it.
>> > > >
>> > > > Let me know if you need some more info
>> > > >
>> > > > Thanks!
>> > > >
>> > > > --
>> > > >
>> > > > Nitay Kufert
>> > > > Backend Team Leader
>> > > > [image: ironSource] <http://www.ironsrc.com>
>> > > >
>> > > > email nita...@ironsrc.com
>> > > > mobile +972-54-5480021
>> > > > fax +972-77-5448273
>> > > > skype nitay.kufert.ssa
>> > > > 121 Menachem Begin St., Tel Aviv, Israel
>> > > > ironsrc.com <http://www.ironsrc.com>
>> > > > [image: linkedin] <https://www.linkedin.com/company/ironsource>
>> > [image:
>> > > > twitter] <https://twitter.com/ironsource> [image: facebook]
>> > > > <https://www.facebook.com/ironSource> [image: googleplus]
>> > > > <https://plus.google.com/+ironsrc>
>> > > > This email (including any attachments) is for the sole use of the
>> > > intended
>> > > > recipient and may contain confidential information which may be
>> > protected
>> > > > by legal privilege. If you are not the intended recipient, or the
>> > > employee
>> > > > or agent responsible for delivering it to the intended recipient,
>> you
>> > are
>> > > > hereby notified that any use, dissemination, distribution or
>> copying of
>> > > > this communication and/or its content is strictly prohibited. If you
>> > are
>> > > > not the intended recipient, please immediately notify us by reply
>> email
>> > > or
>> > > > by telephone, delete this email and destroy any copies. Thank you.
>> > > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>> >
>> > --
>> >
>> > Nitay Kufert
>> > Backend Team Leader
>> > [image: ironSource] <http://www.ironsrc.com>
>> >
>> > email nita...@ironsrc.com
>> > mobile +972-54-5480021
>> > fax +972-77-5448273
>> > skype nitay.kufert.ssa
>> > 121 Menachem Begin St., Tel Aviv, Israel
>> > ironsrc.com <http://www.ironsrc.com>
>> > [image: linkedin] <https://www.linkedin.com/company/ironsource> [image:
>> > twitter] <https://twitter.com/ironsource> [image: facebook]
>> > <https://www.facebook.com/ironSource> [image: googleplus]
>> > <https://plus.google.com/+ironsrc>
>> > This email (including any attachments) is for the sole use of the
>> intended
>> > recipient and may contain confidential information which may be
>> protected
>> > by legal privilege. If you are not the intended recipient, or the
>> employee
>> > or agent responsible for delivering it to the intended recipient, you
>> are
>> > hereby notified that any use, dissemination, distribution or copying of
>> > this communication and/or its content is strictly prohibited. If you are
>> > not the intended recipient, please immediately notify us by reply email
>> or
>> > by telephone, delete this email and destroy any copies. Thank you.
>> >
>>
>>
>> --
>> -- Guozhang
>>
>
>
> --
>
> Nitay Kufert
> Backend Team Leader
> [image: ironSource] <http://www.ironsrc.com>
>
> email nita...@ironsrc.com
> mobile +972-54-5480021
> fax +972-77-5448273
> skype nitay.kufert.ssa
> 121 Menachem Begin St., Tel Aviv, Israel
> ironsrc.com <http://www.ironsrc.com>
> [image: linkedin] <https://www.linkedin.com/company/ironsource> [image:
> twitter] <https://twitter.com/ironsource> [image: facebook]
> <https://www.facebook.com/ironSource> [image: googleplus]
> <https://plus.google.com/+ironsrc>
> This email (including any attachments) is for the sole use of the intended
> recipient and may contain confidential information which may be protected
> by legal privilege. If you are not the intended recipient, or the employee
> or agent responsible for delivering it to the intended recipient, you are
> hereby notified that any use, dissemination, distribution or copying of
> this communication and/or its content is strictly prohibited. If you are
> not the intended recipient, please immediately notify us by reply email or
> by telephone, delete this email and destroy any copies. Thank you.
>
--
Nitay Kufert
Backend Team Leader
[image: ironSource] <http://www.ironsrc.com>
email nita...@ironsrc.com
mobile +972-54-5480021
fax +972-77-5448273
skype nitay.kufert.ssa
121 Menachem Begin St., Tel Aviv, Israel
ironsrc.com <http://www.ironsrc.com>
[image: linkedin] <https://www.linkedin.com/company/ironsource> [image:
twitter] <https://twitter.com/ironsource> [image: facebook]
<https://www.facebook.com/ironSource> [image: googleplus]
<https://plus.google.com/+ironsrc>
This email (including any attachments) is for the sole use of the intended
recipient and may contain confidential information which may be protected
by legal privilege. If you are not the intended recipient, or the employee
or agent responsible for delivering it to the intended recipient, you are
hereby notified that any use, dissemination, distribution or copying of
this communication and/or its content is strictly prohibited. If you are
not the intended recipient, please immediately notify us by reply email or
by telephone, delete this email and destroy any copies. Thank you.
