[jira] [Created] (KAFKA-14125) More senses to application

[Shyam Damodar Bodhare (Jira)]

Shyam Damodar Bodhare created KAFKA-14125:
-

 Summary: More senses to application
 Key: KAFKA-14125
 URL: https://issues.apache.org/jira/browse/KAFKA-14125
 Project: Kafka
  Issue Type: Improvement
Reporter: Shyam Damodar Bodhare


As a human being has 5 senses for sensing and different mechanisms for 
interacting with outside world.

An application should've more than one senses.

If http api is not working, alternate route like messaging should take over.

Like human body has redundancy (2 eyes, hands, legs etc.).

Not only in the event of failure, but under load conditions as well, other 
mechanisms should be brought to use.

This will mainly be useful in micro services.

 

Also micro services shouldn't directly make external web service calls.

Even though individual application can have connection pool, replicating this 
code in all enterprise applications is redundant and inefficient.

Instead, a central application (built using Kafka) can be a single point of 
interface (clustered) nto external world. Here we can adjust firewalls, system 
sockets, timeouts, request/response logging.

Individual enterprise applications will communicate with this application 
asynchronously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14120) Produce Kafka Streams Skipped Records Metrics

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna resolved KAFKA-14120.
---
Resolution: Not A Problem

> Produce Kafka Streams Skipped Records Metrics
> -
>
> Key: KAFKA-14120
> URL: https://issues.apache.org/jira/browse/KAFKA-14120
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.2.0
>Reporter: Yusu Jwa
>Priority: Minor
>
> Hi, I want to monitor "skip records" metrics and find a page that the feature 
> for Skipped Records Metrics is adopted.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-274%3A+Kafka+Streams+Skipped+Records+Metrics]
> However, there is no Skipped Records Metrics in Kafka 3.2 version.
> I found [the 
> metric|https://github.com/apache/kafka/blob/3.2/streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetrics.java#L48]
>  in source code, but it is used in only test case.
> [https://github.com/apache/kafka/blob/8464e366827d4c3a822beff32b8a0123767cbf0e/streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetrics.java#L126-L136]
> [https://github.com/apache/kafka/blob/8464e366827d4c3a822beff32b8a0123767cbf0e/streams/src/test/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetricsTest.java#L52-L68]
> Could you check it and produce the Skipped Records Metrics?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14125) More senses to application

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14125.

Resolution: Invalid

Closing as invalid. If you want to discuss microservice architectures, the 
users mailing list might be a better place.

> More senses to application
> --
>
> Key: KAFKA-14125
> URL: https://issues.apache.org/jira/browse/KAFKA-14125
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Shyam Damodar Bodhare
>Priority: Minor
>
> As a human being has 5 senses for sensing and different mechanisms for 
> interacting with outside world.
> An application should've more than one senses.
> If http api is not working, alternate route like messaging should take over.
> Like human body has redundancy (2 eyes, hands, legs etc.).
> Not only in the event of failure, but under load conditions as well, other 
> mechanisms should be brought to use.
> This will mainly be useful in micro services.
>  
> Also micro services shouldn't directly make external web service calls.
> Even though individual application can have connection pool, replicating this 
> code in all enterprise applications is redundant and inefficient.
> Instead, a central application (built using Kafka) can be a single point of 
> interface (clustered) nto external world. Here we can adjust firewalls, 
> system sockets, timeouts, request/response logging.
> Individual enterprise applications will communicate with this application 
> asynchronously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14126) Convert remaining DynamicBrokerReconfigurationTest tests to KRaft

[David Arthur (Jira)]

David Arthur created KAFKA-14126:


 Summary: Convert remaining DynamicBrokerReconfigurationTest tests 
to KRaft
 Key: KAFKA-14126
 URL: https://issues.apache.org/jira/browse/KAFKA-14126
 Project: Kafka
  Issue Type: Test
Reporter: David Arthur


After the initial conversion in https://github.com/apache/kafka/pull/12455, 
three tests still need to be converted. 

* testKeyStoreAlter
* testTrustStoreAlter
* testThreadPoolResize





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-844: Transactional State Stores

[Bruno Cadonna]

Hi Alex,

Thanks for the updates!

1. Don't you want to deprecate StateStore#flush(). As far as I 
understand, commit() is the new flush(), right? If you do not deprecate 
it, you don't get rid of the error room you describe in your KIP by 
having a flush() and a commit().



2. I would shorten Materialized#withTransactionalityEnabled() to 
Materialized#withTransactionsEnabled().



3. Could you also describe a bit more in detail where the offsets passed 
into commit() and recover() come from?



For my next two points, I need the commit workflow that you were so kind 
to post into this thread:


1. write stuff to the state store
2. producer.sendOffsetsToTransaction(token); producer.commitTransaction();
3. flush (<- that would be call to commit(), right?)
4. checkpoint


4. I do not quite understand how a state store can roll forward. You 
mention in the thread the following:


"If the crash failure happens during #3, the state store can roll 
forward and finish the flush/commit."


How does the state store know where it stopped the flushing when it 
crashed?


This seems an optimization to me. I think in general the state store 
should rollback to the last successfully committed state and restore 
from there until the end of the changelog topic partition. The last 
committed state is the offsets in the checkpoint file.



5. In the same e-mail from point 4, you also state:

"If the crash failure happens between #3 and #4, the state store should 
do nothing during recovery and just proceed with the checkpoint."


How should Streams know that the failure was between #3 and #4 during 
recovery? It just sees a valid state store and a valid checkpoint file. 
Streams does not know that the state of the checkpoint file does not 
match with the committed state of the state store.
Also, how should Streams know what to write into the checkpoint file 
after the crash?
This issue arises because we store the offset outside of the state 
store. Maybe we need an additional method on the state store interface 
that returns the offset at which the state store is.



Best,
Bruno




On 27.07.22 11:51, Alexander Sorokoumov wrote:

Hey Nick,

Thank you for the kind words and the feedback! I'll definitely add an
option to configure the transactional mechanism in Stores factory method
via an argument as John previously suggested and might add the in-memory
option via RocksDB Indexed Batches if I figure why their creation via
rocksdb jni fails with `UnsatisfiedLinkException`.

Best,
Alex

On Wed, Jul 27, 2022 at 11:46 AM Alexander Sorokoumov <
asorokou...@confluent.io> wrote:


Hey Guozhang,

1) About the param passed into the `recover()` function: it seems to me

that the semantics of "recover(offset)" is: recover this state to a
transaction boundary which is at least the passed-in offset. And the only
possibility that the returned offset is different than the passed-in
offset
is that if the previous failure happens after we've done all the commit
procedures except writing the new checkpoint, in which case the returned
offset would be larger than the passed-in offset. Otherwise it should
always be equal to the passed-in offset, is that right?



Right now, the only case when `recover` returns an offset different from
the passed one is when the failure happens *during* commit.


If the failure happens after commit but before the checkpoint, `recover`
might return either a passed or newer committed offset, depending on the
implementation. The `recover` implementation in the prototype returns a
passed offset because it deletes the commit marker that holds that offset
after the commit is done. In that case, the store will replay the last
commit from the changelog. I think it is fine as the changelog replay is
idempotent.

2) It seems the only use for the "transactional()" function is to determine

if we can update the checkpoint file while in EOS.



Right now, there are 2 other uses for `transactional()`:
1. To determine what to do during initialization if the checkpoint is gone
(see [1]). If the state store is transactional, we don't have to wipe the
existing data. Thinking about it now, we do not really need this check
whether the store is `transactional` because if it is not, we'd not have
written the checkpoint in the first place. I am going to remove that check.
2. To determine if the persistent kv store in KStreamImplJoin should be
transactional (see [2], [3]).

I am not sure if we can get rid of the checks in point 2. If so, I'd be
happy to encapsulate `transactional()` logic in `commit/recover`.

Best,
Alex

1.
https://github.com/apache/kafka/pull/12393/files#diff-971d9ef7ea8aef687fc7ee131bd166ced94445f4ab55aa83007541dccfdaL256-R281
2.
https://github.com/apache/kafka/pull/12393/files#diff-9ce43046fdef1233ab762e728abd1d3d44d7c270b28dcf6b63aa31a93a30af07R266-R278
3.
https://github.com/apache/kafka/pull/12393/files#diff-9ce43046fdef1233ab762e728abd1d3d44d7c270b28dcf6b63aa31a93a30af07R348-R354

On Tue, Jul 26, 2022 at 6:

Re: [VOTE] KIP-731: Record Rate Limiting for Kafka Connect

[Sagar]

Hey Ryan,

Just curious what happened to this KIP? I see it never got enough votes.. I
think this could be a useful feature.

Thanks!
Sagar.

On Sat, May 8, 2021 at 1:16 PM Dongjin Lee  wrote:

> Thank you for the KIP. I need this feature.
>
> +1 (non-binding)
>
> Thanks,
> Dongjin
>
> On Thu, Apr 29, 2021 at 9:34 AM Ryanne Dolan 
> wrote:
>
> > Hey y'all, I'd like to start the vote on KIP-731, which enables operators
> > to limit Connector throughput.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect
> >
> > Thanks for your votes!
> >
> > Ryanne
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Sagar]

Thanks Justine,

I thought -1 might be a good default setting for backward compatibility
reasons. I am not too adamant on it either. Probably worth mentioning in
the description that setting it to -1 would disable the feature?

Other than that, LGTM!

Thanks!
Sagar.

On Fri, Jul 29, 2022 at 8:37 AM Luke Chen  wrote:

> Hi Jason,
>
> Thanks for the info. I don't strongly insist on making the default to -1
> for backward compatibility.
> If other people in the community also agree with the change, I'm good with
> that.
>
> Thank you.
> Luke
>
>
>
>
> On Fri, Jul 29, 2022 at 5:35 AM Justine Olshan
> 
> wrote:
>
> > Thanks Jason, Luke, Sagar, and Kirk,
> >
> > Seems like there is still some debate over the default value. I think
> there
> > is a general consensus that we can reduce the default at some point, but
> > exactly when is still not clear. I do think Jason made a good point about
> > applications taking 1 day to retry. I am interested if there are other
> use
> > cases we didn't consider though.
> >
> > I've also updated the description to reference `delivery.timeout.ms.` I'm
> > not sure if we also need that config to reference this one (the
> > bi-directional reference Kirk mentioned). Let me know if something should
> > still be updated or if something is unclear.
> >
> > Thanks again,
> > Justine
> >
> > On Thu, Jul 28, 2022 at 10:46 AM Kirk True  wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. I appreciated the background context and clarity
> you
> > > added.
> > >
> > > On Wed, Jul 27, 2022, at 2:57 AM, Sagar wrote:
> > > > Thanks Justine for the KIP. I think it might be better to document
> the
> > > > correlation between the new config and delivery.timeout.ms in the
> > Public
> > > > Interfaces Description.
> > >
> > > +1.
> > >
> > > A bi-directional reference between the two configuration options would
> be
> > > great for clarity. This is especially true given that the value of `
> > > producer.id.expiration.ms, when left at -1, comes from the value of
> > > transactional.id.expiration.ms.`
> > >
> > > Thanks!
> > > Kirk
> > >
> > > >
> > > > Also, I agree with Luke that for now setting a default to -1 should
> be
> > > > good. We can look to switch to 1 day with major release.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen  wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP.
> > > > > I agree with you that we should try our best to keep backward
> > > > > compatibility, although our intention is to have lower producer id
> > > > > expiration timeout.
> > > > > So, I think we should keep default to -1 IMO.
> > > > > Maybe we change the default to 1 day in next major release (4.0)?
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Thanks for taking a look Jason!
> > > > > >
> > > > > > I wondered if we wanted to have a smaller default but wasn't sure
> > > about
> > > > > the
> > > > > > compatibility story -- especially since there is the chance for
> > > producer
> > > > > > IDs to expire silently.
> > > > > > I do think that 1 day is fairly reasonable. If I don't hear any
> > > > > conflicting
> > > > > > opinions I can go ahead and update the default.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP. Although I hate seeing new configurations,
> I
> > > think
> > > > > > this
> > > > > > > is a good change. Combining these timeout behaviors into a
> single
> > > > > > > configuration was definitely a mistake, but we didn't
> anticipate
> > > the
> > > > > > > problem with the producer id cache. I do wonder if we can make
> > the
> > > > > > default
> > > > > > > a bit lower to reduce the chances that users will hit the same
> > > memory
> > > > > > > issues we have seen. After decoupling this configuration from
> > > > > > > transactional.id.expiration.ms, the new timeout just needs to
> > > cover
> > > > > the
> > > > > > > longest duration that a producer might be retrying the same
> > Produce
> > > > > > > request. 7 days seems too high. Although I think it could go a
> > fair
> > > > > even
> > > > > > > lower, perhaps 1 day is a reasonable place to start?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > > On Mon, Jul 25, 2022 at 9:25 AM Justine Olshan
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Bill,
> > > > > > > > Thanks! I was just going to say that hopefully
> > > > > > > > transactional.id.expiration.ms would also be over the
> delivery
> > > > > > timeout.
> > > > > > > :)
> > > > > > > > Thanks for the +1!
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Mon, Jul 25, 2022 at 9:17 AM Bill Bejeck <
> bbej...@gmail.com
> > >
> 

Re: [VOTE] KIP-731: Record Rate Limiting for Kafka Connect

[Ryanne Dolan]

Thanks Sagar, I have a POC implementation, but I haven't seen much traction
here yet. Happy to pick this back up or hand it off if there is interest.

Ryanne

On Fri, Jul 29, 2022, 9:50 AM Sagar  wrote:

> Hey Ryan,
>
> Just curious what happened to this KIP? I see it never got enough votes.. I
> think this could be a useful feature.
>
> Thanks!
> Sagar.
>
> On Sat, May 8, 2021 at 1:16 PM Dongjin Lee  wrote:
>
> > Thank you for the KIP. I need this feature.
> >
> > +1 (non-binding)
> >
> > Thanks,
> > Dongjin
> >
> > On Thu, Apr 29, 2021 at 9:34 AM Ryanne Dolan 
> > wrote:
> >
> > > Hey y'all, I'd like to start the vote on KIP-731, which enables
> operators
> > > to limit Connector throughput.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect
> > >
> > > Thanks for your votes!
> > >
> > > Ryanne
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Ismael Juma]

+1 for having 1 day as the default and for including this change in the
release notes.

Ismael

On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson 
wrote:

> I don't think a major release is a requirement for a default change for
> what it's worth. I do appreciate that there is a preference for not rocking
> the boat though. For a little bit of background here, the problem we
> have encountered in production since the idempotent producer became the
> default is OOM errors due to huge numbers of producerIds that had to be
> retained in the cache for 7 days. It is hard to prevent use cases from
> emerging where producers are used and discarded rapidly. We will be using a
> lower value for sure, but it would also be nice to reduce the likelihood
> for the community to see this problem. The benefit of the caching
> diminishes quickly over time since it is primarily meant to handle producer
> retry windows. I do not think there is much difference between 1 days and 7
> days from an application perspective, but it is a huge difference for the
> broker's memory usage.
>
> Best,
> Jason
>
> On Wed, Jul 27, 2022 at 2:57 AM Sagar  wrote:
>
> > Thanks Justine for the KIP. I think it might be better to document the
> > correlation between the new config and delivery.timeout.ms in the Public
> > Interfaces Description.
> >
> > Also, I agree with Luke that for now setting a default to -1 should be
> > good. We can look to switch to 1 day with major release.
> >
> > Thanks!
> > Sagar.
> >
> > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen  wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for the KIP.
> > > I agree with you that we should try our best to keep backward
> > > compatibility, although our intention is to have lower producer id
> > > expiration timeout.
> > > So, I think we should keep default to -1 IMO.
> > > Maybe we change the default to 1 day in next major release (4.0)?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Thanks for taking a look Jason!
> > > >
> > > > I wondered if we wanted to have a smaller default but wasn't sure
> about
> > > the
> > > > compatibility story -- especially since there is the chance for
> > producer
> > > > IDs to expire silently.
> > > > I do think that 1 day is fairly reasonable. If I don't hear any
> > > conflicting
> > > > opinions I can go ahead and update the default.
> > > >
> > > > Justine
> > > >
> > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson
> > > > 
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP. Although I hate seeing new configurations, I
> > think
> > > > this
> > > > > is a good change. Combining these timeout behaviors into a single
> > > > > configuration was definitely a mistake, but we didn't anticipate
> the
> > > > > problem with the producer id cache. I do wonder if we can make the
> > > > default
> > > > > a bit lower to reduce the chances that users will hit the same
> memory
> > > > > issues we have seen. After decoupling this configuration from
> > > > > transactional.id.expiration.ms, the new timeout just needs to
> cover
> > > the
> > > > > longest duration that a producer might be retrying the same Produce
> > > > > request. 7 days seems too high. Although I think it could go a fair
> > > even
> > > > > lower, perhaps 1 day is a reasonable place to start?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Mon, Jul 25, 2022 at 9:25 AM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Hey Bill,
> > > > > > Thanks! I was just going to say that hopefully
> > > > > > transactional.id.expiration.ms would also be over the delivery
> > > > timeout.
> > > > > :)
> > > > > > Thanks for the +1!
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Jul 25, 2022 at 9:17 AM Bill Bejeck 
> > > wrote:
> > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > I just took another look at the KIP, and I realize my
> > > > > question/suggestion
> > > > > > > about default values has already been addressed in the
> > > > `Compatibility`
> > > > > > > section.
> > > > > > >
> > > > > > > I'm +1 on the KIP.
> > > > > > >
> > > > > > > -Bill
> > > > > > >
> > > > > > > On Thu, Jul 21, 2022 at 6:20 PM Bill Bejeck  >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Justine,
> > > > > > > >
> > > > > > > > Thanks for the well written KIP, this looks like it will be a
> > > > useful
> > > > > > > > addition.
> > > > > > > >
> > > > > > > > Overall the KIP looks good to me, I have one
> question/comment.
> > > > > > > >
> > > > > > > > You mentioned that setting the `producer.id.expiration.ms`
> > less
> > > > than
> > > > > > the
> > > > > > > > delivery timeout could lead to duplicates, which makes sense.
> > To
> > > > > help
> > > > > > > > avoid this situation, do we want to consider a default value
> > that
> > > > is
> > > > > > the
> > > > > > > > same as the delivery timeout?
> > > > > > > >
> > > > > > 

[GitHub] [kafka-site] bbejeck opened a new pull request, #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck opened a new pull request, #430:
URL: https://github.com/apache/kafka-site/pull/430

   This PR is a follow-up #427.  This PR adds a clickable link that will load 
an `iframe`;  in line with the [ASF privacy 
policy](https://privacy.apache.org/faq/committers.html) on embedded YouTube 
videos.  There will be a separate PR performing the same clickable image 
approach for the Kafka Streams videos.
   
   
   
   ### When the  introduction page loads:
   https://user-images.githubusercontent.com/199238/181817208-11b2c68e-db80-42bd-b90c-ecc05e461c61.png";>
   
   ### When the quickstart page loads:
   https://user-images.githubusercontent.com/199238/181817737-b5773e38-73e2-4649-a5fc-62c6096f1f94.png";>
   
   
   
   ### After clicking the image on the introduction page
   https://user-images.githubusercontent.com/199238/181817763-012224b7-6797-4a77-8f5d-7ad8048bfa9d.png";>
   
   ### After clicking the image on the quickstart page
   https://user-images.githubusercontent.com/199238/181817786-62cc9d48-973e-46d0-8487-d55a962f1197.png";>
   
   I validated the behavior by running the Kafka site locally.
   
   There are some additional times for follow-up:
   
   1.  I am continuing to look into getting the videos hosted on the ASF 
YouTube page which will take care of the branding issue.
   2. I've submitted a request to get the "What is Apache Kafka" video to 
remove the mention of Confluent.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1199820791

   ping @divijvaidya, @mimaison , and @vvcephei 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1199926659

   >One suggestion if it's not too much trouble, and if you agree... It might 
be obvious from context that that image is a button that plays a video, but it 
would be clearer if we superimposed the "play arrow" over the middle on the 
image. What do you think?
   
   I agree, but the "play arrow" on the loaded `iframe` is coming from YouTube. 
 I'm sure there's a way to do it, but that is beyond my CSS skills.  
   
   I've updated the PR to have some text instructing to click to load the 
video.  The text disappears when a user clicks the image.
   
   WDYT?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Justine Olshan]

Hi all,

I've updated the KIP to include the new default of 1 day and information
about -1 in the description of the config.
I wonder though if including -1 makes sense now that it is not the default
value. Is there a benefit for manually setting -1 vs manually setting the
value that transactional.id.expiration.ms has?

Let me know your thoughts.
Thanks,
Justine

On Fri, Jul 29, 2022 at 10:54 AM Ismael Juma  wrote:

> +1 for having 1 day as the default and for including this change in the
> release notes.
>
> Ismael
>
> On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson  >
> wrote:
>
> > I don't think a major release is a requirement for a default change for
> > what it's worth. I do appreciate that there is a preference for not
> rocking
> > the boat though. For a little bit of background here, the problem we
> > have encountered in production since the idempotent producer became the
> > default is OOM errors due to huge numbers of producerIds that had to be
> > retained in the cache for 7 days. It is hard to prevent use cases from
> > emerging where producers are used and discarded rapidly. We will be
> using a
> > lower value for sure, but it would also be nice to reduce the likelihood
> > for the community to see this problem. The benefit of the caching
> > diminishes quickly over time since it is primarily meant to handle
> producer
> > retry windows. I do not think there is much difference between 1 days
> and 7
> > days from an application perspective, but it is a huge difference for the
> > broker's memory usage.
> >
> > Best,
> > Jason
> >
> > On Wed, Jul 27, 2022 at 2:57 AM Sagar  wrote:
> >
> > > Thanks Justine for the KIP. I think it might be better to document the
> > > correlation between the new config and delivery.timeout.ms in the
> Public
> > > Interfaces Description.
> > >
> > > Also, I agree with Luke that for now setting a default to -1 should be
> > > good. We can look to switch to 1 day with major release.
> > >
> > > Thanks!
> > > Sagar.
> > >
> > > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen  wrote:
> > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP.
> > > > I agree with you that we should try our best to keep backward
> > > > compatibility, although our intention is to have lower producer id
> > > > expiration timeout.
> > > > So, I think we should keep default to -1 IMO.
> > > > Maybe we change the default to 1 day in next major release (4.0)?
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan
> > > > 
> > > > wrote:
> > > >
> > > > > Thanks for taking a look Jason!
> > > > >
> > > > > I wondered if we wanted to have a smaller default but wasn't sure
> > about
> > > > the
> > > > > compatibility story -- especially since there is the chance for
> > > producer
> > > > > IDs to expire silently.
> > > > > I do think that 1 day is fairly reasonable. If I don't hear any
> > > > conflicting
> > > > > opinions I can go ahead and update the default.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the KIP. Although I hate seeing new configurations, I
> > > think
> > > > > this
> > > > > > is a good change. Combining these timeout behaviors into a single
> > > > > > configuration was definitely a mistake, but we didn't anticipate
> > the
> > > > > > problem with the producer id cache. I do wonder if we can make
> the
> > > > > default
> > > > > > a bit lower to reduce the chances that users will hit the same
> > memory
> > > > > > issues we have seen. After decoupling this configuration from
> > > > > > transactional.id.expiration.ms, the new timeout just needs to
> > cover
> > > > the
> > > > > > longest duration that a producer might be retrying the same
> Produce
> > > > > > request. 7 days seems too high. Although I think it could go a
> fair
> > > > even
> > > > > > lower, perhaps 1 day is a reasonable place to start?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > > On Mon, Jul 25, 2022 at 9:25 AM Justine Olshan
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Bill,
> > > > > > > Thanks! I was just going to say that hopefully
> > > > > > > transactional.id.expiration.ms would also be over the delivery
> > > > > timeout.
> > > > > > :)
> > > > > > > Thanks for the +1!
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Mon, Jul 25, 2022 at 9:17 AM Bill Bejeck  >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Justine,
> > > > > > > >
> > > > > > > > I just took another look at the KIP, and I realize my
> > > > > > question/suggestion
> > > > > > > > about default values has already been addressed in the
> > > > > `Compatibility`
> > > > > > > > section.
> > > > > > > >
> > > > > > > > I'm +1 on the KIP.
> > > > > > > >
> > > > > > > > -Bill
> > > > > > > >
> > > > > > > > On Thu, Jul 21, 2022 at 6:20 PM Bill Bejeck <
> bbej...@gmail.c

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Sagar]

Hey Justine,

Thanks for that. In a way you are right, that setting the new config to -1
is equivalent to setting it equal to transactional.id.expiration.ms . The
only reason I mentioned the -1 case is it gives the users the ability to
disable this feature (which mayn't even be desirable based on what Jason
described above) and still be able to re-configure
transactional.id.expiration.ms. Otherwise, everytime users change
transactional.id.expiration.ms, they will need to also remember and change
this new config. Again I don't know how frequently this config is changed
or should users even disable this feature, but that was the reasoning I
had. If you think it isn't necessary, we can remove it.

On Sat, Jul 30, 2022 at 3:08 AM Justine Olshan 
wrote:

> Hi all,
>
> I've updated the KIP to include the new default of 1 day and information
> about -1 in the description of the config.
> I wonder though if including -1 makes sense now that it is not the default
> value. Is there a benefit for manually setting -1 vs manually setting the
> value that transactional.id.expiration.ms has?
>
> Let me know your thoughts.
> Thanks,
> Justine
>
> On Fri, Jul 29, 2022 at 10:54 AM Ismael Juma  wrote:
>
> > +1 for having 1 day as the default and for including this change in the
> > release notes.
> >
> > Ismael
> >
> > On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson
>  > >
> > wrote:
> >
> > > I don't think a major release is a requirement for a default change for
> > > what it's worth. I do appreciate that there is a preference for not
> > rocking
> > > the boat though. For a little bit of background here, the problem we
> > > have encountered in production since the idempotent producer became the
> > > default is OOM errors due to huge numbers of producerIds that had to be
> > > retained in the cache for 7 days. It is hard to prevent use cases from
> > > emerging where producers are used and discarded rapidly. We will be
> > using a
> > > lower value for sure, but it would also be nice to reduce the
> likelihood
> > > for the community to see this problem. The benefit of the caching
> > > diminishes quickly over time since it is primarily meant to handle
> > producer
> > > retry windows. I do not think there is much difference between 1 days
> > and 7
> > > days from an application perspective, but it is a huge difference for
> the
> > > broker's memory usage.
> > >
> > > Best,
> > > Jason
> > >
> > > On Wed, Jul 27, 2022 at 2:57 AM Sagar 
> wrote:
> > >
> > > > Thanks Justine for the KIP. I think it might be better to document
> the
> > > > correlation between the new config and delivery.timeout.ms in the
> > Public
> > > > Interfaces Description.
> > > >
> > > > Also, I agree with Luke that for now setting a default to -1 should
> be
> > > > good. We can look to switch to 1 day with major release.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen  wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP.
> > > > > I agree with you that we should try our best to keep backward
> > > > > compatibility, although our intention is to have lower producer id
> > > > > expiration timeout.
> > > > > So, I think we should keep default to -1 IMO.
> > > > > Maybe we change the default to 1 day in next major release (4.0)?
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Thanks for taking a look Jason!
> > > > > >
> > > > > > I wondered if we wanted to have a smaller default but wasn't sure
> > > about
> > > > > the
> > > > > > compatibility story -- especially since there is the chance for
> > > > producer
> > > > > > IDs to expire silently.
> > > > > > I do think that 1 day is fairly reasonable. If I don't hear any
> > > > > conflicting
> > > > > > opinions I can go ahead and update the default.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP. Although I hate seeing new configurations,
> I
> > > > think
> > > > > > this
> > > > > > > is a good change. Combining these timeout behaviors into a
> single
> > > > > > > configuration was definitely a mistake, but we didn't
> anticipate
> > > the
> > > > > > > problem with the producer id cache. I do wonder if we can make
> > the
> > > > > > default
> > > > > > > a bit lower to reduce the chances that users will hit the same
> > > memory
> > > > > > > issues we have seen. After decoupling this configuration from
> > > > > > > transactional.id.expiration.ms, the new timeout just needs to
> > > cover
> > > > > the
> > > > > > > longest duration that a producer might be retrying the same
> > Produce
> > > > > > > request. 7 days seems too high. Although I think it could go a
> > fair
> > > > > even
> > > > > > > lower, perhaps 1 day is a reasona

[GitHub] [kafka-site] vvcephei commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

vvcephei commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1200082801

   Oh, that works! I was just thinking of editing the placeholder image itself 
to paste a stock play arrow on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org