Unable to log in apache kafka
I'm struggling to enable logs in apache kafka and cannot get a hang of why it's failing. Pls help me out!
[DISCUSS] KIP-682: Connect TimestampConverter support for multiple fields and multiple input formats
Hello everyone! (Thanks to Tom Bentley for directing me in this direction!) I have made a series of changes to some of the standard Connect transforms to meet some of the challenges at my company to consume messages using Connect, and have been running them for a few weeks now as custom SMTs. I realized that several of these changes might actually be really good features to be included in the standard transforms so I will open some KIPs to go along with PRs which I already submitted (apologize for going a bit backwards in the process as this was my first time working with the Kafka project). The first one I have created a KIP for is KIP-682 on the TimestampConverter transform. Basically the 2 limitations that I am proposing to remove are the ability to only convert one field at a time, but instead convert multiple fields using a common config, and that if any producer sends a slightly different date/timestamp string format on any message then the transformation fails... so to instead add the ability to support variations in the string input format. More info here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-682%3A+Connect+TimestampConverter+support+for+multiple+fields+and+multiple+input+formats I would appreciate any kind of discussion to make improvements and it would be great to see changes like this being pushed up. Thanks for now! Best regards, Joshua Grisham
Re: [DISCUSS] KIP-682: Connect TimestampConverter support for multiple fields and multiple input formats
I love this idea! I have a current KIP for a hash transform and am working adding multi field/nested support to that one. I can think of a few times we’ve needed functionality like that. I’d add that adding support for transforming nested fields would be a great feature for this. -Brandon Brandon Brown > On Nov 7, 2020, at 11:11 AM, Joshua Grisham wrote: > > Hello everyone! > (Thanks to Tom Bentley for directing me in this direction!) > > I have made a series of changes to some of the standard Connect transforms > to meet some of the challenges at my company to consume messages using > Connect, and have been running them for a few weeks now as custom SMTs. > > I realized that several of these changes might actually be really good > features to be included in the standard transforms so I will open some KIPs > to go along with PRs which I already submitted (apologize for going a bit > backwards in the process as this was my first time working with the Kafka > project). > > The first one I have created a KIP for is KIP-682 on the TimestampConverter > transform. > > Basically the 2 limitations that I am proposing to remove are the ability > to only convert one field at a time, but instead convert multiple fields > using a common config, and that if any producer sends a slightly different > date/timestamp string format on any message then the transformation > fails... so to instead add the ability to support variations in the string > input format. > > More info here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-682%3A+Connect+TimestampConverter+support+for+multiple+fields+and+multiple+input+formats > > I would appreciate any kind of discussion to make improvements and it would > be great to see changes like this being pushed up. > > Thanks for now! > > Best regards, > Joshua Grisham
[jira] [Created] (KAFKA-10694) Implement zero copy for FetchSnapshot
Jose Armando Garcia Sancio created KAFKA-10694: -- Summary: Implement zero copy for FetchSnapshot Key: KAFKA-10694 URL: https://issues.apache.org/jira/browse/KAFKA-10694 Project: Kafka Issue Type: Sub-task Components: replication Reporter: Jose Armando Garcia Sancio -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] KIP-682: Connect TimestampConverter support for multiple fields and multiple input formats
Hi Brandon, I have added what i called "recursive" support to find child fields on any level to the Cast and ReplaceField transforms as well, the PR is here https://github.com/apache/kafka/pull/9493 I plan to try and write up a KIP for that one as well maybe in the next day or two. It is not exactly the same as specifically targeting one nested field based on its full path, but instead will look for the field name at any level of the structure and cast/replace all of them if they match. The same could be added in theory to almost all of the transforms i guess but maybe that is for another discussion 😊 But for now with this TimestampConverter the only thing which I have proposed are the two mentioned before - multiple fields and support for pattern like matching of string timestamps. Joshua Den lör 7 nov. 2020 18:42Brandon Brown skrev: > I love this idea! I have a current KIP for a hash transform and am working > adding multi field/nested support to that one. I can think of a few times > we’ve needed functionality like that. I’d add that adding support for > transforming nested fields would be a great feature for this. > > -Brandon > > Brandon Brown > > On Nov 7, 2020, at 11:11 AM, Joshua Grisham > wrote: > > > > Hello everyone! > > (Thanks to Tom Bentley for directing me in this direction!) > > > > I have made a series of changes to some of the standard Connect > transforms > > to meet some of the challenges at my company to consume messages using > > Connect, and have been running them for a few weeks now as custom SMTs. > > > > I realized that several of these changes might actually be really good > > features to be included in the standard transforms so I will open some > KIPs > > to go along with PRs which I already submitted (apologize for going a bit > > backwards in the process as this was my first time working with the Kafka > > project). > > > > The first one I have created a KIP for is KIP-682 on the > TimestampConverter > > transform. > > > > Basically the 2 limitations that I am proposing to remove are the ability > > to only convert one field at a time, but instead convert multiple fields > > using a common config, and that if any producer sends a slightly > different > > date/timestamp string format on any message then the transformation > > fails... so to instead add the ability to support variations in the > string > > input format. > > > > More info here: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-682%3A+Connect+TimestampConverter+support+for+multiple+fields+and+multiple+input+formats > > > > I would appreciate any kind of discussion to make improvements and it > would > > be great to see changes like this being pushed up. > > > > Thanks for now! > > > > Best regards, > > Joshua Grisham >
[jira] [Created] (KAFKA-10695) Improve interface for sensor expiration
David Mao created KAFKA-10695:
-
Summary: Improve interface for sensor expiration
Key: KAFKA-10695
URL: https://issues.apache.org/jira/browse/KAFKA-10695
Project: Kafka
Issue Type: Improvement
Components: metrics
Reporter: David Mao
To check whether a sensor has expired, a user should be able to check the
return value of {{Sensor.hasExpired()}}.{{ }}However, recording to an expired
sensor isn't prohibited, and if a user does record to an expired sensor, we can
lose the information that the sensor was expired and removed from metrics.
The way we work around this currently, is that all expiring sensors are
retrieved from {{Metrics}} each time.
A potential improvement would be to add some internal state on whether a sensor
has been expired (removed from {{Metrics)}}, and hasExpired should then return
that internal state. To do this, we would need to add a new method in Sensor
that expresses the intent of expiring a sensor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10696) Replace ProduceResponse.PartitionResponse by auto-generated PartitionProduceResponse
Chia-Ping Tsai created KAFKA-10696: -- Summary: Replace ProduceResponse.PartitionResponse by auto-generated PartitionProduceResponse Key: KAFKA-10696 URL: https://issues.apache.org/jira/browse/KAFKA-10696 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai This is a follow-up of KAFKA-9628. related discussion: https://github.com/apache/kafka/pull/9401#discussion_r518976605 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10697) Remove ProduceResponse.responses
Chia-Ping Tsai created KAFKA-10697: -- Summary: Remove ProduceResponse.responses Key: KAFKA-10697 URL: https://issues.apache.org/jira/browse/KAFKA-10697 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai This is a follow-up of KAFKA-9628. related discussion: https://github.com/apache/kafka/pull/9401#discussion_r518984349 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10698) Avoid casting BaseRecords to MemoryRecords
Chia-Ping Tsai created KAFKA-10698: -- Summary: Avoid casting BaseRecords to MemoryRecords Key: KAFKA-10698 URL: https://issues.apache.org/jira/browse/KAFKA-10698 Project: Kafka Issue Type: Improvement Reporter: Chia-Ping Tsai This is a follow-up of KAFKA-9628. related discussion: https://github.com/apache/kafka/pull/9401#discussion_r518987586 -- This message was sent by Atlassian Jira (v8.3.4#803005)
