[jira] [Updated] (KAFKA-677) Retention process gives exception if an empty segment is chosen for collection
[ https://issues.apache.org/jira/browse/KAFKA-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-677: -- Attachment: kafka_677-cleanup.patch Not sure what's causing this. However, in Log.truncateTo(), we have a couple of error cases upon which we only log an error and let it go. Those error cases will bring the log to an inconsistent state. So, it's probably better to throw a KafkaStorageException ( the exception will propagate all the way to KafkaApis.handleLeaderAndIsrRequest() and cause the broker to shut down) instead. Attach a patch to clean this up. > Retention process gives exception if an empty segment is chosen for collection > -- > > Key: KAFKA-677 > URL: https://issues.apache.org/jira/browse/KAFKA-677 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8 > > Attachments: kafka_677-cleanup.patch > > > java.io.FileNotFoundException: > /mnt/u001/kafka_08_long_running_test/kafka-logs/NewsActivityEvent-3/.index > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:212) > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:244) > at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:233) > at kafka.log.Log.rollToOffset(Log.scala:459) > at kafka.log.Log.roll(Log.scala:443) > at kafka.log.Log.markDeletedWhile(Log.scala:395) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:241) > at > kafka.log.LogManager$$anonfun$cleanupLogs$2.apply(LogManager.scala:277) > at > kafka.log.LogManager$$anonfun$cleanupLogs$2.apply(LogManager.scala:275) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) > at > scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:275) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:141) > at kafka.utils.Utils$$anon$2.run(Utils.scala:66) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-678) We should default to NullEncoder for producer
Jay Kreps created KAFKA-678: --- Summary: We should default to NullEncoder for producer Key: KAFKA-678 URL: https://issues.apache.org/jira/browse/KAFKA-678 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jay Kreps Assignee: Jay Kreps Currently we default to using whatever serializer you set for your value to also work for your key. This works if the serializer is of a generic sort (Avro, Java serialization, etc) and both key and value map into this. However if you have a custom serializer this is not the right thing to do. I think it would be better to default this to NullEncoder which defaults to maintain the pre-0.8 behavior of not retaining the key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-677) Retention process gives exception if an empty segment is chosen for collection
[ https://issues.apache.org/jira/browse/KAFKA-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538269#comment-13538269 ] Jay Kreps commented on KAFKA-677: - The first case makes sense. However the intended behavior for truncateTo(offset) is to ensure that logEndOffset < offset. So if this postcondition is trivially satisfied I don't think this is an error, just a no-op. This is important because I think eventually we should expose truncate as a public api and we don't want that shutting down the broker. > Retention process gives exception if an empty segment is chosen for collection > -- > > Key: KAFKA-677 > URL: https://issues.apache.org/jira/browse/KAFKA-677 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps >Assignee: Jay Kreps > Fix For: 0.8 > > Attachments: kafka_677-cleanup.patch > > > java.io.FileNotFoundException: > /mnt/u001/kafka_08_long_running_test/kafka-logs/NewsActivityEvent-3/.index > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:212) > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:244) > at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:233) > at kafka.log.Log.rollToOffset(Log.scala:459) > at kafka.log.Log.roll(Log.scala:443) > at kafka.log.Log.markDeletedWhile(Log.scala:395) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:241) > at > kafka.log.LogManager$$anonfun$cleanupLogs$2.apply(LogManager.scala:277) > at > kafka.log.LogManager$$anonfun$cleanupLogs$2.apply(LogManager.scala:275) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) > at > scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:275) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:141) > at kafka.utils.Utils$$anon$2.run(Utils.scala:66) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-648) Use uniform convention for naming properties keys
[
https://issues.apache.org/jira/browse/KAFKA-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapnil Ghike updated KAFKA-648:
Summary: Use uniform convention for naming properties keys (was: Use
uniform convention for naming properties keys for clientId, groupId,
consumerId, correlationId)
> Use uniform convention for naming properties keys
> --
>
> Key: KAFKA-648
> URL: https://issues.apache.org/jira/browse/KAFKA-648
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Priority: Blocker
> Fix For: 0.8.1
>
>
> Currently, the convention that we seem to use to get a property value in
> *Config is as follows:
> val configVal = property.getType("config.val", ...) // dot is used to
> separate two words in the key and the first letter of second word is
> capitalized in configVal.
> We should use similar convention for groupId, consumerId, clientId,
> correlationId.
> This change will probably be backward non-compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-648) Use uniform convention for naming properties keys
[
https://issues.apache.org/jira/browse/KAFKA-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede reassigned KAFKA-648:
---
Assignee: Sriram Subramanian
> Use uniform convention for naming properties keys
> --
>
> Key: KAFKA-648
> URL: https://issues.apache.org/jira/browse/KAFKA-648
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Sriram Subramanian
>Priority: Blocker
> Fix For: 0.8.1
>
>
> Currently, the convention that we seem to use to get a property value in
> *Config is as follows:
> val configVal = property.getType("config.val", ...) // dot is used to
> separate two words in the key and the first letter of second word is
> capitalized in configVal.
> We should use similar convention for groupId, consumerId, clientId,
> correlationId.
> This change will probably be backward non-compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data
[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538606#comment-13538606 ] Jay Kreps commented on KAFKA-671: - I am concerned this may be a blocker for production usage. If this adds 200k per request with 5000 outstanding requests that is 1G of memory. This is not too far out there for production usage with a high number of producers. > DelayedProduce requests should not hold full producer request data > -- > > Key: KAFKA-671 > URL: https://issues.apache.org/jira/browse/KAFKA-671 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Joel Koshy > Fix For: 0.8.1 > > > Per summary, this leads to unnecessary memory usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-668) Controlled shutdown admin tool should not require controller JMX url/port to be supplied
[ https://issues.apache.org/jira/browse/KAFKA-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-668: - Attachment: KAFKA-668-v1.patch This is a pretty straightforward change. It's slightly hacky in that I'm appending the :jmxport to the zk string, and it is effectively ignored in the Broker class. I preferred this over adding a jmxPort field to the Broker class as that would be cause wide-spread edits. > Controlled shutdown admin tool should not require controller JMX url/port to > be supplied > > > Key: KAFKA-668 > URL: https://issues.apache.org/jira/browse/KAFKA-668 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8, 0.8.1 >Reporter: Joel Koshy > Fix For: 0.8 > > Attachments: KAFKA-668-v1.patch > > > The controlled shutdown admin command takes a zookeeper string and also > requires the user to supply the controller's jmx url/port. This is a bit > annoying since the purpose of the zookeeper string is to discover the > controller. The tool should require exactly one of these options. If > zookeeper is supplied then discover the controller and its jmx port (which > means we will need to add the jmx port information to zk). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-657) Add an API to commit offsets
[ https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-657: --- Attachment: KAFKA-657v5.patch Attaching patch v5. Added ASL header and javaapi. Not really sure how to test the javaapi wrapper - tried writing a Java junit test but SBT was not too happy about that. Jun, regarding point 42, I'm not sure what you mean, I have the "case e => ..." in both handle functions. What more do I need to add for exception handling here? Jay, this does not include the metadata field. I will work on that next. > Add an API to commit offsets > > > Key: KAFKA-657 > URL: https://issues.apache.org/jira/browse/KAFKA-657 > Project: Kafka > Issue Type: New Feature >Reporter: Jay Kreps > Labels: project > Attachments: KAFKA-657v1.patch, KAFKA-657v2.patch, KAFKA-657v3.patch, > KAFKA-657v4.patch, KAFKA-657v5.patch > > > Currently the consumer directly writes their offsets to zookeeper. Two > problems with this: (1) This is a poor use of zookeeper, and we need to > replace it with a more scalable offset store, and (2) it makes it hard to > carry over to clients in other languages. A first step towards accomplishing > that is to add a proper Kafka API for committing offsets. The initial version > of this would just write to zookeeper as we do today, but in the future we > would then have the option of changing this. > This api likely needs to take a sequence of > consumer-group/topic/partition/offset entries and commit them all. > It would be good to do a wiki design on how this would work and consensus on > that first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
