[jira] [Updated] (KAFKA-1109) Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS

[Viktor Kolodrevskiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Kolodrevskiy updated KAFKA-1109:
---

Affects Version/s: 0.8.1

> Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS
> -
>
> Key: KAFKA-1109
> URL: https://issues.apache.org/jira/browse/KAFKA-1109
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8, 0.8.1
> Environment: *nix
>Reporter: Viktor Kolodrevskiy
>
> kafka-run-class.sh contains GC log code:
> # GC options
> GC_FILE_SUFFIX='-gc.log'
> GC_LOG_FILE_NAME=''
> if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS"] ; then
>   shift
>   GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
>   shift
>   KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
> fi
> So when in my scripts I start kafka and want to override KAFKA_GC_LOG_OPTS by 
> exporting new values I get:
> Exception in thread "main" java.lang.NoClassDefFoundError: daemon
> Caused by: java.lang.ClassNotFoundException: daemon
> That's because shift is not done when KAFKA_GC_LOG_OPTS is set and "daemon" 
> is passed as main class.
> I suggest to replace it with this code:
> # GC options
> GC_FILE_SUFFIX='-gc.log'
> GC_LOG_FILE_NAME=''
> if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS" ] ; then
>   shift
>   GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
>   shift
>   KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
> else
> if [ "$1" = "daemon" ] && [ "$KAFKA_GC_LOG_OPTS" != "" ] ; then
>   shift 2
> fi
> fi



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Apache Kafka Release 0.8.0 - Candidate 2

[Jun Rao]

+1. Verified quickstart and unit tests.

Thanks,

Jun


On Sun, Nov 3, 2013 at 5:55 PM, Joe Stein  wrote:

> Hello,
>
> This is the second candidate for release of Apache Kafka 0.8.0.   The only
> change to this release candidate from the last is that this builds against
> Java 6 as expected.
>
> Release Notes for the 0.8.0 release
>
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday November, 7th, 6pm PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to
> the md5 and sha1 checksum
>
> * Release artifacts to be voted upon (source and binary):
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> (i.e. in sbt land this can be added to the build.sbt to use Kafka
> resolvers += "Apache Staging" at "
> https://repository.apache.org/content/groups/staging/";
> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"
> )
>
> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bcbe8de95131259dd9167ffcfbfe0a798b5ec2d2
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

[jira] [Commented] (KAFKA-1118) /solr/select/ returned fields

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813971#comment-13813971
 ] 

Jun Rao commented on KAFKA-1118:


Is this created by mistake? This seems like a solr jira.

> /solr/select/ returned fields
> -
>
> Key: KAFKA-1118
> URL: https://issues.apache.org/jira/browse/KAFKA-1118
> Project: Kafka
>  Issue Type: Bug
> Environment: solr-spec
> 4.5.1
> solr-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:44:41
> lucene-spec
> 4.5.1
> lucene-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:40:03
>Reporter: antoine s
>Priority: Minor
>
> I have a field in schema.xml
>  multiValued="true"/>
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> does not return the 'description'
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> returns it



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Reopened] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede reopened KAFKA-1119:
--


Yes, removing overrides is an issue today. I like option #1 where no value 
means delete the per-topic override.

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Blocker
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1119:
-

Priority: Critical  (was: Blocker)

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1120) Controller could miss a broker state change

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814096#comment-13814096
 ] 

Jun Rao commented on KAFKA-1120:


The root cause of this issue is that the controller missed a broker state 
change that it should have seen and therefore didn't make the correct decision. 
One way to fix that is for the controller to store the creation time of a 
broker registration. That way, on a broker change event, the controller can see 
if there has been any broker whose registration time has changed. We can then 
force a leader election on affected partitions accordingly. 

> Controller could miss a broker state change 
> 
>
> Key: KAFKA-1120
> URL: https://issues.apache.org/jira/browse/KAFKA-1120
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>
> When the controller is in the middle of processing a task (e.g., preferred 
> leader election, broker change), it holds a controller lock. During this 
> time, a broker could have de-registered and re-registered itself in ZK. After 
> the controller finishes processing the current task, it will start processing 
> the logic in the broker change listener. However, it will see no broker 
> change and therefore won't do anything to the restarted broker. This broker 
> will be in a weird state since the controller doesn't inform it to become the 
> leader of any partition. Yet, the cached metadata in other brokers could 
> still list that broker as the leader for some partitions. Client requests 
> routed to that broker will then get a TopicOrPartitionNotExistException. This 
> broker will continue to be in this bad state until it's restarted again.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1120) Controller could miss a broker state change

[Jun Rao (JIRA)]

Jun Rao created KAFKA-1120:
--

 Summary: Controller could miss a broker state change 
 Key: KAFKA-1120
 URL: https://issues.apache.org/jira/browse/KAFKA-1120
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
Reporter: Jun Rao


When the controller is in the middle of processing a task (e.g., preferred 
leader election, broker change), it holds a controller lock. During this time, 
a broker could have de-registered and re-registered itself in ZK. After the 
controller finishes processing the current task, it will start processing the 
logic in the broker change listener. However, it will see no broker change and 
therefore won't do anything to the restarted broker. This broker will be in a 
weird state since the controller doesn't inform it to become the leader of any 
partition. Yet, the cached metadata in other brokers could still list that 
broker as the leader for some partitions. Client requests routed to that broker 
will then get a TopicOrPartitionNotExistException. This broker will continue to 
be in this bad state until it's restarted again.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15201: tool for checking the consistency among replicas

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/#review28186
---



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Default max.message.size is (1MB + header). Default fetch size should be a 
little larger than 1MB. How about 1.1 MB?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


This can throw an exception if the leader doesn't exist

Replicas that don't have a leader when this tool is run cannot set the 
initial offset. Should we skip such partitions?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


It will be useful to know the fetch offset of the chunk that is undergoing 
verification. 



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


map{ -> map {



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


It will be good to include a relevant message with every assert



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Let's add a comment explaining what this loop does. 



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


This is not a messageSet but a message. Can we rename this to 
messageInfoFromFirstReplica ?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


What happens if one replica returns empty message set because it is not 
caught up and other replicas return some valid messages? Then ideally the fetch 
offset should not be updated since we need to repeat the verification for that 
fetch offset



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Let's rename isDone to something meaningful. It is a little confusing that 
we update the fetch offset even when isDone is false.


- Neha Narkhede


On Nov. 4, 2013, 12:34 a.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15201/
> ---
> 
> (Updated Nov. 4, 2013, 12:34 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1117
> https://issues.apache.org/jira/browse/KAFKA-1117
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> kafka-1117
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814110#comment-13814110
 ] 

Neha Narkhede commented on KAFKA-1112:
--

Stack trace -

[2013-11-01 17:46:02,685] INFO Loading log 'foo-4' (kafka.log.LogManager)
[2013-11-01 17:46:04,898] FATAL Fatal error during KafkaServerStable startup. 
Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.IllegalArgumentException: requirement failed: Corrupt index found, 
index file (/mnt/u001/temp/kafka-logs/foo-4/.index) has 
non-zero size but the last offset is 0 and the base offset is 0
at scala.Predef$.require(Predef.scala:145)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:161)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:160)
at scala.collection.Iterator$class.foreach(Iterator.scala:631)
at 
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaCo

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Guozhang Wang
>Priority: Critical
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1112:
-

Assignee: Jay Kreps  (was: Guozhang Wang)

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Reopened] (KAFKA-1116) Need to upgrade sbt-assembly to compile on scala 2.10.2

[Kane Kim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kane Kim reopened KAFKA-1116:
-


Attaching a patch

> Need to upgrade sbt-assembly to compile on scala 2.10.2
> ---
>
> Key: KAFKA-1116
> URL: https://issues.apache.org/jira/browse/KAFKA-1116
> Project: Kafka
>  Issue Type: Bug
>Reporter: Kane Kim
>Priority: Minor
>
> Need to upgrade sbt-assembly to 0.9.0 compile on scala 2.10.2



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814134#comment-13814134
 ] 

Guozhang Wang commented on KAFKA-1112:
--

Attached another stack trace with more debugging turned on. The root cause is 
that when we load the index file, the initial size is set to the limit of the 
file, and hence the position is pointing to the last entry. In most cases the 
last entry will be 0, and the recoveryLog process will skip since it shows the 
latest offset is smaller than the checkpoint. But when doing the sanity check 
it finds the number of entries is non-zero (actually the max number of entries) 
while the last offset is equal to the base offset since reading the last entry 
gives you 0 relative offset.

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1116) Need to upgrade sbt-assembly to compile on scala 2.10.2

[Kane Kim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kane Kim updated KAFKA-1116:


Attachment: kafka-1116-scala.2.10.2.patch

Upgrading sbt-assembly

> Need to upgrade sbt-assembly to compile on scala 2.10.2
> ---
>
> Key: KAFKA-1116
> URL: https://issues.apache.org/jira/browse/KAFKA-1116
> Project: Kafka
>  Issue Type: Bug
>Reporter: Kane Kim
>Priority: Minor
> Attachments: kafka-1116-scala.2.10.2.patch
>
>
> Need to upgrade sbt-assembly to 0.9.0 compile on scala 2.10.2



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1112:
-

Attachment: KAFKA-1112.out

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1116) Need to upgrade sbt-assembly to compile on scala 2.10.2

[Kane Kim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kane Kim updated KAFKA-1116:


Fix Version/s: 0.8.1
   0.8
   Labels: build  (was: )
Affects Version/s: 0.8.1
   0.8
   Status: Patch Available  (was: Reopened)

> Need to upgrade sbt-assembly to compile on scala 2.10.2
> ---
>
> Key: KAFKA-1116
> URL: https://issues.apache.org/jira/browse/KAFKA-1116
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Priority: Minor
>  Labels: build
> Fix For: 0.8, 0.8.1
>
> Attachments: kafka-1116-scala.2.10.2.patch
>
>
> Need to upgrade sbt-assembly to 0.9.0 compile on scala 2.10.2



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1118) /solr/select/ returned fields

[antoine s (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814220#comment-13814220
 ] 

antoine s commented on KAFKA-1118:
--

yes solr. not sure how i ended up here.. sorry

> /solr/select/ returned fields
> -
>
> Key: KAFKA-1118
> URL: https://issues.apache.org/jira/browse/KAFKA-1118
> Project: Kafka
>  Issue Type: Bug
> Environment: solr-spec
> 4.5.1
> solr-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:44:41
> lucene-spec
> 4.5.1
> lucene-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:40:03
>Reporter: antoine s
>Priority: Minor
>
> I have a field in schema.xml
>  multiValued="true"/>
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> does not return the 'description'
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> returns it



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1118) /solr/select/ returned fields

[antoine s (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

antoine s resolved KAFKA-1118.
--

Resolution: Invalid

> /solr/select/ returned fields
> -
>
> Key: KAFKA-1118
> URL: https://issues.apache.org/jira/browse/KAFKA-1118
> Project: Kafka
>  Issue Type: Bug
> Environment: solr-spec
> 4.5.1
> solr-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:44:41
> lucene-spec
> 4.5.1
> lucene-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:40:03
>Reporter: antoine s
>Priority: Minor
>
> I have a field in schema.xml
>  multiValued="true"/>
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> does not return the 'description'
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> returns it



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Closed] (KAFKA-1118) /solr/select/ returned fields

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1118.



> /solr/select/ returned fields
> -
>
> Key: KAFKA-1118
> URL: https://issues.apache.org/jira/browse/KAFKA-1118
> Project: Kafka
>  Issue Type: Bug
> Environment: solr-spec
> 4.5.1
> solr-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:44:41
> lucene-spec
> 4.5.1
> lucene-impl
> 4.5.1 1533280 - mark - 2013-10-17 21:40:03
>Reporter: antoine s
>Priority: Minor
>
> I have a field in schema.xml
>  multiValued="true"/>
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> does not return the 'description'
> http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
> returns it



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Swapnil Ghike (JIRA)]

Swapnil Ghike created KAFKA-1121:


 Summary: DumpLogSegments tool should print absolute file name to 
report inconsistencies
 Key: KAFKA-1121
 URL: https://issues.apache.org/jira/browse/KAFKA-1121
 Project: Kafka
  Issue Type: Bug
Reporter: Swapnil Ghike
Assignee: Swapnil Ghike


Normally, the user would know where the index file lies. But in case of a 
script that continuously checks the index files for consistency, it will help 
to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Swapnil Ghike (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1121:
-

Affects Version/s: 0.8

> DumpLogSegments tool should print absolute file name to report inconsistencies
> --
>
> Key: KAFKA-1121
> URL: https://issues.apache.org/jira/browse/KAFKA-1121
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Fix For: 0.8.1
>
>
> Normally, the user would know where the index file lies. But in case of a 
> script that continuously checks the index files for consistency, it will help 
> to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Swapnil Ghike (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1121:
-

Fix Version/s: 0.8.1

> DumpLogSegments tool should print absolute file name to report inconsistencies
> --
>
> Key: KAFKA-1121
> URL: https://issues.apache.org/jira/browse/KAFKA-1121
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Fix For: 0.8.1
>
>
> Normally, the user would know where the index file lies. But in case of a 
> script that continuously checks the index files for consistency, it will help 
> to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Review Request 15248: DumpLogSegments should print absolute file path while printing errors

[Swapnil Ghike]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15248/
---

Review request for kafka.


Bugs: KAFKA-1121
https://issues.apache.org/jira/browse/KAFKA-1121


Repository: kafka


Description
---

dumplogseg


Diffs
-

  core/src/main/scala/kafka/tools/DumpLogSegments.scala 
89b6cb1d0c3d9a1335184d0fc778246ce47738d3 

Diff: https://reviews.apache.org/r/15248/diff/


Testing
---


Thanks,

Swapnil Ghike

[jira] [Commented] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Swapnil Ghike (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814308#comment-13814308
 ] 

Swapnil Ghike commented on KAFKA-1121:
--

Created reviewboard https://reviews.apache.org/r/15248/
 against branch origin/trunk

> DumpLogSegments tool should print absolute file name to report inconsistencies
> --
>
> Key: KAFKA-1121
> URL: https://issues.apache.org/jira/browse/KAFKA-1121
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Fix For: 0.8.1
>
> Attachments: KAFKA-1121.patch
>
>
> Normally, the user would know where the index file lies. But in case of a 
> script that continuously checks the index files for consistency, it will help 
> to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Swapnil Ghike (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1121:
-

Attachment: KAFKA-1121.patch

> DumpLogSegments tool should print absolute file name to report inconsistencies
> --
>
> Key: KAFKA-1121
> URL: https://issues.apache.org/jira/browse/KAFKA-1121
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Fix For: 0.8.1
>
> Attachments: KAFKA-1121.patch
>
>
> Normally, the user would know where the index file lies. But in case of a 
> script that continuously checks the index files for consistency, it will help 
> to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15201: tool for checking the consistency among replicas

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/#review28162
---



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


We do not need () after withRequiredArg here.



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Ditto as above for these three.



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Why do we ask for all topics from brokers then filter them? Is this because 
the start-up brokers may not have metadata info for all the topics yet?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


In think in this case the verification would stop and probably report the 
max lag for this topic-partition. This is also fine I think.



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


createNewVerificationBarrier here will set the count to 1, and then count 
it down to 0. So the next round the verification barrier count would be 1 and 
hence not holding other threads?


- Guozhang Wang


On Nov. 4, 2013, 12:34 a.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15201/
> ---
> 
> (Updated Nov. 4, 2013, 12:34 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1117
> https://issues.apache.org/jira/browse/KAFKA-1117
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> kafka-1117
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 15248: DumpLogSegments should print absolute file path while printing errors

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15248/#review28234
---

Ship it!



core/src/main/scala/kafka/tools/DumpLogSegments.scala


Good catch!


- Guozhang Wang


On Nov. 5, 2013, 10:08 p.m., Swapnil Ghike wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15248/
> ---
> 
> (Updated Nov. 5, 2013, 10:08 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1121
> https://issues.apache.org/jira/browse/KAFKA-1121
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> dumplogseg
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/tools/DumpLogSegments.scala 
> 89b6cb1d0c3d9a1335184d0fc778246ce47738d3 
> 
> Diff: https://reviews.apache.org/r/15248/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Swapnil Ghike
> 
>

Re: Review Request 15248: DumpLogSegments should print absolute file path while printing errors

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15248/#review28238
---

Ship it!


Ship It!

- Neha Narkhede


On Nov. 5, 2013, 10:08 p.m., Swapnil Ghike wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15248/
> ---
> 
> (Updated Nov. 5, 2013, 10:08 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1121
> https://issues.apache.org/jira/browse/KAFKA-1121
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> dumplogseg
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/tools/DumpLogSegments.scala 
> 89b6cb1d0c3d9a1335184d0fc778246ce47738d3 
> 
> Diff: https://reviews.apache.org/r/15248/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Swapnil Ghike
> 
>

[jira] [Created] (KAFKA-1122) Kafka can log giant log lines

[Jason Rosenberg (JIRA)]

Jason Rosenberg created KAFKA-1122:
--

 Summary: Kafka can log giant log lines
 Key: KAFKA-1122
 URL: https://issues.apache.org/jira/browse/KAFKA-1122
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8
Reporter: Jason Rosenberg
Priority: Minor


There are a number of log lines that the kafka server, and high-level consumer, 
can log, that can end up becoming a giant log line.  This can be cumbersome to 
deal with in a log file.

This happens in my case as I have have a large number of topics (on the order 
of 500-700 topics).  Typically, these giant log lines will say something 
separately about every topic on the broker.  An example:

2013-11-04 23:28:11,148  INFO [kafka-request-handler-0] server.ReplicaManager - 
[Replica Manager on Broker 10]: Handling LeaderAndIsr request 
Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0)
 -> 
(LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.

Imagine that line going on with a separate entry for 700 topics.  There are 
many other examples of this phenomenon in the server, and high-level consumer.

I'd think these log lines could be separated into a single line per topic.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Closed] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede closed KAFKA-1121.



> DumpLogSegments tool should print absolute file name to report inconsistencies
> --
>
> Key: KAFKA-1121
> URL: https://issues.apache.org/jira/browse/KAFKA-1121
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Fix For: 0.8.1
>
> Attachments: KAFKA-1121.patch
>
>
> Normally, the user would know where the index file lies. But in case of a 
> script that continuously checks the index files for consistency, it will help 
> to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1121) DumpLogSegments tool should print absolute file name to report inconsistencies

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede resolved KAFKA-1121.
--

Resolution: Fixed

> DumpLogSegments tool should print absolute file name to report inconsistencies
> --
>
> Key: KAFKA-1121
> URL: https://issues.apache.org/jira/browse/KAFKA-1121
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Fix For: 0.8.1
>
> Attachments: KAFKA-1121.patch
>
>
> Normally, the user would know where the index file lies. But in case of a 
> script that continuously checks the index files for consistency, it will help 
> to have the absolute file path printed in the output.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Assigned] (KAFKA-1063) run log cleanup at startup

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede reassigned KAFKA-1063:


Assignee: Neha Narkhede

> run log cleanup at startup
> --
>
> Key: KAFKA-1063
> URL: https://issues.apache.org/jira/browse/KAFKA-1063
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8
>Reporter: paul mackles
>Assignee: Neha Narkhede
>Priority: Minor
> Fix For: 0.8.1
>
>
> Jun suggested I file this ticket to have the brokers start running cleanup at 
> start. Here is the scenario that precipitated it:
> We ran into a situation on our dev cluster (3 nodes, v0.8) where we ran out 
> of disk on one of the nodes . As expected, the broker shut itself down and 
> all of the clients switched over to the other nodes. So far so good. 
> To free up disk space, I reduced log.retention.hours to something more 
> manageable (from 172 to 12). I did this on all 3 nodes. Since the other 2 
> nodes were running OK, I first tried to restart the node which ran out of 
> disk. Unfortunately, it kept shutting itself down due to the full disk. From 
> the logs, I think this was because it was trying to sync-up the replicas it 
> was responsible for and of course couldn't due to the lack of disk space. My 
> hope was that upon restart, it would see the new retention settings and free 
> up a bunch of disk space before trying to do any syncs.
> I then went and restarted the other 2 nodes. They both picked up the new 
> retention settings and freed up a bunch of storage as a result. I then went 
> back and tried to restart the 3rd node but to no avail. It still had problems 
> with the full disks.
> I thought about trying to reassign partitions so that the node in question 
> had less to manage but that turned out to be a hassle so I wound up manually 
> deleting some of the old log/segment files. The broker seemed to come back 
> fine after that but that's not something I would want to do on a production 
> server.
> We obviously need better monitoring/alerting to avoid this situation 
> altogether, but I am wondering if the order of operations at startup 
> could/should be changed to better account for scenarios like this. Or maybe a 
> utility to remove old logs after changing ttl? Did I miss a better way to 
> handle this?
> Original email thread is here:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201309.mbox/%3cce6365ae.82d66%25pmack...@adobe.com%3e



--
This message was sent by Atlassian JIRA
(v6.1#6144)