[jira] [Commented] (KAFKA-1028) per topic configuration of preference for consistency over availability

2013-11-03 Thread Jason Rosenberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812345#comment-13812345
 ] 

Jason Rosenberg commented on KAFKA-1028:


Agreed on a default setting.  Also, it would be nice if this could be 
configurable for topics identified with a regex, so that all topics that match 
"*.reliable.*" will have this feature applied.

Also, it would be good for it be applicable to existing topics, not only newly 
created ones, so existing topics in a running system can have the benefit of 
this feature.

(In general too, default settings for topics identified via regex, and 
retroactive applicability, are desirable for all per topic config  options).

> per topic configuration of preference for consistency over availability
> ---
>
> Key: KAFKA-1028
> URL: https://issues.apache.org/jira/browse/KAFKA-1028
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Scott Clasen
>Assignee: Neha Narkhede
>
> As discussed with Neha on the ML.
> It should be possible to configure a topic to disallow unclean leader 
> election, thus preventing the situation where committed messages can be 
> discarded once a failed leader comes back online in a situation where it was 
> the only ISR.
> This would open kafka to additional usecases where the possibility of 
> committted messages being discarded is unacceptable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Apache Kafka Release 0.8.0 - Candidate 1

2013-11-03 Thread Joe Stein 
I upgraded recently and forgot.  We should stick with 6 that was my bad.

let's cancel the vote and I will redo with Java 6 hopefully later today/tonight 
or tomorrow and call rc2 vote 


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
/


On Nov 2, 2013, at 11:19 PM, Jun Rao  wrote:

> Thanks for preparing the release. Is the binary release compiled with java
> 7? Got the following error using a java 6 runtime.
> 
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
> --from-beginning
> Exception in thread "main" java.lang.UnsupportedClassVersionError:
> kafka/javaapi/consumer/ConsumerConnector : Unsupported major.minor version
> 51.0
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178)
> at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
> 
> Should we release the binary in java 6 or java 7?
> 
> Jun
> 
> 
> On Thu, Oct 31, 2013 at 12:05 PM, Joe Stein  wrote:
> 
>> Hello,
>> 
>> This is the first candidate for release of Apache Kafka 0.8.0.
>> 
>> Release Notes for the 0.8.0 release
>> 
>> http://people.apache.org/~joestein/kafka-0.8.0-candidate1/RELEASE_NOTES.html
>> 
>> *** Please download, test and vote by Monday November, 4th, 9pm PDT
>> 
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to
>> the md5 and sha1 checksum
>> 
>> * Release artifacts to be voted upon (source and binary):
>> http://people.apache.org/~joestein/kafka-0.8.0-candidate1/
>> 
>> * Maven artifacts to be voted upon prior to release:
>> https://repository.apache.org/content/groups/staging/
>> 
>> (i.e. in sbt land this can be added to the build.sbt to use Kafka
>> resolvers += "Apache Staging" at "
>> https://repository.apache.org/content/groups/staging/";
>> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"
>> )
>> 
>> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag
>> 
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bcbe8de95131259dd9167ffcfbfe0a798b5ec2d2
>> 
>> /***
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop 
>> /
>>

[jira] [Created] (KAFKA-1116) Need to upgrade sbt-assembly to compile on scala 2.10.2

2013-11-03 Thread Kane Kim (JIRA) 
Kane Kim created KAFKA-1116:
---

 Summary: Need to upgrade sbt-assembly to compile on scala 2.10.2
 Key: KAFKA-1116
 URL: https://issues.apache.org/jira/browse/KAFKA-1116
 Project: Kafka
  Issue Type: Bug
Reporter: Kane Kim
Priority: Minor


Need to upgrade sbt-assembly to 0.9.0 compile on scala 2.10.2



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1116) Need to upgrade sbt-assembly to compile on scala 2.10.2

2013-11-03 Thread Kane Kim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kane Kim resolved KAFKA-1116.
-

Resolution: Implemented

> Need to upgrade sbt-assembly to compile on scala 2.10.2
> ---
>
> Key: KAFKA-1116
> URL: https://issues.apache.org/jira/browse/KAFKA-1116
> Project: Kafka
>  Issue Type: Bug
>Reporter: Kane Kim
>Priority: Minor
>
> Need to upgrade sbt-assembly to 0.9.0 compile on scala 2.10.2



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1115) producer performance affected by trace/debug calls

2013-11-03 Thread Jun Rao (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812516#comment-13812516
 ] 

Jun Rao commented on KAFKA-1115:


Thanks for the patch. Are you sure that scala doesn't do lazy evaluation? I did 
the following test. It seems that it does?

scala> import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.atomic.AtomicInteger

scala> val c = new AtomicInteger(1)
c: java.util.concurrent.atomic.AtomicInteger = 1

scala> def a(msg: => String, isRun: Boolean) { if (isRun) println(msg) }
a: (msg: => String,isRun: Boolean)Unit

scala> a("test %d".format(c.incrementAndGet), false)

scala> c
res3: java.util.concurrent.atomic.AtomicInteger = 1


> producer performance affected by trace/debug calls
> --
>
> Key: KAFKA-1115
> URL: https://issues.apache.org/jira/browse/KAFKA-1115
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.8
>Reporter: Francois Saint-Jacques
>Assignee: Francois Saint-Jacques
> Attachments: producer-performance-fix.patch
>
>
> After investigating high CPU usage on some producers in production, we found 
> out that a lot of time was passed in string construction for logging of DEBUG 
> and TRACE level.
> This patch removes most of the logging calls, on our systems it cuts CPU 
> usage down to half of what it used to be.
> Note that this is a significant boost in performance for environment where 
> there are thousands of msg/s.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1109) Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS

2013-11-03 Thread Jun Rao (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812526#comment-13812526
 ] 

Jun Rao commented on KAFKA-1109:


Thanks for pointing this out. Could you submit the proposed changes as a patch?

> Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS
> -
>
> Key: KAFKA-1109
> URL: https://issues.apache.org/jira/browse/KAFKA-1109
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: *nix
>Reporter: Viktor Kolodrevskiy
>
> kafka-run-class.sh contains GC log code:
> # GC options
> GC_FILE_SUFFIX='-gc.log'
> GC_LOG_FILE_NAME=''
> if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS"] ; then
>   shift
>   GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
>   shift
>   KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
> fi
> So when in my scripts I start kafka and want to override KAFKA_GC_LOG_OPTS by 
> exporting new values I get:
> Exception in thread "main" java.lang.NoClassDefFoundError: daemon
> Caused by: java.lang.ClassNotFoundException: daemon
> That's because shift is not done when KAFKA_GC_LOG_OPTS is set and "daemon" 
> is passed as main class.
> I suggest to replace it with this code:
> # GC options
> GC_FILE_SUFFIX='-gc.log'
> GC_LOG_FILE_NAME=''
> if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS" ] ; then
>   shift
>   GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
>   shift
>   KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
> else
> if [ "$1" = "daemon" ] && [ "$KAFKA_GC_LOG_OPTS" != "" ] ; then
>   shift 2
> fi
> fi



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1117) tool for checking the consistency among replicas

2013-11-03 Thread Jun Rao (JIRA) 
Jun Rao created KAFKA-1117:
--

 Summary: tool for checking the consistency among replicas
 Key: KAFKA-1117
 URL: https://issues.apache.org/jira/browse/KAFKA-1117
 Project: Kafka
  Issue Type: New Feature
  Components: core
Affects Versions: 0.8.1
Reporter: Jun Rao
Assignee: Jun Rao
 Fix For: 0.8.1






--
This message was sent by Atlassian JIRA
(v6.1#6144)

Review Request 15201: tool for checking the consistency among replicas

2013-11-03 Thread Jun Rao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/
---

Review request for kafka.


Bugs: KAFKA-1117
https://issues.apache.org/jira/browse/KAFKA-1117


Repository: kafka


Description
---

kafka-1117


Diffs
-

  core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 

Diff: https://reviews.apache.org/r/15201/diff/


Testing
---


Thanks,

Jun Rao

[jira] [Updated] (KAFKA-1117) tool for checking the consistency among replicas

2013-11-03 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1117:
---

Attachment: KAFKA-1117.patch

> tool for checking the consistency among replicas
> 
>
> Key: KAFKA-1117
> URL: https://issues.apache.org/jira/browse/KAFKA-1117
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.1
>
> Attachments: KAFKA-1117.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1117) tool for checking the consistency among replicas

2013-11-03 Thread Jun Rao (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812536#comment-13812536
 ] 

Jun Rao commented on KAFKA-1117:


Created reviewboard https://reviews.apache.org/r/15201/


> tool for checking the consistency among replicas
> 
>
> Key: KAFKA-1117
> URL: https://issues.apache.org/jira/browse/KAFKA-1117
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.1
>
> Attachments: KAFKA-1117.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1118) /solr/select/ returned fields

2013-11-03 Thread antoine s (JIRA) 
antoine s created KAFKA-1118:


 Summary: /solr/select/ returned fields
 Key: KAFKA-1118
 URL: https://issues.apache.org/jira/browse/KAFKA-1118
 Project: Kafka
  Issue Type: Bug
 Environment: 

solr-spec
4.5.1

solr-impl
4.5.1 1533280 - mark - 2013-10-17 21:44:41

lucene-spec
4.5.1

lucene-impl
4.5.1 1533280 - mark - 2013-10-17 21:40:03


Reporter: antoine s
Priority: Minor


I have a field in schema.xml


http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
does not return the 'description'

http://localhost:8983/solr/select/?q=video&defType=edismax&qf=name^20.0+text^0.3
returns it



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1115) producer performance affected by trace/debug calls

2013-11-03 Thread Francois Saint-Jacques (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812550#comment-13812550
 ] 

Francois Saint-Jacques commented on KAFKA-1115:
---

You are right, and by looking at the code the trace/debug logging functions are 
defined with lazy arguments.

By debugging a bit further, I did not have any log4.properties in the producer 
classpath. The library does give me a WARN about missing log4.properties file. 
After this warning message, the program does not output any more log. 
Internally I'd assume the logging still we're set to TRACE without any appender 
to stdout/stderr.

I'll take WARN more seriously next time. Maybe a put message in the producer 
documentation that not providing log4j configuration will have a serious impact 
on performance.

On another note, I believe trace/debug message shouldn't be committed to 
production code, but this is only a personal hunch.

> producer performance affected by trace/debug calls
> --
>
> Key: KAFKA-1115
> URL: https://issues.apache.org/jira/browse/KAFKA-1115
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.8
>Reporter: Francois Saint-Jacques
>Assignee: Francois Saint-Jacques
> Attachments: producer-performance-fix.patch
>
>
> After investigating high CPU usage on some producers in production, we found 
> out that a lot of time was passed in string construction for logging of DEBUG 
> and TRACE level.
> This patch removes most of the logging calls, on our systems it cuts CPU 
> usage down to half of what it used to be.
> Note that this is a significant boost in performance for environment where 
> there are thousands of msg/s.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[VOTE] Apache Kafka Release 0.8.0 - Candidate 2

2013-11-03 Thread Joe Stein 
Hello,

This is the second candidate for release of Apache Kafka 0.8.0.   The only
change to this release candidate from the last is that this builds against
Java 6 as expected.

Release Notes for the 0.8.0 release
http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html

*** Please download, test and vote by Thursday November, 7th, 6pm PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to
the md5 and sha1 checksum

* Release artifacts to be voted upon (source and binary):
http://people.apache.org/~joestein/kafka-0.8.0-candidate2/

* Maven artifacts to be voted upon prior to release:
https://repository.apache.org/content/groups/staging/

(i.e. in sbt land this can be added to the build.sbt to use Kafka
resolvers += "Apache Staging" at "
https://repository.apache.org/content/groups/staging/";
libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"
)

* The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bcbe8de95131259dd9167ffcfbfe0a798b5ec2d2

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/