[
https://issues.apache.org/jira/browse/KAFKA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Anderson updated KAFKA-2927:
----------------------------------
Summary: System tests: reduce storage footprint of collected logs (was:
System tests: reduce storage footprint of KafkaService)
> System tests: reduce storage footprint of collected logs
> --------------------------------------------------------
>
> Key: KAFKA-2927
> URL: https://issues.apache.org/jira/browse/KAFKA-2927
> Project: Kafka
> Issue Type: Bug
> Reporter: Geoff Anderson
> Assignee: Geoff Anderson
>
> Looking at recent night test runs (testing.confluent.io/kafka), the storage
> requirements for log output from the various services has increased
> significantly, up to 7-10G for a single test run, up from hundreds of MB
> Current breakdown:
> 23M Benchmark
> 3.2M ClientCompatibilityTest
> 613M ConnectDistributedTest
> 1.1M ConnectRestApiTest
> 1.5M ConnectStandaloneFileTest
> 2.0M ConsoleConsumerTest
> 440K KafkaVersionTest
> 744K Log4jAppenderTest
> 49M QuotaTest
> 3.0G ReplicationTest
> 1.2G TestMirrorMakerService
> 185M TestUpgrade
> 372K TestVerifiableProducer
> 2.3G VerifiableConsumerTest
> The biggest contributors in these test suites:
> ReplicationTest:
> verifiable_producer.log (currently TRACE level)
> VerifiableConsumerTest:
> kafka server.log
> TestMirrorMakerService:
> verifiable_producer.log
> ConnectDistributedTest:
> kafka server.log
> The worst offenders are therefore
> verifiable_producer.log which is logging at TRACE level, and kafka server.log
> which is logging at debug level
> One solution is to:
> 1) Update the log4j configs to log separately to both an INFO level file, and
> another file for DEBUG at least for the worst offenders.
> 2) Don't collect these DEBUG (and below) logs by default; only mark for
> collection during failure
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
