Hello everyone! New Kafka user here. I am learning Kafka and trying to run it for the first time (part of my CS undergrad thesis) I have the minimum server in AWS EC2 with 500MB of RAM and whenever I try to start Kafka (kraft mode) I get "killed" as a last message.
First I had to change the heap size from default (1GB) because I received an error about it so now It is 256M and now I see the logs of startup process but the process is killed. How can I know the cause of this error? I paste the last part of the log here. I just followed the quickstart https://kafka.apache.org/quickstart I also quickly saw the log directory (server, controller) but any hint to me. Thanks in advance for your assistance. [2023-02-16 01:00:54,082] INFO [LogLoader partition=__cluster_metadata-0, dir=/tmp/kraft-combined-logs] Producer state recovery took 3ms for snapshot load and 0ms for segment recovery from offset 28 (kafka.log.UnifiedLog$) [2023-02-16 01:00:54,131] INFO Initialized snapshots with IDs SortedSet() from /tmp/kraft-combined-logs/__cluster_metadata-0 (kafka.raft.KafkaMetadataLog$) [2023-02-16 01:00:54,183] INFO [raft-expiration-reaper]: Starting (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper) [2023-02-16 01:00:54,523] INFO [RaftManager nodeId=1] Completed transition to ResignedState(localId=1, epoch=9, voters=[1], electionTimeoutMs=1695, unackedVoters=[], preferredSuccessors=[]) (org.apache.kafka.raft.QuorumState) [2023-02-16 01:00:54,533] INFO [RaftManager nodeId=1] Completed transition to CandidateState(localId=1, epoch=10, retries=1, electionTimeoutMs=1539) (org.apache.kafka.raft.QuorumState) [2023-02-16 01:00:54,549] INFO [RaftManager nodeId=1] Completed transition to Leader(localId=1, epoch=10, epochStartOffset=28, highWatermark=Optional.empty, voterStates={1=ReplicaState(nodeId=1, endOffset=Optional.empty, lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1, hasAcknowledgedLeader=true)}) (org.apache.kafka.raft.QuorumState) [2023-02-16 01:00:54,615] INFO [kafka-raft-outbound-request-thread]: Starting (kafka.raft.RaftSendThread) [2023-02-16 01:00:54,620] INFO [kafka-raft-io-thread]: Starting (kafka.raft.KafkaRaftManager$RaftIoThread) [2023-02-16 01:00:54,694] INFO [RaftManager nodeId=1] High watermark set to LogOffsetMetadata(offset=29, metadata=Optional[(segmentBaseOffset=0,relativePositionInSegment=3044)]) for the first time for epoch 10 based on indexOfHw 0 and voters [ReplicaState(nodeId=1, endOffset=Optional[LogOffsetMetadata(offset=29, metadata=Optional[(segmentBaseOffset=0,relativePositionInSegment=3044)])], lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1, hasAcknowledgedLeader=true)] (org.apache.kafka.raft.LeaderState) [2023-02-16 01:00:54,723] INFO [RaftManager nodeId=1] Registered the listener org.apache.kafka.image.loader.MetadataLoader@1633422834 (org.apache.kafka.raft.KafkaRaftClient) [2023-02-16 01:00:54,761] INFO [RaftManager nodeId=1] Registered the listener org.apache.kafka.controller.QuorumController$QuorumMetaLogListener@629125780 (org.apache.kafka.raft.KafkaRaftClient) [2023-02-16 01:00:54,780] INFO [MetadataLoader 1] Publishing initial snapshot at offset 27 to SnapshotGenerator (org.apache.kafka.image.loader.MetadataLoader) [2023-02-16 01:00:54,798] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,799] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,808] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,813] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,849] INFO [ExpirationReaper-1-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:54,865] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Enabling request processing. (kafka.network.SocketServer) [2023-02-16 01:00:54,874] INFO [BrokerServer id=1] Transition from SHUTDOWN to STARTING (kafka.server.BrokerServer) [2023-02-16 01:00:54,877] INFO [BrokerServer id=1] Starting broker (kafka.server.BrokerServer) [2023-02-16 01:00:54,918] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,918] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,919] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,919] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2023-02-16 01:00:54,957] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Starting (kafka.server.BrokerToControllerRequestThread) [2023-02-16 01:00:54,972] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Recorded new controller, from now on will use node localhost:9093 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread) [2023-02-16 01:00:55,059] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [2023-02-16 01:00:55,065] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.DataPlaneAcceptor) [2023-02-16 01:00:55,074] INFO [SocketServer listenerType=BROKER, nodeId=1] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer) [2023-02-16 01:00:55,082] INFO [BrokerToControllerChannelManager broker=1 name=alterPartition]: Starting (kafka.server.BrokerToControllerRequestThread) [2023-02-16 01:00:55,082] INFO [BrokerToControllerChannelManager broker=1 name=alterPartition]: Recorded new controller, from now on will use node localhost:9093 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread) [2023-02-16 01:00:55,101] INFO [ExpirationReaper-1-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,101] INFO [ExpirationReaper-1-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,101] INFO [ExpirationReaper-1-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,112] INFO [ExpirationReaper-1-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,152] INFO [ExpirationReaper-1-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,153] INFO [ExpirationReaper-1-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,282] INFO [RaftManager nodeId=1] Registered the listener kafka.server.metadata.BrokerMetadataListener@1398770581 (org.apache.kafka.raft.KafkaRaftClient) [2023-02-16 01:00:55,287] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Starting (kafka.server.BrokerToControllerRequestThread) [2023-02-16 01:00:55,287] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Recorded new controller, from now on will use node localhost:9093 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread) [2023-02-16 01:00:55,301] INFO [BrokerLifecycleManager id=1] Incarnation K3MwUk8zQUGgIXaeg0yY_w of broker 1 in cluster zodne5JyThyBUzTtQnIADg is now STARTING. (kafka.server.BrokerLifecycleManager) [2023-02-16 01:00:55,393] INFO [ExpirationReaper-1-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2023-02-16 01:00:55,405] INFO [BrokerServer id=1] Waiting for broker metadata to catch up. (kafka.server.BrokerServer) [2023-02-16 01:00:55,537] INFO [BrokerLifecycleManager id=1] Successfully registered broker 1 with broker epoch 30 (kafka.server.BrokerLifecycleManager) [2023-02-16 01:00:55,546] INFO [BrokerLifecycleManager id=1] The broker has caught up. Transitioning from STARTING to RECOVERY. (kafka.server.BrokerLifecycleManager) [2023-02-16 01:00:55,549] INFO [BrokerMetadataListener id=1] Starting to publish metadata events at offset 30. (kafka.server.metadata.BrokerMetadataListener) [2023-02-16 01:00:55,552] INFO [BrokerMetadataPublisher id=1] Publishing initial metadata at offset OffsetAndEpoch(offset=30, epoch=10) with metadata.version 3.4-IV0. (kafka.server.metadata.BrokerMetadataPublisher) [2023-02-16 01:00:55,553] INFO Loading logs from log dirs ArraySeq(/tmp/kraft-combined-logs) (kafka.log.LogManager) [2023-02-16 01:00:55,563] INFO [BrokerLifecycleManager id=1] The broker is in RECOVERY. (kafka.server.BrokerLifecycleManager) [2023-02-16 01:00:55,565] INFO Attempting recovery for all logs in /tmp/kraft-combined-logs since no clean shutdown file was found (kafka.log.LogManager) [2023-02-16 01:00:55,591] INFO Loaded 0 logs in 38ms. (kafka.log.LogManager) [2023-02-16 01:00:55,594] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2023-02-16 01:00:55,599] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) Killed
