If you use Samza's
OutgoingMessageEnvelope<https://samza.apache.org/learn/documentation/0.9/api/javadocs/org/apache/samza/system/OutgoingMessageEnvelope.html>
to send a message using this format:
public OutgoingMessageEnvelope(SystemStream systemStream,
java.lang.Object partitionKey,
java.lang.Object key,
java.lang.Object message)
Constructs a new OutgoingMessageEnvelope from specified components.
Parameters:
systemStream - Object representing the appropriate stream of which this
envelope will be sent on.
partitionKey - A key representing which partition of the systemStream to send
this envelope on.
key - A deserialized key to be used for the message.
message - A deserialized message to be sent in this envelope.
and you call this method within a stream task's process() method and want to
route the incoming messages to an appropriate partition, will Samza create the
partitions for you when you call the method?
E.g.
MessageA = {"id": "idA", "key": "keyA", "body":"some details"}
MessageB = {"id": "idB", "key": "keyB", "body":"some more details"}
If I call within a stream task's process() where msg is a message instance:
public void process(IncomingMessageEnvelope envelope, MessageCollector
collector, TaskCoordinator coordinator) {
// ...
String partition = msg["id"]
String key = msg["key"]
collector.send(new OutgoingMessageEnvelope(new SystemStream("kafka",
"PartitionedMessages"), id, key, msg));
// ...
Will this create partitions idA and idB automatically for me (i.e. do I need to
have created these partitions before I send message to them)?
I want to be able to route a message to an appropriate partition and also to be
able to log compaction with a separate message key. I do not know in advance
how many partitions I will need - is this compatible with the way Samza works?
