So if I have a feed with
{user_id:12345,
ethnicity: asian,
location: "cerritos, ca",
Height:"5.9",
weight: "150 lbs"}
I am referring to https://kafka.apache.org/081/ops.html#topic-config
How do I map the 3 columns - (user_id, ethnicity, and location) to a
partition id. If I map it this way and say create 10 partitions, each
partition will contain a subset of data grouped by these columns - right?
- Shekar
On Thu, Mar 26, 2015 at 9:38 AM, Roger Hoover <roger.hoo...@gmail.com>
wrote:
> Hi Richard,
>
> You can also partition by a key like "user_id" so that all messages for a
> given user would end up in the same partition. This can be useful for
> calculating user-specific aggregations or doing a distributed join where
> the local state is also partitioned on user_id.
>
> Cheers,
>
> Roger
>
> On Thu, Mar 26, 2015 at 9:28 AM, Richard Lee <rd...@tivo.com> wrote:
>
> > Is there a typo below? Are all of these actually in the same topic, just
> > different partitions? Partitioning, AFAIK, is mainly done for
> parallelism
> > & throughput reasons. What is the reason for partitioning your dataset
> by
> > ‘columns’?
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic
> > ?
> >
> > Richard
> >
> > > On Mar 26, 2015, at 8:22 AM, Shekar Tippur <ctip...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > Want to confirm a basic understanding of Kafka.
> > > If I have a dataset that needs to be partitioned by 4 columns, then the
> > > progression is
> > >
> > > {topic1:partition_key1} -> {Group by samza on partition_key1}
> > > ->
> > > {topic2:partition_key2} -> {Group by samza on partition_key2}
> > > ->
> > > {topic3:partition_key3} -> {Group by samza on partition_key3}
> > > ->
> > > {topic4:partition_key4} -> {Group by samza on partition_key4}
> > >
> > > Can you please confirm if my understanding is right?
> > >
> > > - Shekar
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
