Hi Gwen, I like the approach of converting to general forms in most cases, which is an early bound design.
You can also take a late bound approach of leaving data in its original form but adding metadata to enable translation if needed at a later time. This is necessary if you have homogeneous consumers downstream and the translation is "lossy" in any sense. Just as one example floating point numbers can lose precision on conversion. Cheers, Robert On Mon, Aug 25, 2014 at 5:36 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > Personally, I like converting data before writing to Kafka, so I can > easily support many consumers who don't know about EBCDIC. > > A third option is to have a consumer that reads EBCDIC data from one > Kafka topic and writes ASCII to another Kafka topic. This has the > benefits of preserving the raw data in Kafka, in case you need it for > troubleshooting, and also supporting non-EBCDIC consumers. > > The cost is a more complex architecture, but if you already have a > stream processing system around (Storm, Samza, Spark), it can be an > easy addition. > > > On Mon, Aug 25, 2014 at 5:28 PM, <sonali.parthasara...@accenture.com> > wrote: > > Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of > doing an EBCDIC to ASCII conversion before sending to Kafka Vs. using an > ebcdic library after in the consumer > > > > Thanks! > > S > > > > -----Original Message----- > > From: Gwen Shapira [mailto:gshap...@cloudera.com] > > Sent: Monday, August 25, 2014 5:22 PM > > To: users@kafka.apache.org > > Subject: Re: EBCDIC support > > > > Hi Sonali, > > > > Kafka doesn't really care about EBCDIC or any other format - for Kafka > bits are just bits. So they are all supported. > > > > Kafka does not "read" data from a socket though. Well, it does, but the > data has to be sent by a Kafka producer. Most likely you'll need to > implement a producer that will get the data from the socket and send it as > a message to Kafka. The content of the message can be anything, including > EBCDIC -. > > > > Then you'll need a consumer to read the data from Kafka and do > something with this - the consumer will need to know what to do with a > message that contains EBCDIC data. Perhaps you have EBCDIC libraries you > can reuse there. > > > > Hope this helps. > > > > Gwen > > > > On Mon, Aug 25, 2014 at 5:14 PM, <sonali.parthasara...@accenture.com> > wrote: > >> Hey all, > >> > >> This might seem like a silly question, but does kafka have support for > EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket > where the data resides in EBCDIC format, can Kafka read that directly? > >> > >> Thanks, > >> Sonali > >> > >> ________________________________ > >> > >> This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. Where allowed > by local law, electronic communications with Accenture and its affiliates, > including e-mail and instant messaging (including content), may be scanned > by our systems for the purposes of information security and assessment of > internal compliance with Accenture policy. > >> ______________________________________________________________________ > >> ________________ > >> > >> www.accenture.com >
