Having been spared any EBCDIC experience whatsoever (ie from a positio of thorough ignorance), if you are transmitting text or things with a designated textual form (presumably) I would recommend that your conversion be to unicode rather than ascii if you don't already have consumers expecting a given conversion. That way you will avoid losing information, particularly if you expect any of your conversion tools to be of more general use.
Christian On 08/25/2014 05:36 PM, Gwen Shapira wrote: > Personally, I like converting data before writing to Kafka, so I can > easily support many consumers who don't know about EBCDIC. > > A third option is to have a consumer that reads EBCDIC data from one > Kafka topic and writes ASCII to another Kafka topic. This has the > benefits of preserving the raw data in Kafka, in case you need it for > troubleshooting, and also supporting non-EBCDIC consumers. > > The cost is a more complex architecture, but if you already have a > stream processing system around (Storm, Samza, Spark), it can be an > easy addition. > > > On Mon, Aug 25, 2014 at 5:28 PM, <sonali.parthasara...@accenture.com> wrote: >> Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of doing >> an EBCDIC to ASCII conversion before sending to Kafka Vs. using an ebcdic >> library after in the consumer >> >> Thanks! >> S >> >> -----Original Message----- >> From: Gwen Shapira [mailto:gshap...@cloudera.com] >> Sent: Monday, August 25, 2014 5:22 PM >> To: users@kafka.apache.org >> Subject: Re: EBCDIC support >> >> Hi Sonali, >> >> Kafka doesn't really care about EBCDIC or any other format - for Kafka bits >> are just bits. So they are all supported. >> >> Kafka does not "read" data from a socket though. Well, it does, but the data >> has to be sent by a Kafka producer. Most likely you'll need to implement a >> producer that will get the data from the socket and send it as a message to >> Kafka. The content of the message can be anything, including EBCDIC -. >> >> Then you'll need a consumer to read the data from Kafka and do something >> with this - the consumer will need to know what to do with a message that >> contains EBCDIC data. Perhaps you have EBCDIC libraries you can reuse there. >> >> Hope this helps. >> >> Gwen >> >> On Mon, Aug 25, 2014 at 5:14 PM, <sonali.parthasara...@accenture.com> wrote: >>> Hey all, >>> >>> This might seem like a silly question, but does kafka have support for >>> EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket >>> where the data resides in EBCDIC format, can Kafka read that directly? >>> >>> Thanks, >>> Sonali >>> >>> ________________________________ >>> >>> This message is for the designated recipient only and may contain >>> privileged, proprietary, or otherwise confidential information. If you have >>> received it in error, please notify the sender immediately and delete the >>> original. Any other use of the e-mail by you is prohibited. Where allowed >>> by local law, electronic communications with Accenture and its affiliates, >>> including e-mail and instant messaging (including content), may be scanned >>> by our systems for the purposes of information security and assessment of >>> internal compliance with Accenture policy. >>> ______________________________________________________________________ >>> ________________ >>> >>> www.accenture.com
signature.asc
Description: OpenPGP digital signature
