Thanks Ryan. In my case I have around 200 small files in mainframe . Columns are same within a file but vary in number across files .Now I need to get all these data into a *single* hive table . The first three columns are standard in case of all files .Any idea how the schema would look if I use the stingray reader?.I am guessing it would be more like string,string,string,array(strings)?.
-Nishanth On Fri, Jun 2, 2017 at 10:51 AM, Ryan Harris <ryan.har...@zionsbancorp.com> wrote: > I wrote some custom python parsing scripts using StingRay Reader ( > http://stingrayreader.sourceforge.net/cobol.html ) that read in the > copybooks and use the results to automatically generate hive table schema > based on the source copybook. The EBCDIC data is then extracted to TAB > separated ASCII values to load to Hive. > > Some tables had some very sparse column values, so in those cases, I > bundled the sparse data into a catch-all JSON field in the Hive table. > > > > The parser is able to handle both fixed-length records as well as > variable-length VB-type records. > > > > Let me know if you have any questions regarding Stingray…. > > > > *From:* Nishanth S [mailto:nishanth.2...@gmail.com] > *Sent:* Friday, June 02, 2017 10:07 AM > *To:* user@hive.apache.org > *Subject:* Migrating Variable Length Files to Hive > > > > [External Email] > ------------------------------ > > Hello hive users, > > > > We are looking at migrating files(less than 5 Mb of data in total) with > variable record lengths from a mainframe system to hive.You could think of > this as metadata.Each of these records can have columns ranging from 3 to > n( means each record type have different number of columns) based on > record type.What would be the best strategy to migrate this to hive .I was > thinking of converting these files into one variable length csv file and > then importing them to a hive table .Hive table will consist of 4 columns > with the 4th column having comma separated list of values from column > column 4 to n.Are there other alternative or better approaches for this > solution.Appreciate any feedback on this. > > > > Thanks, > > Nishanth > ------------------------------ > THIS ELECTRONIC MESSAGE, INCLUDING ANY ACCOMPANYING DOCUMENTS, IS > CONFIDENTIAL and may contain information that is privileged and exempt from > disclosure under applicable law. If you are neither the intended recipient > nor responsible for delivering the message to the intended recipient, > please note that any dissemination, distribution, copying or the taking of > any action in reliance upon the message is strictly prohibited. If you have > received this communication in error, please notify the sender immediately. > Thank you. >
