[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489794#comment-13489794
]
Mike Naseef commented on PIG-3015:
----------------------------------
We are very excited about this direction, as we were considering a private
re-write to AvroStorage for some of the issues you are addressing. I want to +1
passing the schema into the LoadFunc. The old AvroStorage is very slow and a
resource hog when we have a directory hierarchy to scan - even when we set the
no_schema_check property. Furthermore, we occasionally have issues with pig
jobs picking the old schema when we have a schema update. Manually specifying
the schema would fix this (option 1a should cover this as well) and give us
more flexibility in defining the data we want pig to pull from a file.
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
