Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/4972#issuecomment-78407842
I changed the implementation to not use the code from the ticket, and
instead created a wrapper around arrays, and a factory method that did the
appropriate casting and building of the wrapper object.
I was mildly concerned about the performance of this implementation, as now
I'm instantiating a new object as opposed to before when I was purely invoking
static methods. I ran some more localized unit test benchmarks.
With the 1000 x 1000 2D String array tests I was doing before, the
performance difference seems to be better, but I might as well attribute the
difference to noise since the difference is so small:
Before this PR at all: 209.275 s, 190.107 s, 185.424 s
Before most recent commit: 160.431, 149.487, 151.66
After most recent commit: 151.812, 151.505, 142.953
I was mostly concerned about the overhead to create the object if the
arrays were small, though, because now I'm spending more time on creating the
wrapper itself. It turns out we're still doing better even on small arrays
compared to before the PR however. I tested estimating the size of a
1000-String 1D array:
Before PR: 27.246, 32.124, 28.433
After commit: 20.036, 20.465, 21.38
Most importantly however we should be able to get around the licensing
issues now. I also made the code only use the array get() and getLength()
methods since we never set() on arrays anywhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
