Re: "Unable to load native-hadoop library for your platform" while running Spark jobs

On 19 Jan 2017, at 10:59, Sean Owen mailto:so...@cloudera.com>> wrote: It's a message from Hadoop libs, not Spark. It can be safely ignored. It's just saying you haven't installed the additional (non-Apache-licensed) native libs that can accelerate some operations. This is something you can ea

Re: "Unable to load native-hadoop library for your platform" while running Spark jobs

Thanks, Sean. I will explore online more. Regards, _ *Md. Rezaul Karim*, BSc, MSc PhD Researcher, INSIGHT Centre for Data Analytics National University of Ireland, Galway IDA Business Park, Dangan, Galway, Ireland Web: http://www.reza-analytics.eu/index.html

Re: "Unable to load native-hadoop library for your platform" while running Spark jobs

It's a message from Hadoop libs, not Spark. It can be safely ignored. It's just saying you haven't installed the additional (non-Apache-licensed) native libs that can accelerate some operations. This is something you can easily have read more about online. On Thu, Jan 19, 2017 at 10:57 AM Md. Reza

Re: Unable to load native-hadoop library for your platform

> On 4 Aug 2015, at 12:26, Sean Owen wrote: > > Oh good point, does the Windows integration need native libs for > POSIX-y file system access? I know there are some binaries shipped for > this purpose but wasn't sure if that's part of what's covered in the > native libs message. On unix its a w

Re: Unable to load native-hadoop library for your platform

Oh good point, does the Windows integration need native libs for POSIX-y file system access? I know there are some binaries shipped for this purpose but wasn't sure if that's part of what's covered in the native libs message. On Tue, Aug 4, 2015 at 6:01 PM, Steve Loughran wrote: > Think it may be

Re: Unable to load native-hadoop library for your platform

Think it may be needed on Windows, certainly if you start trying to work with local files. > On 4 Aug 2015, at 00:34, Sean Owen wrote: > > It won't affect you if you're not actually running Hadoop. But it's > mainly things like Snappy/LZO compression which are implemented as > native librarie

Re: Unable to load native-hadoop library for your platform

Hi Sean, Thanks for the information and clarification. On Tue, Aug 4, 2015 at 1:04 PM, Sean Owen wrote: > It won't affect you if you're not actually running Hadoop. But it's > mainly things like Snappy/LZO compression which are implemented as > native libraries under the hood. Spark doesn't ne

Re: Unable to load native-hadoop library for your platform

It won't affect you if you're not actually running Hadoop. But it's mainly things like Snappy/LZO compression which are implemented as native libraries under the hood. Spark doesn't necessarily use these anyway; it's from the Hadoop libs. On Tue, Aug 4, 2015 at 8:30 AM, Deepesh Maheshwari wrote:

Re: Unable to load native-hadoop library for your platform

Can you elaborate about the things this native library covering. One you mentioned accelerated compression. It would be very helpful if you can give any useful to link to read more about it. On Tue, Aug 4, 2015 at 12:56 PM, Sean Owen wrote: > You can ignore it entirely. It just means you haven'

Re: Unable to load native-hadoop library for your platform

You can ignore it entirely. It just means you haven't installed and configured native libraries for things like accelerated compression, but it has no negative impact otherwise. On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari wrote: > Hi, > > When i run the spark locally on windows it gives be