Hadoop native and snappy library

[Yves Weissig]

Hi list,

I'm trying to enable the Hadoop native library and the snappy library
for compression in Hadoop 2.2.0, but I always end up with:

./hadoop/bin/hadoop checknative -a
Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false

I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
.so in hadoop/lib/native/. I also compiled snappy from scratch and put
it there. In a different approach I installed snappy via sudo apt-get
and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
still no luck.

What is going on here? Why won't Hadoop find my native libraries? Is
there any log where I can check what went wrong during loading?

Best regards
Yves

[Ted Yu]

See https://issues.apache.org/jira/browse/HBASE-10303

And https://hbase.apache.org/book.html#snappy.compression

Cheers

Post by Yves Weissig
Hi list,
I'm trying to enable the Hadoop native library and the snappy library
./hadoop/bin/hadoop checknative -a
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
.so in hadoop/lib/native/. I also compiled snappy from scratch and put
it there. In a different approach I installed snappy via sudo apt-get
and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
still no luck.
What is going on here? Why won't Hadoop find my native libraries? Is
there any log where I can check what went wrong during loading?
Best regards
Yves

[Yves Weissig]

Thanks... after looking at the sourcecode of the native library loading
process and activating the DEBUG messages I found this in the "hadoop.log":

2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
`GLIBC_2.14' not found (required by
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)

Is it possible to compile the native library with glibc 2.12, because at
the moment I am not able to upgrade glibc on the servers which run Hadoop.

Best regards
Yves

Post by Ted Yu
See https://issues.apache.org/jira/browse/HBASE-10303
And https://hbase.apache.org/book.html#snappy.compression
Cheers

Post by Yves Weissig
Hi list,
I'm trying to enable the Hadoop native library and the snappy library
./hadoop/bin/hadoop checknative -a
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
.so in hadoop/lib/native/. I also compiled snappy from scratch and put
it there. In a different approach I installed snappy via sudo apt-get
and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
still no luck.
What is going on here? Why won't Hadoop find my native libraries? Is
there any log where I can check what went wrong during loading?
Best regards
Yves

[java8964]

Where do you compile your libhadoop.so.1.0.0?
It is more like that you compiled libhadoop.so.1.0.0 in a environment with glibc 2.14, but tried to use it in an environment only have glibc 2.12.
If you are using a hadoop compiled by yourself, then it is best to compile in an environment matching with your production.
Yong

Date: Tue, 11 Feb 2014 14:12:34 +0100
Subject: Re: Hadoop native and snappy library
Thanks... after looking at the sourcecode of the native library loading
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
`GLIBC_2.14' not found (required by
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
Is it possible to compile the native library with glibc 2.12, because at
the moment I am not able to upgrade glibc on the servers which run Hadoop.
Best regards
Yves

Post by Ted Yu
See https://issues.apache.org/jira/browse/HBASE-10303
And https://hbase.apache.org/book.html#snappy.compression
Cheers

Post by Yves Weissig
Hi list,
I'm trying to enable the Hadoop native library and the snappy library
./hadoop/bin/hadoop checknative -a
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
.so in hadoop/lib/native/. I also compiled snappy from scratch and put
it there. In a different approach I installed snappy via sudo apt-get
and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
still no luck.
What is going on here? Why won't Hadoop find my native libraries? Is
there any log where I can check what went wrong during loading?
Best regards
Yves