HDFS mounting issue using Hadoop-Fuse on Fully Distributed Cluster?

Discussion:

Manu S

2012-04-26 10:50:42 UTC

Dear All,

I have installed *Hadoop-fuse* to mount the HDFS filesystem locally . I
could mount the HDFS without any issues.But I am not able to do any file
operations like *delete, copy, move* etc directly. The directory ownership
automatically changed to *nobody:nobody* while mounting.

*[***@namenode ~]# ls -ld /hdfs_mount/
drwxr-xr-x 2 root root 4096 Apr 3 16:22 /hdfs_mount/

[***@namenode ~]# hadoop-fuse-dfs dfs://namenode:8020 /hdfs_mount/** **
INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount/

[***@namenode ~]# ls -ld /hdfs_mount/* *
drwxrwxr-x 10 nobody nobody 4096 Apr 5 13:22 /hdfs_mount/*

I tried the same with *pseudo-distributed node*,but its working fine. I can
do any normal file operations after mounting the HDFS.

Appreciate your help on the same.

--
Thanks & Regards
----
*Manu S*
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

alo alt

2012-04-26 10:56:06 UTC

Permalink

Hi,

I wrote a small writeup about:
http://mapredit.blogspot.de/2011/11/nfs-exported-hdfs-cdh3.html

As you see, the FS is mounted as nobody and you try as root. Change the permissions in your hdfs:
hadoop -dfs chmod / chown ....

- Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

Post by Manu S
Dear All,
I have installed *Hadoop-fuse* to mount the HDFS filesystem locally . I
could mount the HDFS without any issues.But I am not able to do any file
operations like *delete, copy, move* etc directly. The directory ownership
automatically changed to *nobody:nobody* while mounting.
drwxr-xr-x 2 root root 4096 Apr 3 16:22 /hdfs_mount/
INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount/
drwxrwxr-x 10 nobody nobody 4096 Apr 5 13:22 /hdfs_mount/*
I tried the same with *pseudo-distributed node*,but its working fine. I can
do any normal file operations after mounting the HDFS.
Appreciate your help on the same.
--
Thanks & Regards
----
*Manu S*
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

Manu S

2012-04-26 12:00:59 UTC

Permalink

Thanks a lot Alex.

Actually I didn't tried the NFS option, as I am trying to sort out this
hadoop-fuse mounting issue.
I can't change the ownership of mount directory after hadoop-fuse mount.

[***@namenode ~]# ls -ld /hdfs_mount/
drwxrwxr-x 11 nobody nobody 4096 Apr 9 12:34 /hdfs_mount/

[***@namenode ~]# chown hdfs /hdfs_mount/
chown: changing ownership of `/hdfs_mount/': Input/output error

Any ideas?

Post by alo alt
Hi,
http://mapredit.blogspot.de/2011/11/nfs-exported-hdfs-cdh3.html
As you see, the FS is mounted as nobody and you try as root. Change the
hadoop -dfs chmod / chown ....
- Alex
--
Alexander Lorenz
http://mapredit.blogspot.com

ownership

Post by Manu S
automatically changed to *nobody:nobody* while mounting.
drwxr-xr-x 2 root root 4096 Apr 3 16:22 /hdfs_mount/
INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount/
drwxrwxr-x 10 nobody nobody 4096 Apr 5 13:22 /hdfs_mount/*
I tried the same with *pseudo-distributed node*,but its working fine. I

can

Post by Manu S
do any normal file operations after mounting the HDFS.
Appreciate your help on the same.
--
Thanks & Regards
----
*Manu S*
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Thanks & Regards
----
*Manu S*
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

alo alt

2012-04-26 12:39:44 UTC

Permalink

Manu,

did you mount hdfs over fstab:

hadoop-fuse-dfs#dfs://namenode.local:<PORT> /hdfs-mount fuse usetrash,rw 0 0" ?

You could that do with:
"mkdir -p /hdfs-mount && chmod 777 /hdfs-mount && echo "hadoop-fuse-dfs#dfs://NN.URI:<PORT> /hdfs-mount fuse usetrash,rw 0 0" >> /etc/fstab && mount -a ; mount"

- Alex

Post by Manu S
Thanks a lot Alex.
Actually I didn't tried the NFS option, as I am trying to sort out this hadoop-fuse mounting issue.
I can't change the ownership of mount directory after hadoop-fuse mount.
drwxrwxr-x 11 nobody nobody 4096 Apr 9 12:34 /hdfs_mount/
chown: changing ownership of `/hdfs_mount/': Input/output error
Any ideas?
Hi,
http://mapredit.blogspot.de/2011/11/nfs-exported-hdfs-cdh3.html
hadoop -dfs chmod / chown ....
- Alex
--
Alexander Lorenz
http://mapredit.blogspot.com

--
Thanks & Regards
----
Manu S
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Alexander Lorenz
http://mapredit.blogspot.com

Manu S

2012-04-26 12:53:58 UTC

Permalink

Yeah Alex, I tried. But still I am not able to make it

[***@namenode ~]# *echo "hadoop-fuse-dfs#dfs://namenode:8020 /hdfs_mount
fuse usetrash,rw 0 0" >> /etc/fstab*
[***@namenode ~]# *mount -a
INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount*

[***@namenode ~]# *mount | grep fuse
fuse on /hdfs_mount type fuse
(rw,nosuid,nodev,allow_other,default_permissions)*

[***@namenode ~]# ls -ld /hdfs_mount/
drwxrwxr-x 11 *nobody nobody* 4096 Apr 9 12:34 /hdfs_mount/
[***@namenode ~]# touch /hdfs_mount/file
*touch: cannot touch `/hdfs_mount/file': Permission denied*

Post by alo alt
Manu,
hadoop-fuse-dfs#dfs://namenode.local:<PORT> /hdfs-mount fuse usetrash,rw 0 0" ?
"mkdir -p /hdfs-mount && chmod 777 /hdfs-mount && echo
"hadoop-fuse-dfs#dfs://NN.URI:<PORT> /hdfs-mount fuse usetrash,rw 0 0" >>
/etc/fstab && mount -a ; mount"
- Alex

Post by Manu S
Thanks a lot Alex.
Actually I didn't tried the NFS option, as I am trying to sort out this

hadoop-fuse mounting issue.

Post by Manu S
I can't change the ownership of mount directory after hadoop-fuse mount.
drwxrwxr-x 11 nobody nobody 4096 Apr 9 12:34 /hdfs_mount/
chown: changing ownership of `/hdfs_mount/': Input/output error
Any ideas?
Hi,
http://mapredit.blogspot.de/2011/11/nfs-exported-hdfs-cdh3.html
As you see, the FS is mounted as nobody and you try as root. Change the
hadoop -dfs chmod / chown ....
- Alex
--
Alexander Lorenz
http://mapredit.blogspot.com

ownership

Post by Manu S

Post by Manu S
automatically changed to *nobody:nobody* while mounting.
drwxr-xr-x 2 root root 4096 Apr 3 16:22 /hdfs_mount/

Post by Manu S

Post by Manu S
INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount/
drwxrwxr-x 10 nobody nobody 4096 Apr 5 13:22 /hdfs_mount/*
I tried the same with *pseudo-distributed node*,but its working fine. I

can

Post by Manu S

--
Thanks & Regards
----
Manu S
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Alexander Lorenz
http://mapredit.blogspot.com

--
Thanks & Regards
----
*Manu S*
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

alo alt

2012-04-26 13:58:36 UTC

Permalink

Yes, as I wrote. You can't use root as user for writing, root (or superuser) has another context in hdfs. Just change into hdfs (su - hdfs) and try again. For all user who should have access to the mounted fs you should create a group and chown them in hdfs (maybe /tmp/group or similar)

best,
Alex

Post by Manu S
Yeah Alex, I tried. But still I am not able to make it
INFO fuse_options.c:162 Adding FUSE arg /hdfs_mount
fuse on /hdfs_mount type fuse (rw,nosuid,nodev,allow_other,default_permissions)
drwxrwxr-x 11 nobody nobody 4096 Apr 9 12:34 /hdfs_mount/
touch: cannot touch `/hdfs_mount/file': Permission denied
Manu,
hadoop-fuse-dfs#dfs://namenode.local:<PORT> /hdfs-mount fuse usetrash,rw 0 0" ?
"mkdir -p /hdfs-mount && chmod 777 /hdfs-mount && echo "hadoop-fuse-dfs#dfs://NN.URI:<PORT> /hdfs-mount fuse usetrash,rw 0 0" >> /etc/fstab && mount -a ; mount"
- Alex

--
Thanks & Regards
----
Manu S
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Alexander Lorenz
http://mapredit.blogspot.com
--
Thanks & Regards
----
Manu S
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Alexander Lorenz
http://mapredit.blogspot.com

alo alt

2012-04-26 14:23:25 UTC

Permalink

Manu,

for clarifying:

root has no access to the mounted HDFS. Just follow the howto:

1. create the group and the users on ALL nodes:
groupadd hdfs-user && adduser USERNAME -G hdfs-user

2. sudo into hdfs:
su - hdfs

3. create a directory in hdfs and change the rights:
hadoop fs -mkdir /someone && hadoop fs -chmod 774 /someone && hadoop fs -chgrp hdfs-user /someone

Now the users you created and added into to group are able to write files.

- Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

Post by alo alt
Yes, as I wrote. You can't use root as user for writing, root (or superuser) has another context in hdfs. Just change into hdfs (su - hdfs) and try again. For all user who should have access to the mounted fs you should create a group and chown them in hdfs (maybe /tmp/group or similar)
best,
Alex

--
Thanks & Regards
----
Manu S
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Alexander Lorenz
http://mapredit.blogspot.com
--
Thanks & Regards
----
Manu S
SI Engineer - OpenSource & HPC
Wipro Infotech
Mob: +91 8861302855 Skype: manuspkd
www.opensourcetalk.co.in

--
Alexander Lorenz
http://mapredit.blogspot.com

Harsh J

2014-05-03 19:11:11 UTC

Permalink

Can you check your dmesg | tail output to see if there are any error
messages from the HDFS fuse client?

Hi,
I m also getting the same error i.e. ?--------- ? ? ? ? ? hdfs
after mounting the hadoo file system using root. Please can you advise how
to fix this. This issue is happening irrespective of version of CDH

Post by Manu S
Dear All
I have the same problem !!!!!!!!!
I have a small cluster of hadoop (ersion 1.1.2) and I want to user hdfs
folder like a general directory .
I install fuse version 2.9.2 ,
but when I want to mount this directory, I have a problem
cat /etc/fstab
/usr/local/hadoop/build/
contrib/fuse-dfs/fuse_dfs#dfs://hb:9000 /export/hdfs fuse usetrash,rw 0 0
port=9000,server=hb
fuse-dfs didn't recognize /export/hdfs,-2
fuse-dfs ignoring option dev
fuse-dfs ignoring option suid
fuse: bad mount point `/export/hdfs': Input/output error
total 0
?--------- ? ? ? ? ? hdfs
appreciate your help

--
---
You received this message because you are subscribed to the Google Groups
"CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

--
Harsh J