Discussion:
Replication problem of HDFS
(too old to reply)
ChaoChun Liang
2007-09-05 08:58:15 UTC
Permalink
According the reference of
HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
a file in the HDFS will be split into one or more blocks and these blocks
are stored in
a set of Datanodes.

I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found only
the
namenode increase the block numbers, others nodes keep the same value. It
means
all blocks copied to the namenode, none to datanodes. Is it correct?

ChaoChun
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
Sent from the Hadoop Users mailing list archive at Nabble.com.
Dhruba Borthakur
2007-09-05 21:44:49 UTC
Permalink
Hi ChaoChun,

I do not fully understand your problem. I am guessing that you are running a
Datanode on the same machine as the Namenode. I am also guessing that you
are using the Namenode machine as a client to upload a file into HDFS. In
this case, the only replica of the file will reside on the Datanode that is
local to the client.

Thanks,
dhruba

-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn-***@public.gmane.org]
Sent: Wednesday, September 05, 2007 1:58 AM
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: Replication problem of HDFS


According the reference of
HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
a file in the HDFS will be split into one or more blocks and these blocks
are stored in
a set of Datanodes.

I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found only
the
namenode increase the block numbers, others nodes keep the same value. It
means
all blocks copied to the namenode, none to datanodes. Is it correct?

ChaoChun
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
Sent from the Hadoop Users mailing list archive at Nabble.com.
ChaoChun Liang
2007-09-06 01:26:27 UTC
Permalink
Yes, you are right. the namenode and datanode are in the same machine
and upload data into HDFS in the same one in my environment. I suppose
the HDFS will distribute these blocks to all others datanode(according the
HDFS reference), but it is not actually.
Post by Dhruba Borthakur
Inthis case, the only replica of the file will reside on the Datanode that
is
Post by Dhruba Borthakur
local to the client.
So, does it conflict with the HDFS reference? (a file in the HDFS will be
split into
one or more blocks and these blocks are stored in a set of Datanodes. )

What kind of uploading to let all data/files store into the datanodes(not a
single one)?

ChaoChun
Post by Dhruba Borthakur
Hi ChaoChun,
I do not fully understand your problem. I am guessing that you are running a
Datanode on the same machine as the Namenode. I am also guessing that you
are using the Namenode machine as a client to upload a file into HDFS. In
this case, the only replica of the file will reside on the Datanode that is
local to the client.
Thanks,
dhruba
-----Original Message-----
Sent: Wednesday, September 05, 2007 1:58 AM
Subject: Replication problem of HDFS
According the reference of
HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
a file in the HDFS will be split into one or more blocks and these blocks
are stored in
a set of Datanodes.
I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found only
the
namenode increase the block numbers, others nodes keep the same value. It
means
all blocks copied to the namenode, none to datanodes. Is it correct?
ChaoChun
--
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
Sent from the Hadoop Users mailing list archive at Nabble.com.
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12514090
Sent from the Hadoop Users mailing list archive at Nabble.com.
Stu Hood
2007-09-06 04:45:28 UTC
Permalink
ChaoChun,

Since you set the 'replication = 1' for the file, only 1 copy of the file's blocks will be stored in Hadoop. If you want all 5 machines to have copies of each block, then you would set 'replication = 5' for the file.

The default for replication is 3.

Thanks,
Stu



-----Original Message-----
From: ChaoChun Liang
Sent: Wednesday, September 5, 2007 9:26pm
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: RE: Replication problem of HDFS


Yes, you are right. the namenode and datanode are in the same machine
and upload data into HDFS in the same one in my environment. I suppose
the HDFS will distribute these blocks to all others datanode(according the
HDFS reference), but it is not actually.
Post by Dhruba Borthakur
Inthis case, the only replica of the file will reside on the Datanode that
is
Post by Dhruba Borthakur
local to the client.
So, does it conflict with the HDFS reference? (a file in the HDFS will be
split into
one or more blocks and these blocks are stored in a set of Datanodes. )

What kind of uploading to let all data/files store into the datanodes(not a
single one)?

ChaoChun
Post by Dhruba Borthakur
Hi ChaoChun,
I do not fully understand your problem. I am guessing that you are running
a
Datanode on the same machine as the Namenode. I am also guessing that you
are using the Namenode machine as a client to upload a file into HDFS. In
this case, the only replica of the file will reside on the Datanode that
is
local to the client.
Thanks,
dhruba
-----Original Message-----
Sent: Wednesday, September 05, 2007 1:58 AM
Subject: Replication problem of HDFS
According the reference of
HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
a file in the HDFS will be split into one or more blocks and these blocks
are stored in
a set of Datanodes.
I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found
only
the
namenode increase the block numbers, others nodes keep the same value. It
means
all blocks copied to the namenode, none to datanodes. Is it correct?
ChaoChun
--
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
Sent from the Hadoop Users mailing list archive at Nabble.com.
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12514090
Sent from the Hadoop Users mailing list archive at Nabble.com.
ChaoChun Liang
2007-09-07 02:23:09 UTC
Permalink
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.

ChaoChun
Post by Dhruba Borthakur
ChaoChun,
Since you set the 'replication = 1' for the file, only 1 copy of the
file's blocks will be stored in Hadoop. If you want all 5 machines to have
copies of each block, then you would set 'replication = 5' for the file.
The default for replication is 3.
Thanks,
Stu
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12534839
Sent from the Hadoop Users mailing list archive at Nabble.com.
Earney, Billy C.
2007-09-10 17:43:39 UTC
Permalink
ChoaChun,

I'm new to hadoop, but my understanding is that the data is divided into
blocks, and that not all blocks need be on the same node. So if a file
has 2 blocks the first block of a file could be on node 1 and the second
block could be on node 2. From the link below, it seems that for each
block, the client will contact the namenode and request one or more
datanodes to store the block on.

http://lucene.apache.org/hadoop/hdfs_design.html#Replication+Pipelining

Is my understanding of the documentation correct?


-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn-***@public.gmane.org]
Sent: Thursday, September 06, 2007 9:23 PM
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: RE: Replication problem of HDFS


So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put),
not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N)
and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.

ChaoChun
Post by Dhruba Borthakur
ChaoChun,
Since you set the 'replication = 1' for the file, only 1 copy of the
file's blocks will be stored in Hadoop. If you want all 5 machines to have
copies of each block, then you would set 'replication = 5' for the file.
The default for replication is 3.
Thanks,
Stu
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a125348
39
Sent from the Hadoop Users mailing list archive at Nabble.com.
Dhruba Borthakur
2007-09-10 21:19:47 UTC
Permalink
Hi ChoaChun,

Your explanation sounds right.

Thanks,
dhruba

-----Original Message-----
From: Earney, Billy C. [mailto:earney-Ejzj3ReGi3n2fBVCVOL8/***@public.gmane.org]
Sent: Monday, September 10, 2007 10:44 AM
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: RE: Replication problem of HDFS

ChoaChun,

I'm new to hadoop, but my understanding is that the data is divided into
blocks, and that not all blocks need be on the same node. So if a file
has 2 blocks the first block of a file could be on node 1 and the second
block could be on node 2. From the link below, it seems that for each
block, the client will contact the namenode and request one or more
datanodes to store the block on.

http://lucene.apache.org/hadoop/hdfs_design.html#Replication+Pipelining

Is my understanding of the documentation correct?


-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn-***@public.gmane.org]
Sent: Thursday, September 06, 2007 9:23 PM
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: RE: Replication problem of HDFS


So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put),
not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N)
and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.

ChaoChun
Post by Dhruba Borthakur
ChaoChun,
Since you set the 'replication = 1' for the file, only 1 copy of the
file's blocks will be stored in Hadoop. If you want all 5 machines to have
copies of each block, then you would set 'replication = 5' for the file.
The default for replication is 3.
Thanks,
Stu
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a125348
39
Sent from the Hadoop Users mailing list archive at Nabble.com.
ChaoChun Liang
2007-09-11 03:06:16 UTC
Permalink
According the HDFS reference, it sounds right.

ChaoChun
Post by Earney, Billy C.
ChoaChun,
I'm new to hadoop, but my understanding is that the data is divided into
blocks, and that not all blocks need be on the same node. So if a file
has 2 blocks the first block of a file could be on node 1 and the second
block could be on node 2. From the link below, it seems that for each
block, the client will contact the namenode and request one or more
datanodes to store the block on.
http://lucene.apache.org/hadoop/hdfs_design.html#Replication+Pipelining
Is my understanding of the documentation correct?
-----Original Message-----
Sent: Thursday, September 06, 2007 9:23 PM
Subject: RE: Replication problem of HDFS
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.
ChaoChun
Post by Dhruba Borthakur
ChaoChun,
Since you set the 'replication = 1' for the file, only 1 copy of the
file's blocks will be stored in Hadoop. If you want all 5 machines to
have
Post by Dhruba Borthakur
copies of each block, then you would set 'replication = 5' for the
file.
Post by Dhruba Borthakur
The default for replication is 3.
Thanks,
Stu
--
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a125348
39
Sent from the Hadoop Users mailing list archive at Nabble.com.
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12607024
Sent from the Hadoop Users mailing list archive at Nabble.com.
Stu Hood
2007-09-07 23:03:22 UTC
Permalink
Post by ChaoChun Liang
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes. It will store the blocks in 'replication' datanodes. If "replication == 2" then it will make sure that 2 copies of each of the M blocks exist on datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks.No, it means to replicate the file to N datanodes. The client is only used to transfer files to/from Hadoop: it doesn't do any long term storage.
Thanks,
Stu


-----Original Message-----
From: ChaoChun Liang
Sent: Thursday, September 6, 2007 10:23pm
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: RE: Replication problem of HDFS


So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.

ChaoChun
Post by ChaoChun Liang
ChaoChun,
Since you set the 'replication = 1' for the file, only 1 copy of the
file's blocks will be stored in Hadoop. If you want all 5 machines to have
copies of each block, then you would set 'replication = 5' for the file.
The default for replication is 3.
Thanks,
Stu
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12534839
Sent from the Hadoop Users mailing list archive at Nabble.com.
ChaoChun Liang
2007-09-11 03:04:11 UTC
Permalink
Post by Dhruba Borthakur
Post by ChaoChun Liang
If "replication == 2" then it will make sure that 2 copies of each of the
M blocks exist on datanodes.
No, it means to replicate the file to N datanodes. The client is only used
to transfer files to/from >>Hadoop: it doesn't do any long term storage.

Sorry, the client what I means is the datanodes.

In my application, whether M blocks(described as above) exist in the name
datanode(i.e. each database
owns a completed M block), or shared M blocks for datanodes in the HDFS is
important for us.

If these M blocks could be shared, we may use the HDFS, otherswise we may
condiser the local file system
for the map/reduce processing.

ChaoChun

-----Original Message-----
From: ChaoChun Liang
Sent: Thursday, September 6, 2007 10:23pm
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: RE: Replication problem of HDFS


So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.

ChaoChun
Post by Dhruba Borthakur
ChaoChun,
Since you set the 'replication = 1' for the file, only 1 copy of the
file's blocks will be stored in Hadoop. If you want all 5 machines to have
copies of each block, then you would set 'replication = 5' for the file.
The default for replication is 3.
Thanks,
Stu
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12534839
Sent from the Hadoop Users mailing list archive at Nabble.com.
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12607008
Sent from the Hadoop Users mailing list archive at Nabble.com.
Ted Dunning
2007-09-11 06:34:47 UTC
Permalink
Your question is very hard to understand. The problem may be the names of
the different kinds of server.

There is one namenode and there are many datanodes.

Each file is divided into one or more blocks. By default the block has a
maximum size of 64MB. Each block from a file is stored on one or more
datanodes. The number of datanodes holding each block is called replication
factor. The namenode holds information about what blocks are in each file.
The namenode also contains information about what blocks each datanode
holds.

As an example, consider that you have 3 files called A, B, and C. Each file
is 150MB so they have two full size blocks (A1, A2, B1, B2, C1, C2) and one
partial block that is 22MB in size (A3, B3, C3).

Suppose that replication factor is 1 for A, 2 for B and 3 for C.

One possible state of five datanodes is this:

Datanode1:
A1, B2, C3, C1

Datanode2:
A2, C2, B2

Datanode3:
A3, C1, C3, B1

Datanode4:
B1, C1, C2, B3

Datanode5:
B3, C2, C3

The namenode would contain this information:

A -> (A1, A2, A3)
B -> (B1, B2, B3)
C -> (C1, C2, C3)

A1 -> (Datanode1)
B1 -> (Datanode3, Datanode4)
C1 -> (Datanode1, Datanode3, Datanode4)
... And so on ...

Does that help?
Post by ChaoChun Liang
In my application, whether M blocks(described as above) exist in the name
datanode(i.e. each database
owns a completed M block), or shared M blocks for datanodes in the HDFS is
important for us.
If these M blocks could be shared, we may use the HDFS, otherswise we may
condiser the local file system
for the map/reduce processing.
ChaoChun Liang
2007-09-13 06:11:38 UTC
Permalink
Thanks for your detail example and explanation.

The problem what I met is, all split blocks stored in the same datanode,
that is, (A1, A2, A3) stored in the same datanode in your example.

My test case is putting (by "hadoop fs -put" command) a file about 1GB to
HDFS
with 4 datanodes, where the namenode and datanode are in the same machine).
dfs.block.size=67108864 and dfs.replication=1 in the hadoop-site.xml

And uploading from the namenode(i.e A3 as following) machine.
P.S. the namenode and datanode in the same machine, A3.

The datanode status "before" the uploading process is
------------------------------------------------------------------------
Node Last Contact Admin State Size (GB) Used (%) Blocks
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 70.13 18
A4 1 In Service 39.06 18.52 1

The datanode status "after" the uploading process is
------------------------------------------------------------------------
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 71.95 35
A4 1 In Service 39.06 18.52 1

You can see that blocks increases only in the A3 node (by 17 blocks, from 18
to 35),
and the block numbers is others datanodes are the same and not changed.

Is it look something wrong? Or it is the configuration problem.

ChaoChun
Post by Ted Dunning
Your question is very hard to understand. The problem may be the names of
the different kinds of server.
There is one namenode and there are many datanodes.
Each file is divided into one or more blocks. By default the block has a
maximum size of 64MB. Each block from a file is stored on one or more
datanodes. The number of datanodes holding each block is called replication
factor. The namenode holds information about what blocks are in each file.
The namenode also contains information about what blocks each datanode
holds.
As an example, consider that you have 3 files called A, B, and C. Each file
is 150MB so they have two full size blocks (A1, A2, B1, B2, C1, C2) and one
partial block that is 22MB in size (A3, B3, C3).
Suppose that replication factor is 1 for A, 2 for B and 3 for C.
A1, B2, C3, C1
A2, C2, B2
A3, C1, C3, B1
B1, C1, C2, B3
B3, C2, C3
A -> (A1, A2, A3)
B -> (B1, B2, B3)
C -> (C1, C2, C3)
A1 -> (Datanode1)
B1 -> (Datanode3, Datanode4)
C1 -> (Datanode1, Datanode3, Datanode4)
... And so on ...
Does that help?
Post by ChaoChun Liang
In my application, whether M blocks(described as above) exist in the name
datanode(i.e. each database
owns a completed M block), or shared M blocks for datanodes in the HDFS is
important for us.
If these M blocks could be shared, we may use the HDFS, otherswise we may
condiser the local file system
for the map/reduce processing.
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12649233
Sent from the Hadoop Users mailing list archive at Nabble.com.
Ted Dunning
2007-09-13 15:50:11 UTC
Permalink
This is a knonw behavior (a feature, even). When yu write on a datanode, it
prefers to put the data on that node because it is local.

To avoid this r
un the put on a non-datanode.

Or do the put with a higher replication and drop the replication after the
put.

Or use distcp if all of the data nodes have access to the same data (perhaps
via nfs).
Post by ChaoChun Liang
Thanks for your detail example and explanation.
The problem what I met is, all split blocks stored in the same datanode,
that is, (A1, A2, A3) stored in the same datanode in your example.
My test case is putting (by "hadoop fs -put" command) a file about 1GB to
HDFS
...
Post by ChaoChun Liang
Is it look something wrong? Or it is the configuration problem.
Dhruba Borthakur
2007-09-13 22:55:08 UTC
Permalink
This is expected behaviour. Since you have 4 datanodes, it might make sense
to bump up the replication factor to 2 or higher. Then you would see other
Datanodes getting filled up with blocks.

Thanks,
dhruba

-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn-***@public.gmane.org]
Sent: Wednesday, September 12, 2007 11:12 PM
To: hadoop-user-PPu3vs9EauNd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: Re: Replication problem of HDFS


Thanks for your detail example and explanation.

The problem what I met is, all split blocks stored in the same datanode,
that is, (A1, A2, A3) stored in the same datanode in your example.

My test case is putting (by "hadoop fs -put" command) a file about 1GB to
HDFS
with 4 datanodes, where the namenode and datanode are in the same machine).
dfs.block.size=67108864 and dfs.replication=1 in the hadoop-site.xml

And uploading from the namenode(i.e A3 as following) machine.
P.S. the namenode and datanode in the same machine, A3.

The datanode status "before" the uploading process is
------------------------------------------------------------------------
Node Last Contact Admin State Size (GB) Used (%)
Blocks
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 70.13 18
A4 1 In Service 39.06 18.52 1

The datanode status "after" the uploading process is
------------------------------------------------------------------------
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 71.95 35
A4 1 In Service 39.06 18.52 1

You can see that blocks increases only in the A3 node (by 17 blocks, from 18
to 35),
and the block numbers is others datanodes are the same and not changed.

Is it look something wrong? Or it is the configuration problem.

ChaoChun
Post by Ted Dunning
Your question is very hard to understand. The problem may be the names of
the different kinds of server.
There is one namenode and there are many datanodes.
Each file is divided into one or more blocks. By default the block has a
maximum size of 64MB. Each block from a file is stored on one or more
datanodes. The number of datanodes holding each block is called replication
factor. The namenode holds information about what blocks are in each file.
The namenode also contains information about what blocks each datanode
holds.
As an example, consider that you have 3 files called A, B, and C. Each file
is 150MB so they have two full size blocks (A1, A2, B1, B2, C1, C2) and one
partial block that is 22MB in size (A3, B3, C3).
Suppose that replication factor is 1 for A, 2 for B and 3 for C.
A1, B2, C3, C1
A2, C2, B2
A3, C1, C3, B1
B1, C1, C2, B3
B3, C2, C3
A -> (A1, A2, A3)
B -> (B1, B2, B3)
C -> (C1, C2, C3)
A1 -> (Datanode1)
B1 -> (Datanode3, Datanode4)
C1 -> (Datanode1, Datanode3, Datanode4)
... And so on ...
Does that help?
Post by ChaoChun Liang
In my application, whether M blocks(described as above) exist in the name
datanode(i.e. each database
owns a completed M block), or shared M blocks for datanodes in the HDFS is
important for us.
If these M blocks could be shared, we may use the HDFS, otherswise we may
condiser the local file system
for the map/reduce processing.
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12649233
Sent from the Hadoop Users mailing list archive at Nabble.com.
Continue reading on narkive:
Loading...