Discussion:
a question about dfs.replication
(too old to reply)
Francis.Hu
2013-07-01 06:19:31 UTC
Permalink
Hi, All



I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all
configuration work is done, I started all nodes. Then I saved a file into
HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and
also see the file is already listed in the hdfs list.

My question is: The replication column in HDFS web page is showing as 3,
not 2. Does anyone know What the problem is?



---Actual setting of hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>2</value>

</property>



After that, I typed dfsamdin command to check the file:

hdfs fsck /test3/

The result of above command:

/test3/hello005.txt: Under replicated
BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
Target Replicas is 3 but found 2 replica(s).

Status: HEALTHY

Total size: 35 B

Total dirs: 1

Total files: 1

Total blocks (validated): 1 (avg. block size 35 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 1 (33.333332 %)

Number of data-nodes: 3

Number of racks: 1

FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds





Thanks,

Francis Hu
Емельянов Борис
2013-07-01 07:17:51 UTC
Permalink
Post by Francis.Hu
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode
and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml.
After all configuration work is done, I started all nodes. Then I
saved a file into HDFS through java client. nOW I can access hdfs web
page: x.x.x.x:50070,and also see the file is already listed in the
hdfs list.
My question is: *The replication column in HDFS web page is showing as
3, not 2. Does anyone know What the problem is?*
**
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
hdfs fsck /test3/
/test3/hello005.txt: Under replicated
BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only
default replication factor, which can be overrided when putting file to
hdfs.
Francis.Hu
2013-07-01 07:51:51 UTC
Permalink
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.

So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.





发件人: ЕЌельяМПв БПрОс [mailto:emelyanov-***@public.gmane.org]
发送时闎: Monday, July 01, 2013 15:18
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: a question about dfs.replication



On 01.07.2013 10:19, Francis.Hu wrote:

Hi, All



I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.

My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?



---Actual setting of hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>2</value>

</property>



After that, I typed dfsamdin command to check the file:

hdfs fsck /test3/

The result of above command:

/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).

Status: HEALTHY

Total size: 35 B

Total dirs: 1

Total files: 1

Total blocks (validated): 1 (avg. block size 35 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 1 (33.333332 %)

Number of data-nodes: 3

Number of racks: 1

FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds





Thanks,

Francis Hu



If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
yypvsxf19870706
2013-07-01 15:24:26 UTC
Permalink
Hi

Could you please get the property value by using : hdfs getconf -confkey dfs.replication.

·¢×ÔÎÒµÄ iPhone
Post by Francis.Hu
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
·¢ËÍʱŒä: Monday, July 01, 2013 15:18
Ö÷Ìâ: Re: a question about dfs.replication
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
hdfs fsck /test3/
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Francis.Hu
2013-07-02 03:30:12 UTC
Permalink
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"







but in web page ,it is 3 as below:





发件人: yypvsxf19870706 [mailto:yypvsxf19870706-***@public.gmane.org]
发送时闎: Monday, July 01, 2013 23:24
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: 答倍: a question about dfs.replication



Hi



Could you please get the property value by using : hdfs getconf -confkey dfs.replication.


鍙戣嚜鎎戠殑 iPhone


鍊?2013-7-1锛?5:51锛孎rancis.Hu <francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org> 鍐欓亟锛?br>



Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.

So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.





鍙戜欢浜?span lang="EN-US">: 袝屑械谢耜耟薪芯胁 袘芯耉懈耋 [mailto:emelyanov-***@public.gmane.org]
鍙戊€䜹怂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
涓婚: Re: a question about dfs.replication



On 01.07.2013 10:19, Francis.Hu wrote:

Hi, All



I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.

My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?



---Actual setting of hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>2</value>

</property>



After that, I typed dfsamdin command to check the file:

hdfs fsck /test3/

The result of above command:

/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).

Status: HEALTHY

Total size: 35 B

Total dirs: 1

Total files: 1

Total blocks (validated): 1 (avg. block size 35 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 1 (33.333332 %)

Number of data-nodes: 3

Number of racks: 1

FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds





Thanks,

Francis Hu



If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Francis.Hu
2013-07-02 04:25:32 UTC
Permalink
Thanks all of you, I just get the problem fixed through the command:

hdfs dfs -setrep -R -w 2 /



Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?



Thanks,

Francis.Hu



发件人: Francis.Hu [mailto:francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org]
发送时闎: Tuesday, July 02, 2013 11:30
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: 答倍: 答倍: a question about dfs.replication



Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"







but in web page ,it is 3 as below:





发件人: yypvsxf19870706 [mailto:yypvsxf19870706-***@public.gmane.org]
发送时闎: Monday, July 01, 2013 23:24
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: 答倍: a question about dfs.replication



Hi



Could you please get the property value by using : hdfs getconf -confkey dfs.replication.


鍙戣嚜鎎戠殑 iPhone


鍊?2013-7-1锛?5:51锛孎rancis.Hu <francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org> 鍐欓亟锛?br>



Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.

So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.





鍙戜欢浜?span lang="EN-US">: 袝屑械谢耜耟薪芯胁 袘芯耉懈耋 [mailto:emelyanov-***@public.gmane.org]
鍙戊€䜹怂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
涓婚: Re: a question about dfs.replication



On 01.07.2013 10:19, Francis.Hu wrote:

Hi, All



I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.

My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?



---Actual setting of hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>2</value>

</property>



After that, I typed dfsamdin command to check the file:

hdfs fsck /test3/

The result of above command:

/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).

Status: HEALTHY

Total size: 35 B

Total dirs: 1

Total files: 1

Total blocks (validated): 1 (avg. block size 35 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 1 (33.333332 %)

Number of data-nodes: 3

Number of racks: 1

FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds





Thanks,

Francis Hu



If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Azuryy Yu
2013-07-02 04:39:54 UTC
Permalink
It's not HDFS issue.

dfs.replication is a client side configuration, not server side. so you
need to set it to '2' on your client side( your application running on).
THEN execute command such as : hdfs dfs -put or call HDFS API in java
application.
Thanks all of you, I just get the problem fixed through the command: ****
hdfs dfs -setrep -R -w 2 /****
** **
Is that an issue of HDFS ? Why do i need to execute manually a command to
tell the hadoop the replication factor even it is set in hdfs-site.xml ?**
**
** **
Thanks,****
Francis.Hu****
** **
*·¢ËÍʱŒä:* Tuesday, July 02, 2013 11:30
*Ö÷Ìâ:* ŽðžŽ: ŽðžŽ: a question about dfs.replication****
** **
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
****
** **
****
** **
but in web page ,it is 3 as below:****
****
** **
*·¢ËÍʱŒä:* Monday, July 01, 2013 23:24
*Ö÷Ìâ:* Re: ŽðžŽ: a question about dfs.replication****
** **
Hi ****
** **
Could you please get the property value by using : hdfs getconf
-confkey dfs.replication.****
发自我的 iPhone****
****
Actually, My java client is running with the same configuration as the
hadoop's . The dfs.replication is already set as 2 in my hadoop's
configuration.****
So i think the dfs.replication is already overrided by my configuration in
hdfs-site.xml. but seems it doesn't work even i overrided the parameter
evidently.****
****
****
*发é**€**æ—¶é—**?span lang="EN-US">:* Monday, July 01, 2013 15:18
*äž»é**¢˜:* Re: a question about dfs.replication****
****
On 01.07.2013 10:19, Francis.Hu wrote:****
Hi, All****
****
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
all configuration work is done, I started all nodes. Then I saved a file
x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
*
My question is: *The replication column in HDFS web page is showing as
3, not 2. Does anyone know What the problem is?*****
* *****
---Actual setting of hdfs-site.xml****
<property>****
<name>dfs.replication</name>****
<value>2</value>****
</property>****
****
After that, I typed dfsamdin command to check the file:****
hdfs fsck /test3/****
The result of above command:****
/test3/hello005.txt: Under replicated
BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
Target Replicas is 3 but found 2 replica(s).****
Status: HEALTHY****
Total size: 35 B****
Total dirs: 1****
Total files: 1****
Total blocks (validated): 1 (avg. block size 35 B)****
Minimally replicated blocks: 1 (100.0 %)****
Over-replicated blocks: 0 (0.0 %)****
Under-replicated blocks: 1 (100.0 %)****
Mis-replicated blocks: 0 (0.0 %)****
Default replication factor: 2****
Average block replication: 2.0****
Corrupt blocks: 0****
Missing replicas: 1 (33.333332 %)****
Number of data-nodes: 3****
Number of racks: 1****
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
****
****
Thanks,****
Francis Hu****
****
If I'm not mistaking "dfs.replication" parameter in config sets only
default replication factor, which can be overrided when putting file to
hdfs.****
YouPeng Yang
2013-07-02 04:52:01 UTC
Permalink
HI HU and Yu

Aggree with dfs.replication is a client side configuration, not server
side. It make the point in my last mail sense.

And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can
not change the existed file's replication value.
Post by Azuryy Yu
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you
need to set it to '2' on your client side( your application running on).
THEN execute command such as : hdfs dfs -put or call HDFS API in java
application.
Thanks all of you, I just get the problem fixed through the command: ***
*
hdfs dfs -setrep -R -w 2 /****
** **
Is that an issue of HDFS ? Why do i need to execute manually a command to
tell the hadoop the replication factor even it is set in hdfs-site.xml ?*
***
** **
Thanks,****
Francis.Hu****
** **
*·¢ËÍʱŒä:* Tuesday, July 02, 2013 11:30
*Ö÷Ìâ:* ŽðžŽ: ŽðžŽ: a question about dfs.replication****
** **
Yes , it returns 2 correctly after "hdfs getconf -confkey
dfs.replication"****
** **
****
** **
but in web page ,it is 3 as below:****
****
** **
*·¢ËÍʱŒä:* Monday, July 01, 2013 23:24
*Ö÷Ìâ:* Re: ŽðžŽ: a question about dfs.replication****
** **
Hi ****
** **
Could you please get the property value by using : hdfs getconf
-confkey dfs.replication.****
发自我的 iPhone****
****
Actually, My java client is running with the same configuration as the
hadoop's . The dfs.replication is already set as 2 in my hadoop's
configuration.****
So i think the dfs.replication is already overrided by my configuration
in hdfs-site.xml. but seems it doesn't work even i overrided the parameter
evidently.****
****
****
*发é**€**æ—¶é—**?span lang="EN-US">:* Monday, July 01, 2013 15:18
*äž»é**¢˜:* Re: a question about dfs.replication****
****
On 01.07.2013 10:19, Francis.Hu wrote:****
Hi, All****
****
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode
and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml.
After all configuration work is done, I started all nodes. Then I saved a
x.x.x.x:50070,and also see the file is already listed in the hdfs list.**
**
My question is: *The replication column in HDFS web page is showing as
3, not 2. Does anyone know What the problem is?*****
* *****
---Actual setting of hdfs-site.xml****
<property>****
<name>dfs.replication</name>****
<value>2</value>****
</property>****
****
After that, I typed dfsamdin command to check the file:****
hdfs fsck /test3/****
The result of above command:****
/test3/hello005.txt: Under replicated
BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
Target Replicas is 3 but found 2 replica(s).****
Status: HEALTHY****
Total size: 35 B****
Total dirs: 1****
Total files: 1****
Total blocks (validated): 1 (avg. block size 35 B)****
Minimally replicated blocks: 1 (100.0 %)****
Over-replicated blocks: 0 (0.0 %)****
Under-replicated blocks: 1 (100.0 %)****
Mis-replicated blocks: 0 (0.0 %)****
Default replication factor: 2****
Average block replication: 2.0****
Corrupt blocks: 0****
Missing replicas: 1 (33.333332 %)****
Number of data-nodes: 3****
Number of racks: 1****
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
****
****
Thanks,****
Francis Hu****
****
If I'm not mistaking "dfs.replication" parameter in config sets only
default replication factor, which can be overrided when putting file to
hdfs.****
Francis.Hu
2013-07-02 06:29:40 UTC
Permalink
YouPeng Yang,



you said that may be the answer. Thank you.



发件人: YouPeng Yang [mailto:yypvsxf19870706-***@public.gmane.org]
发送时闎: Tuesday, July 02, 2013 12:52
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: reply: a question about dfs.replication



HI HU and Yu



Aggree with dfs.replication is a client side configuration, not server side. It make the point in my last mail sense.



And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can not change the existed file's replication value.







2013/7/2 Azuryy Yu <azuryyyu-***@public.gmane.org>

It's not HDFS issue.

dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.





On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org> wrote:

Thanks all of you, I just get the problem fixed through the command:

hdfs dfs -setrep -R -w 2 /



Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?



Thanks,

Francis.Hu



发件人: Francis.Hu [mailto:francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org]
发送时闎: Tuesday, July 02, 2013 11:30
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: 答倍: 答倍: a question about dfs.replication



Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"







but in web page ,it is 3 as below:





发件人: yypvsxf19870706 [mailto:yypvsxf19870706-***@public.gmane.org]
发送时闎: Monday, July 01, 2013 23:24
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: 答倍: a question about dfs.replication



Hi



Could you please get the property value by using : hdfs getconf -confkey dfs.replication.


鍙戣嚜鎎戠殑 iPhone


鍊?2013-7-1锛?5:51锛孎rancis.Hu <francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org> 鍐欓亟锛?br>



Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.

So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.





鍙戜欢浜?span lang="EN-US">: 袝屑械谢耜耟薪芯胁 袘芯耉懈耋 [mailto:emelyanov-***@public.gmane.org]
鍙戊€䜹怂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
涓婚: Re: a question about dfs.replication



On 01.07.2013 10:19, Francis.Hu wrote:

Hi, All



I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.

My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?



---Actual setting of hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>2</value>

</property>



After that, I typed dfsamdin command to check the file:

hdfs fsck /test3/

The result of above command:

/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).

Status: HEALTHY

Total size: 35 B

Total dirs: 1

Total files: 1

Total blocks (validated): 1 (avg. block size 35 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 1 (33.333332 %)

Number of data-nodes: 3

Number of racks: 1

FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds





Thanks,

Francis Hu



If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Francis.Hu
2013-07-02 06:22:17 UTC
Permalink
actually, my client side is already set to "2".



发件人: Azuryy Yu [mailto:azuryyyu-***@public.gmane.org]
发送时闎: Tuesday, July 02, 2013 12:40
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: reply: a question about dfs.replication



It's not HDFS issue.

dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.





On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org> wrote:

Thanks all of you, I just get the problem fixed through the command:

hdfs dfs -setrep -R -w 2 /



Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?



Thanks,

Francis.Hu



发件人: Francis.Hu [mailto:francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org]
发送时闎: Tuesday, July 02, 2013 11:30
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: 答倍: 答倍: a question about dfs.replication



Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"







but in web page ,it is 3 as below:





发件人: yypvsxf19870706 [mailto:yypvsxf19870706-***@public.gmane.org]
发送时闎: Monday, July 01, 2013 23:24
收件人: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
䞻题: Re: 答倍: a question about dfs.replication



Hi



Could you please get the property value by using : hdfs getconf -confkey dfs.replication.


鍙戣嚜鎎戠殑 iPhone


鍊?2013-7-1锛?5:51锛孎rancis.Hu <francis.hu-As5zvvbeLCpDRwzJAX7SwQC/***@public.gmane.org> 鍐欓亟锛?br>



Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.

So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.





鍙戜欢浜?span lang="EN-US">: 袝屑械谢耜耟薪芯胁 袘芯耉懈耋 [mailto:emelyanov-***@public.gmane.org]
鍙戊€䜹怂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
涓婚: Re: a question about dfs.replication



On 01.07.2013 10:19, Francis.Hu wrote:

Hi, All



I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.

My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?



---Actual setting of hdfs-site.xml

<property>

<name>dfs.replication</name>

<value>2</value>

</property>



After that, I typed dfsamdin command to check the file:

hdfs fsck /test3/

The result of above command:

/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).

Status: HEALTHY

Total size: 35 B

Total dirs: 1

Total files: 1

Total blocks (validated): 1 (avg. block size 35 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 2

Average block replication: 2.0

Corrupt blocks: 0

Missing replicas: 1 (33.333332 %)

Number of data-nodes: 3

Number of racks: 1

FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds





Thanks,

Francis Hu



If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
YouPeng Yang
2013-07-02 04:41:48 UTC
Permalink
Hi Hu
It comes an point in my mind.And I tested it.
Before you set the dfs.replication to 2,the block may have already
existed with the original replication value 3.
After you changed the value, the replication of the former block was
still 3.

So the file that you created after the changement will be with the
replication 2.

However I do not have the access to change the replication within
the existed
files.
Post by Francis.Hu
Yes , it returns 2 correctly after "hdfs getconf -confkey
dfs.replication"****
** **
****
** **
but in web page ,it is 3 as below:****
****
** **
*·¢ËÍʱŒä:* Monday, July 01, 2013 23:24
*Ö÷Ìâ:* Re: ŽðžŽ: a question about dfs.replication****
** **
Hi ****
** **
Could you please get the property value by using : hdfs getconf
-confkey dfs.replication.****
发自我的 iPhone****
****
Actually, My java client is running with the same configuration as the
hadoop's . The dfs.replication is already set as 2 in my hadoop's
configuration.****
So i think the dfs.replication is already overrided by my configuration in
hdfs-site.xml. but seems it doesn't work even i overrided the parameter
evidently.****
****
****
*发é**€**æ—¶é—**?span lang="EN-US">:* Monday, July 01, 2013 15:18
*äž»é**¢˜:* Re: a question about dfs.replication****
****
On 01.07.2013 10:19, Francis.Hu wrote:****
Hi, All****
****
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
all configuration work is done, I started all nodes. Then I saved a file
x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
*
My question is: *The replication column in HDFS web page is showing as
3, not 2. Does anyone know What the problem is?*****
* *****
---Actual setting of hdfs-site.xml****
<property>****
<name>dfs.replication</name>****
<value>2</value>****
</property>****
****
After that, I typed dfsamdin command to check the file:****
hdfs fsck /test3/****
The result of above command:****
/test3/hello005.txt: Under replicated
BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
Target Replicas is 3 but found 2 replica(s).****
Status: HEALTHY****
Total size: 35 B****
Total dirs: 1****
Total files: 1****
Total blocks (validated): 1 (avg. block size 35 B)****
Minimally replicated blocks: 1 (100.0 %)****
Over-replicated blocks: 0 (0.0 %)****
Under-replicated blocks: 1 (100.0 %)****
Mis-replicated blocks: 0 (0.0 %)****
Default replication factor: 2****
Average block replication: 2.0****
Corrupt blocks: 0****
Missing replicas: 1 (33.333332 %)****
Number of data-nodes: 3****
Number of racks: 1****
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
****
****
Thanks,****
Francis Hu****
****
If I'm not mistaking "dfs.replication" parameter in config sets only
default replication factor, which can be overrided when putting file to
hdfs.****
Continue reading on narkive:
Loading...