HDFS?Balancer負載均衡器及語法詳解

更新時間：2023年03月31日 11:24:22 作者：huan1993

這篇文章主要為大家介紹了HDFS?Balancer負載均衡器及語法詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪

1、背景

當我們的hadoop集群運行了一段時間之后，各個DataNode上的數(shù)據(jù)分布并不一定是均勻分布的。比如說： 我們向現(xiàn)有集群中添加了一個新的DataNode。

2、什么是平衡

此處是我自己的一個簡單的理解

所謂的平衡指的是 每個DataNode的利用率 與 集群的利用率 之間相差不超過給定的閾值百分比。此處的平衡 指的是各個DataNode之間的平衡，同一個DataNode之間的各個磁盤是不會平衡的。

2.1 每個DataNode的利用率計算

DataNode的利用率= dfs已用的空間 / 分配給dfs的空間。

注意： 分配給dfs的空間 不是磁盤的總空間。

2.2 集群的利用率

集群的利用率= 各datanode dfs已使用的空間 / 各datanode總空間

2.3 平衡

假設平衡的閾值是 5%，集群的利用率是 37.5，那么每個節(jié)點的利用率在32.5%到42.5%之間都認為是均衡的。也就是說，極端情況下，DataNode的利用率最大相差10%。

3、hdfs balancer語法

[hadoopdeploy@hadoop01 ~]$ hdfs balancer --help
Usage: hdfs balancer
	[-policy <policy>]	the balancing policy: datanode or blockpool
	[-threshold <threshold>]	Percentage of disk capacity
	[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]	Excludes the specified datanodes.
	[-include [-f <hosts-file> | <comma-separated list of hosts>]]	Includes only the specified datanodes.
	[-source [-f <hosts-file> | <comma-separated list of hosts>]]	Pick only the specified datanodes as source nodes.
	[-blockpools <comma-separated list of blockpool ids>]	The balancer will only run on blockpools included in this list.
	[-idleiterations <idleiterations>]	Number of consecutive idle iterations (-1 for Infinite) before exit.
	[-runDuringUpgrade]	Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.
	[-asService]	Run as a long running service.

Generic options supported are:
-conf <configuration file>        specify an application configuration file
-D <property=value>               define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port>  specify a ResourceManager
-files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines

The general command line syntax is:
command [genericOptions] [commandOptions]

參數(shù)	描述
threshold	磁盤容量的百分比。默認值為10%，表示上下浮動10%。
policy	平衡策略。 datanode（默認）：當每一個DataNode是平衡的時候，集群就是平衡的。 blockpool：當每一個DataNode中的blockpool是平衡的，集群就是平衡的。
exclude	不參與平衡的DataNode節(jié)點
include	參與平衡的DataNode節(jié)點
source	僅選取指定的數(shù)據(jù)節(jié)點作為源節(jié)點
blockpools	Balancer僅在指定的blockpools中運行
idleiterations	退出前的連續(xù)空閑迭代次數(shù)（-1表示無限）
-runDuringUpgrade	是否在正在進行的HDFS升級過程中運行平衡器。通常不需要這樣做，因為這不會影響過度使用的計算機上的已用空間。
-asService	作為長期運行的服務運行

4、運行一個簡單的balance案例

4.1 設置平衡數(shù)據(jù)傳輸帶寬

[hadoopdeploy@hadoop01 ~]$ hdfs dfsadmin  -setBalancerBandwidth 10485760
Balancer bandwidth is set to 10485760
[hadoopdeploy@hadoop01 ~]$

當我們的集群負載需要調(diào)低這個值，當我們的集群負載較低時，可以適當調(diào)高這個值。

4.2 執(zhí)行banalce

[hadoopdeploy@hadoop01 ~]$ hdfs balancer -policy datanode -threshold 5
2023-03-26 14:10:09,785 INFO balancer.Balancer: Using a threshold of 5.0
2023-03-26 14:10:09,786 INFO balancer.Balancer: namenodes  = [hdfs://hadoop01:8020]
2023-03-26 14:10:09,786 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
2023-03-26 14:10:09,786 INFO balancer.Balancer: included nodes = []
2023-03-26 14:10:09,786 INFO balancer.Balancer: excluded nodes = []
2023-03-26 14:10:09,786 INFO balancer.Balancer: source nodes = []
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved  NameNode
2023-03-26 14:10:09,787 INFO balancer.NameNodeConnector: getBlocks calls for hdfs://hadoop01:8020 will be rate-limited to 20 per second
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.namenode.get-blocks.max-qps = 20 (default=20)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 100 (default=100)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.bandwidthPerSec = 104857600 (default=104857600)
2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)
2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.141:9866
2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.140:9866
2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.142:9866
2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 over-utilized: []
2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 underutilized: []
2023-3-26 14:10:10                0                  0 B                 0 B                0 B                  0  hdfs://hadoop01:8020
The cluster is balanced. Exiting...
2023-3-26 14:10:10       Balancing took 810.0 milliseconds
[hadoopdeploy@hadoop01 ~]$