09.Yarn配置多队列的公平调度器

Leefs 2021-09-11 PM 1510℃ 0条

# 09.Yarn配置多队列的公平调度器

### 一、配置

（1）修改`yarn-site.xml`文件，加入以下参数

```xml
<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description>配置使用公平调度器</description>
</property>

<property>
    <name>yarn.scheduler.fair.allocation.file</name>
    <value>/opt/module/hadoop-3.1.3/etc/hadoop/fair-scheduler.xml</value>
    <description>指明公平调度器队列分配配置文件</description>
</property>

<property>
    <name>yarn.scheduler.fair.preemption</name>
    <value>false</value>
    <description>禁止队列间资源抢占</description>
</property>
```

（2）配置`fair-scheduler.xml`

```xml
<?xml version="1.0"?>
<allocations>
  
  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
  
  <queueMaxResourcesDefault>4096mb,4vcores</queueMaxResourcesDefault>

<queue name="test">
    
    <minResources>2048mb,2vcores</minResources>
    
    <maxResources>4096mb,4vcores</maxResources>
    
    <maxRunningApps>4</maxRunningApps>
    
    <maxAMShare>0.5</maxAMShare>
    
    <weight>1.0</weight>
    
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>
  
  <queue name="demo" type="parent">
    
    <minResources>2048mb,2vcores</minResources>
    
    <maxResources>4096mb,4vcores</maxResources>
    
    <maxRunningApps>4</maxRunningApps>
    
    <maxAMShare>0.5</maxAMShare>
    
    <weight>1.0</weight>
    
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>

<queuePlacementPolicy>
    
    <rule name="specified" create="false"/>
    
    <rule name="nestedUserQueue" create="true">
        <rule name="primaryGroup" create="false"/>
    </rule>
    
    <rule name="reject" />
  </queuePlacementPolicy>
</allocations>
```

（3）分发配置并重启Yarn

```
[root@hadoop102 hadoop]# xsync yarn-site.xml
[root@hadoop102 hadoop]# xsync fair-scheduler.xml

[root@hadoop103 hadoop-3.1.3]# sbin/stop-yarn.sh
[root@hadoop103 hadoop-3.1.3]# sbin/start-yarn.sh
```

### 二、提交测试任务

（1）提交任务时指定队列，按照配置规则，任务会到指定的`root.test`队列

```
[root@hadoop102 hadoop-3.1.3]# hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi -Dmapreduce.job.queuename=root.test 1 1
```

（2）提交任务时不指定队列，按照配置规则，任务会到`root.demo.demo`队列

```
[root@hadoop102 hadoop-3.1.3]# hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 1 1
```

*附：*

*文章来源《尚硅谷大数据之Hadoop》*

标签: Hadoop, Yarn

非特殊说明，本博所有文章均为博主原创。

如若转载，请注明出处：https://lilinchao.com/archives/1437.html

上一篇 08.Yarn容量调度器多队列配置案例

下一篇 VMware16安装Ubuntu16.04教程

09.Yarn配置多队列的公平调度器

评论啦~

栏目分类

标签云

友情链接申请

09.Yarn配置多队列的公平调度器

 评论啦~

 栏目分类

标签云

友情链接申请

评论啦~

栏目分类

标签云

友情链接申请