使用 spark-submit 部署应用程序：应用程序已添加到调度程序，但尚未激活

Question

我有带有 Linux Centos 12G 内存的 VirtualBox。我需要将 2 个应用程序部署到以非分布式配置运行的 hadoop。这是我的 YARN 配置：<configuration><property>    <name>yarn.nodemanager.pmem-check-enabled</name>    <value>false</value></property><property>   <name>yarn.nodemanager.aux-services</name>   <value>mapreduce_shuffle</value></property><property>   <name>yarn.resourcemanager.address</name>   <value>0.0.0.0:8032</value></property><property>  <name>yarn.scheduler.maximum-allocation-vcores</name>  <value>130</value></property><property>   <name>yarn.nodemanager.vmem-check-enabled</name>   <value>false</value>   <description>Whether virtual memory limits will be enforced for containers</description></property><property>   <name>yarn.scheduler.maximum-allocation-mb</name>   <value>4048</value></property><property>   <name>yarn.nodemanager.vmem-pmem-ratio</name>   <value>1</value>   <description>Ratio between virtual memory to physical memory whensetting memory limits for containers</description></property><property>   <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>   <value>1</value></property></configuration>我部署了第一个应用程序，它运行正常：spark-submit --master yarn --deploy-mode 客户端 --name OryxBatchLayer-ALSExample --class com.cloudera.oryx.batch.Main --files oryx.conf --driver-memory 500m --driver-java-options "-Dconfig.file=oryx.conf" --executor-memory 500m --executor-cores 1 --conf spark.executor.extraJavaOptions="-Dconfig.file=oryx.conf" --conf spark.ui.port= 4040 --conf spark.io.compression.codec=lzf --conf spark.logConf=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.speculation=true --conf spark. ui.showConsoleProgress=false --conf spark.dynamicAllocation.enabled=false --num-executors=1 oryx-batch-2.8.0-SNAPSHOT.jar8088 处的 YARN 管理器表示我正在使用 8 个 vcore 中的 2 个和 8g 内存中的 2 个：

Answer 1

修复是编辑

~/hadoop-3.1.0/etc/hadoop/capacity-scheduler.xml

并将 .1 更新为 1：

<property>

    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>

    <value>1</value>

    <description>

      Maximum percent of resources in the cluster which can be used to run

      application masters i.e. controls number of concurrent running

      applications.

    </description>

  </property>