hdfs 显示本地文件列表

Question

我在 OS X 中安装了 Hadoop，并且运行良好。我的经验是最近的，并且正在努力学习有关使用 Hadoop 进行应用程序开发的更多信息。昨天，当我需要在 Hadoop 中查找目录和/或文件列表时，我只需键入$ hadoop fs -ls 并且，它会显示集群中的所有内容。今天，它显示了文件系统中的所有本地内容。我必须提供 hdfs 的确切地址才能获得内容列表，$ hadoop fs -ls hdfs://localhost:8020/user/myName我的core-site.xml文件和以前一样，<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--  Licensed under the Apache License, Version 2.0 (the "License");  you may not use this file except in compliance with the License.  You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0  Unless required by applicable law or agreed to in writing, software  distributed under the License is distributed on an "AS IS" BASIS,  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and  limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><!-- Put site-specific property overrides in this file. --> <configuration>    <property>        <name>hadoop.tmp.dir</name>        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>        <description>A base for other temporary directories.</description>    </property>    <property>        <name>fs.default.name</name>        <value>hdfs://localhost:8020</value>    </property></configuration>在启动 hadoop 守护进程之前，我停止了集群并再次使用以下命令重新格式化分布式文件系统，以便我们可以在执行 map-reduce 作业时将数据源放入 hdfs 文件系统$ hdfs namenode -format我收到管理员报告通知 FileSystem file:/// 不是 HDFS 文件系统，$ hadoop dfsadmin -reportWARNING: Use of this script to execute dfsadmin is deprecated.WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.2018-10-18 18:01:27,316 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicablereport: FileSystem file:/// is not an HDFS file systemUsage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]

Answer 1

您需要确保已添加了一个环境变量HADOOP_CONF_DIR，该变量被调用以设置到包含来自 Hadoop 的 XML 文件的目录中。

您可以.bashrc在主文件夹中执行此操作

否则，您将获得默认文件系统 ，file://它仍然有效并且仍然可以正常运行 MapReduce 作业

FWIW，这是我的核心站点

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>file:///tmp/hadoop/hdfs/tmp</value>

        <description>A base for other temporary directories.</description>

    </property>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://localhost:9000</value>

  </property>

</configuration>

和 hdfs 站点

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

        <name>dfs.replication</name>

        <value>1</value>

    </property>

  <property>

      <name>dfs.namenode.name.dir</name>

      <value>file:///tmp/hadoop/hdfs/names</value>

  </property>

  <property>

    <name>fs.checkpoint.dir</name>

    <value>file:///tmp/hadoop/hdfs/checkpoint</value>

  </property>

  <property>

    <name>fs.checkpoint.edits.dir</name>

    <value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>

  </property>

  <property>

      <name>dfs.datanode.data.dir</name>

      <value>file:///tmp/hadoop/hdfs/data</value>

  </property>

</configuration>

Answer 2

在 core-site.xml 文件中进行如下编辑。

<value>hdfs://localhost.localdomain:8020/</value>

我相信缺少斜线 (8020/) 会造成这个问题。尝试一下。