Centos7部署hadoop（单机）

1.修改主机名称（需要在root用户下执行）

hostnamectl set-hostname 需要修改的主机名称

或者修改配置文件 vim /etc/hostname

2.如果主机没有固定IP，需要固定IP（这一步自行查询）

3.关闭防火墙

systemctl start firewalld.service  #开启防火墙
systemctl restart firewalld.service #重启防火墙
systemctl stop firewalld.service #关闭防火墙
systemctl status firewalld.service # 防火墙状态

4.禁用selinux

永久关闭selinux 安全策略，可以修改/etc/selinux/config, 将SELINUX=enforcing 修改为SELINUX=disabled

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

5.设置ssh免密登录

进入/root/.ssh储存密钥文件夹，通过ls -l指令查看是否有旧密钥

cd /root/.ssh #进入秘钥存放目录

rm -rf * #删除旧秘钥

使用ssh-keygen -t dsa 命令生成秘钥，在这个过程中需要多次回车键选取默认配置

[root@localhost .ssh]# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
SHA256:QNHQYbzj9rWNmAItP5x root@hadoopmaster
The key's randomart image is:
+---[DSA 1024]----+
|      +*oo . .   |
|     . 
|       B * * 
|         +  oo.oo|
+----[SHA256]-----+

将生成的密钥文件id_dsa.pub 复制到SSH指定的密钥文件中authorized_keys中

cat id_dsa.pub >>authorized_keys

测试秘钥是否登入成功

[root@localhost .ssh]# ssh hadoopmaster
The authenticity of host 'hadoopmaster (fe80::7468:4a91:e381:bd03%eth0)' can't be established.
ECDSA key fingerprint is SHA256:SOi/rsJBsRn/zcHQ/gtT0Bg.
ECDSA key fingerprint is MD5:6a:0:88:38:fc:e0:bf:4b6:bf:59:b0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoopmaster,f8a91:e3' (ECDSA) to the list of known hosts.
Last login: Fri Feb  2 10:17:45 2024 from 192.168

6.重启

修改主机名等相关配置，必须重启主机

[root@hadoopmaster ~]# reboot

7.安装jdk

将jdk-8u341-linux-x64.rpm上传到/user/local文件夹中，并执行

rpm -ivh jdk-8u341-linux-x64.rpm

这样安装的jdk路径为：/usr/java/jdk1.8.0_341-amd64

8.安装hadoop（hadoop用户下操作）

将hadoop-3.3.6.tar.gz文件上传到/home/hadoop文件夹，然后使用tar -xvf hadoop-3.3.6.tar.gz 解压文件，并使用mv hadoop-3.3.6 hadoop更改文件夹名

9.配置hadoop环境配置（root下操作）

vim /etc/profile

export HADOOP_HOME=/home/hadoop/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib
export HDFS_NAMENODE_USER=root

export HDFS_DATANODE_USER=root

export HDFS_SECONDARYNAMENODE_USER=root

export HDFS_JOURNALNODE_USER=root

export HDFS_ZKFC_USER=root

export YARN_RESOURCEMANAGER_USER=root

export YARN_NODEMANAGER_USER=root

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

修改完之后，执行source /etc/profile使变更环境变量生效

[root@hadoopmaster local]# source /etc/profile

在hadoop目录创建data目录

mkdir ./data

11.修改配置文件

进入/home/hadoop/hadoop/etc/hadoop查看目录下的文件，配置几个必要的文件

（1）配置core-site.xml

vim ./core-site.xml

	
                fs.defaultFS
                hdfs://hadoopmaster:9000
                NameNode URI
        

	
      		hadoop.tmp.dir
      		/home/hadoop/hadoop/data
    	

    	
      		hadoop.http.staticuser.user
      		root
    	

    	
      		io.file.buffer.size
      		131073
    	

    	
      		hadoop.proxyuser.root.hosts
      		*
    	

    	
      		hadoop.proxyuser.root.groups
      		*
    	

（2）配置hdfs-site.xml

vim ./hdfs-site.xml

        
                dfs.replication
                1
        

        
                dfs.namenode.name.dir
                /home/hadoop/hadoop/data/dfs/name
        

        
                dfs.datanode.data.dir
                /home/hadoop/hadoop/data/dfs/data
        

        
                dfs.permissions.enabled
                false
        

（3）配置mapred-site.xml

vim ./mapred-site.xml

	
      		mapreduce.framework.name
      		yarn
    	

    	
      		mapreduce.jobhistory.address
      		hadoopmaster:10020
    	

    	
      		mapreduce.jobhistory.webapp.address
      		hadoopmaster:19888
    	

    	
      		mapreduce.map.memory.mb
      		2048
    	

    	
      		mapreduce.reduce.memory.mb
      		2048
    	

	
      		mapreduce.application.classpath
      		/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/*:/home/hadoop/hadoop/share/hadoop/common/lib/*:/home/hadoop/hadoop/share/hadoop/hdfs/*:/home/hadoop/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop/share/hadoop/mapreduce/*:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*

    	

（4）配置yarn-site.xml

vim ./yarn-site.xml

     yarn.resourcemanager.connect.retry-interval.ms

     20000

   

   

     yarn.resourcemanager.scheduler.class

     org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

   

   

     yarn.nodemanager.localizer.address

     hadoopmaster:8040

   

   

     yarn.nodemanager.address

     hadoopmaster:8050

   

   

     yarn.nodemanager.webapp.address

     hadoopmaster:8042

   

  

     yarn.nodemanager.aux-services

     mapreduce_shuffle

   

   

     yarn.nodemanager.local-dirs

     /home/hadoop/hadoop/yarndata/yarn

   

   

     yarn.nodemanager.log-dirs

     /home/hadoop/hadoop/yarndata/log

   

   

     yarn.nodemanager.vmem-check-enabled

     false

   

（5）配置hadoop-env.sh

vim ./hadoop-env.sh

修改第54行

export JAVA_HOME=/usr/java/jdk1.8.0_341-amd64

（6）配置workers

vim ./workers

[hadoop@hadoopmaster hadoop]$ vim ./workers
[hadoop@hadoopmaster hadoop]$ cat ./workers
hadoopmaster

11.初始化hadoop

进入/home/hadoop/hadoop/bin路径

执行：hadoop namenode -format

12.Hadoop3 验证

Hadoop 使用之前必须进行格式化，可以使用如下指令进行格式化：

hadoop namenode -format

如果在使用Hadoop的过程中出错，或者Hadoop 无法正常启动，可能需要重新格式化

重新格式化的流程步骤：

停止Hadoop

删除Hadoop 下的data和logs文件夹

重新格式化

13.启动hadoop

start-all.sh

查看进程

jps

13.停止hadoop

stop-all.sh