Hadoop+Spark+Zookeeper+Hbase集群搭建

• 端口
• Hadoop
• Spark
• Zookeeper
• Hbase

端口开放问题
关闭防火墙systemctl stop firewalld，并在服务器开放以下端口：

使用docker部署hadoop集群-手把手复现

vim core-site.xml

    
        fs.defaultFS
        hdfs://hadoop1:9000
    
    
        hadoop.tmp.dir
        file:/home/hadoop/tmp
    

vim hdfs-site.xml

	
        dfs.namenode.name.dir
        file:/home/hadoop/hdfs_name
    
    
        dfs.datanode.data.dir
        file:/home/hadoop/hdfs_data
    
    
        dfs.replication
        2
    
    
        dfs.namenode.secondary.http-address
        hadoop1:9001
    
    
        dfs.webhdfs.enabled
        true
    
    
        dfs.permissions
        false
    

vim mapred-site.xml

	
        mapreduce.framework.name
        yarn
    
    
        mapreduce.jobhistory.address
        hadoop1:10020
    
    
        mapreduce.jobhistory.webapp.address
        hadoop1:19888
    
    
                yarn.app.mapreduce.am.env
                HADOOP_MAPRED_HOME=${HADOOP_HOME}
        
        
         mapreduce.map.env
         HADOOP_MAPRED_HOME=${HADOOP_HOME}
        
        
          mapreduce.reduce.env
          HADOOP_MAPRED_HOME=${HADOOP_HOME}
        

vim yarn-site.xml

        
                yarn.resourcemanager.hostname
                hadoop1
        
        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
        
                yarn.log-aggregation-enable
                true
        
        
                yarn.log-aggregation.retain-seconds
                604800
        

vim hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.8

Spark集群安装-基于hadoop集群

（插播反爬信息 ）博主CSDN地址：https://wzlodq.blog.csdn.net/

1. 下载解压
   https://downloads.apache.org/zookeeper/stable/apache-zookeeper-3.6.3-bin.tar.gz

wget https://downloads.apache.org/zookeeper/stable/apache-zookeeper-3.6.3-bin.tar.gz
tar -zxvf apache-zookeeper-3.6.3-bin.tar.gz
mv apache-zookeeper-3.6.3-bin /usr/local/zookeeper

2. 配置zoo.cfg

cd /usr/local/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
vim zoo.cfg

dataDir=/data/zookeeper/data/
dataLogDir=/data/zookeeper/logs/
quorumListenonAllIPs=true
admin.serverPort=8181
server.1=hadoop1:2888:3888
server.2=hadoop2:2888:3888

3. 配置myid

mkdir -p /data/zookeeper/
cd /data/zookeeper/
mkdir data logs
cd data
vim myid

4. 同步

scp -r /usr/local/zookeeper/ hadoop2:/usr/local/

5. 启动
   在各个节点启动zookeeper

cd /usr/local/zookeeper/bin
./zkServer.sh start

IDEA远程连接Hbase及其Java API实战当时这篇是单机的，没介绍集群，步骤基本一致。

1. 下载解压

wget https://downloads.apache.org/hbase/2.3.5/hbase-2.3.5-src.tar.gz
tar -zxvf hbase-2.3.5-src.tar.gz
mv hbase-2.3.5 /usr/local/hbase

2. 配置hbase-site.xml

cd /usr/local/hbase/conf
vi hbase-site.xml

  
    hbase.cluster.distributed
    true
  
  
    hbase.tmp.dir
    ./tmp
  
  
    hbase.unsafe.stream.capability.enforce
    true
  
  
    hbase.rootdir
    hdfs://hadoop1:9000/hbase
  
  
    hbase.zookeeper.quorum
    hadoop1:2181,hadoop2:2181
    The directory shared by RegionServers. 
  
  
    hbase.master.info.port
    16010
  
  
    zookeeper.session.timeout
    90000
    hbase-default.xml
  

3. 配置hbase-env.sh

cd /usr/local/hbase/conf
echo $JAVA_HOME
vi hbase-env.sh #添加和你输出的JAVA_HOME一致

export JAVA_HOME=/usr/local/jdk1.8
export Hbase_MANAGES_ZK=false

4. 设置从节点

cd /usr/local/hbase/conf
vi regionservers

5. 同步
   将主节点Hbase配置同步给从节点

scp -r /usr/local/hbase/ hadoop2:/usr/local/

6. 启动测试
   记得先启动zookeeper和hadoop

cd /usr/local/hbase/bin
./start-hbase.sh

原创不易，请勿转载（本不富裕的访问量雪上加霜 ）
博主首页：https://wzlodq.blog.csdn.net/
来都来了，不评论两句吗
如果文章对你有帮助，记得一键三连❤