Big Data Cluster Distributed Deployment Guide (Fully Verified)
#

A copy-paste-ready deployment guide for a complete big data cluster. Every configuration file is shown in full. All steps are verified on a 3-node CentOS 7.9 cluster.

Variable Substitution Reference
#

When deploying to a new cluster, replace these values:

Variable	Tutorial Value	Replace With
Domain suffix	`pblh123.cn`	Your domain, or `localdomain` if none
Node count	3	Actual node count
Master node	`hd2` / `192.168.137.155`	Your master hostname/IP
Node 2	`hd3` / `192.168.137.156`	Your second node
Node 3	`hd4` / `192.168.137.157`	Your third node
Hadoop version	`hadoop-3.3.6`	Actual version directory name
JDK version	`jdk1.8.0_471`	Actual JDK directory name
HDFS replication	`2`	min(node_count, 3)
MySQL password	`Hive@12345`	Your MySQL hive user password
Install root	`/opt/module`	Components install directory
Data root	`/opt/data`	HDFS/ZK/Kafka data directory
NM memory	hd2:2800, hd3:2000, hd4:1200	70-80% of physical RAM (MB)
SSH user	`hadoop`	Service user account
SSH password	`hadoop`	Corresponding password

Table of Contents
#

1. Architecture
#

1.1 Topology
#

               ┌─────────────────────────────────────┐
               │           hd2 (4G/2C)                │
               │  192.168.137.155                     │
               │  NameNode, ResourceManager           │
               │  HBase Master, Hive Metastore        │
               │  HiveServer2, JobHistoryServer       │
               └──────────────┬──────────────────────┘
                              │
            ┌─────────────────┼─────────────────┐
            │                 │                 │
   ┌────────┴────────┐ ┌──────┴──────┐ ┌───────┴───────┐
   │  hd3 (3G/2C)    │ │ hd4 (2G/2C) │ │ (scalable)    │
   │  DataNode, NM   │ │ DataNode, NM│ │               │
   │  SecondaryNN    │ │ ZK, Kafka   │ │               │
   │  ZK, Kafka      │ │ HBase RS    │ │               │
   │  HBase RS       │ │             │ │               │
   └─────────────────┘ └─────────────┘ └───────────────┘

1.2 Service Assignment
#

Service	hd2	hd3	hd4	Notes
NameNode	Y			HDFS metadata
DataNode	Y	Y	Y	Data storage
SecondaryNameNode		Y		Checkpoint merge
ResourceManager	Y			YARN scheduling
NodeManager	Y	Y	Y	Task execution
JobHistoryServer	Y			MR history
ZooKeeper	Y	Y	Y	Coordination
Hive Metastore	Y			Metadata service
HiveServer2	Y			SQL query
HBase Master	Y			Region management
HBase RegionServer	Y	Y	Y	Data read/write
Kafka Broker	Y	Y	Y	Message queue
Spark HistoryServer	Y			Task history

1.3 Startup Order
#

MySQL ──► HDFS ──┬──► YARN ──► JobHistory
                 │
                 ├──► ZooKeeper ──┬──► HBase
                 │                │
                 │                └──► Kafka
                 │
                 └──► Hive Metastore ──► HiveServer2 ──► Spark HistoryServer

Start: HDFS → (ZooKeeper + YARN) → (Hive → HBase → Kafka → Spark)
Stop: Spark → Kafka → HBase → Hive → YARN → ZooKeeper → HDFS

2. Environment Setup
#

2.1 Hardware Requirements
#

Node	Memory	CPU	Disk
hd2	≥4GB	≥2 cores	≥40GB
hd3	≥3GB	≥2 cores	≥40GB
hd4	≥2GB	≥2 cores	≥40GB

2.2 Software Versions
#

Component	Version	Path
JDK	1.8.0_471	`/opt/module/jdk1.8.0_471`
Hadoop	3.3.6	`/opt/module/hadoop-3.3.6`
ZooKeeper	3.5.9	`/opt/module/apache-zookeeper-3.5.9-bin`
Hive	3.1.3	`/opt/module/apache-hive-3.1.3-bin`
HBase	2.6.4	`/opt/module/hbase-2.6.4-hadoop3`
Kafka	2.8.0	`/opt/module/kafka_2.13-2.8.0`
Spark	3.5.8	`/opt/module/spark-3.5.8-bin-hadoop3-scala2.13`
MySQL	5.7+	System

2.3 Create User (All Nodes, as root)
#

useradd hadoop
echo "hadoop" | passwd --stdin hadoop
echo "hadoop ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

2.4 /etc/hosts (All Nodes, as root)
#

cat >> /etc/hosts << 'EOF'
192.168.137.155 hd2.pblh123.cn hd2
192.168.137.156 hd3.pblh123.cn hd3
192.168.137.157 hd4.pblh123.cn hd4
EOF

Adapt: Replace IPs and hostnames with your actual values.

2.5 Password-less SSH (as hadoop user)
#

su - hadoop
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

for host in hd2 hd3 hd4; do
    ssh-copy-id -o StrictHostKeyChecking=no hadoop@$host
done

# Verify
ssh hd3 "hostname"
ssh hd4 "hostname"

2.6 Firewall & SELinux (All Nodes, as root)
#

systemctl stop firewalld && systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

2.7 JDK (All Nodes, as root)
#

tar -zxf jdk-8u471-linux-x64.tar.gz -C /opt/module/
/opt/module/jdk1.8.0_471/bin/java -version
# Expected: java version "1.8.0_471"

2.8 Create Directories (All Nodes, as hadoop)
#

mkdir -p /opt/data/{namenode,datanode,hadoop-tmp,zookeeper,kafkadata}

2.9 Global Environment Script (All Nodes, as root)
#

Create /etc/profile.d/hadoop_env.sh:

sudo tee /etc/profile.d/hadoop_env.sh << 'EOF'
JAVA_HOME=/opt/module/jdk1.8.0_471
HADOOP_HOME=/opt/module/hadoop-3.3.6
KAFKA_HOME=/opt/module/kafka_2.13-2.8.0
HIVE_HOME=/opt/module/apache-hive-3.1.3-bin
SPARK_HOME=/opt/module/spark-3.5.8-bin-hadoop3-scala2.13
HBASE_HOME=/opt/module/hbase-2.6.4-hadoop3
ZOOKEEPER_HOME=/opt/module/apache-zookeeper-3.5.9-bin

PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$KAFKA_HOME/bin:$HIVE_HOME/bin:$SPARK_HOME/bin:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin
EOF

source /etc/profile.d/hadoop_env.sh

Adapt: Update version directory names to match your installation.

3. MySQL Setup
#

On hd2 only, for Hive metadata storage.

yum install -y mariadb-server mariadb
systemctl start mariadb && systemctl enable mariadb

mysql -u root << EOF
CREATE DATABASE IF NOT EXISTS hive_metastore DEFAULT CHARACTER SET utf8;
CREATE USER IF NOT EXISTS 'hive'@'%' IDENTIFIED BY 'Hive@12345';
GRANT ALL PRIVILEGES ON hive_metastore.* TO 'hive'@'%';
FLUSH PRIVILEGES;
EOF

Adapt: Replace Hive@12345 with your actual password.

4. Hadoop HDFS Deployment
#

4.1 core-site.xml
#

Path: ${HADOOP_HOME}/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hd2.pblh123.cn:9000</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hadoop.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hive.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hive.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/data/hadoop-tmp</value>
  </property>
</configuration>

4.2 hdfs-site.xml
#

Path: ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///opt/data/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///opt/data/datanode</value>
  </property>
  <property>
    <name>dfs.blocksize</name>
    <value>64m</value>
  </property>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>hd2.pblh123.cn:9870</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hd3.pblh123.cn:9868</value>
  </property>
  <property>
    <name>dfs.permissions.enabled</name>
    <value>false</value>
  </property>
</configuration>

4.3 workers
#

Path: ${HADOOP_HOME}/etc/hadoop/workers

hd2.pblh123.cn
hd3.pblh123.cn
hd4.pblh123.cn

4.4 hadoop-env.sh
#

Path: ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/opt/module/jdk1.8.0_471
export HADOOP_HOME=/opt/module/hadoop-3.3.6
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export HADOOP_PID_DIR=$HADOOP_HOME/pids

export HADOOP_HEAPSIZE_MAX=512m
export HADOOP_HEAPSIZE_MIN=256m
export HDFS_NAMENODE_OPTS="-Xmx512m -Xms256m"
export HDFS_DATANODE_OPTS="-Xmx256m -Xms128m"
export YARN_RESOURCEMANAGER_HEAPSIZE=512
export YARN_NODEMANAGER_HEAPSIZE=256

4.5 Sync to All Nodes
#

HADOOP_CONF=/opt/module/hadoop-3.3.6/etc/hadoop
for host in hd3 hd4; do
    scp $HADOOP_CONF/{core-site.xml,hdfs-site.xml,workers,hadoop-env.sh} hadoop@$host:$HADOOP_CONF/
done

4.6 Format NameNode (hd2 only, once)
#

source /etc/profile.d/hadoop_env.sh
hdfs namenode -format
# Expected: ...successfully formatted

4.7 Start HDFS
#

start-dfs.sh
sleep 5

4.8 Verify HDFS
#

jps | grep -E "NameNode|DataNode"
hdfs dfsadmin -report                    # Should show 3 live datanodes
echo "Hello HDFS" | hdfs dfs -put - /test.txt && hdfs dfs -cat /test.txt && hdfs dfs -rm /test.txt
# Web UI: http://192.168.137.155:9870

5. YARN Deployment
#

5.1 yarn-site.xml (Per-Node Memory)
#

hd2 version (/opt/module/hadoop-3.3.6/etc/hadoop/yarn-site.xml):

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hd2.pblh123.cn</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hd2.pblh123.cn:8088</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>2800</value><!-- hd2: 4G×70%≈2800MB -->
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>2048</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>256</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>2</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.log.server.url</name>
    <value>http://hd2.pblh123.cn:19888/jobhistory/logs</value>
  </property>
</configuration>

hd3: Change yarn.nodemanager.resource.memory-mb to 2000, scheduler max to 1744.
hd4: Change to 1200 and 944.
Formula: NM_memory = physical_RAM × 70%, scheduler_max = min(NM_memory - 256, 2048)

5.2 mapred-site.xml
#

Path: ${HADOOP_HOME}/etc/hadoop/mapred-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>hd2.pblh123.cn:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hd2.pblh123.cn:19888</value>
  </property>
</configuration>

5.3 Sync and Start
#

for host in hd3 hd4; do
    scp $HADOOP_CONF/{yarn-site.xml,mapred-site.xml} hadoop@$host:$HADOOP_CONF/
done

start-yarn.sh
mapred --daemon start historyserver
sleep 5

5.4 Verify YARN
#

yarn node -list -all | grep RUNNING    # Should show 3 nodes
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi 2 5
# Web UI: http://192.168.137.155:8088

6. ZooKeeper Deployment
#

6.1 zoo.cfg (Same on All Nodes)
#

Path: ${ZOOKEEPER_HOME}/conf/zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/data/zookeeper
clientPort=2181
maxClientCnxns=60
autopurge.snapRetainCount=3
autopurge.purgeInterval=1

server.1=hd2.pblh123.cn:2888:3888
server.2=hd3.pblh123.cn:2888:3888
server.3=hd4.pblh123.cn:2888:3888

6.2 myid (Different Per Node)
#

# hd2
echo 1 > /opt/data/zookeeper/myid
# hd3
echo 2 > /opt/data/zookeeper/myid
# hd4
echo 3 > /opt/data/zookeeper/myid

6.3 Sync, Start and Verify
#

ZK_CONF=/opt/module/apache-zookeeper-3.5.9-bin/conf
for host in hd3 hd4; do scp $ZK_CONF/zoo.cfg hadoop@$host:$ZK_CONF/; done

# Start on all nodes
for host in hd2 hd3 hd4; do ssh $host "source /etc/profile.d/hadoop_env.sh && zkServer.sh start"; done
sleep 3

# Verify
for host in hd2 hd3 hd4; do ssh $host "zkServer.sh status"; done
# Expected: 1 leader, 2 followers

7. Hive Deployment
#

Hive services on hd2 only.

7.1 MySQL JDBC Driver
#

HIVE_LIB=/opt/module/apache-hive-3.1.3-bin/lib
wget -P $HIVE_LIB https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.33/mysql-connector-java-8.0.33.jar

7.2 hive-site.xml
#

Path: /opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://hd2.pblh123.cn:9083</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive_metastore?useSSL=false&amp;createDatabaseIfNotExist=true</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>Hive@12345</value>
  </property>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
  </property>
  <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>hd2.pblh123.cn</value>
  </property>
  <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
  </property>
</configuration>

7.3 Init Metastore Schema
#

source /etc/profile.d/hadoop_env.sh
schematool -dbType mysql -initSchema
# Expected: schemaTool completed

7.4 Start Hive Services
#

nohup hive --service metastore > /opt/data/hive-metastore.log 2>&1 &
sleep 3
nohup hiveserver2 > /opt/data/hiveserver2.log 2>&1 &
sleep 2

7.5 Verify Hive
#

jps | grep RunJar  # Should see 2 RunJar processes
beeline -u jdbc:hive2://hd2.pblh123.cn:10000 -n hadoop -e "SHOW DATABASES;"

8. HBase Deployment
#

8.1 hbase-site.xml (Same on All Nodes)
#

Path: ${HBASE_HOME}/conf/hbase-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://hd2.pblh123.cn:9000/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>hd2.pblh123.cn,hd3.pblh123.cn,hd4.pblh123.cn</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
  <property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
  </property>
  <property>
    <name>hbase.wal.provider</name>
    <value>filesystem</value>
  </property>
</configuration>

8.2 regionservers
#

Path: ${HBASE_HOME}/conf/regionservers

hd2.pblh123.cn
hd3.pblh123.cn
hd4.pblh123.cn

8.3 Sync, Start and Verify
#

HBASE_CONF=/opt/module/hbase-2.6.4-hadoop3/conf
for host in hd3 hd4; do
    scp $HBASE_CONF/{hbase-site.xml,regionservers} hadoop@$host:$HBASE_CONF/
done

start-hbase.sh
sleep 5

echo "status" | hbase shell -n
# Expected: 1 active master, 3 region servers
# Web UI: http://192.168.137.155:16010

9. Kafka Deployment
#

9.1 server.properties (Per-Node broker.id)
#

Common settings for all nodes:

zookeeper.connect=hd2.pblh123.cn:2181,hd3.pblh123.cn:2181,hd4.pblh123.cn:2181
listeners=PLAINTEXT://:9092

Per-node broker.id:

# hd2
sed -i 's/^broker.id=.*/broker.id=1/' /opt/module/kafka_2.13-2.8.0/config/server.properties
sed -i 's|^zookeeper.connect=.*|zookeeper.connect=hd2.pblh123.cn:2181,hd3.pblh123.cn:2181,hd4.pblh123.cn:2181|' /opt/module/kafka_2.13-2.8.0/config/server.properties

# hd3 (broker.id=2), hd4 (broker.id=3) — same pattern

9.2 Cloned Node Fix
#

# If nodes were cloned, delete stale meta.properties
ssh hd3 "rm -f /opt/data/kafkadata/meta.properties"
ssh hd4 "rm -f /opt/data/kafkadata/meta.properties"

9.3 Start and Verify
#

for host in hd2 hd3 hd4; do
    ssh $host "source /etc/profile.d/hadoop_env.sh && nohup kafka-server-start.sh -daemon \$KAFKA_HOME/config/server.properties > /dev/null 2>&1 &"
done
sleep 3

kafka-topics.sh --bootstrap-server hd2.pblh123.cn:9092 --list

# Producer/consumer test
echo "test message" | kafka-console-producer.sh --bootstrap-server hd2.pblh123.cn:9092 --topic verify-topic
kafka-console-consumer.sh --bootstrap-server hd2.pblh123.cn:9092 --topic verify-topic --from-beginning --max-messages 1

10. Spark Deployment
#

10.1 spark-env.sh
#

Path: ${SPARK_HOME}/conf/spark-env.sh

#!/usr/bin/env bash
export JAVA_HOME=/opt/module/jdk1.8.0_471
export HADOOP_HOME=/opt/module/hadoop-3.3.6
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/module/spark-3.5.8-bin-hadoop3-scala2.13
export SPARK_CONF_DIR=$SPARK_HOME/conf

export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 \
  -Dspark.history.fs.logDirectory=hdfs://hd2.pblh123.cn:9000/spark-history \
  -Dspark.history.retainedApplications=30"

export SPARK_DAEMON_MEMORY=256m

10.2 spark-defaults.conf
#

Path: ${SPARK_HOME}/conf/spark-defaults.conf

spark.master                       yarn
spark.submit.deployMode            cluster
spark.eventLog.enabled             true
spark.eventLog.dir                 hdfs://hd2.pblh123.cn:9000/spark-history
spark.history.fs.logDirectory      hdfs://hd2.pblh123.cn:9000/spark-history
spark.serializer                   org.apache.spark.serializer.KryoSerializer
spark.driver.memory                512m
spark.executor.memory              512m
spark.executor.cores               1
spark.executor.instances           2
spark.yarn.am.memory               512m
spark.yarn.maxAppAttempts          2
spark.sql.shuffle.partitions       2
spark.driver.maxResultSize         256m

10.3 Link hive-site.xml
#

ln -sf /opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml \
    /opt/module/spark-3.5.8-bin-hadoop3-scala2.13/conf/hive-site.xml

10.4 Create HDFS Directories
#

hdfs dfs -mkdir -p /spark-history /user/hive/warehouse
hdfs dfs -chmod 777 /spark-history

10.5 Sync, Start and Verify
#

SPARK_CONF=/opt/module/spark-3.5.8-bin-hadoop3-scala2.13/conf
for host in hd3 hd4; do
    scp $SPARK_CONF/{spark-env.sh,spark-defaults.conf} hadoop@$host:$SPARK_CONF/
    ssh $host "ln -sf /opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml $SPARK_CONF/hive-site.xml"
done

$SPARK_HOME/sbin/start-history-server.sh

# Spark Pi test
spark-submit --master yarn --deploy-mode cluster \
    --class org.apache.spark.examples.SparkPi \
    $SPARK_HOME/examples/jars/spark-examples_2.13-3.5.8.jar 100
# Expected: Pi is roughly 3.14...

# Web UI: http://192.168.137.155:18080

11. Cluster Verification
#

11.1 Process Check
#

for host in hd2 hd3 hd4; do
    echo "=== $host ==="
    ssh $host "source /etc/profile.d/hadoop_env.sh && jps -l | grep -v Jps | sort"
    echo ""
done

Expected processes:

Node	Expected
hd2	NameNode, DataNode, ResourceManager, NodeManager, QuorumPeerMain, JobHistoryServer, HMaster, HRegionServer, Kafka, HistoryServer, 2×RunJar
hd3	DataNode, SecondaryNameNode, NodeManager, QuorumPeerMain, HRegionServer, Kafka
hd4	DataNode, NodeManager, QuorumPeerMain, HRegionServer, Kafka

11.2 Quick Test Suite
#

# HDFS
echo "verify" | hdfs dfs -put - /verify.txt && hdfs dfs -cat /verify.txt && hdfs dfs -rm /verify.txt

# YARN
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi 2 5

# ZooKeeper
echo stat | nc hd2.pblh123.cn 2181 | grep Mode

# Hive
beeline -u jdbc:hive2://hd2.pblh123.cn:10000 -n hadoop -e "SHOW DATABASES;"

# HBase
echo "status" | hbase shell -n

# Kafka
kafka-topics.sh --bootstrap-server hd2.pblh123.cn:9092 --list

# Spark
spark-submit --master yarn --deploy-mode cluster \
    --class org.apache.spark.examples.SparkPi \
    $SPARK_HOME/examples/jars/spark-examples_2.13-3.5.8.jar 10

12. Cluster Management
#

12.1 One-Click Start/Stop
#

# Shell (on cluster node)
bash cluster_ctl.sh start|stop|restart|status
bash cluster_ctl.sh start kafka       # Single service

# Python (from management machine)
python cluster_ctl.py start|stop|restart|status
python cluster_ctl.py start hdfs

12.2 Cluster Shutdown
#

# Shell — stop services then power off
bash shutdown_cluster.sh

# Skip service stop, force power off
bash shutdown_cluster.sh --force

# Python (from management machine)
python shutdown_cluster.py
python shutdown_cluster.py --force

Flow: ① Confirm → ② Stop all services → ③ Shut down slave nodes → ④ Shut down master node.

12.3 Manual Service Control
#

# HDFS: start-dfs.sh / stop-dfs.sh
# YARN: start-yarn.sh / stop-yarn.sh
# ZK:   zkServer.sh start / zkServer.sh stop
# HBase: start-hbase.sh / stop-hbase.sh
# Kafka: kafka-server-start.sh / kafka-server-stop.sh
# Spark: $SPARK_HOME/sbin/start-history-server.sh / stop-history-server.sh

12.3 Log Paths
#

Component	Log Path
HDFS NN	`$HADOOP_HOME/logs/hadoop-hadoop-namenode-*.log`
YARN RM	`$HADOOP_HOME/logs/yarn-hadoop-resourcemanager-*.log`
ZooKeeper	`$ZOOKEEPER_HOME/logs/zookeeper.out`
Kafka	`$KAFKA_HOME/logs/server.log`
HBase	`/opt/data/hbase/logs/`
Hive	`/opt/data/hive-metastore.log`, `/opt/data/hiveserver2.log`

12.4 Port Reference
#

Port	Service
9000	HDFS NameNode RPC
9870	HDFS NameNode Web
8088	YARN RM Web
19888	JobHistory Web
2181	ZooKeeper Client
9083	Hive Metastore
10000	HiveServer2
9092	Kafka Broker
16010	HBase Master Web
18080	Spark History Server

12.5 Web UIs
#

Service	URL
HDFS	http://192.168.137.155:9870
YARN	http://192.168.137.155:8088
Spark	http://192.168.137.155:18080
HBase	http://192.168.137.155:16010
JobHistory	http://192.168.137.155:19888

13. One-Click Scripts
#

13.1 config.ini Template
#

Edit config.ini then run bash deploy.sh:

[NODES]
hd2=192.168.137.155
hd3=192.168.137.156
hd4=192.168.137.157

[NODE_MEMORY]
hd2=2800
hd3=2000
hd4=1200

[NODE_CPU]
hd2=2
hd3=2
hd4=2

[SSH]
user=hadoop
password=hadoop

[PATHS]
base_dir=/opt/module
data_dir=/opt/data

[VERSIONS]
jdk=jdk1.8.0_471
hadoop=hadoop-3.3.6
zookeeper=apache-zookeeper-3.5.9-bin
hive=apache-hive-3.1.3-bin
hbase=hbase-2.6.4-hadoop3
kafka=kafka_2.13-2.8.0
spark=spark-3.5.8-bin-hadoop3-scala2.13

[MYSQL]
host=localhost
port=3306
database=hive_metastore
user=hive
password=Hive@12345

[PORTS]
hdfs_namenode=9000
hdfs_web=9870
yarn_web=8088
jobhistory=19888
zookeeper_client=2181
hive_metastore=9083
hive_server2=10000
hbase_master_web=16010
kafka=9092
spark_history=18080

[HDFS]
replication=2
block_size_mb=64
namenode_dir=/opt/data/namenode
datanode_dir=/opt/data/datanode
hadoop_tmp=/opt/data/hadoop-tmp

[ZOOKEEPER]
tick_time=2000
init_limit=10
sync_limit=5
data_dir=/opt/data/zookeeper

[SPARK]
driver_memory=512m
executor_memory=512m
executor_cores=1
executor_instances=2
am_memory=512m

[JAVA_HEAP]
namenode=512m
datanode=256m
rm=512
nm=256

13.2 Deployment Flow
#

vi config.ini                # 1. Edit configuration
sudo yum install -y sshpass  # 2. Install sshpass
bash deploy.sh               # 3. One-click deploy
bash cluster_ctl.sh status   # 4. Verify

13.3 Remote Management Machine
#

pip install paramiko
python deploy.py
python cluster_ctl.py start
python cluster_ctl.py status

14. Troubleshooting
#

DataNode Not Registering
#

cat $HADOOP_HOME/etc/hadoop/workers   # Hostnames must match
cat /etc/hosts                        # Consistent across all nodes
tail -100 $HADOOP_HOME/logs/hadoop-hadoop-datanode-*.log | grep -i error

NodeManager Not Starting (Low Memory)
#

free -h
# Reduce yarn.nodemanager.resource.memory-mb to 70-80% of physical RAM

ZooKeeper Election Failure
#

cat /opt/data/zookeeper/myid          # Must match server.X in zoo.cfg
grep "server\." $ZOOKEEPER_HOME/conf/zoo.cfg
telnet hd3 2888                       # Check connectivity

Kafka Cloned Node Failure
#

# Error: broker.id already registered
rm -f /opt/data/kafkadata/meta.properties
kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

HBase Master Exits After Start
#

zkServer.sh status                    # ZK must be running
hdfs dfs -ls /hbase                   # Directory must exist
cat /opt/data/hbase/logs/hbase-hadoop-master-*.log | grep ERROR

HDFS Safemode
#

hdfs dfsadmin -safemode get
hdfs dfsadmin -safemode leave

Appendix: New Cluster Checklist
#

[ ] Update /etc/hosts on all nodes
[ ] Update config.ini: nodes, memory, SSH credentials
[ ] Verify version directory names in [VERSIONS] match /opt/module/
[ ] Update MySQL credentials in [MYSQL]
[ ] Replace domain suffix: .pblh123.cn hardcoded in scripts → your domain
[ ] Set replication factor ≤ DataNode count
[ ] Configure per-node memory: 70-80% of physical RAM
[ ] Run bash deploy.sh
[ ] Run bash cluster_ctl.sh status
[ ] Verify each component: HDFS → YARN → ZK → Hive → HBase → Kafka → Spark

Domain Suffix Note
#

The scripts hardcode .pblh123.cn as the domain suffix. For a new cluster:

Option A: Configure /etc/hosts with matching FQDN entries
Option B: Globally replace .pblh123.cn with your domain (or remove suffix for short hostnames) in: deploy.sh, deploy.py, cluster_ctl.sh, cluster_ctl.py

Appendix: Script Files
#

File	Purpose	Run On
`config.ini`	Global configuration	Edit before deploy
`deploy.sh`	Shell one-click deploy	Master node
`deploy.py`	Python deploy script	Management machine
`cluster_ctl.sh`	Shell start/stop	Cluster node
`cluster_ctl.py`	Python start/stop	Management machine
`ssh_exec.py`	SSH connection module	Dependency
`verify.py`	Cluster verification	Management machine
`cluster_check.py`	Quick status check	Management machine
`deep_check.py`	Deep diagnostics	Management machine
`shutdown_cluster.sh`	Shell cluster shutdown	Cluster node
`shutdown_cluster.py`	Python cluster shutdown	Management machine

Big Data Cluster Distributed Deployment Guide (Fully Verified)#

Variable Substitution Reference#

Table of Contents#

1. Architecture#

1.1 Topology#

1.2 Service Assignment#

1.3 Startup Order#

2. Environment Setup#

2.1 Hardware Requirements#

2.2 Software Versions#

2.3 Create User (All Nodes, as root)#

2.4 /etc/hosts (All Nodes, as root)#

2.5 Password-less SSH (as hadoop user)#

2.6 Firewall & SELinux (All Nodes, as root)#

2.7 JDK (All Nodes, as root)#

2.8 Create Directories (All Nodes, as hadoop)#

2.9 Global Environment Script (All Nodes, as root)#

3. MySQL Setup#

4. Hadoop HDFS Deployment#

4.1 core-site.xml#

4.2 hdfs-site.xml#

4.3 workers#

4.4 hadoop-env.sh#

4.5 Sync to All Nodes#

4.6 Format NameNode (hd2 only, once)#

4.7 Start HDFS#

4.8 Verify HDFS#

5. YARN Deployment#

5.1 yarn-site.xml (Per-Node Memory)#

5.2 mapred-site.xml#

5.3 Sync and Start#

5.4 Verify YARN#

6. ZooKeeper Deployment#

6.1 zoo.cfg (Same on All Nodes)#

6.2 myid (Different Per Node)#

6.3 Sync, Start and Verify#

7. Hive Deployment#

7.1 MySQL JDBC Driver#

7.2 hive-site.xml#

7.3 Init Metastore Schema#

7.4 Start Hive Services#

7.5 Verify Hive#

8. HBase Deployment#

8.1 hbase-site.xml (Same on All Nodes)#

8.2 regionservers#

8.3 Sync, Start and Verify#

9. Kafka Deployment#

9.1 server.properties (Per-Node broker.id)#

9.2 Cloned Node Fix#

9.3 Start and Verify#

10. Spark Deployment#

10.1 spark-env.sh#

10.2 spark-defaults.conf#

10.3 Link hive-site.xml#

10.4 Create HDFS Directories#

10.5 Sync, Start and Verify#

11. Cluster Verification#

11.1 Process Check#

11.2 Quick Test Suite#

12. Cluster Management#

12.1 One-Click Start/Stop#

12.2 Cluster Shutdown#

12.3 Manual Service Control#

12.3 Log Paths#

12.4 Port Reference#

12.5 Web UIs#

13. One-Click Scripts#

13.1 config.ini Template#

13.2 Deployment Flow#

13.3 Remote Management Machine#

14. Troubleshooting#

DataNode Not Registering#

NodeManager Not Starting (Low Memory)#

ZooKeeper Election Failure#

Kafka Cloned Node Failure#

HBase Master Exits After Start#

HDFS Safemode#

Appendix: New Cluster Checklist#

Domain Suffix Note#

Appendix: Script Files#