Big-Data on 李先生的树洞

Big Data Cluster Ansible Automated Deployment Tutorial

pblh123@126.com (John Lee) — Fri, 29 May 2026 00:00:00 +0000

Big Data Cluster Ansible Automated Deployment Tutorial
#

An Ansible-based declarative automation deployment solution, coexisting with existing Shell/Python scripts, for 3-10 node clusters.

1. Prerequisites
#

1.1 Control Node Requirements
#

Requirement	Details
OS	Linux / macOS (Windows WSL compatible)
Ansible	>= 2.15
sshpass	`yum install sshpass` (CentOS) or `apt install sshpass` (Ubuntu)
Network	SSH access to all cluster nodes

1.2 Cluster Node Requirements
#

Requirement	Details
OS	CentOS 7.9+ / Rocky Linux 8+
SSH	Enabled with password or key authentication
Sudo	hadoop user has sudo privileges
Disk	`/opt/module` at least 10GB free

1.3 Install Ansible
#

# CentOS / Rocky
sudo yum install -y epel-release && sudo yum install -y ansible

# Ubuntu / Debian
sudo apt update && sudo apt install -y ansible

# macOS
brew install ansible

# Verify
ansible --version # Should show >= 2.15

1.4 Pre-cache Component Archives (Recommended)
#

cd ansible/files/cache/

# Download all components (~1.8 GB total)
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.9/apache-zookeeper-3.5.9-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.6.4/hbase-2.6.4-hadoop3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.8/spark-3.5.8-bin-hadoop3-scala2.13.tgz
wget -c https://maven.aliyun.com/repository/public/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar

JDK should be pre-installed to /opt/module/jdk1.8.0_471 on all nodes, or can be auto-downloaded via the common role.

Big Data Cluster Distributed Deployment Guide

pblh123@126.com (John Lee) — Fri, 29 May 2026 00:00:00 +0000

Big Data Cluster Distributed Deployment Guide (Fully Verified)
#

A copy-paste-ready deployment guide for a complete big data cluster. Every configuration file is shown in full. All steps are verified on a 3-node CentOS 7.9 cluster.

Variable Substitution Reference
#

When deploying to a new cluster, replace these values:

Docker Big Data Cluster HA Deployment

pblh123@126.com (John Lee) — Fri, 29 May 2026 00:00:00 +0000

Docker Big Data Cluster HA Deployment
#

Docker Compose orchestrated 32-container full-stack HA big data cluster for local testing and learning.

Architecture Overview
#

 ┌─────────────────────────────────────────────────┐
 │ Docker Bridge: bigdata-net │
 │ 172.25.0.0/16 │
 ├─────────────────────────────────────────────────┤
 │ Layer 0 (base): MySQL + ZooKeeper x3 │
 │ Layer 1 (storage): JournalNode x3 │
 │ Layer 2 (storage): NameNode x2 + ZKFC x2 │
 │ Layer 3 (storage): DataNode x3 │
 │ Layer 4 (compute): ResourceManager x2 + NM x3 │
 │ Layer 5 (analytics): Hive Metastore/Server2 │
 │ Layer 6 (analytics): HBase x5 + Kafka x3 │
 │ Layer 7 (analytics): Spark + Flink History │
 └─────────────────────────────────────────────────┘

Component	Version	HA Strategy	Containers
HDFS	3.3.6	Dual NameNode + 3 JN + ZKFC	10
YARN	3.3.6	Dual RM + ZK State Store	6
ZooKeeper	3.7.1	3-node quorum	3
HBase	2.6.4	Dual Master + ZK discovery	5
Hive	3.1.3	Metastore + Server2 (separate)	2
Kafka	2.8.0	3 Broker + ZK coordination	3
Spark	3.5.8	History Server (YARN on-demand)	1
Flink	1.19.1	History Server (YARN on-demand)	1
MySQL	5.7	—	1

Total: 32 containers, ~10-11GB RAM

Big-Data on 李先生的树洞

Big Data Cluster Ansible Automated Deployment Tutorial

Big Data Cluster Ansible Automated Deployment Tutorial #

1. Prerequisites #

1.1 Control Node Requirements #

1.2 Cluster Node Requirements #

1.3 Install Ansible #

1.4 Pre-cache Component Archives (Recommended) #

Big Data Cluster Distributed Deployment Guide

Big Data Cluster Distributed Deployment Guide (Fully Verified) #

Variable Substitution Reference #

Docker Big Data Cluster HA Deployment

Docker Big Data Cluster HA Deployment #

Architecture Overview #

Big Data Cluster Ansible Automated Deployment Tutorial
#

1. Prerequisites
#

1.1 Control Node Requirements
#

1.2 Cluster Node Requirements
#

1.3 Install Ansible
#

1.4 Pre-cache Component Archives (Recommended)
#

Big Data Cluster Distributed Deployment Guide (Fully Verified)
#

Variable Substitution Reference
#

Docker Big Data Cluster HA Deployment
#

Architecture Overview
#