Big Data Cluster Ansible Automated Deployment Tutorial
#

An Ansible-based declarative automation deployment solution, coexisting with existing Shell/Python scripts, for 3-10 node clusters.

1. Prerequisites
#

1.1 Control Node Requirements
#

Requirement	Details
OS	Linux / macOS (Windows WSL compatible)
Ansible	>= 2.15
sshpass	`yum install sshpass` (CentOS) or `apt install sshpass` (Ubuntu)
Network	SSH access to all cluster nodes

1.2 Cluster Node Requirements
#

Requirement	Details
OS	CentOS 7.9+ / Rocky Linux 8+
SSH	Enabled with password or key authentication
Sudo	hadoop user has sudo privileges
Disk	`/opt/module` at least 10GB free

1.3 Install Ansible
#

# CentOS / Rocky
sudo yum install -y epel-release && sudo yum install -y ansible

# Ubuntu / Debian
sudo apt update && sudo apt install -y ansible

# macOS
brew install ansible

# Verify
ansible --version    # Should show >= 2.15

1.4 Pre-cache Component Archives (Recommended)
#

cd ansible/files/cache/

# Download all components (~1.8 GB total)
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.9/apache-zookeeper-3.5.9-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.6.4/hbase-2.6.4-hadoop3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.8/spark-3.5.8-bin-hadoop3-scala2.13.tgz
wget -c https://maven.aliyun.com/repository/public/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar

JDK should be pre-installed to /opt/module/jdk1.8.0_471 on all nodes, or can be auto-downloaded via the common role.

2. Cluster Configuration
#

2.1 Edit Inventory
#

Edit ansible/inventory/hosts.yml with your node IPs and hostnames:

cluster:
  hosts:
    node1:
      ansible_host: 192.168.137.155  # Master node IP
      hostname: hd2                   # Master hostname
    node2:
      ansible_host: 192.168.137.156
      hostname: hd3
    node3:
      ansible_host: 192.168.137.157
      hostname: hd4

2.2 Modify Global Variables
#

Edit the all.vars section:

# Network mode: "hostname" (hostname.domain) or "ip" (direct IP)
network_mode: "hostname"
domain_suffix: "pblh123.cn"     # Your domain suffix

# SSH credentials
ansible_user: hadoop
ansible_password: hadoop        # Your password

# MySQL (Hive Metastore)
mysql:
  host: localhost
  database: hive_metastore
  user: hive
  password: "Hive@12345"        # Your MySQL password

3. Deployment
#

3.1 Pre-flight Check
#

# Test SSH connectivity
ansible -i ansible/inventory/hosts.yml cluster -m ping
# Expected: SUCCESS / "pong" for each node

# Check node memory
ansible -i ansible/inventory/hosts.yml cluster -m setup -a "filter=ansible_memtotal_mb"

3.2 Step-by-Step (Recommended for First Use)
#

cd ansible/

# Step 1: Install components
ansible-playbook -i inventory/hosts.yml playbooks/install.yml

# Step 2: Distribute configs
ansible-playbook -i inventory/hosts.yml playbooks/configure.yml

# Step 3: Format HDFS and start services
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags format,hdfs,zookeeper,yarn

3.3 One-Click Deployment
#

cd ansible/

# Full deployment (with HDFS formatting)
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml

# Re-deploy (skip HDFS format, preserve data)
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --skip-tags format

# Deploy specific component only
ansible-playbook -i inventory/hosts.yml playbooks/deploy.yml --tags hadoop,hive

4. Cluster Management
#

4.1 Service Status
#

ansible-playbook -i inventory/hosts.yml playbooks/status.yml

4.2 Start / Stop / Restart
#

# Start all services
ansible-playbook -i inventory/hosts.yml playbooks/start.yml

# Stop all services
ansible-playbook -i inventory/hosts.yml playbooks/stop.yml

# Restart all services
ansible-playbook -i inventory/hosts.yml playbooks/restart.yml

# Start/stop individual service
ansible-playbook -i inventory/hosts.yml playbooks/start.yml --tags kafka
ansible-playbook -i inventory/hosts.yml playbooks/stop.yml --tags hbase

4.3 Functional Verification (13 checks)
#

ansible-playbook -i inventory/hosts.yml playbooks/verify.yml

#	Check	Method
1	Node connectivity	ping
2	HDFS read/write	Write → Read → Verify → Delete
3	HDFS capacity	`hdfs dfsadmin -report`
4	YARN nodes	`yarn node -list`
5	ZooKeeper roles	`echo stat \| nc localhost 2181` (each node)
6	Hive Metastore	TCP 9083
7	HiveServer2	TCP 10000
8	HBase Shell	`echo "status" \| hbase shell`
9	Kafka E2E	Create → Produce → Consume → Delete topic
10	Spark Pi	`spark-submit --class SparkPi` (cluster mode)
11	Spark History	TCP 18080
12	Web UIs (4x)	HDFS / YARN / Spark / HBase HTTP

5. Web UIs
#

Service	URL
HDFS NameNode	`http://<master_ip>:9870`
YARN ResourceManager	`http://<master_ip>:8088`
Spark History Server	`http://<master_ip>:18080`
HBase Master	`http://<master_ip>:16010`

6. Troubleshooting
#

SSH Connection Failed
#

ssh hadoop@192.168.137.155 "hostname"
sudo grep PasswordAuthentication /etc/ssh/sshd_config
sudo sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl restart sshd

Port Already In Use
#

sudo netstat -tlnp | grep <port>
jps -l | grep -v Jps | awk '{print $1}' | xargs -r kill -9

Service Start Timeout
#

ls /opt/module/
tail -100 /opt/module/hadoop-3.3.6/logs/*.log
# Manual startup for debugging:
source /etc/profile.d/hadoop_env.sh
hdfs namenode

Adding New Nodes
#

Add node to inventory/hosts.yml
Assign zk_myid and kafka_broker_id (increment)
Run: ansible-playbook -i inventory/hosts.yml playbooks/configure.yml --limit <new_node>
Restart affected services (ZooKeeper, HBase, Kafka)

Slow Downloads
#

# Option 1: Pre-download to local cache (recommended)
cd ansible/files/cache/
# Download all components via Chinese mirrors

# Option 2: Change mirror source
# Edit mirrors.apache in inventory/hosts.yml to use Aliyun mirror

7. Relationship with Shell/Python Scripts
#

Both solutions run independently:

Scenario	Recommended
Initial deploy, config consistency	Ansible
Quick start/stop, daily ops	Shell (`cluster_ctl.sh`)
Remote management	Python (`cluster_ctl.py`)
Fix config drift	Ansible `configure.yml`

Ansible and existing scripts share the same configuration semantics from config.ini, but use a separate inventory/hosts.yml variable file. Changes to configuration must be synchronized in both places.

Big Data Cluster Ansible Automated Deployment Tutorial#

1. Prerequisites#

1.1 Control Node Requirements#

1.2 Cluster Node Requirements#

1.3 Install Ansible#

1.4 Pre-cache Component Archives (Recommended)#

2. Cluster Configuration#

2.1 Edit Inventory#

2.2 Modify Global Variables#

3. Deployment#

3.1 Pre-flight Check#

3.2 Step-by-Step (Recommended for First Use)#

3.3 One-Click Deployment#

4. Cluster Management#

4.1 Service Status#

4.2 Start / Stop / Restart#

4.3 Functional Verification (13 checks)#

5. Web UIs#

6. Troubleshooting#

SSH Connection Failed#

Port Already In Use#

Service Start Timeout#

Adding New Nodes#

Slow Downloads#

7. Relationship with Shell/Python Scripts#