Big Data Cluster Ansible Automated Deployment Tutorial

pblh123@126.com (John Lee) — Fri, 29 May 2026 00:00:00 +0000

Big Data Cluster Ansible Automated Deployment Tutorial
#

An Ansible-based declarative automation deployment solution, coexisting with existing Shell/Python scripts, for 3-10 node clusters.

1. Prerequisites
#

1.1 Control Node Requirements
#

Requirement	Details
OS	Linux / macOS (Windows WSL compatible)
Ansible	>= 2.15
sshpass	`yum install sshpass` (CentOS) or `apt install sshpass` (Ubuntu)
Network	SSH access to all cluster nodes

1.2 Cluster Node Requirements
#

Requirement	Details
OS	CentOS 7.9+ / Rocky Linux 8+
SSH	Enabled with password or key authentication
Sudo	hadoop user has sudo privileges
Disk	`/opt/module` at least 10GB free

1.3 Install Ansible
#

# CentOS / Rocky
sudo yum install -y epel-release && sudo yum install -y ansible

# Ubuntu / Debian
sudo apt update && sudo apt install -y ansible

# macOS
brew install ansible

# Verify
ansible --version # Should show >= 2.15

1.4 Pre-cache Component Archives (Recommended)
#

cd ansible/files/cache/

# Download all components (~1.8 GB total)
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.9/apache-zookeeper-3.5.9-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.6.4/hbase-2.6.4-hadoop3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.8/spark-3.5.8-bin-hadoop3-scala2.13.tgz
wget -c https://maven.aliyun.com/repository/public/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar

JDK should be pre-installed to /opt/module/jdk1.8.0_471 on all nodes, or can be auto-downloaded via the common role.

大数据集群 Ansible 自动化部署教程

pblh123@126.com (John Lee) — Fri, 29 May 2026 00:00:00 +0000

大数据集群 Ansible 自动化部署教程
#

基于 Ansible 的声明式自动化部署方案，与现有 Shell/Python 脚本并存，适用于 3-10 节点集群。

一、前置准备
#

1.1 管理机要求
#

需求	说明
OS	Linux / macOS（Windows WSL 兼容）
Ansible	>= 2.15（安装方法见下）
sshpass	`yum install sshpass`（CentOS）或 `apt install sshpass`（Ubuntu）
网络	可 SSH 到达所有集群节点

1.2 集群节点要求
#

需求	说明
OS	CentOS 7.9+ / Rocky Linux 8+
SSH	已启用、密码认证或密钥认证
Sudo	hadoop 用户有 sudo 权限
磁盘	`/opt/module` 至少 10GB 可用空间

1.3 安装 Ansible
#

# CentOS / Rocky
sudo yum install -y epel-release
sudo yum install -y ansible

# Ubuntu / Debian
sudo apt update && sudo apt install -y ansible

# macOS
brew install ansible

# 验证安装
ansible --version # 应显示 >= 2.15

1.4 本地缓存安装包（推荐，加速部署）
#

cd ansible/files/cache/

# 下载 Hadoop 各组件，已下载的跳过（共约 1.8GB）
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.9/apache-zookeeper-3.5.9-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.6.4/hbase-2.6.4-hadoop3-bin.tar.gz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz
wget -c https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.8/spark-3.5.8-bin-hadoop3-scala2.13.tgz
wget -c https://maven.aliyun.com/repository/public/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar

JDK 需预先手动解压到所有节点的 /opt/module/jdk1.8.0_471（或通过 common role 自动下载，但 Adoptium 镜像速度较慢，建议手动安装）。

Ansible on 李先生的树洞

Big Data Cluster Ansible Automated Deployment Tutorial

Big Data Cluster Ansible Automated Deployment Tutorial #

1. Prerequisites #

1.1 Control Node Requirements #

1.2 Cluster Node Requirements #

1.3 Install Ansible #

1.4 Pre-cache Component Archives (Recommended) #

大数据集群 Ansible 自动化部署教程

大数据集群 Ansible 自动化部署教程 #

一、前置准备 #

1.1 管理机要求 #

1.2 集群节点要求 #

1.3 安装 Ansible #

1.4 本地缓存安装包（推荐，加速部署） #