Docker Big Data Cluster HA Deployment
#Docker Compose orchestrated 32-container full-stack HA big data cluster for local testing and learning.
Architecture Overview
# ┌─────────────────────────────────────────────────┐
│ Docker Bridge: bigdata-net │
│ 172.25.0.0/16 │
├─────────────────────────────────────────────────┤
│ Layer 0 (base): MySQL + ZooKeeper x3 │
│ Layer 1 (storage): JournalNode x3 │
│ Layer 2 (storage): NameNode x2 + ZKFC x2 │
│ Layer 3 (storage): DataNode x3 │
│ Layer 4 (compute): ResourceManager x2 + NM x3 │
│ Layer 5 (analytics): Hive Metastore/Server2 │
│ Layer 6 (analytics): HBase x5 + Kafka x3 │
│ Layer 7 (analytics): Spark + Flink History │
└─────────────────────────────────────────────────┘
| Component | Version | HA Strategy | Containers |
|---|
| HDFS | 3.3.6 | Dual NameNode + 3 JN + ZKFC | 10 |
| YARN | 3.3.6 | Dual RM + ZK State Store | 6 |
| ZooKeeper | 3.7.1 | 3-node quorum | 3 |
| HBase | 2.6.4 | Dual Master + ZK discovery | 5 |
| Hive | 3.1.3 | Metastore + Server2 (separate) | 2 |
| Kafka | 2.8.0 | 3 Broker + ZK coordination | 3 |
| Spark | 3.5.8 | History Server (YARN on-demand) | 1 |
| Flink | 1.19.1 | History Server (YARN on-demand) | 1 |
| MySQL | 5.7 | — | 1 |
Total: 32 containers, ~10-11GB RAM
Prerequisites
#- Docker Desktop 24+ (Windows/Mac/Linux)
- 16GB+ RAM (close other heavy apps recommended)
- 50GB+ disk space
- Component binaries cached in
ansible/files/cache/
Quick Start
#cd docker
# 1. Build the unified image (~10-20 min first time)
make build
# 2. Start by layer (or make all for everything)
make base # Layer 0: MySQL + ZKx3
make storage # Layer 1-3: HDFS HA
make compute # Layer 4: YARN HA
make analytics # Layer 5-7: Hive + HBase + Kafka + Spark/Flink
# Or one-shot
make all
# 3. First-time deployment: initialize HDFS
make init-hdfs
# 4. Verification
make verify # Level 1 Smoke Test
make verify-spark # Level 2a Spark
make verify-flink # Level 2b Flink
make verify-ha # Level 3 HA Failover
Layered Startup
#Start on demand — low-memory environments don’t need everything:
| Layer | Command | Services | Memory |
|---|
| Layer 0 | make base | MySQL, ZKx3 | ~1.5GB |
| Layer 1-3 | make storage | JNx3, NNx2, ZKFCx2, DNx3 | ~3.5GB |
| Layer 4 | make compute | RMx2, NMx3, JobHistory | ~2.5GB |
| Layer 5-7 | make analytics | Hivex2, HBasex5, Kafkax3, Spark/Flinkx2 | ~3.5GB |
Command Reference
#| Command | Description |
|---|
make build | Build unified image |
make all | Start all 32 containers |
make base | Start infrastructure layer |
make storage | Start HDFS HA storage layer |
make compute | Start YARN HA compute layer |
make analytics | Start analytics layer |
make status | Container resource usage |
make ps | Container list |
make verify | Level 1 Smoke Test (7 items) |
make verify-spark | Level 2a Spark verification |
make verify-flink | Level 2b Flink verification |
make verify-ha | Level 3 HA failover test (10 items) |
make init-hdfs | First-time HDFS format + init |
make stop | Stop all services (reverse order) |
make restart | Restart all |
make clean | Remove containers and network |
make logs | View all logs |
make logs-<name> | View specific container logs |
make shell-<name> | Enter container shell |
Verification Tests
#Level 1: Smoke Test (7 items)
#| # | Test | What it verifies |
|---|
| S1 | HDFS read/write | Write → Read → Delete |
| S2 | YARN nodes | >=3 NM RUNNING |
| S3 | ZK cluster | 1 leader + 2 follower |
| S4 | Hive Metastore | Port 9083 reachable |
| S5 | HiveServer2 | Port 10000 reachable |
| S6 | HBase Shell | status command works |
| S7 | Kafka E2E | Create → Produce → Consume → Delete |
Level 2: Spark/Flink Functional (6 items)
#| # | Test |
|---|
| SP1 | Spark Pi (YARN cluster mode) |
| SP2 | Spark SQL query |
| SP3 | Spark History Server HTTP |
| FL1 | Flink YARN Session creation |
| FL2 | Flink WordCount batch job |
| FL3 | Flink History Server HTTP |
Level 3: HA Failover (10 items)
#| # | Test | Fault Injection |
|---|
| HA1 | NN failover | Stop namenode-active |
| HA2 | NN recovery | Start namenode-active |
| HA3 | HDFS fault tolerance | Read/write after NN switch |
| HA4 | RM failover | Stop resourcemanager-1 |
| HA5 | Spark on HA | Submit after RM switch |
| HA6 | Flink on HA | Session after RM switch |
| HA7 | HMaster failover | Stop hbase-master-1 |
| HA8 | JN fault tolerance | Stop 1/3 JN |
| HA9 | ZK fault tolerance | Stop 1/3 ZK |
| HA10 | Kafka fault tolerance | Stop 1/3 Broker |
Web UI Access
#| Service | URL |
|---|
| NameNode Active | http://localhost:9870 |
| NameNode Standby | http://localhost:9871 |
| YARN RM1 | http://localhost:8088 |
| YARN RM2 | http://localhost:8089 |
| JobHistory | http://localhost:19888 |
| HBase Master1 | http://localhost:16010 |
| HBase Master2 | http://localhost:16011 |
| Spark History | http://localhost:18080 |
| Flink History | http://localhost:8082 |
Troubleshooting
#Container fails to start
#docker compose --profile <name> logs # View layer logs
make logs-<container> # View specific container
make shell-<container> # Enter container for debugging
Out of memory
#- Reduce JVM heap in
.env (e.g., NAMENODE_HEAP=128m) - Start by layer instead of all at once
- Close other heavy apps on Windows
HDFS not healthy
#make init-hdfs # Re-format (destroys data)
docker compose --profile storage restart # Restart storage layer
Port conflicts
#Edit ports mappings in docker-compose.yml (host port is on the left).
Comparison with Bare-Metal/Ansible
#| Feature | Docker | Ansible |
|---|
| Purpose | Local testing/learning | Production bare-metal |
| Memory | 10-11GB | 12GB+ |
| HA | Full production-grade | Full production-grade |
| Config | Standalone config/ dir | Jinja2 templates |
| Management | Makefile + docker compose | ansible-playbook |
| Network | Bridge (172.25.0.0/16) | Physical NIC |