https://blog.pblhlyr.top/tags/high-availability/Recent content in High-Availability on 李先生的树洞Hugo -- gohugo.iozh-cnpblh123@126.com (John Lee)pblh123@126.com (John Lee)© 2026 John LeeFri, 29 May 2026 00:00:00 +0000https://blog.pblhlyr.top/en/posts/docker-big-data-cluster-ha-deployment/Fri, 29 May 2026 00:00:00 +0000pblh123@126.com (John Lee)https://blog.pblhlyr.top/en/posts/docker-big-data-cluster-ha-deployment/<h1 class="relative group">Docker Big Data Cluster HA Deployment <div id="docker-big-data-cluster-ha-deployment" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#docker-big-data-cluster-ha-deployment" aria-label="锚点">#</a> </span> </h1> <p>Docker Compose orchestrated 32-container full-stack HA big data cluster for local testing and learning.</p> <h2 class="relative group">Architecture Overview <div id="architecture-overview" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#architecture-overview" aria-label="锚点">#</a> </span> </h2> <div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> ┌─────────────────────────────────────────────────┐ </span></span><span class="line"><span class="cl"> │ Docker Bridge: bigdata-net │ </span></span><span class="line"><span class="cl"> │ 172.25.0.0/16 │ </span></span><span class="line"><span class="cl"> ├─────────────────────────────────────────────────┤ </span></span><span class="line"><span class="cl"> │ Layer 0 (base): MySQL + ZooKeeper x3 │ </span></span><span class="line"><span class="cl"> │ Layer 1 (storage): JournalNode x3 │ </span></span><span class="line"><span class="cl"> │ Layer 2 (storage): NameNode x2 + ZKFC x2 │ </span></span><span class="line"><span class="cl"> │ Layer 3 (storage): DataNode x3 │ </span></span><span class="line"><span class="cl"> │ Layer 4 (compute): ResourceManager x2 + NM x3 │ </span></span><span class="line"><span class="cl"> │ Layer 5 (analytics): Hive Metastore/Server2 │ </span></span><span class="line"><span class="cl"> │ Layer 6 (analytics): HBase x5 + Kafka x3 │ </span></span><span class="line"><span class="cl"> │ Layer 7 (analytics): Spark + Flink History │ </span></span><span class="line"><span class="cl"> └─────────────────────────────────────────────────┘</span></span></code></pre></div></div> <table> <thead> <tr> <th>Component</th> <th>Version</th> <th>HA Strategy</th> <th>Containers</th> </tr> </thead> <tbody> <tr> <td>HDFS</td> <td>3.3.6</td> <td>Dual NameNode + 3 JN + ZKFC</td> <td>10</td> </tr> <tr> <td>YARN</td> <td>3.3.6</td> <td>Dual RM + ZK State Store</td> <td>6</td> </tr> <tr> <td>ZooKeeper</td> <td>3.7.1</td> <td>3-node quorum</td> <td>3</td> </tr> <tr> <td>HBase</td> <td>2.6.4</td> <td>Dual Master + ZK discovery</td> <td>5</td> </tr> <tr> <td>Hive</td> <td>3.1.3</td> <td>Metastore + Server2 (separate)</td> <td>2</td> </tr> <tr> <td>Kafka</td> <td>2.8.0</td> <td>3 Broker + ZK coordination</td> <td>3</td> </tr> <tr> <td>Spark</td> <td>3.5.8</td> <td>History Server (YARN on-demand)</td> <td>1</td> </tr> <tr> <td>Flink</td> <td>1.19.1</td> <td>History Server (YARN on-demand)</td> <td>1</td> </tr> <tr> <td>MySQL</td> <td>5.7</td> <td>—</td> <td>1</td> </tr> </tbody> </table> <p><strong>Total: 32 containers, ~10-11GB RAM</strong></p>