https://blog.pblhlyr.top/tags/docker/Recent content in Docker on 李先生的树洞Hugo -- gohugo.iozh-cnpblh123@126.com (John Lee)pblh123@126.com (John Lee)© 2026 John LeeFri, 29 May 2026 00:00:00 +0000https://blog.pblhlyr.top/en/posts/docker-big-data-cluster-ha-deployment/Fri, 29 May 2026 00:00:00 +0000pblh123@126.com (John Lee)https://blog.pblhlyr.top/en/posts/docker-big-data-cluster-ha-deployment/<h1 class="relative group">Docker Big Data Cluster HA Deployment <div id="docker-big-data-cluster-ha-deployment" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#docker-big-data-cluster-ha-deployment" aria-label="锚点">#</a> </span> </h1> <p>Docker Compose orchestrated 32-container full-stack HA big data cluster for local testing and learning.</p> <h2 class="relative group">Architecture Overview <div id="architecture-overview" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#architecture-overview" aria-label="锚点">#</a> </span> </h2> <div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> ┌─────────────────────────────────────────────────┐ </span></span><span class="line"><span class="cl"> │ Docker Bridge: bigdata-net │ </span></span><span class="line"><span class="cl"> │ 172.25.0.0/16 │ </span></span><span class="line"><span class="cl"> ├─────────────────────────────────────────────────┤ </span></span><span class="line"><span class="cl"> │ Layer 0 (base): MySQL + ZooKeeper x3 │ </span></span><span class="line"><span class="cl"> │ Layer 1 (storage): JournalNode x3 │ </span></span><span class="line"><span class="cl"> │ Layer 2 (storage): NameNode x2 + ZKFC x2 │ </span></span><span class="line"><span class="cl"> │ Layer 3 (storage): DataNode x3 │ </span></span><span class="line"><span class="cl"> │ Layer 4 (compute): ResourceManager x2 + NM x3 │ </span></span><span class="line"><span class="cl"> │ Layer 5 (analytics): Hive Metastore/Server2 │ </span></span><span class="line"><span class="cl"> │ Layer 6 (analytics): HBase x5 + Kafka x3 │ </span></span><span class="line"><span class="cl"> │ Layer 7 (analytics): Spark + Flink History │ </span></span><span class="line"><span class="cl"> └─────────────────────────────────────────────────┘</span></span></code></pre></div></div> <table> <thead> <tr> <th>Component</th> <th>Version</th> <th>HA Strategy</th> <th>Containers</th> </tr> </thead> <tbody> <tr> <td>HDFS</td> <td>3.3.6</td> <td>Dual NameNode + 3 JN + ZKFC</td> <td>10</td> </tr> <tr> <td>YARN</td> <td>3.3.6</td> <td>Dual RM + ZK State Store</td> <td>6</td> </tr> <tr> <td>ZooKeeper</td> <td>3.7.1</td> <td>3-node quorum</td> <td>3</td> </tr> <tr> <td>HBase</td> <td>2.6.4</td> <td>Dual Master + ZK discovery</td> <td>5</td> </tr> <tr> <td>Hive</td> <td>3.1.3</td> <td>Metastore + Server2 (separate)</td> <td>2</td> </tr> <tr> <td>Kafka</td> <td>2.8.0</td> <td>3 Broker + ZK coordination</td> <td>3</td> </tr> <tr> <td>Spark</td> <td>3.5.8</td> <td>History Server (YARN on-demand)</td> <td>1</td> </tr> <tr> <td>Flink</td> <td>1.19.1</td> <td>History Server (YARN on-demand)</td> <td>1</td> </tr> <tr> <td>MySQL</td> <td>5.7</td> <td>—</td> <td>1</td> </tr> </tbody> </table> <p><strong>Total: 32 containers, ~10-11GB RAM</strong></p>https://blog.pblhlyr.top/posts/docker-%E5%A4%A7%E6%95%B0%E6%8D%AE%E9%9B%86%E7%BE%A4%E9%AB%98%E5%8F%AF%E7%94%A8%E9%83%A8%E7%BD%B2/Fri, 29 May 2026 00:00:00 +0000pblh123@126.com (John Lee)https://blog.pblhlyr.top/posts/docker-%E5%A4%A7%E6%95%B0%E6%8D%AE%E9%9B%86%E7%BE%A4%E9%AB%98%E5%8F%AF%E7%94%A8%E9%83%A8%E7%BD%B2/<h1 class="relative group">Docker 大数据集群高可用部署 <div id="docker-大数据集群高可用部署" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#docker-%e5%a4%a7%e6%95%b0%e6%8d%ae%e9%9b%86%e7%be%a4%e9%ab%98%e5%8f%af%e7%94%a8%e9%83%a8%e7%bd%b2" aria-label="锚点">#</a> </span> </h1> <p>Docker Compose 编排的 32 容器全组件 HA 大数据集群，用于本地测试与学习。</p> <h2 class="relative group">架构概述 <div id="架构概述" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%9e%b6%e6%9e%84%e6%a6%82%e8%bf%b0" aria-label="锚点">#</a> </span> </h2> <div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> ┌─────────────────────────────────────────────────┐ </span></span><span class="line"><span class="cl"> │ Docker Bridge: bigdata-net │ </span></span><span class="line"><span class="cl"> │ 172.25.0.0/16 │ </span></span><span class="line"><span class="cl"> ├─────────────────────────────────────────────────┤ </span></span><span class="line"><span class="cl"> │ Layer 0 (base): MySQL + ZooKeeper x3 │ </span></span><span class="line"><span class="cl"> │ Layer 1 (storage): JournalNode x3 │ </span></span><span class="line"><span class="cl"> │ Layer 2 (storage): NameNode x2 + ZKFC x2 │ </span></span><span class="line"><span class="cl"> │ Layer 3 (storage): DataNode x3 │ </span></span><span class="line"><span class="cl"> │ Layer 4 (compute): ResourceManager x2 + NM x3 │ </span></span><span class="line"><span class="cl"> │ Layer 5 (analytics): Hive Metastore/Server2 │ </span></span><span class="line"><span class="cl"> │ Layer 6 (analytics): HBase x5 + Kafka x3 │ </span></span><span class="line"><span class="cl"> │ Layer 7 (analytics): Spark + Flink History │ </span></span><span class="line"><span class="cl"> └─────────────────────────────────────────────────┘</span></span></code></pre></div></div> <table> <thead> <tr> <th>组件</th> <th>版本</th> <th>HA 方案</th> <th>容器数</th> </tr> </thead> <tbody> <tr> <td>HDFS</td> <td>3.3.6</td> <td>双 NameNode + 3 JournalNode + ZKFC</td> <td>10</td> </tr> <tr> <td>YARN</td> <td>3.3.6</td> <td>双 ResourceManager + ZK 状态存储</td> <td>6</td> </tr> <tr> <td>ZooKeeper</td> <td>3.7.1</td> <td>3 节点仲裁</td> <td>3</td> </tr> <tr> <td>HBase</td> <td>2.6.4</td> <td>双 Master + ZK 发现</td> <td>5</td> </tr> <tr> <td>Hive</td> <td>3.1.3</td> <td>Metastore + Server2（独立容器）</td> <td>2</td> </tr> <tr> <td>Kafka</td> <td>2.8.0</td> <td>3 Broker + ZK 协调</td> <td>3</td> </tr> <tr> <td>Spark</td> <td>3.5.8</td> <td>History Server（YARN 模式按需提交）</td> <td>1</td> </tr> <tr> <td>Flink</td> <td>1.19.1</td> <td>History Server（YARN 模式按需提交）</td> <td>1</td> </tr> <tr> <td>MySQL</td> <td>5.7</td> <td>—</td> <td>1</td> </tr> </tbody> </table> <p><strong>总计：32 容器，~10-11GB 内存</strong></p>