-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
149 lines (77 loc) · 208 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>石臻臻的杂货铺</title>
<link href="http://example.com/atom.xml" rel="self"/>
<link href="http://example.com/"/>
<updated>2021-08-23T09:56:22.583Z</updated>
<id>http://example.com/</id>
<author>
<name>石臻臻</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>test</title>
<link href="http://example.com/2021/08/23/test/"/>
<id>http://example.com/2021/08/23/test/</id>
<published>2021-08-23T09:45:04.000Z</published>
<updated>2021-08-23T09:56:22.583Z</updated>
<content type="html"><![CDATA[<h1 id="Tabs"><a href="#Tabs" class="headerlink" title="Tabs"></a>Tabs</h1><div class="tabs" id="test1"><ul class="nav-tabs"><li class="tab active"><button type="button" data-href="#test1-1">test1 1</button></li><li class="tab"><button type="button" data-href="#test1-2">test1 2</button></li><li class="tab"><button type="button" data-href="#test1-3">test1 3</button></li></ul><div class="tab-contents"><div class="tab-item-content active" id="test1-1"><p><strong>This is Tab 1.</strong></p><button type="button" class="tab-to-top" aria-label="scroll to top"><i class="fas fa-arrow-up"></i></button></div><div class="tab-item-content" id="test1-2"><p><strong>This is Tab 2.</strong></p><button type="button" class="tab-to-top" aria-label="scroll to top"><i class="fas fa-arrow-up"></i></button></div><div class="tab-item-content" id="test1-3"><p>臣亮言：<mark class="hl-label default">先帝</mark> 創業未半，而<mark class="hl-label blue">中道崩殂</mark> 。今天下三分，<mark class="hl-label pink">益州疲敝</mark> ，此誠<mark class="hl-label red">危急存亡之秋</mark> 也！然侍衞之臣，不懈於內；<mark class="hl-label purple">忠志之士</mark> ，忘身於外者，蓋追先帝之殊遇，欲報之於陛下也。誠宜開張聖聽，以光先帝遺德，恢弘志士之氣；不宜妄自菲薄，引喻失義，以塞忠諫之路也。<br>宮中、府中，俱為一體；陟罰臧否，不宜異同。若有<mark class="hl-label orange">作奸</mark> 、<mark class="hl-label green">犯科</mark> ，及為忠善者，宜付有司，論其刑賞，以昭陛下平明之治；不宜偏私，使內外異法也。</p><button type="button" class="tab-to-top" aria-label="scroll to top"><i class="fas fa-arrow-up"></i></button></div></div></div><h1 id="tag-hide-隐藏内容"><a href="#tag-hide-隐藏内容" class="headerlink" title="tag-hide 隐藏内容"></a>tag-hide 隐藏内容</h1><div class="hide-block"><button type="button" class="hide-button button--animated" style="background-color: bg;color: color">display </button><div class="hide-content"><p>哟 现实了隐藏的内容呢！！！</p></div></div><div class="hide-toggle" style="border: 1px solid bg"><div class="hide-button toggle-title" style="background-color: bg;color: color"><i class="fas fa-caret-right fa-fw"></i><span>display</span></div> <div class="hide-content"><p>content</p></div></div><h1 id="高亮背景颜色"><a href="#高亮背景颜色" class="headerlink" title="高亮背景颜色"></a>高亮背景颜色</h1><p>臣亮言：<mark class="hl-label default">先帝</mark> 創業未半，而<mark class="hl-label blue">中道崩殂</mark> 。今天下三分，<mark class="hl-label pink">益州疲敝</mark> ，此誠<mark class="hl-label red">危急存亡之秋</mark> 也！然侍衞之臣，不懈於內；<mark class="hl-label purple">忠志之士</mark> ，忘身於外者，蓋追先帝之殊遇，欲報之於陛下也。誠宜開張聖聽，以光先帝遺德，恢弘志士之氣；不宜妄自菲薄，引喻失義，以塞忠諫之路也。<br>宮中、府中，俱為一體；陟罰臧否，不宜異同。若有<mark class="hl-label orange">作奸</mark> 、<mark class="hl-label green">犯科</mark> ，及為忠善者，宜付有司，論其刑賞，以昭陛下平明之治；不宜偏私，使內外異法也。</p><h1 id="Note-Bootstrap-Callout"><a href="#Note-Bootstrap-Callout" class="headerlink" title="Note (Bootstrap Callout)"></a>Note (Bootstrap Callout)</h1><h2 id="simple"><a href="#simple" class="headerlink" title="simple"></a>simple</h2><div class="note simple"><p>默認 提示塊標籤</p></div><div class="note default simple"><p>default 提示塊標籤</p></div><div class="note primary simple"><p>primary 提示塊標籤</p></div><div class="note success simple"><p>success 提示塊標籤</p></div><div class="note info simple"><p>info 提示塊標籤</p></div><div class="note warning simple"><p>warning 提示塊標籤</p></div><div class="note danger simple"><p>danger 提示塊標籤</p></div><h1 id="modern"><a href="#modern" class="headerlink" title="modern"></a>modern</h1><div class="note modern"><p>默認 提示塊標籤</p></div><div class="note default modern"><p>default 提示塊標籤</p></div><div class="note primary modern"><p>primary 提示塊標籤</p></div><div class="note success modern"><p>success 提示塊標籤</p></div><div class="note info modern"><p>info 提示塊標籤</p></div><div class="note warning modern"><p>warning 提示塊標籤</p></div><div class="note danger modern"><p>danger 提示塊標籤</p></div><h1 id="flat"><a href="#flat" class="headerlink" title="flat"></a>flat</h1><div class="note flat"><p>默認 提示塊標籤</p></div><div class="note default flat"><p>default 提示塊標籤</p></div><div class="note primary flat"><p>primary 提示塊標籤</p></div><div class="note success flat"><p>success 提示塊標籤</p></div><div class="note info flat"><p>info 提示塊標籤</p></div><div class="note warning flat"><p>warning 提示塊標籤</p></div><div class="note danger flat"><p>danger 提示塊標籤</p></div><h1 id="disabled"><a href="#disabled" class="headerlink" title="disabled"></a>disabled</h1><div class="note disabled"><p>默認 提示塊標籤</p></div><div class="note default disabled"><p>default 提示塊標籤</p></div><div class="note primary disabled"><p>primary 提示塊標籤</p></div><div class="note success disabled"><p>success 提示塊標籤</p></div><div class="note info disabled"><p>info 提示塊標籤</p></div><div class="note warning disabled"><p>warning 提示塊標籤</p></div><div class="note danger disabled"><p>danger 提示塊標籤</p></div>]]></content>
<summary type="html"><h1 id="Tabs"><a href="#Tabs" class="headerlink" title="Tabs"></a>Tabs</h1><div class="tabs" id="test1"><ul class="nav-tabs"><li class="tab </summary>
</entry>
<entry>
<title>【kafka源码】kafka分区副本的分配规则</title>
<link href="http://example.com/2021/08/21/partition-reasignment/"/>
<id>http://example.com/2021/08/21/partition-reasignment/</id>
<published>2021-08-21T06:05:53.000Z</published>
<updated>2021-08-23T09:33:31.936Z</updated>
<content type="html"><![CDATA[<p>哪個英文字母最酷？ <span class="hide-inline"><button type="button" class="hide-button button--animated" style="background-color: #FF7242;color: #fff">查看答案 </button><span class="hide-content">因為西裝褲(C裝酷)</span></span></p><p>門裏站着一個人? <span class="hide-inline"><button type="button" class="hide-button button--animated" style="">Click </button><span class="hide-content">閃</span></span></p><h2 id="源码分析"><a href="#源码分析" class="headerlink" title="源码分析"></a>源码分析</h2><p><strong>创建Topic的源码入口 <code>AdminManager.createTopics()</code></strong></p><p>以下只列出了分区分配相关代码其他省略</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createTopics</span></span>(timeout: <span class="type">Int</span>,</span><br><span class="line"> validateOnly: <span class="type">Boolean</span>,</span><br><span class="line"> toCreate: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">CreatableTopic</span>],</span><br><span class="line"> includeConfigsAndMetatadata: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">CreatableTopicResult</span>],</span><br><span class="line"> responseCallback: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">ApiError</span>] => <span class="type">Unit</span>): <span class="type">Unit</span> = {</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 1. map over topics creating assignment and calling zookeeper</span></span><br><span class="line"> <span class="keyword">val</span> brokers = metadataCache.getAliveBrokers.map { b => kafka.admin.<span class="type">BrokerMetadata</span>(b.id, b.rack) }</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">val</span> metadata = toCreate.values.map(topic =></span><br><span class="line"> <span class="keyword">try</span> { </span><br><span class="line"> <span class="keyword">val</span> assignments = <span class="keyword">if</span> (topic.assignments().isEmpty) {</span><br><span class="line"> <span class="type">AdminUtils</span>.assignReplicasToBrokers(</span><br><span class="line"> brokers, resolvedNumPartitions, resolvedReplicationFactor)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">val</span> assignments = <span class="keyword">new</span> mutable.<span class="type">HashMap</span>[<span class="type">Int</span>, <span class="type">Seq</span>[<span class="type">Int</span>]]</span><br><span class="line"> <span class="comment">// Note: we don't check that replicaAssignment contains unknown brokers - unlike in add-partitions case,</span></span><br><span class="line"> <span class="comment">// this follows the existing logic in TopicCommand</span></span><br><span class="line"> topic.assignments.asScala.foreach {</span><br><span class="line"> <span class="keyword">case</span> assignment => assignments(assignment.partitionIndex()) =</span><br><span class="line"> assignment.brokerIds().asScala.map(a => a: <span class="type">Int</span>)</span><br><span class="line"> }</span><br><span class="line"> assignments</span><br><span class="line"> }</span><br><span class="line"> trace(<span class="string">s"Assignments for topic <span class="subst">$topic</span> are <span class="subst">$assignments</span> "</span>)</span><br><span class="line"> </span><br><span class="line"> }</span><br><span class="line"></span><br></pre></td></tr></table></figure><ol><li>以上有两种方式,一种是我们没有指定分区分配的情况也就是没有使用参数<code>--replica-assignment</code>；一种是自己指定了分区分配</li></ol><h3 id="1-自己指定了分区分配规则"><a href="#1-自己指定了分区分配规则" class="headerlink" title="1. 自己指定了分区分配规则"></a>1. 自己指定了分区分配规则</h3><p>从源码中得知, 会把我们指定的规则进行了包装,<strong>注意它并没有去检查你指定的Broker是否存在;</strong></p><h3 id="2-自动分配-AdminUtils-assignReplicasToBrokers"><a href="#2-自动分配-AdminUtils-assignReplicasToBrokers" class="headerlink" title="2. 自动分配 AdminUtils.assignReplicasToBrokers"></a>2. 自动分配 AdminUtils.assignReplicasToBrokers</h3><p><img src="https://img-blog.csdnimg.cn/20210609202822549.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="图片描述是啥"></p><ol><li>参数检查: 分区数>0; 副本数>0; 副本数<=Broker数 (如果自己未定义会直接使用Broker中个配置)</li><li>根据是否有 机架信息来进行不同方式的分配;</li><li>要么整个集群都有机架信息,要么整个集群都没有机架信息; 否则抛出异常</li></ol><p><strong>副本分配的几个原则:</strong></p><ol><li>将副本平均分布在所有的 Broker 上; </li><li>partition 的多个副本应该分配在不同的 Broker 上;</li><li>如果所有的 Broker 有机架信息的话, partition 的副本应该分配到不同的机架上。</li></ol><h4 id="无机架方式分配"><a href="#无机架方式分配" class="headerlink" title="无机架方式分配"></a>无机架方式分配</h4><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">AdminUtils</span>.assignReplicasToBrokersRackUnaware</span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 副本分配时,有三个原则:</span></span><br><span class="line"><span class="comment"> * 1. 将副本平均分布在所有的 Broker 上;</span></span><br><span class="line"><span class="comment"> * 2. partition 的多个副本应该分配在不同的 Broker 上;</span></span><br><span class="line"><span class="comment"> * 3. 如果所有的 Broker 有机架信息的话, partition 的副本应该分配到不同的机架上。</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * 为实现上面的目标,在没有机架感知的情况下，应该按照下面两个原则分配 replica:</span></span><br><span class="line"><span class="comment"> * 1. 从 broker.list 随机选择一个 Broker,使用 round-robin 算法分配每个 partition 的第一个副本;</span></span><br><span class="line"><span class="comment"> * 2. 对于这个 partition 的其他副本,逐渐增加 Broker.id 来选择 replica 的分配。</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">assignReplicasToBrokersRackUnaware</span></span>(nPartitions: <span class="type">Int</span>,</span><br><span class="line"> replicationFactor: <span class="type">Int</span>,</span><br><span class="line"> brokerList: <span class="type">Seq</span>[<span class="type">Int</span>],</span><br><span class="line"> fixedStartIndex: <span class="type">Int</span>,</span><br><span class="line"> startPartitionId: <span class="type">Int</span>): <span class="type">Map</span>[<span class="type">Int</span>, <span class="type">Seq</span>[<span class="type">Int</span>]] = {</span><br><span class="line"> <span class="keyword">val</span> ret = mutable.<span class="type">Map</span>[<span class="type">Int</span>, <span class="type">Seq</span>[<span class="type">Int</span>]]()</span><br><span class="line"> <span class="comment">// 这里是上一层传递过了的所有 存活的Broker列表的ID</span></span><br><span class="line"> <span class="keyword">val</span> brokerArray = brokerList.toArray</span><br><span class="line"> <span class="comment">//默认随机选一个index开始</span></span><br><span class="line"> <span class="keyword">val</span> startIndex = <span class="keyword">if</span> (fixedStartIndex >= <span class="number">0</span>) fixedStartIndex <span class="keyword">else</span> rand.nextInt(brokerArray.length)</span><br><span class="line"> <span class="comment">//默认从0这个分区号开始</span></span><br><span class="line"> <span class="keyword">var</span> currentPartitionId = math.max(<span class="number">0</span>, startPartitionId)</span><br><span class="line"> <span class="keyword">var</span> nextReplicaShift = <span class="keyword">if</span> (fixedStartIndex >= <span class="number">0</span>) fixedStartIndex <span class="keyword">else</span> rand.nextInt(brokerArray.length)</span><br><span class="line"> <span class="keyword">for</span> (_ <- <span class="number">0</span> until nPartitions) {</span><br><span class="line"> <span class="keyword">if</span> (currentPartitionId > <span class="number">0</span> && (currentPartitionId % brokerArray.length == <span class="number">0</span>))</span><br><span class="line"> nextReplicaShift += <span class="number">1</span></span><br><span class="line"> <span class="keyword">val</span> firstReplicaIndex = (currentPartitionId + startIndex) % brokerArray.length</span><br><span class="line"> <span class="keyword">val</span> replicaBuffer = mutable.<span class="type">ArrayBuffer</span>(brokerArray(firstReplicaIndex))</span><br><span class="line"> <span class="keyword">for</span> (j <- <span class="number">0</span> until replicationFactor - <span class="number">1</span>)</span><br><span class="line"> replicaBuffer += brokerArray(replicaIndex(firstReplicaIndex, nextReplicaShift, j, brokerArray.length))</span><br><span class="line"> ret.put(currentPartitionId, replicaBuffer)</span><br><span class="line"> currentPartitionId += <span class="number">1</span></span><br><span class="line"> }</span><br><span class="line"> ret</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">//主要的计算间隔数的方法</span></span><br><span class="line"> <span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">replicaIndex</span></span>(firstReplicaIndex: <span class="type">Int</span>, secondReplicaShift: <span class="type">Int</span>, replicaIndex: <span class="type">Int</span>, nBrokers: <span class="type">Int</span>): <span class="type">Int</span> = {</span><br><span class="line"> <span class="keyword">val</span> shift = <span class="number">1</span> + (secondReplicaShift + replicaIndex) % (nBrokers - <span class="number">1</span>)</span><br><span class="line"> (firstReplicaIndex + shift) % nBrokers</span><br><span class="line"> }</span><br><span class="line"> </span><br></pre></td></tr></table></figure><ol><li>从 broker.list 随机选择一个 Broker,使用 round-robin 算法分配每个 partition 的第一个副本;</li><li>对于这个 partition 的其他副本,逐渐增加 Broker.id 来选择 replica 的分配。</li><li>对于副本分配来说,每经历一次Broker的遍历,则第一个副本跟后面的副本直接的间隔+1;</li></ol><p>从代码和描述来看,可能理解不是很简单,但是下面的图我相信会让你非常快速的理解;</p><p>我们稍微在这段代码里面节点日志 <img src="https://img-blog.csdnimg.cn/2021070515393097.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"> 然后写段单元测试,执行一下,看看分配过程</p><h5 id="Broker列表-0-1-2-3-4-分区数-10-副本数3-起始随机BrokerId-0-起始随机nextReplicaShift-0"><a href="#Broker列表-0-1-2-3-4-分区数-10-副本数3-起始随机BrokerId-0-起始随机nextReplicaShift-0" class="headerlink" title="Broker列表{0,1,2,3,4} 分区数 10 副本数3 起始随机BrokerId=0; 起始随机nextReplicaShift=0"></a>Broker列表{0,1,2,3,4} 分区数 10 副本数3 起始随机BrokerId=0; 起始随机nextReplicaShift=0</h5><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">@Test</span><br><span class="line">def testReplicaAssignment2(): Unit = {</span><br><span class="line"> val brokerMetadatas = (0 to 4).map(new BrokerMetadata(_, None))</span><br><span class="line"> AdminUtils.assignReplicasToBrokers(brokerMetadatas, 10, 3, 0)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>输出:</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">起始随机startIndex:0;起始随机nextReplicaShift：0</span><br><span class="line">(p-0,ArrayBuffer(0, 1, 2))</span><br><span class="line">(p-1,ArrayBuffer(1, 2, 3))</span><br><span class="line">(p-2,ArrayBuffer(2, 3, 4))</span><br><span class="line">(p-3,ArrayBuffer(3, 4, 0))</span><br><span class="line">(p-4,ArrayBuffer(4, 0, 1))</span><br><span class="line">变更nextReplicaShift:1</span><br><span class="line">(p-5,ArrayBuffer(0, 2, 3))</span><br><span class="line">(p-6,ArrayBuffer(1, 3, 4))</span><br><span class="line">(p-7,ArrayBuffer(2, 4, 0))</span><br><span class="line">(p-8,ArrayBuffer(3, 0, 1))</span><br><span class="line">(p-9,ArrayBuffer(4, 1, 2))</span><br></pre></td></tr></table></figure><p>看图 <img src="https://img-blog.csdnimg.cn/20210705172124255.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><p>上面是分配的情况,我们每一行每一行看, 每次都是先把每个分区的副本分配好的; </p><ol><li>最开始的时候,随机一个Broker作为第一个来接受P0; 这里我们假设随机到了 broker-0; 所以第一个P0在broker-0上; 那么第二个<code>p0-2</code>的位置跟<code>nextReplicaShit</code>有关,这个值也是随机的,这里假设随机的起始值也是0; 这个值意思可以简单的理解为,第一个副本和第二个副本的间隔; </li><li>因为<code>nextReplicaShit=0</code>; 所以p0的分配分别再 {0,1,2}</li><li>然后再分配后面的分区,分区的第一个副本位置都是按照broker顺序遍历的; </li><li>直到这一次的broker遍历完了,那么就要重头再进行遍历了, 同时<code>nextReplicaShit=nextReplicaShit+1=1</code>;</li><li>P5-1 再broker-0上,然后p5-2要跟p5-1间隔<code>nextReplicaShit=1</code>个位置,所以p5-2这时候在broker-2上,P5-3则在P5-2基础上顺推一位就行了,如果顺推的位置上已经有了副本,则继续顺推到没有当前分区副本的Broker</li><li>如果分区过多,有可能nextReplicaShift就变的挺大,在算第一个跟第二个副本的间隔的时候,不用把第一个副本算进去; 假如下面起始是 5，其中经历过的间隔就是 （ 1->2->3->4->1 ）所以PN-2就落在 BrokerLIst[2]上了 <img src="https://img-blog.csdnimg.cn/20210706143509859.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li></ol><h5 id="Broker列表-0-1-2-3-4-分区数-11-副本数3-起始随机BrokerId-0-起始随机nextReplicaShift-0"><a href="#Broker列表-0-1-2-3-4-分区数-11-副本数3-起始随机BrokerId-0-起始随机nextReplicaShift-0" class="headerlink" title="Broker列表{0,1,2,3,4} 分区数 11 副本数3 起始随机BrokerId=0; 起始随机nextReplicaShift=0"></a>Broker列表{0,1,2,3,4} 分区数 11 副本数3 起始随机BrokerId=0; 起始随机nextReplicaShift=0</h5><p>在上面基础上,再增加1个分区,你知道会怎么分配么 结果：</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">起始随机startIndex:0;起始随机nextReplicaShift：0</span><br><span class="line">(p-0,ArrayBuffer(0, 1, 2))</span><br><span class="line">(p-1,ArrayBuffer(1, 2, 3))</span><br><span class="line">(p-2,ArrayBuffer(2, 3, 4))</span><br><span class="line">(p-3,ArrayBuffer(3, 4, 0))</span><br><span class="line">(p-4,ArrayBuffer(4, 0, 1))</span><br><span class="line">变更nextReplicaShift:1</span><br><span class="line">(p-5,ArrayBuffer(0, 2, 3))</span><br><span class="line">(p-6,ArrayBuffer(1, 3, 4))</span><br><span class="line">(p-7,ArrayBuffer(2, 4, 0))</span><br><span class="line">(p-8,ArrayBuffer(3, 0, 1))</span><br><span class="line">(p-9,ArrayBuffer(4, 1, 2))</span><br><span class="line">变更nextReplicaShift:2</span><br><span class="line">(p-10,ArrayBuffer(0, 3, 4))</span><br><span class="line">(p-11,ArrayBuffer(1, 4, 0))</span><br></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20210706100426171.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><h5 id="Broker列表-0-1-2-3-4-分区数-10-副本数4-起始随机BrokerId-0-起始随机nextReplicaShift-0"><a href="#Broker列表-0-1-2-3-4-分区数-10-副本数4-起始随机BrokerId-0-起始随机nextReplicaShift-0" class="headerlink" title="Broker列表{0,1,2,3,4} 分区数 10 副本数4 起始随机BrokerId=0; 起始随机nextReplicaShift=0"></a>Broker列表{0,1,2,3,4} 分区数 10 副本数4 起始随机BrokerId=0; 起始随机nextReplicaShift=0</h5><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">起始随机startIndex:0;起始随机nextReplicaShift：0</span><br><span class="line">(p-0,ArrayBuffer(0, 1, 2, 3))</span><br><span class="line">(p-1,ArrayBuffer(1, 2, 3, 4))</span><br><span class="line">(p-2,ArrayBuffer(2, 3, 4, 0))</span><br><span class="line">(p-3,ArrayBuffer(3, 4, 0, 1))</span><br><span class="line">(p-4,ArrayBuffer(4, 0, 1, 2))</span><br><span class="line">变更nextReplicaShift:1</span><br><span class="line">(p-5,ArrayBuffer(0, 2, 3, 4))</span><br><span class="line">(p-6,ArrayBuffer(1, 3, 4, 0))</span><br><span class="line">(p-7,ArrayBuffer(2, 4, 0, 1))</span><br><span class="line">(p-8,ArrayBuffer(3, 0, 1, 2))</span><br><span class="line">(p-9,ArrayBuffer(4, 1, 2, 3))</span><br></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20210706100327200.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><p>看看这里, 在上面的的副本=3的基础上,新增了一个副本=4, 原有的分配都基本没有变化, 只是在之前的分配基础上,按照顺序再新增了一个副本,见图中的 浅黄色区域 ，如果想缩小副本数量也是同样的道理;</p><p>上面预设的<code>nextReplicaShift=0</code>,并且BrokerList顺序也是 {0,1,2,3,4} ; 这样的情况理解起来稍微容易一点; 但是再实际的分配过程中,这个BrokerList并不是总是按照顺序来的,很可能都是乱的; 所以排列的位置是按照 BrokerList的下标来进行的; 看下图</p><h5 id="Broker列表-1-2-0-4-3-分区数-10-副本数3-起始随机startIndex-0-起始随机nextReplicaShift-3"><a href="#Broker列表-1-2-0-4-3-分区数-10-副本数3-起始随机startIndex-0-起始随机nextReplicaShift-3" class="headerlink" title="Broker列表{1,2,0,4,3} 分区数 10 副本数3 起始随机startIndex=0; 起始随机nextReplicaShift=3"></a>Broker列表{1,2,0,4,3} 分区数 10 副本数3 起始随机startIndex=0; 起始随机nextReplicaShift=3</h5><p><img src="https://img-blog.csdnimg.cn/20210706100237544.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><ol><li>注意BrokerList列表离元素的顺序,会影响分配结果, 这里分析的分配是指列表的顺序,不是Broker的ID</li><li><code>nextReplicaShift</code>是第一个分区副本跟第二个副本间隔的Broker数量,后面的副本则与上一个副本顺推就行如果顺推遇到已经存在副本,则再顺推</li><li>通过这里你也可以看出来,同一个副本不可能在同一个Broker中存在</li></ol><h4 id="有机架方式分配"><a href="#有机架方式分配" class="headerlink" title="有机架方式分配"></a>有机架方式分配</h4><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">assignReplicasToBrokersRackAware</span></span>(nPartitions: <span class="type">Int</span>,</span><br><span class="line"> replicationFactor: <span class="type">Int</span>,</span><br><span class="line"> brokerMetadatas: <span class="type">Seq</span>[<span class="type">BrokerMetadata</span>],</span><br><span class="line"> fixedStartIndex: <span class="type">Int</span>,</span><br><span class="line"> startPartitionId: <span class="type">Int</span>): <span class="type">Map</span>[<span class="type">Int</span>, <span class="type">Seq</span>[<span class="type">Int</span>]] = {</span><br><span class="line"> <span class="keyword">val</span> brokerRackMap = brokerMetadatas.collect { <span class="keyword">case</span> <span class="type">BrokerMetadata</span>(id, <span class="type">Some</span>(rack)) =></span><br><span class="line"> id -> rack</span><br><span class="line"> }.toMap</span><br><span class="line"> <span class="keyword">val</span> numRacks = brokerRackMap.values.toSet.size</span><br><span class="line"> <span class="keyword">val</span> arrangedBrokerList = getRackAlternatedBrokerList(brokerRackMap)</span><br><span class="line"> <span class="keyword">val</span> numBrokers = arrangedBrokerList.size</span><br><span class="line"> <span class="keyword">val</span> ret = mutable.<span class="type">Map</span>[<span class="type">Int</span>, <span class="type">Seq</span>[<span class="type">Int</span>]]()</span><br><span class="line"> <span class="keyword">val</span> startIndex = <span class="keyword">if</span> (fixedStartIndex >= <span class="number">0</span>) fixedStartIndex <span class="keyword">else</span> rand.nextInt(arrangedBrokerList.size)</span><br><span class="line"> <span class="keyword">var</span> currentPartitionId = math.max(<span class="number">0</span>, startPartitionId)</span><br><span class="line"> <span class="keyword">var</span> nextReplicaShift = <span class="keyword">if</span> (fixedStartIndex >= <span class="number">0</span>) fixedStartIndex <span class="keyword">else</span> rand.nextInt(arrangedBrokerList.size)</span><br><span class="line"> <span class="keyword">for</span> (_ <- <span class="number">0</span> until nPartitions) {</span><br><span class="line"> <span class="keyword">if</span> (currentPartitionId > <span class="number">0</span> && (currentPartitionId % arrangedBrokerList.size == <span class="number">0</span>))</span><br><span class="line"> nextReplicaShift += <span class="number">1</span></span><br><span class="line"> <span class="keyword">val</span> firstReplicaIndex = (currentPartitionId + startIndex) % arrangedBrokerList.size</span><br><span class="line"> <span class="keyword">val</span> leader = arrangedBrokerList(firstReplicaIndex)</span><br><span class="line"> <span class="keyword">val</span> replicaBuffer = mutable.<span class="type">ArrayBuffer</span>(leader)</span><br><span class="line"> <span class="keyword">val</span> racksWithReplicas = mutable.<span class="type">Set</span>(brokerRackMap(leader))</span><br><span class="line"> <span class="keyword">val</span> brokersWithReplicas = mutable.<span class="type">Set</span>(leader)</span><br><span class="line"> <span class="keyword">var</span> k = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span> (_ <- <span class="number">0</span> until replicationFactor - <span class="number">1</span>) {</span><br><span class="line"> <span class="keyword">var</span> done = <span class="literal">false</span></span><br><span class="line"> <span class="keyword">while</span> (!done) {</span><br><span class="line"> <span class="keyword">val</span> broker = arrangedBrokerList(replicaIndex(firstReplicaIndex, nextReplicaShift * numRacks, k, arrangedBrokerList.size))</span><br><span class="line"> <span class="keyword">val</span> rack = brokerRackMap(broker)</span><br><span class="line"> <span class="comment">// Skip this broker if</span></span><br><span class="line"> <span class="comment">// 1. there is already a broker in the same rack that has assigned a replica AND there is one or more racks</span></span><br><span class="line"> <span class="comment">// that do not have any replica, or</span></span><br><span class="line"> <span class="comment">// 2. the broker has already assigned a replica AND there is one or more brokers that do not have replica assigned</span></span><br><span class="line"> <span class="keyword">if</span> ((!racksWithReplicas.contains(rack) || racksWithReplicas.size == numRacks)</span><br><span class="line"> && (!brokersWithReplicas.contains(broker) || brokersWithReplicas.size == numBrokers)) {</span><br><span class="line"> replicaBuffer += broker</span><br><span class="line"> racksWithReplicas += rack</span><br><span class="line"> brokersWithReplicas += broker</span><br><span class="line"> done = <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> k += <span class="number">1</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ret.put(currentPartitionId, replicaBuffer)</span><br><span class="line"> currentPartitionId += <span class="number">1</span></span><br><span class="line"> }</span><br><span class="line"> ret</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><h3 id="分区扩容是如何分配的"><a href="#分区扩容是如何分配的" class="headerlink" title="分区扩容是如何分配的"></a>分区扩容是如何分配的</h3><blockquote><p>之前有分析过 <a href="">【kafka源码】TopicCommand之alter源码解析(分区扩容)</a> 我们知道扩容的过程是不会对之前的分区副本有所改动的,但是你新增的分区并不是会按照之前的策略再进行分配; </p></blockquote><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">AdminZKClient.addPartitions</span><br><span class="line"> val proposedAssignmentForNewPartitions = replicaAssignment.getOrElse {</span><br><span class="line"> val startIndex = math.max(0, allBrokers.indexWhere(_.id >= existingAssignmentPartition0.head))</span><br><span class="line"> AdminUtils.assignReplicasToBrokers(allBrokers, partitionsToAdd, existingAssignmentPartition0.size,</span><br><span class="line"> startIndex, existingAssignment.size)</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>看代码, startIndex 获取的是<code>partition-0</code>的第一个副本; allBrokers也是 按照顺序排列好的{0,1,2,3…}； startPartition=当前分区数;</p><p>例如我有个topic 2分区 3副本; 分配情况</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">起始随机startIndex:0currentPartitionId:0;起始随机nextReplicaShift：2;brokerArray:ArrayBuffer(0, 1, 4, 2, 3)</span><br><span class="line">(p-0,ArrayBuffer(0, 2, 3))</span><br><span class="line">(p-1,ArrayBuffer(1, 3, 0))</span><br></pre></td></tr></table></figure><p>我们来计算一下,第3个分区如果同样条件的话应该分配到哪里</p><ol><li>先确定一下分配当时的BrokerList; 按照顺序的关系0->2->3 , 1->3->0 至少 我们可以画出下面的图 <img src="https://img-blog.csdnimg.cn/20210706154343401.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>又根据2->3(2下一个是3) 3->0(3下一个是0)这样的关系可以知道<img src="https://img-blog.csdnimg.cn/20210706154733521.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>又要满足 0->2 和 1->3的跨度要满足一致(当然说的是在同一个遍历范围内<code>currentPartitionId / brokerArray.length 相等</code>)</li><li>又要满足0->1是连续的那么Broker4只能放在1-2之间了;(正常分配的时候，每个分区的第一个副本都是按照brokerList顺序下去的,比如P1(0,2,3),P2(1,3,0), 那么0->1之间肯定是连续的; )</li></ol><p>结果算出来是<code>BrokerList={0,1,4,2,3}</code> 跟我们打印出来的相符合; 那么同样可以计算出来, startIndex=0;(P1的第一个副本id在BrokerList中的索引位置，刚好是索引0，<code>起始随机 nextReplicaShift = 2</code>（P1 0->2 中间隔了1->4>2 ）)</p><p>指定这些我们就可以算出来新增一个分区P3的位置了吧? P3（4,0,1）</p><p>然后执行新增一个分区脚本之后,并不是按照上面分配之后的 {4,0,1} ; 而是如下</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">起始随机startIndex:0 currentPartitionId:2;起始随机nextReplicaShift：0;brokerArray:ArrayBuffer(0, 1, 2, 3, 4)</span><br><span class="line">(p-2,ArrayBuffer(2, 3, 4))</span><br></pre></td></tr></table></figure><h2 id="源码总结"><a href="#源码总结" class="headerlink" title="源码总结"></a>源码总结</h2><h2 id="Q-amp-A"><a href="#Q-amp-A" class="headerlink" title="Q&A"></a>Q&A</h2><h3 id="BrokerList顺序是由什么决定的"><a href="#BrokerList顺序是由什么决定的" class="headerlink" title="BrokerList顺序是由什么决定的"></a>BrokerList顺序是由什么决定的</h3><blockquote><p><img src="https://img-blog.csdnimg.cn/20210706094827190.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"> 最终的地方是 <code>KafkaZkClient.getAllBrokerAndEpochsInCluster</code> ; 从zk中获取Broker列表之后,虽然sort排序了,但是后面又放到一个Map里面,map并不具有顺序性质; 所以只要是<code>Controller</code>有变更之后,都会调用这个接口重新获取的Broker列表,并且每次并不一定完全一致，然后发起<code>UPDATEMETA</code>给所有Broker更新列表;</p></blockquote><p>下面是反推调用路径; <img src="https://img-blog.csdnimg.cn/20210706092220199.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><p><img src="https://img-blog.csdnimg.cn/20210706093242454.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><p>定位 发送<code>UPDATEMEDATA</code>请求</p><p><img src="https://img-blog.csdnimg.cn/20210706094510705.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"> <img src="https://img-blog.csdnimg.cn/20210706094742336.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"> <img src="https://img-blog.csdnimg.cn/20210706094827190.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p>]]></content>
<summary type="html"><p>哪個英文字母最酷？ <span class="hide-inline"><button type="button" class="hide-button button--animated" style="background-color: #FF7242;color: #f</summary>
<category term="kafka" scheme="http://example.com/categories/kafka/"/>
<category term="kafka" scheme="http://example.com/tags/kafka/"/>
<category term="大数据" scheme="http://example.com/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="源码" scheme="http://example.com/tags/%E6%BA%90%E7%A0%81/"/>
</entry>
<entry>
<title>【kafka源码】kafka-topics.sh之alter源码解析(分区扩容)</title>
<link href="http://example.com/2021/08/21/alter-topic/"/>
<id>http://example.com/2021/08/21/alter-topic/</id>
<published>2021-08-21T06:03:44.000Z</published>
<updated>2021-08-23T08:23:02.958Z</updated>
<content type="html"><![CDATA[<p>[TOC]</p><h2 id="脚本参数"><a href="#脚本参数" class="headerlink" title="脚本参数"></a>脚本参数</h2><p><code>sh bin/kafka-topic -help</code> 查看更具体参数</p><p>下面只是列出了跟<code> --alter</code> 相关的参数</p><table><thead><tr><th>参数</th><th>描述</th><th>例子</th></tr></thead><tbody><tr><td><code>--bootstrap-server </code> 指定kafka服务</td><td>指定连接到的kafka服务; 如果有这个参数,则 <code>--zookeeper</code>可以不需要</td><td>–bootstrap-server localhost:9092</td></tr><tr><td><code>--replica-assignment </code></td><td>副本分区分配方式;修改topic的时候可以自己指定副本分配情况;</td><td><code>--replica-assignment id0:id1:id2,id3:id4:id5,id6:id7:id8 </code>；其中，“id0:id1:id2,id3:id4:id5,id6:id7:id8”表示Topic TopicName一共有3个Partition（以“,”分隔），每个Partition均有3个Replica（以“:”分隔），Topic Partition Replica与Kafka Broker之间的对应关系如下:<img src="https://img-blog.csdnimg.cn/20210617140207438.png" alt="在这里插入图片描述"></td></tr><tr><td><code>--topic</code></td><td></td><td></td></tr><tr><td><code>--partitions</code></td><td>扩展到新的分区数</td><td></td></tr></tbody></table><h2 id="Alert-Topic脚本"><a href="#Alert-Topic脚本" class="headerlink" title="Alert Topic脚本"></a>Alert Topic脚本</h2><h2 id="分区扩容"><a href="#分区扩容" class="headerlink" title="分区扩容"></a>分区扩容</h2><p><strong>zk方式(不推荐)</strong></p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 --partitions 2</span><br></pre></td></tr></table></figure><p><strong>kafka版本 >= 2.2 支持下面方式（推荐）</strong><br><strong>单个Topic扩容</strong></p><blockquote><p><code>bin/kafka-topics.sh --bootstrap-server broker_host:port --alter --topic test_create_topic1 --partitions 4</code></p></blockquote><p><strong>批量扩容</strong> (将所有正则表达式匹配到的Topic分区扩容到4个)</p><blockquote><p><code>sh bin/kafka-topics.sh --topic ".*?" --bootstrap-server 172.23.248.85:9092 --alter --partitions 4</code></p><p><code>".*?"</code> 正则表达式的意思是匹配所有; 您可按需匹配</p></blockquote><p><strong>PS:</strong> 当某个Topic的分区少于指定的分区数时候,他会抛出异常;但是不会影响其他Topic正常进行;</p><hr><p>相关可选参数</p><table><thead><tr><th>参数</th><th>描述</th><th>例子</th></tr></thead><tbody><tr><td><code>--replica-assignment </code></td><td>副本分区分配方式;创建topic的时候可以自己指定副本分配情况;</td><td><code>--replica-assignment</code> BrokerId-0:BrokerId-1:BrokerId-2,BrokerId-1:BrokerId-2:BrokerId-0,BrokerId-2:BrokerId-1:BrokerId-0 ; 这个意思是有三个分区和三个副本,对应分配的Broker; 逗号隔开标识分区;冒号隔开表示副本</td></tr></tbody></table><p><strong>PS: 虽然这里配置的是全部的分区副本分配配置,但是真正生效的是新增的分区;</strong><br>比如: 以前3分区1副本是这样的</p><table><thead><tr><th>Broker-1</th><th>Broker-2</th><th>Broker-3</th><th>Broker-4</th></tr></thead><tbody><tr><td>0</td><td>1</td><td>2</td><td></td></tr><tr><td>现在新增一个分区,<code>--replica-assignment</code> 2,1,3,4 ; 看这个意思好像是把0，1号分区互相换个Broker</td><td></td><td></td><td></td></tr><tr><td>Broker-1</td><td>Broker-2</td><td>Broker-3</td><td>Broker-4</td></tr><tr><td>–</td><td>–</td><td>–</td><td>–</td></tr><tr><td>1</td><td>0</td><td>2</td><td>3</td></tr><tr><td>但是实际上不会这样做,Controller在处理的时候会把前面3个截掉; 只取新增的分区分配方式,原来的还是不会变</td><td></td><td></td><td></td></tr><tr><td>Broker-1</td><td>Broker-2</td><td>Broker-3</td><td>Broker-4</td></tr><tr><td>–</td><td>–</td><td>–</td><td>–</td></tr><tr><td>0</td><td>1</td><td>2</td><td>3</td></tr></tbody></table><h2 id="源码解析"><a href="#源码解析" class="headerlink" title="源码解析"></a>源码解析</h2><blockquote><p><font color=red>如果觉得源码解析过程比较枯燥乏味,可以直接如果 <strong>源码总结及其后面部分</strong></font></p></blockquote><p>因为在 <a href="">【kafka源码】TopicCommand之创建Topic源码解析</a> 里面分析的比较详细; 故本文就着重点分析了;</p><h3 id="1-TopicCommand-alterTopic"><a href="#1-TopicCommand-alterTopic" class="headerlink" title="1. TopicCommand.alterTopic"></a>1. <code>TopicCommand.alterTopic</code></h3><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">alterTopic</span></span>(opts: <span class="type">TopicCommandOptions</span>): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="keyword">val</span> topic = <span class="keyword">new</span> <span class="type">CommandTopicPartition</span>(opts)</span><br><span class="line"> <span class="keyword">val</span> topics = getTopics(opts.topic, opts.excludeInternalTopics)</span><br><span class="line"> <span class="comment">//校验Topic是否存在</span></span><br><span class="line"> ensureTopicExists(topics, opts.topic)</span><br><span class="line"> <span class="comment">//获取一下该topic的一些基本信息</span></span><br><span class="line"> <span class="keyword">val</span> topicsInfo = adminClient.describeTopics(topics.asJavaCollection).values()</span><br><span class="line"> adminClient.createPartitions(topics.map {topicName =></span><br><span class="line"> <span class="comment">//判断是否有参数 replica-assignment 指定分区分配方式</span></span><br><span class="line"> <span class="keyword">if</span> (topic.hasReplicaAssignment) {</span><br><span class="line"> <span class="keyword">val</span> startPartitionId = topicsInfo.get(topicName).get().partitions().size()</span><br><span class="line"> <span class="keyword">val</span> newAssignment = {</span><br><span class="line"> <span class="keyword">val</span> replicaMap = topic.replicaAssignment.get.drop(startPartitionId)</span><br><span class="line"> <span class="keyword">new</span> util.<span class="type">ArrayList</span>(replicaMap.map(p => p._2.asJava).asJavaCollection).asInstanceOf[util.<span class="type">List</span>[util.<span class="type">List</span>[<span class="type">Integer</span>]]]</span><br><span class="line"> }</span><br><span class="line"> topicName -> <span class="type">NewPartitions</span>.increaseTo(topic.partitions.get, newAssignment)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> </span><br><span class="line"> topicName -> <span class="type">NewPartitions</span>.increaseTo(topic.partitions.get)</span><br><span class="line"> }}.toMap.asJava).all().get()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ol><li>校验Topic是否存在</li><li>如果设置了<code>--replica-assignment </code>参数, 则会算出新增的分区数的分配; 这个并不会修改原本已经分配好的分区结构.从源码就可以看出来,假如我之前的分配方式是3,3,3(3分区一个副本都在BrokerId-3上)现在我传入的参数是: <code>3,3,3,3</code>(多出来一个分区),这个时候会把原有的给截取掉;只传入3，(表示在Broker3新增一个分区)<img src="https://img-blog.csdnimg.cn/20210617142452499.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>如果没有传入参数<code>--replica-assignment</code>，则后面会用默认分配策略分配</li></ol><h4 id="客户端发起请求createPartitions"><a href="#客户端发起请求createPartitions" class="headerlink" title="客户端发起请求createPartitions"></a>客户端发起请求createPartitions</h4><p><code>KafkaAdminClient.createPartitions</code> 省略部分代码</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> CreatePartitionsResult <span class="title">createPartitions</span><span class="params">(Map<String, NewPartitions> newPartitions,</span></span></span><br><span class="line"><span class="params"><span class="function"> <span class="keyword">final</span> CreatePartitionsOptions options)</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> Map<String, KafkaFutureImpl<Void>> futures = <span class="keyword">new</span> HashMap<>(newPartitions.size());</span><br><span class="line"> <span class="keyword">for</span> (String topic : newPartitions.keySet()) {</span><br><span class="line"> futures.put(topic, <span class="keyword">new</span> KafkaFutureImpl<>());</span><br><span class="line"> }</span><br><span class="line">runnable.call(<span class="keyword">new</span> Call(<span class="string">"createPartitions"</span>, calcDeadlineMs(now, options.timeoutMs()),</span><br><span class="line"> <span class="keyword">new</span> ControllerNodeProvider()) {</span><br><span class="line"> <span class="comment">//省略部分代码</span></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">handleFailure</span><span class="params">(Throwable throwable)</span> </span>{</span><br><span class="line"> completeAllExceptionally(futures.values(), throwable);</span><br><span class="line"> }</span><br><span class="line"> }, now);</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> CreatePartitionsResult(<span class="keyword">new</span> HashMap<>(futures));</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><ol><li>从源码中可以看到向<code>ControllerNodeProvider</code> 发起来<code>createPartitions</code>请求</li></ol><h3 id="2-Controller角色的服务端接受createPartitions请求处理逻辑"><a href="#2-Controller角色的服务端接受createPartitions请求处理逻辑" class="headerlink" title="2. Controller角色的服务端接受createPartitions请求处理逻辑"></a>2. Controller角色的服务端接受createPartitions请求处理逻辑</h3><blockquote><p><code>KafkaApis.handleCreatePartitionsRequest</code></p></blockquote><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">handleCreatePartitionsRequest</span></span>(request: <span class="type">RequestChannel</span>.<span class="type">Request</span>): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="keyword">val</span> createPartitionsRequest = request.body[<span class="type">CreatePartitionsRequest</span>]</span><br><span class="line"></span><br><span class="line"> <span class="comment">//部分代码省略..</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">//如果当前不是Controller角色直接抛出异常</span></span><br><span class="line"> <span class="keyword">if</span> (!controller.isActive) {</span><br><span class="line"> <span class="keyword">val</span> result = createPartitionsRequest.data.topics.asScala.map { topic =></span><br><span class="line"> (topic.name, <span class="keyword">new</span> <span class="type">ApiError</span>(<span class="type">Errors</span>.<span class="type">NOT_CONTROLLER</span>, <span class="literal">null</span>))</span><br><span class="line"> }.toMap</span><br><span class="line"> sendResponseCallback(result)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// Special handling to add duplicate topics to the response</span></span><br><span class="line"> <span class="keyword">val</span> topics = createPartitionsRequest.data.topics.asScala</span><br><span class="line"> <span class="keyword">val</span> dupes = topics.groupBy(_.name)</span><br><span class="line"> .filter { _._2.size > <span class="number">1</span> }</span><br><span class="line"> .keySet</span><br><span class="line"> <span class="keyword">val</span> notDuped = topics.filterNot(topic => dupes.contains(topic.name))</span><br><span class="line"> <span class="keyword">val</span> authorizedTopics = filterAuthorized(request, <span class="type">ALTER</span>, <span class="type">TOPIC</span>, notDuped.map(_.name))</span><br><span class="line"> <span class="keyword">val</span> (authorized, unauthorized) = notDuped.partition { topic => authorizedTopics.contains(topic.name) }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> (queuedForDeletion, valid) = authorized.partition { topic =></span><br><span class="line"> controller.topicDeletionManager.isTopicQueuedUpForDeletion(topic.name)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> errors = dupes.map(_ -> <span class="keyword">new</span> <span class="type">ApiError</span>(<span class="type">Errors</span>.<span class="type">INVALID_REQUEST</span>, <span class="string">"Duplicate topic in request."</span>)) ++</span><br><span class="line"> unauthorized.map(_.name -> <span class="keyword">new</span> <span class="type">ApiError</span>(<span class="type">Errors</span>.<span class="type">TOPIC_AUTHORIZATION_FAILED</span>, <span class="string">"The topic authorization is failed."</span>)) ++</span><br><span class="line"> queuedForDeletion.map(_.name -> <span class="keyword">new</span> <span class="type">ApiError</span>(<span class="type">Errors</span>.<span class="type">INVALID_TOPIC_EXCEPTION</span>, <span class="string">"The topic is queued for deletion."</span>))</span><br><span class="line"></span><br><span class="line"> adminManager.createPartitions(createPartitionsRequest.data.timeoutMs,</span><br><span class="line"> valid,</span><br><span class="line"> createPartitionsRequest.data.validateOnly,</span><br><span class="line"> request.context.listenerName, result => sendResponseCallback(result ++ errors))</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br></pre></td></tr></table></figure><ol><li><p>检验自身是不是Controller角色,不是的话就抛出异常终止流程</p></li><li><p>鉴权</p></li><li><p>调用<code> adminManager.createPartitions</code><br>3.1 从zk中获取<code>/brokers/ids/</code>Brokers列表的元信息的<br>3.2 从zk获取<code>/brokers/topics/{topicName}</code>已经存在的副本分配方式,并判断是否有正在进行副本重分配的进程在执行,如果有的话就抛出异常结束流程<br>3.3 如果从zk获取<code>/brokers/topics/{topicName}</code>数据不存在则抛出异常 <code>The topic '$topic' does not exist</code><br>3.4 检查修改的分区数是否比原来的分区数大，如果比原来还小或者等于原来分区数则抛出异常结束流程<br>3.5 如果传入的参数<code>--replica-assignment</code> 中有不存在的BrokerId;则抛出异常<code>Unknown broker(s) in replica assignment</code>结束流程<br>3.5 如果传入的<code>--partitions</code>数量 与<code>--replica-assignment</code>中新增的部分数量不匹配则抛出异常<code>Increasing the number of partitions by...</code> 结束流程<br>3.6 调用<code> adminZkClient.addPartitions</code></p></li></ol><h4 id="adminZkClient-addPartitions-添加分区"><a href="#adminZkClient-addPartitions-添加分区" class="headerlink" title=" adminZkClient.addPartitions 添加分区"></a><code> adminZkClient.addPartitions</code> 添加分区</h4><ol><li>校验<code>--partitions</code>数量是否比存在的分区数大,否则异常<code>The number of partitions for a topic can only be increased</code></li><li>如果传入了<code>--replica-assignment</code> ,则对副本进行一些简单的校验</li><li>调用<code>AdminUtils.assignReplicasToBrokers</code>分配副本 ; 这个我们在<a href="">【kafka源码】TopicCommand之创建Topic源码解析</a> 也分析过; 具体请看<a href="">【kafka源码】创建Topic的时候是如何分区和副本的分配规则</a>; 当然这里由于我们是新增的分区,只会将新增的分区进行分配计算</li><li>得到分配规则只后,调用<code>adminZkClient.writeTopicPartitionAssignment</code> 写入</li></ol><h4 id="adminZkClient-writeTopicPartitionAssignment将分区信息写入zk中"><a href="#adminZkClient-writeTopicPartitionAssignment将分区信息写入zk中" class="headerlink" title="adminZkClient.writeTopicPartitionAssignment将分区信息写入zk中"></a>adminZkClient.writeTopicPartitionAssignment将分区信息写入zk中</h4><p><img src="https://img-blog.csdnimg.cn/20210617154406685.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><p>我们在 <a href="">【kafka源码】TopicCommand之创建Topic源码解析</a>的时候也分析过这段代码,但是那个时候调用的是<code>zkClient.createTopicAssignment</code> 创建接口<br>这里我们是调用<code> zkClient.setTopicAssignment</code> 写入接口, 写入当然会覆盖掉原有的信息,所以写入的时候会把原来分区信息获取到,重新写入;</p><ol><li>获取Topic原有分区副本分配信息</li><li>将原有的和现在要添加的组装成一个数据对象写入到zk节点<code>/brokers/topics/{topicName}</code>中</li></ol><h3 id="3-Controller监控节点-brokers-topics-topicName-真正在Broker上将分区写入磁盘"><a href="#3-Controller监控节点-brokers-topics-topicName-真正在Broker上将分区写入磁盘" class="headerlink" title="3. Controller监控节点/brokers/topics/{topicName} ,真正在Broker上将分区写入磁盘"></a>3. Controller监控节点<code>/brokers/topics/{topicName}</code> ,真正在Broker上将分区写入磁盘</h3><p>监听到节点信息变更之后调用下面的接口;<br><code>KafkaController.processPartitionModifications</code></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">processPartitionModifications</span></span>(topic: <span class="type">String</span>): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">restorePartitionReplicaAssignment</span></span>(</span><br><span class="line"> topic: <span class="type">String</span>,</span><br><span class="line"> newPartitionReplicaAssignment: <span class="type">Map</span>[<span class="type">TopicPartition</span>, <span class="type">ReplicaAssignment</span>]</span><br><span class="line"> ): <span class="type">Unit</span> = {</span><br><span class="line"> info(<span class="string">"Restoring the partition replica assignment for topic %s"</span>.format(topic))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> existingPartitions = zkClient.getChildren(<span class="type">TopicPartitionsZNode</span>.path(topic))</span><br><span class="line"> <span class="keyword">val</span> existingPartitionReplicaAssignment = newPartitionReplicaAssignment</span><br><span class="line"> .filter(p => existingPartitions.contains(p._1.partition.toString))</span><br><span class="line"> .map { <span class="keyword">case</span> (tp, _) =></span><br><span class="line"> tp -> controllerContext.partitionFullReplicaAssignment(tp)</span><br><span class="line"> }.toMap</span><br><span class="line"></span><br><span class="line"> zkClient.setTopicAssignment(topic,</span><br><span class="line"> existingPartitionReplicaAssignment,</span><br><span class="line"> controllerContext.epochZkVersion)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (!isActive) <span class="keyword">return</span></span><br><span class="line"> <span class="keyword">val</span> partitionReplicaAssignment = zkClient.getFullReplicaAssignmentForTopics(immutable.<span class="type">Set</span>(topic))</span><br><span class="line"> <span class="keyword">val</span> partitionsToBeAdded = partitionReplicaAssignment.filter { <span class="keyword">case</span> (topicPartition, _) =></span><br><span class="line"> controllerContext.partitionReplicaAssignment(topicPartition).isEmpty</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (topicDeletionManager.isTopicQueuedUpForDeletion(topic)) {</span><br><span class="line"> <span class="keyword">if</span> (partitionsToBeAdded.nonEmpty) {</span><br><span class="line"> warn(<span class="string">"Skipping adding partitions %s for topic %s since it is currently being deleted"</span></span><br><span class="line"> .format(partitionsToBeAdded.map(_._1.partition).mkString(<span class="string">","</span>), topic))</span><br><span class="line"></span><br><span class="line"> restorePartitionReplicaAssignment(topic, partitionReplicaAssignment)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// This can happen if existing partition replica assignment are restored to prevent increasing partition count during topic deletion</span></span><br><span class="line"> info(<span class="string">"Ignoring partition change during topic deletion as no new partitions are added"</span>)</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (partitionsToBeAdded.nonEmpty) {</span><br><span class="line"> info(<span class="string">s"New partitions to be added <span class="subst">$partitionsToBeAdded</span>"</span>)</span><br><span class="line"> partitionsToBeAdded.foreach { <span class="keyword">case</span> (topicPartition, assignedReplicas) =></span><br><span class="line"> controllerContext.updatePartitionFullReplicaAssignment(topicPartition, assignedReplicas)</span><br><span class="line"> }</span><br><span class="line"> onNewPartitionCreation(partitionsToBeAdded.keySet)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br></pre></td></tr></table></figure><ol><li>判断是否Controller,不是则直接结束流程</li><li>获取<code>/brokers/topics/{topicName}</code> 节点信息, 然后再对比一下当前该节点的分区分配信息; 看看有没有是新增的分区; 如果是新增的分区这个时候是还没有<code>/brokers/topics/{topicName}/partitions/{分区号}/state</code> ; </li><li>如果当前的TOPIC正在被删除中,那么就没有必要执行扩分区了</li><li>将新增加的分区信息加载到内存中</li><li>调用接口<code>KafkaController.onNewPartitionCreation</code></li></ol><h4 id="KafkaController-onNewPartitionCreation-新增分区"><a href="#KafkaController-onNewPartitionCreation-新增分区" class="headerlink" title="KafkaController.onNewPartitionCreation 新增分区"></a>KafkaController.onNewPartitionCreation 新增分区</h4><p>从这里开始 , 后面的流程就跟创建Topic的对应流程一样了; </p><blockquote><p>该接口主要是针对新增分区和副本的一些状态流转过程; 在<a href="">【kafka源码】TopicCommand之创建Topic源码解析</a> 也同样分析过</p></blockquote><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * This callback is invoked by the topic change callback with the list of failed brokers as input.</span></span><br><span class="line"><span class="comment"> * It does the following -</span></span><br><span class="line"><span class="comment"> * 1. Move the newly created partitions to the NewPartition state</span></span><br><span class="line"><span class="comment"> * 2. Move the newly created partitions from NewPartition->OnlinePartition state</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">onNewPartitionCreation</span></span>(newPartitions: <span class="type">Set</span>[<span class="type">TopicPartition</span>]): <span class="type">Unit</span> = {</span><br><span class="line"> info(<span class="string">s"New partition creation callback for <span class="subst">${newPartitions.mkString(",")}</span>"</span>)</span><br><span class="line"> partitionStateMachine.handleStateChanges(newPartitions.toSeq, <span class="type">NewPartition</span>)</span><br><span class="line"> replicaStateMachine.handleStateChanges(controllerContext.replicasForPartition(newPartitions).toSeq, <span class="type">NewReplica</span>)</span><br><span class="line"> partitionStateMachine.handleStateChanges(</span><br><span class="line"> newPartitions.toSeq,</span><br><span class="line"> <span class="type">OnlinePartition</span>,</span><br><span class="line"> <span class="type">Some</span>(<span class="type">OfflinePartitionLeaderElectionStrategy</span>(<span class="literal">false</span>))</span><br><span class="line"> )</span><br><span class="line"> replicaStateMachine.handleStateChanges(controllerContext.replicasForPartition(newPartitions).toSeq, <span class="type">OnlineReplica</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ol><li>将待创建的分区状态流转为<code>NewPartition</code>;<br><img src="https://img-blog.csdnimg.cn/20210616180239988.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>将待创建的副本 状态流转为<code>NewReplica</code>;<br><img src="https://img-blog.csdnimg.cn/20210616180940961.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>将分区状态从刚刚的<code>NewPartition</code>流转为<code>OnlinePartition</code></li><li>获取<code>leaderIsrAndControllerEpochs</code>; Leader为副本的第一个;<pre><code>1. 向zk中写入`/brokers/topics/{topicName}/partitions/` 持久节点; 无数据2. 向zk中写入`/brokers/topics/{topicName}/partitions/{分区号}` 持久节点; 无数据3. 向zk中写入`/brokers/topics/{topicName}/partitions/{分区号}/state` 持久节点; 数据为`leaderIsrAndControllerEpoch`</code></pre><ol start="4"><li>向副本所属Broker发送<a href=""><code>leaderAndIsrRequest</code></a>请求</li><li>向所有Broker发送<a href=""><code>UPDATE_METADATA</code> </a>请求</li></ol></li><li>将副本状态从刚刚的<code>NewReplica</code>流转为<code>OnlineReplica</code> ,更新下内存</li></ol><p>关于分区状态机和副本状态机详情请看<a href="TODO">【kafka源码】Controller中的状态机</a></p><h3 id="4-Broker收到LeaderAndIsrRequest-创建本地Log"><a href="#4-Broker收到LeaderAndIsrRequest-创建本地Log" class="headerlink" title="4. Broker收到LeaderAndIsrRequest 创建本地Log"></a>4. Broker收到LeaderAndIsrRequest 创建本地Log</h3><blockquote><p>上面步骤中有说到向副本所属Broker发送<a href=""><code>leaderAndIsrRequest</code></a>请求,那么这里做了什么呢<br>其实主要做的是 创建本地Log</p><p>代码太多,这里我们直接定位到只跟创建Topic相关的关键代码来分析<br><code>KafkaApis.handleLeaderAndIsrRequest->replicaManager.becomeLeaderOrFollower->ReplicaManager.makeLeaders...LogManager.getOrCreateLog</code></p></blockquote><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 如果日志已经存在，只返回现有日志的副本否则如果 isNew=true 或者如果没有离线日志目录，则为给定的主题和给定的分区创建日志 否则抛出 KafkaStorageException</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getOrCreateLog</span></span>(topicPartition: <span class="type">TopicPartition</span>, config: <span class="type">LogConfig</span>, isNew: <span class="type">Boolean</span> = <span class="literal">false</span>, isFuture: <span class="type">Boolean</span> = <span class="literal">false</span>): <span class="type">Log</span> = {</span><br><span class="line"> logCreationOrDeletionLock synchronized {</span><br><span class="line"> getLog(topicPartition, isFuture).getOrElse {</span><br><span class="line"> <span class="comment">// create the log if it has not already been created in another thread</span></span><br><span class="line"> <span class="keyword">if</span> (!isNew && offlineLogDirs.nonEmpty)</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">KafkaStorageException</span>(<span class="string">s"Can not create log for <span class="subst">$topicPartition</span> because log directories <span class="subst">${offlineLogDirs.mkString(",")}</span> are offline"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> logDirs: <span class="type">List</span>[<span class="type">File</span>] = {</span><br><span class="line"> <span class="keyword">val</span> preferredLogDir = preferredLogDirs.get(topicPartition)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (isFuture) {</span><br><span class="line"> <span class="keyword">if</span> (preferredLogDir == <span class="literal">null</span>)</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">IllegalStateException</span>(<span class="string">s"Can not create the future log for <span class="subst">$topicPartition</span> without having a preferred log directory"</span>)</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">if</span> (getLog(topicPartition).get.dir.getParent == preferredLogDir)</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">IllegalStateException</span>(<span class="string">s"Can not create the future log for <span class="subst">$topicPartition</span> in the current log directory of this partition"</span>)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (preferredLogDir != <span class="literal">null</span>)</span><br><span class="line"> <span class="type">List</span>(<span class="keyword">new</span> <span class="type">File</span>(preferredLogDir))</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> nextLogDirs()</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> logDirName = {</span><br><span class="line"> <span class="keyword">if</span> (isFuture)</span><br><span class="line"> <span class="type">Log</span>.logFutureDirName(topicPartition)</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="type">Log</span>.logDirName(topicPartition)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> logDir = logDirs</span><br><span class="line"> .toStream <span class="comment">// to prevent actually mapping the whole list, lazy map</span></span><br><span class="line"> .map(createLogDirectory(_, logDirName))</span><br><span class="line"> .find(_.isSuccess)</span><br><span class="line"> .getOrElse(<span class="type">Failure</span>(<span class="keyword">new</span> <span class="type">KafkaStorageException</span>(<span class="string">"No log directories available. Tried "</span> + logDirs.map(_.getAbsolutePath).mkString(<span class="string">", "</span>))))</span><br><span class="line"> .get <span class="comment">// If Failure, will throw</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> log = <span class="type">Log</span>(</span><br><span class="line"> dir = logDir,</span><br><span class="line"> config = config,</span><br><span class="line"> logStartOffset = <span class="number">0</span>L,</span><br><span class="line"> recoveryPoint = <span class="number">0</span>L,</span><br><span class="line"> maxProducerIdExpirationMs = maxPidExpirationMs,</span><br><span class="line"> producerIdExpirationCheckIntervalMs = <span class="type">LogManager</span>.<span class="type">ProducerIdExpirationCheckIntervalMs</span>,</span><br><span class="line"> scheduler = scheduler,</span><br><span class="line"> time = time,</span><br><span class="line"> brokerTopicStats = brokerTopicStats,</span><br><span class="line"> logDirFailureChannel = logDirFailureChannel)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (isFuture)</span><br><span class="line"> futureLogs.put(topicPartition, log)</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> currentLogs.put(topicPartition, log)</span><br><span class="line"></span><br><span class="line"> info(<span class="string">s"Created log for partition <span class="subst">$topicPartition</span> in <span class="subst">$logDir</span> with properties "</span> + <span class="string">s"{<span class="subst">${config.originals.asScala.mkString(", ")}</span>}."</span>)</span><br><span class="line"> <span class="comment">// Remove the preferred log dir since it has already been satisfied</span></span><br><span class="line"> preferredLogDirs.remove(topicPartition)</span><br><span class="line"></span><br><span class="line"> log</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ol><li>如果日志已经存在，只返回现有日志的副本否则如果 isNew=true 或者如果没有离线日志目录，则为给定的主题和给定的分区创建日志 否则抛出<code> KafkaStorageException</code></li></ol><p>详细请看 <a href="">【kafka源码】LeaderAndIsrRequest请求</a></p><h2 id="源码总结"><a href="#源码总结" class="headerlink" title="源码总结"></a>源码总结</h2><p>看图说话<br><img src="https://img-blog.csdnimg.cn/2021061718435568.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><h2 id="Q-amp-A"><a href="#Q-amp-A" class="headerlink" title="Q&A"></a>Q&A</h2><h3 id="如果自定义的分配Broker不存在会怎么样"><a href="#如果自定义的分配Broker不存在会怎么样" class="headerlink" title="如果自定义的分配Broker不存在会怎么样"></a>如果自定义的分配Broker不存在会怎么样</h3><blockquote><p>会抛出异常<code>Unknown broker(s) in replica assignment</code>, 因为在执行的时候会去zk获取当前的在线Broker列表，然后判断是否在线;</p></blockquote><h3 id="如果设置的分区数不等于-replica-assignment中新增的数目会怎么样"><a href="#如果设置的分区数不等于-replica-assignment中新增的数目会怎么样" class="headerlink" title="如果设置的分区数不等于 --replica-assignment中新增的数目会怎么样"></a>如果设置的分区数不等于 <code>--replica-assignment</code>中新增的数目会怎么样</h3><blockquote><p>会抛出异常<code>Increasing the number of partitions by..</code>结束流程</p></blockquote><h3 id="如果写入-brokers-topics-topicName-之后-Controller监听到请求正好挂掉怎么办"><a href="#如果写入-brokers-topics-topicName-之后-Controller监听到请求正好挂掉怎么办" class="headerlink" title="如果写入/brokers/topics/{topicName}之后 Controller监听到请求正好挂掉怎么办"></a>如果写入<code>/brokers/topics/{topicName}</code>之后 Controller监听到请求正好挂掉怎么办</h3><blockquote><p>Controller挂掉会发生重新选举,选举成功之后, 检查到<code>/brokers/topics/{topicName}</code>之后发现没有生成对应的分区,会自动执行接下来的流程;</p></blockquote><h3 id="如果我手动在zk中写入节点-brokers-topics-topicName-partitions-分区号-state-会怎么样"><a href="#如果我手动在zk中写入节点-brokers-topics-topicName-partitions-分区号-state-会怎么样" class="headerlink" title="如果我手动在zk中写入节点/brokers/topics/{topicName}/partitions/{分区号}/state 会怎么样"></a>如果我手动在zk中写入节点<code>/brokers/topics/{topicName}/partitions/{分区号}/state</code> 会怎么样</h3><blockquote><p>Controller并没有监听这个节点,所以不会有变化; 但是当Controller发生重新选举的时候,<br><strong>被删除的节点会被重新添加回来;</strong><br>但是<strong>写入的节点 就不会被删除了</strong>;写入的节点信息会被保存在Controller内存中;<br>同样这会影响到分区扩容</p><hr><p>例子🌰：<br>当前分区3个,副本一个,手贱在zk上添加了一个节点如下图:<br><img src="https://img-blog.csdnimg.cn/20210617175311911.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br>这个时候我想扩展一个分区; 然后执行了脚本, 虽然<code>/brokers/topics/test_create_topic3</code>节点数据变; 但是Broker真正在<code>LeaderAndIsrRequest</code>请求里面没有执行创建本地Log文件; 这是因为源码读取到zk下面partitions的节点数量和新增之后的节点数量没有变更,那么它就认为本次请求没有变更就不会执行创建本地Log文件了;<br>如果判断有变更,还是会去创建的;<br>手贱zk写入N个partition节点 + 扩充N个分区 = Log文件不会被创建<br>手贱zk写入N个partition节点 + 扩充>N个分区 = 正常扩容</p></blockquote><h3 id="如果直接修改节点-brokers-topics-topicName-中的配置会怎么样"><a href="#如果直接修改节点-brokers-topics-topicName-中的配置会怎么样" class="headerlink" title="如果直接修改节点/brokers/topics/{topicName}中的配置会怎么样"></a>如果直接修改节点/brokers/topics/{topicName}中的配置会怎么样</h3><blockquote><p>如果该节点信息是<code>{"version":2,"partitions":{"2":[1],"1":[1],"0":[1]},"adding_replicas":{},"removing_replicas":{}}</code> 看数据,说明3个分区1个副本都在Broker-1上;<br>我在zk上修改成<code>{"version":2,"partitions":{"2":[2],"1":[1],"0":[0]},"adding_replicas":{},"removing_replicas":{}}</code><br>想将分区分配到 Broker-0，Broker-1，Broker-2上<br>TODO。。。</p></blockquote><hr>]]></content>
<summary type="html">描述描述描述描述描述描述描述描述描述</summary>
<category term="kafka" scheme="http://example.com/categories/kafka/"/>
<category term="kafka" scheme="http://example.com/tags/kafka/"/>
<category term="大数据" scheme="http://example.com/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="源码" scheme="http://example.com/tags/%E6%BA%90%E7%A0%81/"/>
</entry>
<entry>
<title>【kafka源码】kafka-topics.sh之删除Topic源码解析</title>
<link href="http://example.com/2021/08/21/delete-topic/"/>
<id>http://example.com/2021/08/21/delete-topic/</id>
<published>2021-08-21T06:00:44.000Z</published>
<updated>2021-08-21T06:02:35.762Z</updated>
<content type="html"><![CDATA[<p>阅读本文之前 你可以先思考一下以下几个问题，然后再带着问题去阅读本文</p><ul><li>什么时候在/admin/delete_topics写入节点的？</li><li>什么时候真正执行删除Topic磁盘日志？</li><li>Controller通知Brokers 执行StopReplica是通知所有的Broker还是只通知跟被删除Topic有关联的Broker？</li><li>删除过程有Broker不在线 或者执行失败怎么办</li><li>在重新分配的过程中,如果执行删除操作会怎么样</li><li>如果正在删除中发生了分区副本重分配的操作怎么办?</li><li>如果直接删除ZK上的<code>/brokers/topics/{topicName}</code>节点会怎样</li></ul><h2 id="删除Topic命令"><a href="#删除Topic命令" class="headerlink" title="删除Topic命令"></a>删除Topic命令</h2><blockquote><p>bin/kafka-topics.sh –bootstrap-server localhost:9092 –delete –topic test</p></blockquote><p>支持正则表达式匹配Topic来进行删除,只需要将topic 用双引号包裹起来<br>例如: 删除以<code>create_topic_byhand_zk</code>为开头的topic;</p><blockquote><blockquote><p>bin/kafka-topics.sh –bootstrap-server localhost:9092 –delete –topic “create_topic_byhand_zk.*”<br><code>.</code>表示任意匹配除换行符 \n 之外的任何单字符。要匹配 . ，请使用 . 。<br><code>·*·</code>：匹配前面的子表达式零次或多次。要匹配 * 字符，请使用 *。<br><code>.*</code> : 任意字符</p></blockquote></blockquote><p><strong>删除任意Topic (慎用)</strong></p><blockquote><p> bin/kafka-topics.sh –bootstrap-server localhost:9092 –delete –topic “.*?” </p><p> 更多的用法请<a href="https://www.runoob.com/regexp/regexp-syntax.html">参考正则表达式</a></p></blockquote><h2 id="相关配置"><a href="#相关配置" class="headerlink" title="相关配置"></a>相关配置</h2><table><thead><tr><th>配置</th><th>描述</th><th>默认</th></tr></thead><tbody><tr><td>file.delete.delay.ms</td><td>topic删除被标记为–delete文件之后延迟多长时间删除正在的Log文件</td><td>60000</td></tr><tr><td>delete.topic.enable</td><td>是否能够删除topic</td><td>true</td></tr></tbody></table><h2 id="源码解析"><a href="#源码解析" class="headerlink" title="源码解析"></a>源码解析</h2><p><font color="red">如果觉得阅读源码解析太枯燥,请直接看 <strong>源码总结及其后面部分</strong></font></p><h3 id="1-客户端发起删除Topic的请求"><a href="#1-客户端发起删除Topic的请求" class="headerlink" title="1. 客户端发起删除Topic的请求"></a>1. 客户端发起删除Topic的请求</h3><p>在<a href="">【kafka源码】TopicCommand之创建Topic源码解析</a> 里面已经分析过了整个请求流程; 所以这里就不再详细的分析请求的过程了,直接看重点;<br><img src="https://img-blog.csdnimg.cn/20210613133230944.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br><strong>向Controller发起 <code>deleteTopics</code>请求</strong></p><h3 id="2-Controller处理deleteTopics的请求"><a href="#2-Controller处理deleteTopics的请求" class="headerlink" title="2. Controller处理deleteTopics的请求"></a>2. Controller处理deleteTopics的请求</h3><p><code>KafkaApis.handle</code><br><code>AdminManager.deleteTopics</code></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Delete topics and wait until the topics have been completely deleted.</span></span><br><span class="line"><span class="comment"> * The callback function will be triggered either when timeout, error or the topics are deleted.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deleteTopics</span></span>(timeout: <span class="type">Int</span>,</span><br><span class="line"> topics: <span class="type">Set</span>[<span class="type">String</span>],</span><br><span class="line"> responseCallback: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">Errors</span>] => <span class="type">Unit</span>): <span class="type">Unit</span> = {</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 1. map over topics calling the asynchronous delete</span></span><br><span class="line"> <span class="keyword">val</span> metadata = topics.map { topic =></span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="comment">// zk中写入数据 标记要被删除的topic /admin/delete_topics/Topic名称</span></span><br><span class="line"> adminZkClient.deleteTopic(topic)</span><br><span class="line"> <span class="type">DeleteTopicMetadata</span>(topic, <span class="type">Errors</span>.<span class="type">NONE</span>)</span><br><span class="line"> } <span class="keyword">catch</span> {</span><br><span class="line"> <span class="keyword">case</span> _: <span class="type">TopicAlreadyMarkedForDeletionException</span> =></span><br><span class="line"> <span class="comment">// swallow the exception, and still track deletion allowing multiple calls to wait for deletion</span></span><br><span class="line"> <span class="type">DeleteTopicMetadata</span>(topic, <span class="type">Errors</span>.<span class="type">NONE</span>)</span><br><span class="line"> <span class="keyword">case</span> e: <span class="type">Throwable</span> =></span><br><span class="line"> error(<span class="string">s"Error processing delete topic request for topic <span class="subst">$topic</span>"</span>, e)</span><br><span class="line"> <span class="type">DeleteTopicMetadata</span>(topic, <span class="type">Errors</span>.forException(e))</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 2. 如果客户端传过来的timeout<=0或者 写入zk数据过程异常了 则执行下面的,直接返回异常</span></span><br><span class="line"> <span class="keyword">if</span> (timeout <= <span class="number">0</span> || !metadata.exists(_.error == <span class="type">Errors</span>.<span class="type">NONE</span>)) {</span><br><span class="line"> <span class="keyword">val</span> results = metadata.map { deleteTopicMetadata =></span><br><span class="line"> <span class="comment">// ignore topics that already have errors</span></span><br><span class="line"> <span class="keyword">if</span> (deleteTopicMetadata.error == <span class="type">Errors</span>.<span class="type">NONE</span>) {</span><br><span class="line"> (deleteTopicMetadata.topic, <span class="type">Errors</span>.<span class="type">REQUEST_TIMED_OUT</span>)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> (deleteTopicMetadata.topic, deleteTopicMetadata.error)</span><br><span class="line"> }</span><br><span class="line"> }.toMap</span><br><span class="line"> responseCallback(results)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// 3. else pass the topics and errors to the delayed operation and set the keys</span></span><br><span class="line"> <span class="keyword">val</span> delayedDelete = <span class="keyword">new</span> <span class="type">DelayedDeleteTopics</span>(timeout, metadata.toSeq, <span class="keyword">this</span>, responseCallback)</span><br><span class="line"> <span class="keyword">val</span> delayedDeleteKeys = topics.map(<span class="keyword">new</span> <span class="type">TopicKey</span>(_)).toSeq</span><br><span class="line"> <span class="comment">// try to complete the request immediately, otherwise put it into the purgatory</span></span><br><span class="line"> topicPurgatory.tryCompleteElseWatch(delayedDelete, delayedDeleteKeys)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br></pre></td></tr></table></figure><ol><li>zk中写入数据topic<code> /admin/delete_topics/Topic名称</code>； 标记要被删除的Topic</li><li>如果客户端传过来的timeout<=0或者 写入zk数据过程异常了 则直接返回异常</li></ol><h3 id="3-Controller监听zk变更-执行删除Topic流程"><a href="#3-Controller监听zk变更-执行删除Topic流程" class="headerlink" title="3. Controller监听zk变更 执行删除Topic流程"></a>3. Controller监听zk变更 执行删除Topic流程</h3><p><code>KafkaController.processTopicDeletion</code></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">processTopicDeletion</span></span>(): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="keyword">if</span> (!isActive) <span class="keyword">return</span></span><br><span class="line"> <span class="keyword">var</span> topicsToBeDeleted = zkClient.getTopicDeletions.toSet</span><br><span class="line"> <span class="keyword">val</span> nonExistentTopics = topicsToBeDeleted -- controllerContext.allTopics</span><br><span class="line"> <span class="keyword">if</span> (nonExistentTopics.nonEmpty) {</span><br><span class="line"> warn(<span class="string">s"Ignoring request to delete non-existing topics <span class="subst">${nonExistentTopics.mkString(",")}</span>"</span>)</span><br><span class="line"> zkClient.deleteTopicDeletions(nonExistentTopics.toSeq, controllerContext.epochZkVersion)</span><br><span class="line"> }</span><br><span class="line"> topicsToBeDeleted --= nonExistentTopics</span><br><span class="line"> <span class="keyword">if</span> (config.deleteTopicEnable) {</span><br><span class="line"> <span class="keyword">if</span> (topicsToBeDeleted.nonEmpty) {</span><br><span class="line"> info(<span class="string">s"Starting topic deletion for topics <span class="subst">${topicsToBeDeleted.mkString(",")}</span>"</span>)</span><br><span class="line"> <span class="comment">// 标记暂时不可删除的Topic</span></span><br><span class="line"> topicsToBeDeleted.foreach { topic =></span><br><span class="line"> <span class="keyword">val</span> partitionReassignmentInProgress =</span><br><span class="line"> controllerContext.partitionsBeingReassigned.map(_.topic).contains(topic)</span><br><span class="line"> <span class="keyword">if</span> (partitionReassignmentInProgress)</span><br><span class="line"> topicDeletionManager.markTopicIneligibleForDeletion(<span class="type">Set</span>(topic),</span><br><span class="line"> reason = <span class="string">"topic reassignment in progress"</span>)</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// add topic to deletion list</span></span><br><span class="line"> topicDeletionManager.enqueueTopicsForDeletion(topicsToBeDeleted)</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// If delete topic is disabled remove entries under zookeeper path : /admin/delete_topics</span></span><br><span class="line"> info(<span class="string">s"Removing <span class="subst">$topicsToBeDeleted</span> since delete topic is disabled"</span>)</span><br><span class="line"> zkClient.deleteTopicDeletions(topicsToBeDeleted.toSeq, controllerContext.epochZkVersion)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br></pre></td></tr></table></figure><ol><li>如果<code>/admin/delete_topics/</code>下面的节点有不存在的Topic,则清理掉</li><li>如果配置了<code>delete.topic.enable=false</code>不可删除Topic的话,则将<code>/admin/delete_topics/</code>下面的节点全部删除,然后流程结束</li><li><code>delete.topic.enable=true</code>; 将主题标记为不符合删除条件,放到<code>topicsIneligibleForDeletion</code>中; 不符合删除条件的是:<strong>Topic分区正在进行分区重分配</strong></li><li>将Topic添加到删除Topic列表<code>topicsToBeDeleted</code>中;</li><li>然后调用<code>TopicDeletionManager.resumeDeletions()</code>方法执行删除操作</li></ol><h4 id="3-1-resumeDeletions-执行删除方法"><a href="#3-1-resumeDeletions-执行删除方法" class="headerlink" title="3.1 resumeDeletions 执行删除方法"></a>3.1 resumeDeletions 执行删除方法</h4><p><code>TopicDeletionManager.resumeDeletions()</code></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"> <span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">resumeDeletions</span></span>(): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="keyword">val</span> topicsQueuedForDeletion = <span class="type">Set</span>.empty[<span class="type">String</span>] ++ controllerContext.topicsToBeDeleted</span><br><span class="line"> <span class="keyword">val</span> topicsEligibleForRetry = mutable.<span class="type">Set</span>.empty[<span class="type">String</span>]</span><br><span class="line"> <span class="keyword">val</span> topicsEligibleForDeletion = mutable.<span class="type">Set</span>.empty[<span class="type">String</span>]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (topicsQueuedForDeletion.nonEmpty)</span><br><span class="line"> topicsQueuedForDeletion.foreach { topic =></span><br><span class="line"> <span class="comment">// if all replicas are marked as deleted successfully, then topic deletion is done</span></span><br><span class="line"> <span class="comment">//如果所有副本都被标记为删除成功了,然后执行删除Topic成功操作; </span></span><br><span class="line"> <span class="keyword">if</span> (controllerContext.areAllReplicasInState(topic, <span class="type">ReplicaDeletionSuccessful</span>)) {</span><br><span class="line"> <span class="comment">// clear up all state for this topic from controller cache and zookeeper</span></span><br><span class="line"> <span class="comment">//执行删除Topic成功之后的操作; </span></span><br><span class="line"> completeDeleteTopic(topic)</span><br><span class="line"> info(<span class="string">s"Deletion of topic <span class="subst">$topic</span> successfully completed"</span>)</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (!controllerContext.isAnyReplicaInState(topic, <span class="type">ReplicaDeletionStarted</span>)) {</span><br><span class="line"> <span class="comment">// if you come here, then no replica is in TopicDeletionStarted and all replicas are not in</span></span><br><span class="line"> <span class="comment">// TopicDeletionSuccessful. That means, that either given topic haven't initiated deletion</span></span><br><span class="line"> <span class="comment">// or there is at least one failed replica (which means topic deletion should be retried).</span></span><br><span class="line"> <span class="keyword">if</span> (controllerContext.isAnyReplicaInState(topic, <span class="type">ReplicaDeletionIneligible</span>)) {</span><br><span class="line"> topicsEligibleForRetry += topic</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Add topic to the eligible set if it is eligible for deletion.</span></span><br><span class="line"> <span class="keyword">if</span> (isTopicEligibleForDeletion(topic)) {</span><br><span class="line"> info(<span class="string">s"Deletion of topic <span class="subst">$topic</span> (re)started"</span>)</span><br><span class="line"> topicsEligibleForDeletion += topic</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// topic deletion retry will be kicked off</span></span><br><span class="line"> <span class="keyword">if</span> (topicsEligibleForRetry.nonEmpty) {</span><br><span class="line"> retryDeletionForIneligibleReplicas(topicsEligibleForRetry)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// topic deletion will be kicked off</span></span><br><span class="line"> <span class="keyword">if</span> (topicsEligibleForDeletion.nonEmpty) {</span><br><span class="line"> <span class="comment">//删除Topic,发送UpdataMetaData请求</span></span><br><span class="line"> onTopicDeletion(topicsEligibleForDeletion)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ol><li><p>重点看看<code>onTopicDeletion</code>方法,标记所有待删除分区;向Brokers发送<code>updateMetadataRequest</code>请求,告知Brokers这个主题正在被删除,并将Leader设置为<code>LeaderAndIsrLeaderDuringDelete</code>；</p><ol><li>将待删除的Topic的所有分区,执行分区状态机的转换 ;当前状态–><code>OfflinePartition</code>-><code>NonExistentPartition</code> ; 这两个状态转换只是在当前Controller内存中更新了一下状态; 关于状态机请看 <a href="">【kafka源码】Controller中的状态机TODO….</a>; </li><li><code>client.sendMetadataUpdate(topics.flatMap(controllerContext.partitionsForTopic))</code> 向待删除Topic分区发送<code>UpdateMetadata</code>请求; 这个时候更新了什么数据呢? <img src="https://img-blog.csdnimg.cn/20210615213621790.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br>看上面图片源码, 发送<code>UpdateMetadata</code>请求的时候把分区的Leader= -2; 表示这个分区正在被删除;那么所有正在被删除的分区就被找到了;拿到这些待删除分区之后干嘛呢？<ol><li>更新一下限流相关信息 </li><li>调用<code>groupCoordinator.handleDeletedPartitions(deletedPartitions)</code>: 清除给定的<code>deletedPartitions</code>的组偏移量以及执行偏移量删除的函数；就是现在该分区不能提供服务啦,不能被消费啦</li></ol></li></ol><p> 详细请看 <a href="">Kafka的元数据更新UpdateMetadata</a></p><ol start="4"><li>调用<code>TopicDeletionManager.onPartitionDeletion</code>接口如下;</li></ol></li></ol><h4 id="3-2-TopicDeletionManager-onPartitionDeletion"><a href="#3-2-TopicDeletionManager-onPartitionDeletion" class="headerlink" title="3.2 TopicDeletionManager.onPartitionDeletion"></a>3.2 TopicDeletionManager.onPartitionDeletion</h4><ol><li>将所有Dead replicas 副本直接移动到<code>ReplicaDeletionIneligible</code>状态，如果某些副本已死，也将相应的主题标记为不适合删除，因为它无论如何都不会成功完成 </li><li>副本状态转换成<code>OfflineReplica</code>; 这个时候会对该Topic的所有副本所在Broker发起<a href=""><code>StopReplicaRequest</code> </a>请求;（参数<code>deletePartitions = false</code>,表示还不执行删除操作）; 以便他们停止向<code>Leader</code>发送<code>fetch</code>请求; 关于状态机请看 <a href="">【kafka源码】Controller中的状态机TODO….</a>; </li><li>副本状态转换成 <code>ReplicaDeletionStarted</code>状态，这个时候会对该Topic的所有副本所在Broker发起<a href=""><code>StopReplicaRequest</code> </a>请求;（参数<code>deletePartitions = true</code>,表示执行删除操作）。这将发送带有 deletePartition=true 的 <a href=""><code>StopReplicaRequest</code> </a>。并将删除相应分区的所有副本中的所有持久数据</li></ol><h3 id="4-Brokers-接受StopReplica请求"><a href="#4-Brokers-接受StopReplica请求" class="headerlink" title="4. Brokers 接受StopReplica请求"></a>4. Brokers 接受StopReplica请求</h3><p>最终调用的是接口<br><code>ReplicaManager.stopReplica</code> ==> <code>LogManager.asyncDelete</code></p><blockquote><p>将给定主题分区“logdir”的目录重命名为“logdir.uuid.delete”，并将其添加到删除队列中<br>例如 :<br><img src="https://img-blog.csdnimg.cn/20210615124118290.png" alt="在这里插入图片描述"></p></blockquote><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">asyncDelete</span></span>(topicPartition: <span class="type">TopicPartition</span>, isFuture: <span class="type">Boolean</span> = <span class="literal">false</span>): <span class="type">Log</span> = {</span><br><span class="line"> <span class="keyword">val</span> removedLog: <span class="type">Log</span> = logCreationOrDeletionLock synchronized {</span><br><span class="line"> <span class="comment">//将待删除的partition在 Logs中删除掉</span></span><br><span class="line"> <span class="keyword">if</span> (isFuture)</span><br><span class="line"> futureLogs.remove(topicPartition)</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> currentLogs.remove(topicPartition)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (removedLog != <span class="literal">null</span>) {</span><br><span class="line"> <span class="comment">//我们需要等到要删除的日志上没有更多的清理任务，然后才能真正删除它。</span></span><br><span class="line"> <span class="keyword">if</span> (cleaner != <span class="literal">null</span> && !isFuture) {</span><br><span class="line"> cleaner.abortCleaning(topicPartition)</span><br><span class="line"> cleaner.updateCheckpoints(removedLog.dir.getParentFile)</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">//重命名topic副本文件夹 命名规则 topic-uuid-delete</span></span><br><span class="line"> removedLog.renameDir(<span class="type">Log</span>.logDeleteDirName(topicPartition))</span><br><span class="line"> checkpointRecoveryOffsetsAndCleanSnapshot(removedLog.dir.getParentFile, <span class="type">ArrayBuffer</span>.empty)</span><br><span class="line"> checkpointLogStartOffsetsInDir(removedLog.dir.getParentFile)</span><br><span class="line"> <span class="comment">//将Log添加到待删除Log队列中,等待删除</span></span><br><span class="line"> addLogToBeDeleted(removedLog)</span><br><span class="line"></span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (offlineLogDirs.nonEmpty) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">KafkaStorageException</span>(<span class="string">s"Failed to delete log for <span class="subst">${if (isFuture) "future" else ""}</span> <span class="subst">$topicPartition</span> because it may be in one of the offline directories <span class="subst">${offlineLogDirs.mkString(",")}</span>"</span>)</span><br><span class="line"> }</span><br><span class="line"> removedLog</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><h4 id="4-1-日志清理定时线程"><a href="#4-1-日志清理定时线程" class="headerlink" title="4.1 日志清理定时线程"></a>4.1 日志清理定时线程</h4><blockquote><p>上面我们知道最终是将待删除的Log添加到了<code>logsToBeDeleted</code>这个队列中; 这个队列就是待删除Log队列，有一个线程 <code>kafka-delete-logs</code>专门来处理的;我们来看看这个线程怎么工作的</p></blockquote><p><code>LogManager.startup</code> 启动的时候 ,启动了一个定时线程</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">scheduler.schedule(<span class="string">"kafka-delete-logs"</span>, <span class="comment">// will be rescheduled after each delete logs with a dynamic period</span></span><br><span class="line"> deleteLogs _,</span><br><span class="line"> delay = <span class="type">InitialTaskDelayMs</span>,</span><br><span class="line"> unit = <span class="type">TimeUnit</span>.<span class="type">MILLISECONDS</span>)</span><br></pre></td></tr></table></figure><p><strong>删除日志的线程</strong></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Delete logs marked for deletion. Delete all logs for which `currentDefaultConfig.fileDeleteDelayMs`</span></span><br><span class="line"><span class="comment"> * has elapsed after the delete was scheduled. Logs for which this interval has not yet elapsed will be</span></span><br><span class="line"><span class="comment"> * considered for deletion in the next iteration of `deleteLogs`. The next iteration will be executed</span></span><br><span class="line"><span class="comment"> * after the remaining time for the first log that is not deleted. If there are no more `logsToBeDeleted`,</span></span><br><span class="line"><span class="comment"> * `deleteLogs` will be executed after `currentDefaultConfig.fileDeleteDelayMs`.</span></span><br><span class="line"><span class="comment"> * 删除标记为删除的日志文件;</span></span><br><span class="line"><span class="comment"> * file.delete.delay.ms 文件延迟删除时间 默认60000毫秒</span></span><br><span class="line"><span class="comment"> * </span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">deleteLogs</span></span>(): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="keyword">var</span> nextDelayMs = <span class="number">0</span>L</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">nextDeleteDelayMs</span></span>: <span class="type">Long</span> = {</span><br><span class="line"> <span class="keyword">if</span> (!logsToBeDeleted.isEmpty) {</span><br><span class="line"> <span class="keyword">val</span> (_, scheduleTimeMs) = logsToBeDeleted.peek()</span><br><span class="line"> scheduleTimeMs + currentDefaultConfig.fileDeleteDelayMs - time.milliseconds()</span><br><span class="line"> } <span class="keyword">else</span></span><br><span class="line"> currentDefaultConfig.fileDeleteDelayMs</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">while</span> ({nextDelayMs = nextDeleteDelayMs; nextDelayMs <= <span class="number">0</span>}) {</span><br><span class="line"> <span class="keyword">val</span> (removedLog, _) = logsToBeDeleted.take()</span><br><span class="line"> <span class="keyword">if</span> (removedLog != <span class="literal">null</span>) {</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="comment">//立即彻底删除此日志目录和文件系统中的所有内容</span></span><br><span class="line"> removedLog.delete()</span><br><span class="line"> info(<span class="string">s"Deleted log for partition <span class="subst">${removedLog.topicPartition}</span> in <span class="subst">${removedLog.dir.getAbsolutePath}</span>."</span>)</span><br><span class="line"> } <span class="keyword">catch</span> {</span><br><span class="line"> <span class="keyword">case</span> e: <span class="type">KafkaStorageException</span> =></span><br><span class="line"> error(<span class="string">s"Exception while deleting <span class="subst">$removedLog</span> in dir <span class="subst">${removedLog.dir.getParent}</span>."</span>, e)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">catch</span> {</span><br><span class="line"> <span class="keyword">case</span> e: <span class="type">Throwable</span> =></span><br><span class="line"> error(<span class="string">s"Exception in kafka-delete-logs thread."</span>, e)</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> scheduler.schedule(<span class="string">"kafka-delete-logs"</span>,</span><br><span class="line"> deleteLogs _,</span><br><span class="line"> delay = nextDelayMs,</span><br><span class="line"> unit = <span class="type">TimeUnit</span>.<span class="type">MILLISECONDS</span>)</span><br><span class="line"> } <span class="keyword">catch</span> {</span><br><span class="line"> <span class="keyword">case</span> e: <span class="type">Throwable</span> =></span><br><span class="line"> <span class="keyword">if</span> (scheduler.isStarted) {</span><br><span class="line"> <span class="comment">// No errors should occur unless scheduler has been shutdown</span></span><br><span class="line"> error(<span class="string">s"Failed to schedule next delete in kafka-delete-logs thread"</span>, e)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br></pre></td></tr></table></figure><p><code>file.delete.delay.ms</code> 决定延迟多久删除</p><h3 id="5-StopReplica-请求成功-执行回调接口"><a href="#5-StopReplica-请求成功-执行回调接口" class="headerlink" title="5.StopReplica 请求成功 执行回调接口"></a>5.StopReplica 请求成功 执行回调接口</h3><blockquote><p>Topic删除完成, 清理相关信息<br>触发这个接口的地方是: 每个Broker执行删除<code>StopReplica</code>成功之后,都会执行一个回调函数;<code>TopicDeletionStopReplicaResponseReceived</code> ; 当然调用方是Controller,回调到的也就是Controller;</p></blockquote><p>传入回调函数的地方<br><img src="https://img-blog.csdnimg.cn/20210615122649613.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><p>执行回调函数 <code>KafkaController.processTopicDeletionStopReplicaResponseReceived</code></p><ol><li><p>如果回调有异常,删除失败则将副本状态转换成==》<code>ReplicaDeletionIneligible</code>，并且重新执行<code>resumeDeletions</code>方法; </p></li><li><p>如果回调正常,则变更状态 <code>ReplicaDeletionStarted</code>==》<code>ReplicaDeletionSuccessful</code>；并且重新执行<code>resumeDeletions</code>方法; </p></li><li><p><code>resumeDeletions</code>方法会判断所有副本是否均被删除,如果全部删除了就会执行下面的<code>completeDeleteTopic</code>代码;否则会继续删除未被成功删除的副本</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">completeDeleteTopic</span></span>(topic: <span class="type">String</span>): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="comment">// deregister partition change listener on the deleted topic. This is to prevent the partition change listener</span></span><br><span class="line"> <span class="comment">// firing before the new topic listener when a deleted topic gets auto created</span></span><br><span class="line"> client.mutePartitionModifications(topic)</span><br><span class="line"> <span class="keyword">val</span> replicasForDeletedTopic = controllerContext.replicasInState(topic, <span class="type">ReplicaDeletionSuccessful</span>)</span><br><span class="line"> <span class="comment">// controller will remove this replica from the state machine as well as its partition assignment cache</span></span><br><span class="line"> replicaStateMachine.handleStateChanges(replicasForDeletedTopic.toSeq, <span class="type">NonExistentReplica</span>)</span><br><span class="line"> controllerContext.topicsToBeDeleted -= topic</span><br><span class="line"> controllerContext.topicsWithDeletionStarted -= topic</span><br><span class="line"> client.deleteTopic(topic, controllerContext.epochZkVersion)</span><br><span class="line"> controllerContext.removeTopic(topic)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ol><li>清理内存中相关信息</li><li>取消注册被删除Topic的相关节点监听器;节点是<code>/brokers/topics/Topic名称</code></li><li>删除zk中的数据包括;<code>/brokers/topics/Topic名称</code>、<code>/config/topics/Topic名称</code> 、<code>/admin/delete_topics/Topic名称</code></li></ol></li></ol><h3 id="6-Controller启动时候-尝试继续处理待删除的Topic"><a href="#6-Controller启动时候-尝试继续处理待删除的Topic" class="headerlink" title="6. Controller启动时候 尝试继续处理待删除的Topic"></a>6. Controller启动时候 尝试继续处理待删除的Topic</h3><p>我们之前分析Controller上线的时候有看到<br><code>KafkaController.onControllerFailover</code><br>以下省略部分代码</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">onControllerFailover</span></span>(): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="comment">// 获取哪些Topic需要被删除，哪些暂时还不能删除</span></span><br><span class="line"> <span class="keyword">val</span> (topicsToBeDeleted, topicsIneligibleForDeletion) = fetchTopicDeletionsInProgress()</span><br><span class="line"></span><br><span class="line"> info(<span class="string">"Initializing topic deletion manager"</span>)</span><br><span class="line"> <span class="comment">//Topic删除管理器初始化</span></span><br><span class="line"> topicDeletionManager.init(topicsToBeDeleted, topicsIneligibleForDeletion)</span><br><span class="line"></span><br><span class="line"> <span class="comment">//Topic删除管理器 尝试开始删除Topi</span></span><br><span class="line"> topicDeletionManager.tryTopicDeletion()</span><br><span class="line"></span><br></pre></td></tr></table></figure><h4 id="6-1-获取需要被删除的Topic和暂时不能删除的Topic"><a href="#6-1-获取需要被删除的Topic和暂时不能删除的Topic" class="headerlink" title="6.1 获取需要被删除的Topic和暂时不能删除的Topic"></a>6.1 获取需要被删除的Topic和暂时不能删除的Topic</h4><p><code> fetchTopicDeletionsInProgress</code></p><ol><li><code>topicsToBeDeleted</code>所有需要被删除的Topic从zk中<code>/admin/delete_topics</code> 获取</li><li><code>topicsIneligibleForDeletion</code>有一部分Topic还暂时不能被删除:<br>①. Topic任意分区正在进行副本重分配<br>②. Topic任意分区副本存在不在线的情况(只有topic有一个副本所在的Broker异常就不能能删除)</li><li>将得到的数据存在在<code>controllerContext</code>内存中</li></ol><h4 id="6-2-topicDeletionManager-init初始化删除管理器"><a href="#6-2-topicDeletionManager-init初始化删除管理器" class="headerlink" title="6.2 topicDeletionManager.init初始化删除管理器"></a>6.2 topicDeletionManager.init初始化删除管理器</h4><ol><li>如果服务器配置<code>delete.topic.enable=false</code>不允许删除topic的话,则删除<code>/admin/delete_topics</code> 中的节点; 这个节点下面的数据是标记topic需要被删除的意思;</li></ol><h4 id="6-3-topicDeletionManager-tryTopicDeletion尝试恢复删除"><a href="#6-3-topicDeletionManager-tryTopicDeletion尝试恢复删除" class="headerlink" title="6.3 topicDeletionManager.tryTopicDeletion尝试恢复删除"></a>6.3 topicDeletionManager.tryTopicDeletion尝试恢复删除</h4><p>这里又回到了上面分析过的<code>resumeDeletions</code>啦；恢复删除操作</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">tryTopicDeletion</span></span>(): <span class="type">Unit</span> = { <span class="keyword">if</span> (isDeleteTopicEnabled) { resumeDeletions() } }</span><br></pre></td></tr></table></figure><h2 id="源码总结"><a href="#源码总结" class="headerlink" title="源码总结"></a>源码总结</h2><p>整个Topic删除, 请看下图<br><img src="https://img-blog.csdnimg.cn/0aed882130c0478b97f518c25ee0a815.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><p>几个注意点:</p><ol><li>Controller 也是Broker</li><li>Controller发起删除请求的时候,只是跟相关联的Broker发起删除请求;</li><li>Broker不在线或者删除失败,Controller会持续进行删除操作; 或者Broker上线之后继续进行删除操作</li></ol><h2 id="Q-amp-A"><a href="#Q-amp-A" class="headerlink" title="Q&A"></a>Q&A</h2><p><font color="red">列举在此主题下比较常见的问题; 如果读者有其他问题可以在评论区评论, 博主会不定期更新</font></p><h3 id="什么时候在-admin-delete-topics写入节点的"><a href="#什么时候在-admin-delete-topics写入节点的" class="headerlink" title="什么时候在/admin/delete_topics写入节点的"></a>什么时候在/admin/delete_topics写入节点的</h3><blockquote><p>客户端发起删除操作deleteTopics的时候,Controller响应deleteTopics请求, 这个时候Controller就将待删除Topic写入了zk的<code>/admin/delete_topics/Topic名称</code>节点中了; </p></blockquote><h3 id="什么时候真正执行删除Topic磁盘日志"><a href="#什么时候真正执行删除Topic磁盘日志" class="headerlink" title="什么时候真正执行删除Topic磁盘日志"></a>什么时候真正执行删除Topic磁盘日志</h3><blockquote><p>Controller监听到zk节点<code>/admin/delete_topics</code>之后,向所有存活的Broker发送删除Topic的请求; Broker收到请求之后将待删除副本标记为–delete后缀; 然后会有专门日志清理现场来进行真正的删除操作; 延迟多久删除是靠<code>file.delete.delay.ms</code>来决定的；默认是60000毫秒 = 一分钟</p></blockquote><h3 id="为什么正在重新分配的Topic不能被删除"><a href="#为什么正在重新分配的Topic不能被删除" class="headerlink" title="为什么正在重新分配的Topic不能被删除"></a>为什么正在重新分配的Topic不能被删除</h3><blockquote><p>正在重新分配的Topic,你都不知道它具体会落在哪个地方,所以肯定也就不知道啥时候删除啊;<br>等分配完毕之后,就会继续删除流程</p></blockquote><h3 id="如果在-admin-delete-topics-中手动写入一个节点会不会正常删除"><a href="#如果在-admin-delete-topics-中手动写入一个节点会不会正常删除" class="headerlink" title="如果在/admin/delete_topics/中手动写入一个节点会不会正常删除"></a>如果在<code>/admin/delete_topics/</code>中手动写入一个节点会不会正常删除</h3><blockquote><p>如果写入的节点,并不是一个真实存在的Topic；则将会直接被删除<br>当然要注意如果配置了<code>delete.topic.enable=false</code>不可删除Topic的话,则将<code>/admin/delete_topics/</code>下面的节点全部删除,然后流程结束<br>如果写入的节点是一个真实存在的Topic; 则将会执行删除Topic的流程; 本质上跟用Kafka客户端执行删除Topic操作没有什么不同</p></blockquote><h3 id="如果直接删除ZK上的-brokers-topics-topicName-节点会怎样"><a href="#如果直接删除ZK上的-brokers-topics-topicName-节点会怎样" class="headerlink" title="如果直接删除ZK上的/brokers/topics/{topicName}节点会怎样"></a>如果直接删除ZK上的<code>/brokers/topics/{topicName}</code>节点会怎样</h3><blockquote><p>TODO…</p></blockquote><h3 id="Controller通知Brokers-执行StopReplica是通知所有的Broker还是只通知跟被删除Topic有关联的Broker？"><a href="#Controller通知Brokers-执行StopReplica是通知所有的Broker还是只通知跟被删除Topic有关联的Broker？" class="headerlink" title="Controller通知Brokers 执行StopReplica是通知所有的Broker还是只通知跟被删除Topic有关联的Broker？"></a>Controller通知Brokers 执行StopReplica是通知所有的Broker还是只通知跟被删除Topic有关联的Broker？</h3><blockquote><p><strong>只是通知跟被删除Topic有关联的Broker;</strong><br>请看下图源码,可以看到所有需要被<code>StopReplica</code>的副本都是被过滤了一遍,获取它们所在的BrokerId; 最后调用的时候也是<code>sendRequest(brokerId, stopReplicaRequest)</code> ;根据获取到的BrokerId发起的请求<br><img src="https://img-blog.csdnimg.cn/20210615141430911.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p></blockquote><h3 id="删除过程有Broker不在线-或者执行失败怎么办"><a href="#删除过程有Broker不在线-或者执行失败怎么办" class="headerlink" title="删除过程有Broker不在线 或者执行失败怎么办"></a>删除过程有Broker不在线 或者执行失败怎么办</h3><blockquote><p>Controller会继续删除操作；或者等Broker上线然后继续删除操作; 反正就是一定会保证所有的分区都被删除(被标记了–delete)之后才会把zk上的数据清理掉;</p></blockquote><h3 id="ReplicaStateMachine-副本状态机"><a href="#ReplicaStateMachine-副本状态机" class="headerlink" title="ReplicaStateMachine 副本状态机"></a>ReplicaStateMachine 副本状态机</h3><blockquote><p>请看 <a href="">【kafka源码】Controller中的状态机TODO</a></p></blockquote><h3 id="在重新分配的过程中-如果执行删除操作会怎么样"><a href="#在重新分配的过程中-如果执行删除操作会怎么样" class="headerlink" title="在重新分配的过程中,如果执行删除操作会怎么样"></a>在重新分配的过程中,如果执行删除操作会怎么样</h3><blockquote><p>删除操作会等待,等待重新分配完成之后,继续进行删除操作<br><img src="https://img-blog.csdnimg.cn/20210621172944227.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p></blockquote><p>Finally: 本文阅读源码为 <code>Kafka-2.5</code></p>]]></content>
<summary type="html"><p>阅读本文之前 你可以先思考一下以下几个问题，然后再带着问题去阅读本文</p>
<ul>
<li>什么时候在/admin/delete_topics写入节点的？</li>
<li>什么时候真正执行删除Topic磁盘日志？</li>
<li>Controller通知Broker</summary>
<category term="kafka" scheme="http://example.com/categories/kafka/"/>
<category term="kafka" scheme="http://example.com/tags/kafka/"/>
<category term="大数据" scheme="http://example.com/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="源码" scheme="http://example.com/tags/%E6%BA%90%E7%A0%81/"/>
</entry>
<entry>
<title>【kafka源码】kafka-topics.sh之创建Topic源码解析</title>
<link href="http://example.com/2021/08/21/create-topic/"/>
<id>http://example.com/2021/08/21/create-topic/</id>
<published>2021-08-21T03:00:34.000Z</published>
<updated>2021-08-23T08:25:13.940Z</updated>
<content type="html"><![CDATA[<p><font size=5 color=red>【kafka源码】kafka-topics.sh之创建Topic源码解析</font></p><blockquote><p><font color=green size=4><strong>日常运维</strong></font> 、<font color=red size=4><strong>问题排查</strong></font> 怎么能够少了滴滴开源的<br><a href="https://github.com/didi/LogiKM"><font color=blue size=4>滴滴开源LogiKM一站式Kafka监控与管控平台</font></a> </p></blockquote><h2 id="脚本参数"><a href="#脚本参数" class="headerlink" title="脚本参数"></a>脚本参数</h2><p><code>sh bin/kafka-topic -help</code> 查看更具体参数</p><p>下面只是列出了跟<code> --create</code> 相关的参数</p><table><thead><tr><th>参数</th><th>描述</th><th>例子</th></tr></thead><tbody><tr><td><code>--bootstrap-server </code> 指定kafka服务</td><td>指定连接到的kafka服务; 如果有这个参数,则 <code>--zookeeper</code>可以不需要</td><td>–bootstrap-server localhost:9092</td></tr><tr><td><code>--zookeeper</code></td><td>弃用, 通过zk的连接方式连接到kafka集群;</td><td>–zookeeper localhost:2181 或者localhost:2181/kafka</td></tr><tr><td><code>--replication-factor </code></td><td>副本数量,注意不能大于broker数量;如果不提供,则会用集群中默认配置</td><td>–replication-factor 3</td></tr><tr><td><code>--partitions</code></td><td>分区数量</td><td>当创建或者修改topic的时候,用这个来指定分区数;如果创建的时候没有提供参数,则用集群中默认值; 注意如果是修改的时候,分区比之前小会有问题</td></tr><tr><td><code>--replica-assignment </code></td><td>副本分区分配方式;创建topic的时候可以自己指定副本分配情况;</td><td><code>--replica-assignment</code> BrokerId-0:BrokerId-1:BrokerId-2,BrokerId-1:BrokerId-2:BrokerId-0,BrokerId-2:BrokerId-1:BrokerId-0 ; 这个意思是有三个分区和三个副本,对应分配的Broker; 逗号隔开标识分区;冒号隔开表示副本</td></tr><tr><td><code>--config </code><String: name=value></td><td>用来设置topic级别的配置以覆盖默认配置;<strong>只在–create 和–bootstrap-server 同时使用时候生效</strong>; 可以配置的参数列表请看文末附件</td><td>例如覆盖两个配置 <code>--config retention.bytes=123455 --config retention.ms=600001</code></td></tr><tr><td><code>--command-config</code> <String: command 文件路径></td><td>用来配置客户端Admin Client启动配置,<strong>只在–bootstrap-server 同时使用时候生效</strong>;</td><td>例如:设置请求的超时时间 <code>--command-config config/producer.proterties </code>; 然后在文件中配置 request.timeout.ms=300000</td></tr><tr><td><code>--create</code></td><td>命令方式; 表示当前请求是创建Topic</td><td><code>--create</code></td></tr></tbody></table><h2 id="创建Topic脚本"><a href="#创建Topic脚本" class="headerlink" title="创建Topic脚本"></a>创建Topic脚本</h2><p><strong>zk方式(不推荐)</strong></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic test</span><br></pre></td></tr></table></figure><p><font color="red">需要注意的是–zookeeper后面接的是kafka的zk配置, 假如你配置的是localhost:2181/kafka 带命名空间的这种,不要漏掉了 </font></p><p><strong>kafka版本 >= 2.2 支持下面方式（推荐）</strong> </p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 3 --topic test</span><br></pre></td></tr></table></figure><p>更多TopicCommand相关命令请看 </p><p><a href="../运维脚本/1.【kafka运维】TopicCommand运维脚本.md">1.【kafka运维】TopicCommand运维脚本</a></p><p>当前分析的kafka源码版本为 <code>kafka-2.5</code></p><h2 id="创建Topic-源码分析"><a href="#创建Topic-源码分析" class="headerlink" title="创建Topic 源码分析"></a>创建Topic 源码分析</h2><p><font color="red">温馨提示: 如果阅读源码略显枯燥,你可以直接看源码总结以及后面部分</font></p><p>首先我们找到源码入口处, 查看一下 <code>kafka-topic.sh</code>脚本的内容<br><code>exec $(dirname $0)/kafka-run-class.sh kafka.admin.TopicCommand "$@"</code><br>最终是执行了<code>kafka.admin.TopicCommand</code>这个类，找到这个地方之后就可以断点调试源码了,用IDEA启动<br><img src="https://img-blog.csdnimg.cn/20210608151956926.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br>记得配置一下入参<br>比如: <code>--create --bootstrap-server 127.0.0.1:9092 --partitions 3 --topic test_create_topic3</code><br> <img src="https://img-blog.csdnimg.cn/20210608152149713.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><h3 id="1-源码入口"><a href="#1-源码入口" class="headerlink" title="1. 源码入口"></a>1. 源码入口</h3><p><img src="https://img-blog.csdnimg.cn/2021060815275820.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br>上面的源码主要作用是</p><ol><li>根据是否有传入参数<code>--zookeeper</code> 来判断创建哪一种 对象<code>topicService</code><br>如果传入了<code>--zookeeper</code> 则创建 类 <code>ZookeeperTopicService</code>的对象<br>否则创建类<code>AdminClientTopicService</code>的对象(我们主要分析这个对象)</li><li>根据传入的参数类型判断是创建topic还是删除等等其他 判断依据是 是否在参数里传入了<code>--create</code></li></ol><h3 id="2-创建AdminClientTopicService-对象"><a href="#2-创建AdminClientTopicService-对象" class="headerlink" title="2. 创建AdminClientTopicService 对象"></a>2. 创建AdminClientTopicService 对象</h3><blockquote><p> <code>val topicService = new AdminClientTopicService(createAdminClient(commandConfig, bootstrapServer))</code></p></blockquote><h4 id="2-1-先创建-Admin"><a href="#2-1-先创建-Admin" class="headerlink" title="2.1 先创建 Admin"></a>2.1 先创建 Admin</h4><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">AdminClientTopicService</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">createAdminClient</span></span>(commandConfig: <span class="type">Properties</span>, bootstrapServer: <span class="type">Option</span>[<span class="type">String</span>]): <span class="type">Admin</span> = {</span><br><span class="line"> bootstrapServer <span class="keyword">match</span> {</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Some</span>(serverList) => commandConfig.put(<span class="type">CommonClientConfigs</span>.<span class="type">BOOTSTRAP_SERVERS_CONFIG</span>, serverList)</span><br><span class="line"> <span class="keyword">case</span> <span class="type">None</span> =></span><br><span class="line"> }</span><br><span class="line"> <span class="type">Admin</span>.create(commandConfig)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">apply</span></span>(commandConfig: <span class="type">Properties</span>, bootstrapServer: <span class="type">Option</span>[<span class="type">String</span>]): <span class="type">AdminClientTopicService</span> =</span><br><span class="line"> <span class="keyword">new</span> <span class="type">AdminClientTopicService</span>(createAdminClient(commandConfig, bootstrapServer))</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><ol><li>如果有入参<code>--command-config</code> ,则将这个文件里面的参数都放到map <code>commandConfig</code>里面, 并且也加入<code>bootstrap.servers</code>的参数;假如配置文件里面已经有了<code>bootstrap.servers</code>配置，那么会将其覆盖</li><li>将上面的<code>commandConfig</code> 作为入参调用<code>Admin.create(commandConfig)</code>创建 Admin; 这个时候调用的Client模块的代码了, 从这里我们就可以看出,我们调用<code>kafka-topic.sh</code>脚本实际上是kafka模拟了一个客户端<code>Client</code>来创建Topic的过程;<br><img src="https://img-blog.csdnimg.cn/20210608160130820.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li></ol><h3 id="3-AdminClientTopicService-createTopic-创建Topic"><a href="#3-AdminClientTopicService-createTopic-创建Topic" class="headerlink" title="3. AdminClientTopicService.createTopic 创建Topic"></a>3. AdminClientTopicService.createTopic 创建Topic</h3><p><code> topicService.createTopic(opts)</code></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">AdminClientTopicService</span> <span class="title">private</span> (<span class="params">adminClient: <span class="type">Admin</span></span>) <span class="keyword">extends</span> <span class="title">TopicService</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">createTopic</span></span>(topic: <span class="type">CommandTopicPartition</span>): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="comment">//如果配置了副本副本数--replication-factor 一定要大于0</span></span><br><span class="line"> <span class="keyword">if</span> (topic.replicationFactor.exists(rf => rf > <span class="type">Short</span>.<span class="type">MaxValue</span> || rf < <span class="number">1</span>))</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">IllegalArgumentException</span>(<span class="string">s"The replication factor must be between 1 and <span class="subst">${Short.MaxValue}</span> inclusive"</span>)</span><br><span class="line"> <span class="comment">//如果配置了--partitions 分区数 必须大于0</span></span><br><span class="line"> <span class="keyword">if</span> (topic.partitions.exists(partitions => partitions < <span class="number">1</span>))</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">IllegalArgumentException</span>(<span class="string">s"The partitions must be greater than 0"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment">//查询是否已经存在该Topic</span></span><br><span class="line"> <span class="keyword">if</span> (!adminClient.listTopics().names().get().contains(topic.name)) {</span><br><span class="line"> <span class="keyword">val</span> newTopic = <span class="keyword">if</span> (topic.hasReplicaAssignment)</span><br><span class="line"> <span class="comment">//如果指定了--replica-assignment参数；则按照指定的来分配副本</span></span><br><span class="line"> <span class="keyword">new</span> <span class="type">NewTopic</span>(topic.name, asJavaReplicaReassignment(topic.replicaAssignment.get))</span><br><span class="line"> <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">new</span> <span class="type">NewTopic</span>(</span><br><span class="line"> topic.name,</span><br><span class="line"> topic.partitions.asJava,</span><br><span class="line"> topic.replicationFactor.map(_.toShort).map(<span class="type">Short</span>.box).asJava)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 将配置--config 解析成一个配置map</span></span><br><span class="line"> <span class="keyword">val</span> configsMap = topic.configsToAdd.stringPropertyNames()</span><br><span class="line"> .asScala</span><br><span class="line"> .map(name => name -> topic.configsToAdd.getProperty(name))</span><br><span class="line"> .toMap.asJava</span><br><span class="line"></span><br><span class="line"> newTopic.configs(configsMap)</span><br><span class="line"> <span class="comment">//调用adminClient创建Topic</span></span><br><span class="line"> <span class="keyword">val</span> createResult = adminClient.createTopics(<span class="type">Collections</span>.singleton(newTopic))</span><br><span class="line"> createResult.all().get()</span><br><span class="line"> println(<span class="string">s"Created topic <span class="subst">${topic.name}</span>."</span>)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> <span class="type">IllegalArgumentException</span>(<span class="string">s"Topic <span class="subst">${topic.name}</span> already exists"</span>)</span><br><span class="line"> }</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><ol><li>检查各项入参是否有问题</li><li><code>adminClient.listTopics()</code>,然后比较是否已经存在待创建的Topic;如果存在抛出异常; </li><li>判断是否配置了参数<code>--replica-assignment</code> ; 如果配置了,那么Topic就会按照指定的方式来配置副本情况</li><li>解析配置<code>--config </code> 配置放到<code> configsMap</code>中; <code>configsMap</code>给到<code>NewTopic</code>对象</li><li>调用<code>adminClient.createTopics</code>创建Topic; 它是如何创建Topic的呢？往下分析源码</li></ol><h4 id="3-1-KafkaAdminClient-createTopics-NewTopic-创建Topic"><a href="#3-1-KafkaAdminClient-createTopics-NewTopic-创建Topic" class="headerlink" title="3.1 KafkaAdminClient.createTopics(NewTopic) 创建Topic"></a>3.1 KafkaAdminClient.createTopics(NewTopic) 创建Topic</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> CreateTopicsResult <span class="title">createTopics</span><span class="params">(<span class="keyword">final</span> Collection<NewTopic> newTopics,</span></span></span><br><span class="line"><span class="params"><span class="function"> <span class="keyword">final</span> CreateTopicsOptions options)</span> </span>{</span><br><span class="line"> </span><br><span class="line"> <span class="comment">//省略部分源码...</span></span><br><span class="line"> Call call = <span class="keyword">new</span> Call(<span class="string">"createTopics"</span>, calcDeadlineMs(now, options.timeoutMs()),</span><br><span class="line"> <span class="keyword">new</span> ControllerNodeProvider()) {</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="keyword">public</span> CreateTopicsRequest.<span class="function">Builder <span class="title">createRequest</span><span class="params">(<span class="keyword">int</span> timeoutMs)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> CreateTopicsRequest.Builder(</span><br><span class="line"> <span class="keyword">new</span> CreateTopicsRequestData().</span><br><span class="line"> setTopics(topics).</span><br><span class="line"> setTimeoutMs(timeoutMs).</span><br><span class="line"> setValidateOnly(options.shouldValidateOnly()));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">handleResponse</span><span class="params">(AbstractResponse abstractResponse)</span> </span>{</span><br><span class="line"> <span class="comment">//省略</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">void</span> <span class="title">handleFailure</span><span class="params">(Throwable throwable)</span> </span>{</span><br><span class="line"> completeAllExceptionally(topicFutures.values(), throwable);</span><br><span class="line"> }</span><br><span class="line"> };</span><br><span class="line"> </span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>这个代码里面主要看下Call里面的接口; 先不管Kafka如何跟服务端进行通信的细节; 我们主要关注创建Topic的逻辑;</p><ol><li><code>createRequest</code>会构造一个请求参数<code>CreateTopicsRequest</code> 例如下图<br><img src="https://img-blog.csdnimg.cn/20210609174617186.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>选择ControllerNodeProvider这个节点发起网络请求<br><img src="https://img-blog.csdnimg.cn/20210609200925505.png" alt="在这里插入图片描述"><br>可以清楚的看到, 创建Topic这个操作是需要Controller来执行的;<br><img src="https://img-blog.csdnimg.cn/20210609200938586.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li></ol><h3 id="4-发起网络请求"><a href="#4-发起网络请求" class="headerlink" title="4. 发起网络请求"></a>4. 发起网络请求</h3><p> <a href="TODO">==>服务端客户端网络模型 </a></p><h3 id="5-Controller角色的服务端接受请求处理逻辑"><a href="#5-Controller角色的服务端接受请求处理逻辑" class="headerlink" title="5. Controller角色的服务端接受请求处理逻辑"></a>5. Controller角色的服务端接受请求处理逻辑</h3><p>首先找到服务端处理客户端请求的 <strong>源码入口</strong> ⇒ <code>KafkaRequestHandler.run()</code></p><p>主要看里面的 <code>apis.handle(request)</code> 方法; 可以看到客户端的请求都在<code>request.bodyAndSize()</code>里面<br><img src="https://img-blog.csdnimg.cn/2021060917574268.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><h4 id="5-1-KafkaApis-handle-request-根据请求传递Api调用不同接口"><a href="#5-1-KafkaApis-handle-request-根据请求传递Api调用不同接口" class="headerlink" title="5.1 KafkaApis.handle(request) 根据请求传递Api调用不同接口"></a>5.1 KafkaApis.handle(request) 根据请求传递Api调用不同接口</h4><p>进入方法可以看到根据<code>request.header.apiKey</code> 调用对应的方法,客户端传过来的是<code>CreateTopics</code><br><img src="https://img-blog.csdnimg.cn/2021060918000338.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><h4 id="5-2-KafkaApis-handleCreateTopicsRequest-处理创建Topic的请求"><a href="#5-2-KafkaApis-handleCreateTopicsRequest-处理创建Topic的请求" class="headerlink" title="5.2 KafkaApis.handleCreateTopicsRequest 处理创建Topic的请求"></a>5.2 KafkaApis.handleCreateTopicsRequest 处理创建Topic的请求</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">def <span class="title">handleCreateTopicsRequest</span><span class="params">(request: RequestChannel.Request)</span>: Unit </span>= {</span><br><span class="line"> <span class="comment">// 部分代码省略</span></span><br><span class="line"><span class="comment">//如果当前Broker不是属于Controller的话,就抛出异常</span></span><br><span class="line"> <span class="keyword">if</span> (!controller.isActive) {</span><br><span class="line"> createTopicsRequest.data.topics.asScala.foreach { topic =></span><br><span class="line"> results.add(<span class="keyword">new</span> CreatableTopicResult().setName(topic.name).</span><br><span class="line"> setErrorCode(Errors.NOT_CONTROLLER.code))</span><br><span class="line"> }</span><br><span class="line"> sendResponseCallback(results)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// 部分代码省略</span></span><br><span class="line"> }</span><br><span class="line"> adminManager.createTopics(createTopicsRequest.data.timeoutMs,</span><br><span class="line"> createTopicsRequest.data.validateOnly,</span><br><span class="line"> toCreate,</span><br><span class="line"> authorizedForDescribeConfigs,</span><br><span class="line"> handleCreateTopicsResults)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br></pre></td></tr></table></figure><ol><li>判断当前处理的broker是不是Controller,如果不是Controller的话直接抛出异常,从这里可以看出,CreateTopic这个操作必须是Controller来进行, 出现这种情况有可能是客户端发起请求的时候Controller已经变更; </li><li>鉴权 <a href="">【Kafka源码】kafka鉴权机制</a></li><li>调用<code>adminManager.createTopics()</code> </li></ol><h4 id="5-3-adminManager-createTopics"><a href="#5-3-adminManager-createTopics" class="headerlink" title="5.3 adminManager.createTopics()"></a>5.3 adminManager.createTopics()</h4><blockquote><p>创建主题并等等主题完全创建,回调函数将会在超时、错误、或者主题创建完成时触发</p></blockquote><p>该方法过长,省略部分代码</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createTopics</span></span>(timeout: <span class="type">Int</span>,</span><br><span class="line"> validateOnly: <span class="type">Boolean</span>,</span><br><span class="line"> toCreate: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">CreatableTopic</span>],</span><br><span class="line"> includeConfigsAndMetatadata: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">CreatableTopicResult</span>],</span><br><span class="line"> responseCallback: <span class="type">Map</span>[<span class="type">String</span>, <span class="type">ApiError</span>] => <span class="type">Unit</span>): <span class="type">Unit</span> = {</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 1. map over topics creating assignment and calling zookeeper</span></span><br><span class="line"> <span class="keyword">val</span> brokers = metadataCache.getAliveBrokers.map { b => kafka.admin.<span class="type">BrokerMetadata</span>(b.id, b.rack) }</span><br><span class="line"> <span class="keyword">val</span> metadata = toCreate.values.map(topic =></span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="comment">//省略部分代码</span></span><br><span class="line"> <span class="comment">//检查Topic是否存在</span></span><br><span class="line"> <span class="comment">//检查 --replica-assignment参数和 (--partitions|| --replication-factor ) 不能同时使用</span></span><br><span class="line"> <span class="comment">// 如果(--partitions|| --replication-factor ) 没有设置,则使用 Broker的配置(这个Broker肯定是Controller)</span></span><br><span class="line"><span class="comment">// 计算分区副本分配方式</span></span><br><span class="line"></span><br><span class="line"> createTopicPolicy <span class="keyword">match</span> {</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Some</span>(policy) =></span><br><span class="line"> <span class="comment">//省略部分代码</span></span><br><span class="line"> adminZkClient.validateTopicCreate(topic.name(), assignments, configs)</span><br><span class="line"> <span class="keyword">if</span> (!validateOnly)</span><br><span class="line"> adminZkClient.createTopicWithAssignment(topic.name, configs, assignments)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">case</span> <span class="type">None</span> =></span><br><span class="line"> <span class="keyword">if</span> (validateOnly)</span><br><span class="line"> <span class="comment">//校验创建topic的参数准确性</span></span><br><span class="line"> adminZkClient.validateTopicCreate(topic.name, assignments, configs)</span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="comment">//把topic相关数据写入到zk中</span></span><br><span class="line"> adminZkClient.createTopicWithAssignment(topic.name, configs, assignments)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> </span><br><span class="line"> }</span><br></pre></td></tr></table></figure><ol><li><p>做一些校验检查<br>①.检查Topic是否存在<br>②. 检查<code> --replica-assignment</code>参数和 (<code>--partitions || --replication-factor</code> ) 不能同时使用<br>③.如果(<code>--partitions || --replication-factor</code> ) 没有设置,则使用 Broker的配置(这个Broker肯定是Controller)<br>④.计算分区副本分配方式</p></li><li><p><code>createTopicPolicy</code> 根据Broker是否配置了创建Topic的自定义校验策略; 使用方式是自定义实现<code>org.apache.kafka.server.policy.CreateTopicPolicy</code>接口;并 在服务器配置 <code>create.topic.policy.class.name=自定义类</code>; 比如我就想所有创建Topic的请求分区数都要大于10; 那么这里就可以实现你的需求了</p></li><li><p><code>createTopicWithAssignment</code>把topic相关数据写入到zk中; 进去分析一下</p></li></ol><h4 id="5-4-写入zookeeper数据"><a href="#5-4-写入zookeeper数据" class="headerlink" title="5.4 写入zookeeper数据"></a>5.4 写入zookeeper数据</h4><p>我们进入到<code> adminZkClient.createTopicWithAssignment(topic.name, configs, assignments)</code>看看有哪些数据写入到了zk中;</p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createTopicWithAssignment</span></span>(topic: <span class="type">String</span>,</span><br><span class="line"> config: <span class="type">Properties</span>,</span><br><span class="line"> partitionReplicaAssignment: <span class="type">Map</span>[<span class="type">Int</span>, <span class="type">Seq</span>[<span class="type">Int</span>]]): <span class="type">Unit</span> = {</span><br><span class="line"> validateTopicCreate(topic, partitionReplicaAssignment, config)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 将topic单独的配置写入到zk中</span></span><br><span class="line"> zkClient.setOrCreateEntityConfigs(<span class="type">ConfigType</span>.<span class="type">Topic</span>, topic, config)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 将topic分区相关信息写入zk中</span></span><br><span class="line"> writeTopicPartitionAssignment(topic, partitionReplicaAssignment.mapValues(<span class="type">ReplicaAssignment</span>(_)).toMap, isUpdate = <span class="literal">false</span>)</span><br><span class="line">}</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>源码就不再深入了,这里直接详细说明一下</p><p><strong>写入Topic配置信息</strong></p><ol><li>先调用<code>SetDataRequest</code>请求往节点<code> /config/topics/Topic名称</code> 写入数据; 这里<br>一般这个时候都会返回 <code>NONODE (NoNode)</code>;节点不存在; 假如zk已经存在节点就直接覆盖掉</li><li>节点不存在的话,就发起<code>CreateRequest</code>请求,写入数据; 并且节点类型是<strong>持久节点</strong></li></ol><p>这里写入的数据,是我们入参时候传的topic配置<code>--config</code>; 这里的配置会覆盖默认配置</p><p><strong>写入Topic分区副本信息</strong></p><ol><li>将已经分配好的副本分配策略写入到 <code>/brokers/topics/Topic名称</code> 中; 节点类型 <strong>持久节点</strong><br><img src="https://img-blog.csdnimg.cn/20210610152129161.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li></ol><p><strong>具体跟zk交互的地方在</strong><br><code>ZookeeperClient.send()</code> 这里包装了很多跟zk的交互;<br><img src="https://img-blog.csdnimg.cn/20210610151032490.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p><h3 id="6-Controller监听-brokers-topics-Topic名称-通知Broker将分区写入磁盘"><a href="#6-Controller监听-brokers-topics-Topic名称-通知Broker将分区写入磁盘" class="headerlink" title="6. Controller监听 /brokers/topics/Topic名称, 通知Broker将分区写入磁盘"></a>6. Controller监听 <code>/brokers/topics/Topic名称</code>, 通知Broker将分区写入磁盘</h3><blockquote><p>Controller 有监听zk上的一些节点; 在上面的流程中已经在zk中写入了 <code>/brokers/topics/Topic名称</code> ; 这个时候Controller就监听到了这个变化并相应;</p></blockquote><p><code>KafkaController.processTopicChange</code></p><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">processTopicChange</span></span>(): <span class="type">Unit</span> = { <span class="comment">//如果处理的不是Controller角色就返回 if (!isActive) return //从zk中获取 `/brokers/topics 所有Topic val topics = zkClient.getAllTopicsInCluster //找出哪些是新增的 val newTopics = topics -- controllerContext.allTopics //找出哪些Topic在zk上被删除了 val deletedTopics = controllerContext.allTopics -- topics controllerContext.allTopics = topics registerPartitionModificationsHandlers(newTopics.toSeq) val addedPartitionReplicaAssignment = zkClient.getFullReplicaAssignmentForTopics(newTopics) deletedTopics.foreach(controllerContext.removeTopic) addedPartitionReplicaAssignment.foreach { case (topicAndPartition, newReplicaAssignment) => controllerContext.updatePartitionFullReplicaAssignment(topicAndPartition, newReplicaAssignment) } info(s"New topics: [$newTopics], deleted topics: [$deletedTopics], new partition replica assignment " + s"[$addedPartitionReplicaAssignment]") if (addedPartitionReplicaAssignment.nonEmpty) onNewPartitionCreation(addedPartitionReplicaAssignment.keySet) }</span></span><br></pre></td></tr></table></figure><ol><li><p>从zk中获取 <code>/brokers/topics</code> 所有Topic跟当前Broker内存中所有Broker<code>controllerContext.allTopics</code>的差异; 就可以找到我们新增的Topic; 还有在zk中被删除了的Broker(该Topic会在当前内存中remove掉)</p></li><li><p>从zk中获取<code>/brokers/topics/{TopicName}</code> 给定主题的副本分配。并保存在内存中<img src="https://img-blog.csdnimg.cn/20210616175718504.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></p></li><li><p>执行<code>onNewPartitionCreation</code>;分区状态开始流转</p></li></ol><h4 id="6-1-onNewPartitionCreation-状态流转"><a href="#6-1-onNewPartitionCreation-状态流转" class="headerlink" title="6.1 onNewPartitionCreation 状态流转"></a>6.1 onNewPartitionCreation 状态流转</h4><blockquote><p>关于Controller的状态机 详情请看: <a href="TODO">【kafka源码】Controller中的状态机</a></p></blockquote><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/** * This callback is invoked by the topic change callback with the list of failed brokers as input. * It does the following - * 1. Move the newly created partitions to the NewPartition state * 2. Move the newly created partitions from NewPartition->OnlinePartition state */</span> <span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">onNewPartitionCreation</span></span>(newPartitions: <span class="type">Set</span>[<span class="type">TopicPartition</span>]): <span class="type">Unit</span> = { info(<span class="string">s"New partition creation callback for <span class="subst">${newPartitions.mkString(",")}</span>"</span>) partitionStateMachine.handleStateChanges(newPartitions.toSeq, <span class="type">NewPartition</span>) replicaStateMachine.handleStateChanges(controllerContext.replicasForPartition(newPartitions).toSeq, <span class="type">NewReplica</span>) partitionStateMachine.handleStateChanges( newPartitions.toSeq, <span class="type">OnlinePartition</span>, <span class="type">Some</span>(<span class="type">OfflinePartitionLeaderElectionStrategy</span>(<span class="literal">false</span>)) ) replicaStateMachine.handleStateChanges(controllerContext.replicasForPartition(newPartitions).toSeq, <span class="type">OnlineReplica</span>) }</span><br></pre></td></tr></table></figure><ol><li>将待创建的分区状态流转为<code>NewPartition</code>;<br><img src="https://img-blog.csdnimg.cn/20210616180239988.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>将待创建的副本 状态流转为<code>NewReplica</code>;<br><img src="https://img-blog.csdnimg.cn/20210616180940961.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"></li><li>将分区状态从刚刚的<code>NewPartition</code>流转为<code>OnlinePartition</code></li><li>获取<code>leaderIsrAndControllerEpochs</code>; Leader为副本的第一个;<pre><code>1. 向zk中写入`/brokers/topics/{topicName}/partitions/` 持久节点; 无数据 2. 向zk中写入`/brokers/topics/{topicName}/partitions/{分区号}` 持久节点; 无数据 3. 向zk中写入`/brokers/topics/{topicName}/partitions/{分区号}/state` 持久节点; 数据为`leaderIsrAndControllerEpoch`</code></pre><ol start="4"><li>向副本所属Broker发送<a href=""><code>leaderAndIsrRequest</code></a>请求</li><li>向所有Broker发送<a href=""><code>UPDATE_METADATA</code> </a>请求</li></ol></li><li>将副本状态从刚刚的<code>NewReplica</code>流转为<code>OnlineReplica</code> ,更新下内存</li></ol><p>关于分区状态机和副本状态机详情请看<a href="TODO">【kafka源码】Controller中的状态机</a></p><h3 id="7-Broker收到LeaderAndIsrRequest-创建本地Log"><a href="#7-Broker收到LeaderAndIsrRequest-创建本地Log" class="headerlink" title="7. Broker收到LeaderAndIsrRequest 创建本地Log"></a>7. Broker收到LeaderAndIsrRequest 创建本地Log</h3><blockquote><p>上面步骤中有说到向副本所属Broker发送<a href=""><code>leaderAndIsrRequest</code></a>请求,那么这里做了什么呢<br>其实主要做的是 创建本地Log</p><p>代码太多,这里我们直接定位到只跟创建Topic相关的关键代码来分析<br><code>KafkaApis.handleLeaderAndIsrRequest->replicaManager.becomeLeaderOrFollower->ReplicaManager.makeLeaders...LogManager.getOrCreateLog</code></p></blockquote><figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/** * 如果日志已经存在，只返回现有日志的副本否则如果 isNew=true 或者如果没有离线日志目录，则为给定的主题和给定的分区创建日志 否则抛出 KafkaStorageException */</span> <span class="function"><span class="keyword">def</span> <span class="title">getOrCreateLog</span></span>(topicPartition: <span class="type">TopicPartition</span>, config: <span class="type">LogConfig</span>, isNew: <span class="type">Boolean</span> = <span class="literal">false</span>, isFuture: <span class="type">Boolean</span> = <span class="literal">false</span>): <span class="type">Log</span> = { logCreationOrDeletionLock synchronized { getLog(topicPartition, isFuture).getOrElse { <span class="comment">// create the log if it has not already been created in another thread if (!isNew && offlineLogDirs.nonEmpty) throw new KafkaStorageException(s"Can not create log for $topicPartition because log directories ${offlineLogDirs.mkString(",")} are offline") val logDirs: List[File] = { val preferredLogDir = preferredLogDirs.get(topicPartition) if (isFuture) { if (preferredLogDir == null) throw new IllegalStateException(s"Can not create the future log for $topicPartition without having a preferred log directory") else if (getLog(topicPartition).get.dir.getParent == preferredLogDir) throw new IllegalStateException(s"Can not create the future log for $topicPartition in the current log directory of this partition") } if (preferredLogDir != null) List(new File(preferredLogDir)) else nextLogDirs() } val logDirName = { if (isFuture) Log.logFutureDirName(topicPartition) else Log.logDirName(topicPartition) } val logDir = logDirs .toStream // to prevent actually mapping the whole list, lazy map .map(createLogDirectory(_, logDirName)) .find(_.isSuccess) .getOrElse(Failure(new KafkaStorageException("No log directories available. Tried " + logDirs.map(_.getAbsolutePath).mkString(", ")))) .get // If Failure, will throw val log = Log( dir = logDir, config = config, logStartOffset = 0L, recoveryPoint = 0L, maxProducerIdExpirationMs = maxPidExpirationMs, producerIdExpirationCheckIntervalMs = LogManager.ProducerIdExpirationCheckIntervalMs, scheduler = scheduler, time = time, brokerTopicStats = brokerTopicStats, logDirFailureChannel = logDirFailureChannel) if (isFuture) futureLogs.put(topicPartition, log) else currentLogs.put(topicPartition, log) info(s"Created log for partition $topicPartition in $logDir with properties " + s"{${config.originals.asScala.mkString(", ")}}.") // Remove the preferred log dir since it has already been satisfied preferredLogDirs.remove(topicPartition) log } } }</span></span><br></pre></td></tr></table></figure><ol><li>如果日志已经存在，只返回现有日志的副本否则如果 isNew=true 或者如果没有离线日志目录，则为给定的主题和给定的分区创建日志 否则抛出<code> KafkaStorageException</code></li></ol><p>详细请看 <a href="">【kafka源码】LeaderAndIsrRequest请求</a></p><h2 id="源码总结"><a href="#源码总结" class="headerlink" title="源码总结"></a>源码总结</h2><blockquote><p>如果上面的源码分析,你不想看,那么你可以直接看这里的简洁叙述</p></blockquote><ol><li>根据是否有传入参数<code>--zookeeper</code> 来判断创建哪一种 对象<code>topicService</code><br>如果传入了<code>--zookeeper</code> 则创建 类 <code>ZookeeperTopicService</code>的对象<br>否则创建类<code>AdminClientTopicService</code>的对象(我们主要分析这个对象)</li><li>如果有入参<code>--command-config</code> ,则将这个文件里面的参数都放到mapl类型 <code>commandConfig</code>里面, 并且也加入<code>bootstrap.servers</code>的参数;假如配置文件里面已经有了<code>bootstrap.servers</code>配置，那么会将其覆盖</li><li>将上面的<code>commandConfig </code>作为入参调用<code>Admin.create(commandConfig)</code>创建 Admin; 这个时候调用的Client模块的代码了, 从这里我们就可以猜测,我们调用<code>kafka-topic.sh</code>脚本实际上是kafka模拟了一个客户端Client来创建Topic的过程;</li><li>一些异常检查<br>①.如果配置了副本副本数–replication-factor 一定要大于0<br>②.如果配置了–partitions 分区数 必须大于0<br>③.去zk查询是否已经存在该Topic</li><li>判断是否配置了参数<code>--replica-assignment</code> ; 如果配置了,那么Topic就会按照指定的方式来配置副本情况</li><li>解析配置<code>--config </code> 配置放到<code>configsMap</code>中; configsMap给到NewTopic对象</li><li><strong>将上面所有的参数包装成一个请求参数<code>CreateTopicsRequest</code> ；然后找到是<code>Controller</code>的节点发起请求(<code>ControllerNodeProvider</code>)</strong></li><li>服务端收到请求之后,开始根据<code>CreateTopicsRequest</code>来调用创建Topic的方法; 不过首先要判断一下自己这个时候是不是<code>Controller</code>; 有可能这个时候Controller重新选举了; 这个时候要抛出异常</li><li>服务端进行一下请求参数检查<br>①.检查Topic是否存在<br>②.检查 <code>--replica-assignment</code>参数和 (<code>--partitions</code> || <code>--replication-factor</code> ) 不能同时使用</li><li>如果(<code>--partitions</code> || <code>--replication-factor</code> ) 没有设置,则使用 Broker的默认配置(这个Broker肯定是Controller) </li><li>计算分区副本分配方式;如果是传入了 <code>--replica-assignment</code>;则会安装自定义参数进行组装;否则的话系统会自动计算分配方式; 具体详情请看 <a href="">【kafka源码】创建Topic的时候是如何分区和副本的分配规则 </a></li><li><code>createTopicPolicy </code>根据Broker是否配置了创建Topic的自定义校验策略; 使用方式是自定义实现<code>org.apache.kafka.server.policy.CreateTopicPolicy</code>接口;并 在服务器配置 <code>create.topic.policy.class.name</code>=自定义类; 比如我就想所有创建Topic的请求分区数都要大于10; 那么这里就可以实现你的需求了</li><li><strong>zk中写入Topic配置信息</strong> 发起<code>CreateRequest</code>请求,这里写入的数据,是我们入参时候传的topic配置<code>--config</code>; 这里的配置会覆盖默认配置；并且节点类型是持久节点;<strong>path</strong> = <code>/config/topics/Topic名称</code></li><li><strong>zk中写入Topic分区副本信息</strong> 发起<code>CreateRequest</code>请求 ,将已经分配好的副本分配策略 写入到 <code>/brokers/topics/Topic名称 </code>中; 节点类型 持久节点</li><li><code>Controller</code>监听zk上面的topic信息; 根据zk上变更的topic信息;计算出新增/删除了哪些Topic; 然后拿到新增Topic的 副本分配信息; 并做一些状态流转</li><li>向新增Topic所在Broker发送<code>leaderAndIsrRequest</code>请求, </li><li>Broker收到<code>发送leaderAndIsrRequest请求</code>; 创建副本Log文件;</li></ol><p><img src="https://img-blog.csdnimg.cn/9ffb4366024d4ff3992231f73cea1c02.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70#pic_center" alt="在这里插入图片描述"></p><h2 id="Q-amp-A"><a href="#Q-amp-A" class="headerlink" title="Q&A"></a>Q&A</h2><h3 id="创建Topic的时候-在Zk上创建了哪些节点"><a href="#创建Topic的时候-在Zk上创建了哪些节点" class="headerlink" title="创建Topic的时候 在Zk上创建了哪些节点"></a>创建Topic的时候 在Zk上创建了哪些节点</h3><blockquote><p>接受客户端请求阶段：</p><ol><li>topic的配置信息 <code> /config/topics/Topic名称</code> 持久节点</li><li>topic的分区信息<code>/brokers/topics/Topic名称</code> 持久节点</li></ol><p>Controller监听zk节点<code>/brokers/topics</code>变更阶段</p><ol><li><code>/brokers/topics/{topicName}/partitions/ </code>持久节点; 无数据</li><li>向zk中写入<code>/brokers/topics/{topicName}/partitions/{分区号}</code> 持久节点; 无数据</li><li>向zk中写入<code>/brokers/topics/{topicName}/partitions/{分区号}/state</code> 持久节点;</li></ol></blockquote><h3 id="创建Topic的时候-什么时候在Broker磁盘上创建的日志文件"><a href="#创建Topic的时候-什么时候在Broker磁盘上创建的日志文件" class="headerlink" title="创建Topic的时候 什么时候在Broker磁盘上创建的日志文件"></a>创建Topic的时候 什么时候在Broker磁盘上创建的日志文件</h3><blockquote><p>当Controller监听zk节点<code>/brokers/topics</code>变更之后,将新增的Topic 解析好的分区状态流转<br><code>NonExistentPartition</code>-><code>NewPartition</code>-><code>OnlinePartition</code> 当流转到<code>OnlinePartition</code>的时候会像分区分配到的Broker发送一个<code>leaderAndIsrRequest</code>请求,当Broker们收到这个请求之后,根据请求参数做一些处理,其中就包括检查自身有没有这个分区副本的本地Log;如果没有的话就重新创建;</p></blockquote><h3 id="如果我没有指定分区数或者副本数-那么会如何创建"><a href="#如果我没有指定分区数或者副本数-那么会如何创建" class="headerlink" title="如果我没有指定分区数或者副本数,那么会如何创建"></a>如果我没有指定分区数或者副本数,那么会如何创建</h3><blockquote><p>我们都知道,如果我们没有指定分区数或者副本数, 则默认使用Broker的配置, 那么这么多Broker,假如不小心默认值配置不一样,那究竟使用哪一个呢？ 那肯定是哪台机器执行创建topic的过程,就是使用谁的配置;<br><strong>所以是谁执行的？</strong> 那肯定是Controller啊! 上面的源码我们分析到了,创建的过程,会指定Controller这台机器去进行;</p></blockquote><h3 id="如果我手动删除了-brokers-topics-下的某个节点会怎么样？"><a href="#如果我手动删除了-brokers-topics-下的某个节点会怎么样？" class="headerlink" title="如果我手动删除了/brokers/topics/下的某个节点会怎么样？"></a>如果我手动删除了<code>/brokers/topics/</code>下的某个节点会怎么样？</h3><blockquote><p>详情请看 <a href="">【kafka实战】一不小心删除了<code>/brokers/topics/</code>下的某个Topic</a></p></blockquote><h3 id="如果我手动在zk中添加-brokers-topics-TopicName-节点会怎么样"><a href="#如果我手动在zk中添加-brokers-topics-TopicName-节点会怎么样" class="headerlink" title="如果我手动在zk中添加/brokers/topics/{TopicName}节点会怎么样"></a>如果我手动在zk中添加<code>/brokers/topics/{TopicName}</code>节点会怎么样</h3><blockquote><p><strong>先说结论:</strong> 根据上面分析过的源码画出的时序图可以指定; 客户端发起创建Topic的请求,本质上是去zk里面写两个数据</p><ol><li>topic的配置信息 <code> /config/topics/Topic名称</code> 持久节点</li><li>topic的分区信息<code>/brokers/topics/Topic名称</code> 持久节点<br> 所以我们绕过这一步骤直接去写入数据,可以达到一样的效果;不过我们的数据需要保证准确<br> 因为在这一步已经没有了一些基本的校验了; 假如这一步我们写入的副本Brokerid不存在会怎样,从时序图中可以看到，<code>leaderAndIsrRequest请求</code>; 就不会正确的发送的不存在的BrokerId上,那么那台机器就不会创建Log文件;</li></ol><p><strong>下面不妨让我们来验证一下;</strong><br>创建一个节点<code>/brokers/topics/create_topic_byhand_zk</code> 节点数据为下面数据; </p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">>{"version":2,"partitions":{"2":[3],"1":[3],"0":[3]},"adding_replicas":{},"removing_replicas":{}}</span><br></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20210617112646965.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br>这里我用的工具<code>PRETTYZOO</code>手动创建的,你也可以用命令行创建;<br>创建完成之后我们再看看本地有没有生成一个Log文件<br><img src="https://img-blog.csdnimg.cn/20210617112806599.png" alt="在这里插入图片描述"><br>可以看到我们指定的Broker,已经生成了对应的分区副本Log文件;<br>而且zk中也写入了其他的数据<img src="https://img-blog.csdnimg.cn/20210617113415168.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY=,size_16,color_FFFFFF,t_70" alt="在这里插入图片描述"><br><code>在我们写入zk数据的时候，就已经确定好了哪个每个分区的Leader是谁了,那就是第一个副本默认为Leader</code></p></blockquote><h3 id="如果写入-brokers-topics-TopicName-节点之后Controller挂掉了会怎么样"><a href="#如果写入-brokers-topics-TopicName-节点之后Controller挂掉了会怎么样" class="headerlink" title="如果写入/brokers/topics/{TopicName}节点之后Controller挂掉了会怎么样"></a>如果写入<code>/brokers/topics/{TopicName}</code>节点之后Controller挂掉了会怎么样</h3><blockquote><p><strong>先说结论</strong>：Controller 重新选举的时候,会有一些初始化的操作; 会把创建过程继续下去</p></blockquote><blockquote><p>然后我们来模拟这么一个过程,先停止集群,然后再zk中写入<code>/brokers/topics/{TopicName}</code>节点数据; 然后再启动一台Broker;<br><strong>源码分析:</strong> 我们之前分析过<a href="">Controller的启动过程与选举</a> 有提到过,这里再提一下Controller当选之后有一个地方处理这个事情</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">replicaStateMachine.startup()partitionStateMachine.startup()</span><br></pre></td></tr></table></figure><p>启动状态机的过程是不是跟上面的<strong>6.1 onNewPartitionCreation 状态流转</strong> 的过程很像; 最终都把状态流转到了<code>OnlinePartition</code>; 伴随着是不发起了<code>leaderAndIsrRequest</code>请求; 是不是Broker收到请求之后,创建本地Log文件了</p></blockquote><h2 id="附件"><a href="#附件" class="headerlink" title="附件"></a>附件</h2><h3 id="–config-可生效参数"><a href="#–config-可生效参数" class="headerlink" title="–config 可生效参数"></a>–config 可生效参数</h3><p>请以<code>sh bin/kafka-topic -help</code> 为准</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">configurations: </span><br><span class="line"> cleanup.policy </span><br><span class="line"> compression.type </span><br><span class="line"> delete.retention.ms </span><br><span class="line"> file.delete.delay.ms </span><br><span class="line"> flush.messages </span><br><span class="line"> flush.ms </span><br><span class="line"> follower.replication.throttled. </span><br><span class="line">replicas </span><br><span class="line"> index.interval.bytes </span><br><span class="line"> leader.replication.throttled.replicas </span><br><span class="line"> max.compaction.lag.ms </span><br><span class="line"> max.message.bytes </span><br><span class="line"> message.downconversion.enable </span><br><span class="line"> message.format.version </span><br><span class="line"> message.timestamp.difference.max.ms </span><br><span class="line"> message.timestamp.type </span><br><span class="line"> min.cleanable.dirty.ratio </span><br><span class="line"> min.compaction.lag.ms </span><br><span class="line"> min.insync.replicas </span><br><span class="line"> preallocate </span><br><span class="line"> retention.bytes </span><br><span class="line"> retention.ms </span><br><span class="line"> segment.bytes </span><br><span class="line"> segment.index.bytes </span><br><span class="line"> segment.jitter.ms </span><br><span class="line"> segment.ms </span><br><span class="line"> unclean.leader.election.enable</span><br></pre></td></tr></table></figure><hr><p><font color=red size=5>Tips:如果关于本篇文章你有疑问,可以在评论区留下,我会在<strong>Q&A</strong>部分进行解答 </font></p><p><font color=red size=2>PS: 文章阅读的源码版本是kafka-2.5 </font></p>]]></content>
<summary type="html">一文带你搞懂kafka创建topic的整个流程</summary>
<category term="kafka" scheme="http://example.com/categories/kafka/"/>
<category term="kafka" scheme="http://example.com/tags/kafka/"/>
<category term="大数据" scheme="http://example.com/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="源码" scheme="http://example.com/tags/%E6%BA%90%E7%A0%81/"/>
</entry>
</feed>