当es Node刚启动或Node ping master节点超时时,会触发当前节点重新加入集群。本文就是对加入集群这一过程的描述。
es中加入集群在org.elasticsearch.discovery.zen.ZenDiscovery#innerJoinCluster中实现。
private void innerJoinCluster() {
DiscoveryNode masterNode = null;
final Thread currentThread = Thread.currentThread();
① nodeJoinController.startElectionContext();
② while (masterNode == null && joinThreadControl.joinThreadActive(currentThread)) {
masterNode = findMaster();
}
if (!joinThreadControl.joinThreadActive(currentThread)) {
logger.trace("thread is no longer in currentJoinThread. Stopping.");
return;
}
if (transportService.getLocalNode().equals(masterNode)) {
final int requiredJoins = Math.max(, electMaster.minimumMasterNodes() - 1); // we count as one
logger.debug("elected as master, waiting for incoming joins ([{}] needed)", requiredJoins);
nodeJoinController.waitToBeElectedAsMaster(requiredJoins, masterElectionWaitForJoinsTimeout,
new NodeJoinController.ElectionCallback() {
@Override
public void onElectedAsMaster(ClusterState state) {
synchronized (stateMutex) {
joinThreadControl.markThreadAsDone(currentThread);
}
}
@Override
public void onFailure(Throwable t) {
logger.trace("failed while waiting for nodes to join, rejoining", t);
synchronized (stateMutex) {
joinThreadControl.markThreadAsDoneAndStartNew(currentThread);
}
}
}
);
} else {
// process any incoming joins (they will fail because we are not the master)
nodeJoinController.stopElectionContext(masterNode + " elected");
// send join request
final boolean success = joinElectedMaster(masterNode);
synchronized (stateMutex) {
if (success) {
DiscoveryNode currentMasterNode = this.clusterState().getNodes().getMasterNode();
if (currentMasterNode == null) {
// Post 1.3.0, the master should publish a new cluster state before acking our join request. we now should have
// a valid master.
logger.debug("no master node is set, despite of join request completing. retrying pings.");
joinThreadControl.markThreadAsDoneAndStartNew(currentThread);
} else if (currentMasterNode.equals(masterNode) == false) {
// update cluster state
joinThreadControl.stopRunningThreadAndRejoin("master_switched_while_finalizing_join");
}
joinThreadControl.markThreadAsDone(currentThread);
} else {
// failed to join. Try again...
joinThreadControl.markThreadAsDoneAndStartNew(currentThread);
}
}
}
}具体步骤如下
1、nodeJoinController创建新的ElectionContext实例,electionContext作用后续说明
2、此方法的核心处findMaster,此方法会找到当前集群中的masterNode,如果不存在则选举一个。详见findMaster节。
下面2步实现非常繁琐且跨网络协调,只说下基本过程:
没有被选举为master的节点向master节点发送join请求并阻塞。
如果当前节点是master,则等待非master节点的join至最大数量并阻塞;同时master节点收到join请求时,如果当前master不处于选举情形,则直接向Join节点发送集群状态,然后响应Join请求,如果处于等待join状态时,则等到达法定节点数量时,则向所有Join节点发送集群状态,然后响应所有的Join请求。
功能:寻找当前集群中活着的Node,如果这些Node不存在masterNode则选举一个。
private DiscoveryNode findMaster() {
logger.trace("starting to ping");
① List<ZenPing.PingResponse> fullPingResponses = pingAndWait(pingTimeout).toList();
final DiscoveryNode localNode = transportService.getLocalNode();
fullPingResponses.add(new ZenPing.PingResponse(localNode, null, this.clusterState()));
// filter responses
final List<ZenPing.PingResponse> pingResponses = filterPingResponses(fullPingResponses, masterElectionIgnoreNonMasters, logger);
② List<DiscoveryNode> activeMasters = new ArrayList<>();
for (ZenPing.PingResponse pingResponse : pingResponses) {
// We can't include the local node in pingMasters list, otherwise we may up electing ourselves without
// any check / verifications from other nodes in ZenDiscover#innerJoinCluster()
if (pingResponse.master() != null && !localNode.equals(pingResponse.master())) {
activeMasters.add(pingResponse.master());
}
}
// nodes discovered during pinging
③ List<ElectMasterService.MasterCandidate> masterCandidates = new ArrayList<>();
for (ZenPing.PingResponse pingResponse : pingResponses) {
if (pingResponse.node().isMasterNode()) {
masterCandidates.add(new ElectMasterService.MasterCandidate(pingResponse.node(), pingResponse.getClusterStateVersion()));
}
}
if (activeMasters.isEmpty()) {
if (electMaster.hasEnoughCandidates(masterCandidates)) {
④ final ElectMasterService.MasterCandidate winner = electMaster.electMaster(masterCandidates);
logger.trace("candidate {} won election", winner);
return winner.getNode();
} else {
// if we don't have enough master nodes, we bail, because there are not enough master to elect from
logger.warn("not enough master nodes discovered during pinging (found [{}], but needed [{}]), pinging again",
masterCandidates, electMaster.minimumMasterNodes());
return null;
}
} else {
assert !activeMasters.contains(localNode) : "local node should never be elected as master when other nodes indicate an active master";
// lets tie break between discovered nodes
⑤ return electMaster.tieBreakActiveMasters(activeMasters);
}
}1、当前node获取集群中其他节点的ZenPing.PingResponse列表。ZenPing.PingResponse主要属性如下:
DiscoveryNode node; :响应node信息
DiscoveryNode master;:响应node master
long clusterStateVersion; :响应node 集群状态版本
2、pingResponses中筛选出其他节点中标记的master node集合:activeMasters
3、构造候选master列表:masterCandidates。(候选master是通过配置选项 node.master(true)来指定的,默认是true。)
4、如果集群中activeMasters为空(当节点加入集群情形),且masterCandidates的数量不小于法定个数(即候选master节点的一半,阻止split brain)(hasEnoughCandidates)(法定个数是有配置选项discovery.zen.minimum_master_nodes指定,默认1,当候选master大于3时必须修改。)时,此时进行master节点的选举(electMaster.electMaster)。
选举的策略非常简单,
public MasterCandidate electMaster(Collection<MasterCandidate> candidates) {
List<MasterCandidate> sortedCandidates = new ArrayList<>(candidates);
sortedCandidates.sort(MasterCandidate::compare);
return sortedCandidates.get();
}取最大的候选节点排序结果值。排序实现如下:
public static int compare(MasterCandidate c1, MasterCandidate c2) {
int ret = Long.compare(c2.clusterStateVersion, c1.clusterStateVersion);
if (ret == ) {
ret = compareNodes(c1.getNode(), c2.getNode());
}
return ret;
}取最大集群状态版本号(每次集群状态变更,版本号也会递增),相同则取节点ID(节点ID为64位UUID值)。
5、如果master node存活,则同样选取上面排序号的activeMasters的第一个。
pingAndWait的实现在org.elasticsearch.discovery.zen.UnicastZenPing#pin 中。
protected void ping(final Consumer<PingCollection> resultsConsumer,
final TimeValue scheduleDuration,
final TimeValue requestDuration) {
final List<DiscoveryNode> seedNodes;
try {
① seedNodes = resolveHostsLists(
unicastZenPingExecutorService,
logger,
configuredHosts,
limitPortCounts,
transportService,
UNICAST_NODE_PREFIX,
resolveTimeout);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
seedNodes.addAll(hostsProvider.buildDynamicNodes());
final DiscoveryNodes nodes = contextProvider.clusterState().nodes();
// add all possible master nodes that were active in the last known cluster configuration
for (ObjectCursor<DiscoveryNode> masterNode : nodes.getMasterNodes().values()) {
seedNodes.add(masterNode.value);
}
final ConnectionProfile connectionProfile =
ConnectionProfile.buildSingleChannelProfile(TransportRequestOptions.Type.REG, requestDuration, requestDuration);
② final PingingRound pingingRound = new PingingRound(pingingRoundIdGenerator.incrementAndGet(), seedNodes, resultsConsumer,
nodes.getLocalNode(), connectionProfile);
activePingingRounds.put(pingingRound.id(), pingingRound);
final AbstractRunnable pingSender = new AbstractRunnable() {
@Override
public void onFailure(Exception e) {
if (e instanceof AlreadyClosedException == false) {
logger.warn("unexpected error while pinging", e);
}
}
@Override
protected void doRun() throws Exception {
sendPings(requestDuration, pingingRound);
}
};
③ threadPool.generic().execute(pingSender);
threadPool.schedule(TimeValue.timeValueMillis(scheduleDuration.millis() / 3), ThreadPool.Names.GENERIC, pingSender);
threadPool.schedule(TimeValue.timeValueMillis(scheduleDuration.millis() / 3 * 2), ThreadPool.Names.GENERIC, pingSender);
threadPool.schedule(scheduleDuration, ThreadPool.Names.GENERIC, new AbstractRunnable() {
@Override
protected void doRun() throws Exception {
finishPingingRound(pingingRound);
}
@Override
public void onFailure(Exception e) {
logger.warn("unexpected error while finishing pinging round", e);
}
});
}1、node ip port发现。不细展开,基本过程如下:
根据配置选项 发现域名:discovery.zen.ping.unicast.hosts(默认["127.0.0.1", "[::1]"])和发现端口transport.profiles.default.port(默认9300-9400),来候选节点位置,如果配置的IP则取前5个端口,如果是域名,则DNS获取所有对应IP下的1个端口集合。
2、构造PingingRound实例,PingingRound主要属性:
List<DiscoveryNode> seedNodes:要发现的Node列表
PingCollection pingCollection:ping 响应集合
Consumer<PingCollection> pingListener:当ping结束后回调方法。
3、3次线程调度pingSender,并在scheduleDuration(超时)结束后AbstractRunnable回调。
在pingSender中向每个发现节点Ping,如果有响应并且是同一个是集群则计入PingingRound.pingCollection实例中。
来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/31509949/viewspace-3124168/ ,如需转载,请注明出处,否则将追究法律责任。