behave test will hang on the third node

[liangxin1300]

# behave bootstrap_sbd_normal.feature 
USING RUNNER: behave.runner:Runner
@sbd
Feature: crmsh bootstrap sbd management # bootstrap_sbd_normal.feature:2
  Tag @clean means need to stop cluster service if the service is available
  @clean
  Scenario: Configure diskless sbd(bsc#1181907)                                           # bootstrap_sbd_normal.feature:7
    Given Cluster service is "stopped" on "sle16-1"                                       # steps/step_implementation.py:45 0.046s
    Given Cluster service is "stopped" on "sle16-2"                                       # steps/step_implementation.py:45 0.042s
    When Run "crm cluster init -S -y" on "sle16-1"                                        # steps/step_implementation.py:109 29.595s
    Then Expected "Diskless SBD requires cluster with three or more nodes." in stderr     # steps/step_implementation.py:170 0.000s
    Then Cluster service is "started" on "sle16-1"                                        # steps/step_implementation.py:237 0.038s
    And Service "sbd" is "started" on "sle16-1"                                           # steps/step_implementation.py:242 0.034s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-2"                                # steps/step_implementation.py:109
    When Run "crm cluster join -c sle16-1 -y" on "sle16-2"                                # steps/step_implementation.py:109 29.681s
    Then Expected "Diskless SBD requires cluster with three or more nodes." in stderr     # steps/step_implementation.py:170 0.000s
    Then Cluster service is "started" on "sle16-2"                                        # steps/step_implementation.py:237 0.039s
    And Service "sbd" is "started" on "sle16-2"                                           # steps/step_implementation.py:242 0.035s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-3"                                # steps/step_implementation.py:109

As stated in the title, the behave test hangs on the third node (the same hang occurs regardless of whether sle16-2 or sle16-3 is the third node). In reality, crmsh has already completed the join process on the third node.

[liangxin1300]

The same issue when I simplify the test script:

    When Run "crm cluster init -s /dev/sda5 -y" on "sle16-1"         # steps/step_implementation.py:109 30.469s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-2"           # steps/step_implementation.py:109 30.708s
    When Run "crm cluster join ssh -c sle16-1 -y" on "sle16-3"       # steps/step_implementation.py:109 10.471s
    When Run "crm cluster join firewalld -c sle16-1 -y" on "sle16-3" # steps/step_implementation.py:109 1.030s
    When Run "crm cluster join cluster -c sle16-1 -y" on "sle16-3"   # steps/step_implementation.py:109

The above test hangs at the last line, and the third node already finished the join process

[liangxin1300]

Passed without sbd:

    When Run "crm cluster init -y" on "sle16-1"             # steps/step_implementation.py:109 25.228s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-2"  # steps/step_implementation.py:109 26.111s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-3"  # steps/step_implementation.py:109 32.607s

Further findings: The process hangs at the last line if I insert time.sleep(200) before the join process completes. In the previous cases where the third node was hanging, the issue occurred because the pacemaker service took a long time to start(with sbd configured), causing the entire join process to take longer to finish.

[liangxin1300]

# behave bootstrap_sbd_normal.feature 
USING RUNNER: behave.runner:Runner
@sbd
Feature: crmsh bootstrap sbd management # bootstrap_sbd_normal.feature:2
  Tag @clean means need to stop cluster service if the service is available
  @clean
  Scenario: Configure diskless sbd(bsc#1181907)                                           # bootstrap_sbd_normal.feature:7
    Given Cluster service is "stopped" on "sle16-1"                                       # steps/step_implementation.py:45 0.046s
    Given Cluster service is "stopped" on "sle16-2"                                       # steps/step_implementation.py:45 0.042s
    When Run "crm cluster init -S -y" on "sle16-1"                                        # steps/step_implementation.py:109 29.595s
    Then Expected "Diskless SBD requires cluster with three or more nodes." in stderr     # steps/step_implementation.py:170 0.000s
    Then Cluster service is "started" on "sle16-1"                                        # steps/step_implementation.py:237 0.038s
    And Service "sbd" is "started" on "sle16-1"                                           # steps/step_implementation.py:242 0.034s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-2"                                # steps/step_implementation.py:109
    When Run "crm cluster join -c sle16-1 -y" on "sle16-2"                                # steps/step_implementation.py:109 29.681s
    Then Expected "Diskless SBD requires cluster with three or more nodes." in stderr     # steps/step_implementation.py:170 0.000s
    Then Cluster service is "started" on "sle16-2"                                        # steps/step_implementation.py:237 0.039s
    And Service "sbd" is "started" on "sle16-2"                                           # steps/step_implementation.py:242 0.035s
    When Run "crm cluster join -c sle16-1 -y" on "sle16-3"                                # steps/step_implementation.py:109

As stated in the title, the behave test hangs on the third node (the same hang occurs regardless of whether sle16-2 or sle16-3 is the third node). In reality, crmsh has already completed the join process on the third node.

It works if I close firewalld on each node, it's weird that I already opened 1122/tcp port before