Using the original execution date of: 2020-06-16 19:23:21Z
Current cluster status:
RemoteNode remote-rhel7-2: UNCLEAN (offline)
Node rhel7-4 (4): UNCLEAN (offline)
Online: [ rhel7-1 rhel7-3 rhel7-5 ]
GuestOnline: [ lxc1:container1 stateful-bundle-1:stateful-bundle-docker-1 ]
Fencing (stonith:fence_xvm): Started rhel7-4 (UNCLEAN)
Clone Set: rsc1-clone [rsc1] (promotable)
rsc1 (ocf::pacemaker:Stateful): Master rhel7-4 (UNCLEAN)
rsc1 (ocf::pacemaker:Stateful): Slave remote-rhel7-2 (UNCLEAN)
Slaves: [ lxc1 rhel7-1 rhel7-3 rhel7-5 ]
Clone Set: rsc2-master [rsc2] (promotable)
rsc2 (ocf::pacemaker:Stateful): Slave rhel7-4 (UNCLEAN)
rsc2 (ocf::pacemaker:Stateful): Master remote-rhel7-2 (UNCLEAN)
Slaves: [ lxc1 rhel7-1 rhel7-3 rhel7-5 ]
remote-rhel7-2 (ocf::pacemaker:remote): FAILED rhel7-1
container1 (ocf::heartbeat:VirtualDomain): Started rhel7-3
container2 (ocf::heartbeat:VirtualDomain): FAILED rhel7-3
Clone Set: lxc-ms-master [lxc-ms] (promotable)
Slaves: [ lxc1 ]
Stopped: [ remote-rhel7-2 rhel7-1 rhel7-3 rhel7-4 rhel7-5 ]
Container bundle set: stateful-bundle [pcmktest:http]
stateful-bundle-0 (192.168.122.131) (ocf::pacemaker:Stateful): FAILED Master rhel7-5
stateful-bundle-1 (192.168.122.132) (ocf::pacemaker:Stateful): Slave rhel7-1
stateful-bundle-2 (192.168.122.133) (ocf::pacemaker:Stateful): FAILED rhel7-4 (UNCLEAN)
Transition Summary:
* Fence (reboot) stateful-bundle-2 (resource: stateful-bundle-docker-2) 'guest is unclean'
* Fence (reboot) stateful-bundle-0 (resource: stateful-bundle-docker-0) 'guest is unclean'
* Fence (reboot) lxc2 (resource: container2) 'guest is unclean'
* Fence (reboot) remote-rhel7-2 'remote connection is unrecoverable'
* Fence (reboot) rhel7-4 'peer is no longer part of the cluster'
* Move Fencing ( rhel7-4 -> rhel7-5 )
* Stop rsc1:0 ( Master rhel7-4 ) due to node availability
* Promote rsc1:1 ( Slave -> Master rhel7-3 )
* Stop rsc1:4 ( Slave remote-rhel7-2 ) due to node availability
* Recover rsc1:5 ( Slave lxc2 )
* Stop rsc2:0 ( Slave rhel7-4 ) due to node availability
* Promote rsc2:1 ( Slave -> Master rhel7-3 )
* Stop rsc2:4 ( Master remote-rhel7-2 ) due to node availability
* Recover rsc2:5 ( Slave lxc2 )
* Recover remote-rhel7-2 ( rhel7-1 )
* Recover container2 ( rhel7-3 )
* Recover lxc-ms:0 ( Master lxc2 )
* Recover stateful-bundle-docker-0 ( rhel7-5 )
* Restart stateful-bundle-0 ( rhel7-5 ) due to required stateful-bundle-docker-0 start
* Recover bundled:0 ( Master stateful-bundle-0 )
* Move stateful-bundle-ip-192.168.122.133 ( rhel7-4 -> rhel7-3 )
* Recover stateful-bundle-docker-2 ( rhel7-4 -> rhel7-3 )
* Move stateful-bundle-2 ( rhel7-4 -> rhel7-3 )
* Recover bundled:2 ( Slave stateful-bundle-2 )
* Restart lxc2 ( rhel7-3 ) due to required container2 start
Executing cluster transition:
* Pseudo action: Fencing_stop_0
* Resource action: rsc1 cancel=11000 on rhel7-3
* Pseudo action: rsc1-clone_demote_0
* Resource action: rsc2 cancel=11000 on rhel7-3
* Pseudo action: rsc2-master_demote_0
* Pseudo action: lxc-ms-master_demote_0
* Resource action: stateful-bundle-0 stop on rhel7-5
* Pseudo action: stateful-bundle-2_stop_0
* Resource action: lxc2 stop on rhel7-3
* Pseudo action: stateful-bundle_demote_0
* Fencing remote-rhel7-2 (reboot)
* Fencing rhel7-4 (reboot)
* Pseudo action: rsc1_demote_0
* Pseudo action: rsc1-clone_demoted_0
* Pseudo action: rsc2_demote_0
* Pseudo action: rsc2-master_demoted_0
* Resource action: container2 stop on rhel7-3
* Pseudo action: stateful-bundle-master_demote_0
* Pseudo action: stonith-stateful-bundle-2-reboot on stateful-bundle-2
* Pseudo action: stonith-lxc2-reboot on lxc2
* Resource action: Fencing start on rhel7-5
* Pseudo action: rsc1-clone_stop_0
* Pseudo action: rsc2-master_stop_0
* Pseudo action: lxc-ms_demote_0
* Pseudo action: lxc-ms-master_demoted_0
* Pseudo action: lxc-ms-master_stop_0
* Pseudo action: bundled_demote_0
* Pseudo action: stateful-bundle-master_demoted_0
* Pseudo action: stateful-bundle_demoted_0
* Pseudo action: stateful-bundle_stop_0
* Resource action: Fencing monitor=120000 on rhel7-5
* Pseudo action: rsc1_stop_0
* Pseudo action: rsc1_stop_0
* Pseudo action: rsc1_stop_0
* Pseudo action: rsc1-clone_stopped_0
* Pseudo action: rsc1-clone_start_0
* Pseudo action: rsc2_stop_0
* Pseudo action: rsc2_stop_0
* Pseudo action: rsc2_stop_0
* Pseudo action: rsc2-master_stopped_0
* Pseudo action: rsc2-master_start_0
* Resource action: remote-rhel7-2 stop on rhel7-1
* Pseudo action: lxc-ms_stop_0
* Pseudo action: lxc-ms-master_stopped_0
* Pseudo action: lxc-ms-master_start_0
* Resource action: stateful-bundle-docker-0 stop on rhel7-5
* Pseudo action: stateful-bundle-docker-2_stop_0
* Pseudo action: stonith-stateful-bundle-0-reboot on stateful-bundle-0
* Resource action: remote-rhel7-2 start on rhel7-1
* Resource action: remote-rhel7-2 monitor=60000 on rhel7-1
* Resource action: container2 start on rhel7-3
* Resource action: container2 monitor=20000 on rhel7-3
* Pseudo action: stateful-bundle-master_stop_0
* Pseudo action: stateful-bundle-ip-192.168.122.133_stop_0
* Resource action: lxc2 start on rhel7-3
* Resource action: lxc2 monitor=30000 on rhel7-3
* Resource action: rsc1 start on lxc2
* Pseudo action: rsc1-clone_running_0
* Resource action: rsc2 start on lxc2
* Pseudo action: rsc2-master_running_0
* Resource action: lxc-ms start on lxc2
* Pseudo action: lxc-ms-master_running_0
* Pseudo action: bundled_stop_0
* Resource action: stateful-bundle-ip-192.168.122.133 start on rhel7-3
* Resource action: rsc1 monitor=11000 on lxc2
* Pseudo action: rsc1-clone_promote_0
* Resource action: rsc2 monitor=11000 on lxc2
* Pseudo action: rsc2-master_promote_0
* Pseudo action: lxc-ms-master_promote_0
* Pseudo action: bundled_stop_0
* Pseudo action: stateful-bundle-master_stopped_0
* Resource action: stateful-bundle-ip-192.168.122.133 monitor=60000 on rhel7-3
* Pseudo action: stateful-bundle_stopped_0
* Pseudo action: stateful-bundle_start_0
* Resource action: rsc1 promote on rhel7-3
* Pseudo action: rsc1-clone_promoted_0
* Resource action: rsc2 promote on rhel7-3
* Pseudo action: rsc2-master_promoted_0
* Resource action: lxc-ms promote on lxc2
* Pseudo action: lxc-ms-master_promoted_0
* Pseudo action: stateful-bundle-master_start_0
* Resource action: stateful-bundle-docker-0 start on rhel7-5
* Resource action: stateful-bundle-docker-0 monitor=60000 on rhel7-5
* Resource action: stateful-bundle-0 start on rhel7-5
* Resource action: stateful-bundle-0 monitor=30000 on rhel7-5
* Resource action: stateful-bundle-docker-2 start on rhel7-3
* Resource action: stateful-bundle-2 start on rhel7-3
* Resource action: rsc1 monitor=10000 on rhel7-3
* Resource action: rsc2 monitor=10000 on rhel7-3
* Resource action: lxc-ms monitor=10000 on lxc2
* Resource action: bundled start on stateful-bundle-0
* Resource action: bundled start on stateful-bundle-2
* Pseudo action: stateful-bundle-master_running_0
* Resource action: stateful-bundle-docker-2 monitor=60000 on rhel7-3
* Resource action: stateful-bundle-2 monitor=30000 on rhel7-3
* Pseudo action: stateful-bundle_running_0
* Resource action: bundled monitor=11000 on stateful-bundle-2
* Pseudo action: stateful-bundle_promote_0
* Pseudo action: stateful-bundle-master_promote_0
* Resource action: bundled promote on stateful-bundle-0
* Pseudo action: stateful-bundle-master_promoted_0
* Pseudo action: stateful-bundle_promoted_0
* Resource action: bundled monitor=10000 on stateful-bundle-0
Using the original execution date of: 2020-06-16 19:23:21Z
Revised cluster status:
Online: [ rhel7-1 rhel7-3 rhel7-5 ]
OFFLINE: [ rhel7-4 ]
RemoteOnline: [ remote-rhel7-2 ]
GuestOnline: [ lxc1:container1 lxc2:container2 stateful-bundle-0:stateful-bundle-docker-0 stateful-bundle-1:stateful-bundle-docker-1 stateful-bundle-2:stateful-bundle-docker-2 ]
Fencing (stonith:fence_xvm): Started rhel7-5
Clone Set: rsc1-clone [rsc1] (promotable)
Masters: [ rhel7-3 ]
Slaves: [ lxc1 lxc2 rhel7-1 rhel7-5 ]
Stopped: [ remote-rhel7-2 rhel7-4 ]
Clone Set: rsc2-master [rsc2] (promotable)
Masters: [ rhel7-3 ]
Slaves: [ lxc1 lxc2 rhel7-1 rhel7-5 ]
Stopped: [ remote-rhel7-2 rhel7-4 ]
remote-rhel7-2 (ocf::pacemaker:remote): Started rhel7-1
container1 (ocf::heartbeat:VirtualDomain): Started rhel7-3
container2 (ocf::heartbeat:VirtualDomain): Started rhel7-3
Clone Set: lxc-ms-master [lxc-ms] (promotable)
Masters: [ lxc2 ]
Slaves: [ lxc1 ]
Container bundle set: stateful-bundle [pcmktest:http]
stateful-bundle-0 (192.168.122.131) (ocf::pacemaker:Stateful): Master rhel7-5
stateful-bundle-1 (192.168.122.132) (ocf::pacemaker:Stateful): Slave rhel7-1
stateful-bundle-2 (192.168.122.133) (ocf::pacemaker:Stateful): Slave rhel7-3