SAP HANA HA cluster maintenance - Pacemaker failover recovery - SAP

Maintain SAP HANA HA clusters deployed with SUSE HAE and Pacemaker on Alibaba Cloud ECS instances, including failover recovery, planned shutdowns, and HSR re-registration.

Covered scenarios:

Primary or standby node failure recovery
Planned shutdown maintenance on one or both nodes
ECS instance type upgrades or downgrades
SAP HANA or SAP application upgrades

Scope

Applies to SAP HANA systems deployed on SUSE HAE 12 clusters that require ECS instance type changes, SAP application or database upgrades, node maintenance, or failover recovery.

Maintenance on SUSE HAE cluster nodes may require stopping resources, migrating them to another node, or shutting down the node entirely. You may also need to temporarily take manual control of cluster resources.

All scenarios use SAP HANA HA as an example. SAP application HA maintenance follows similar procedures.

Prerequisites

Ensure you have:

SSH access to both cluster nodes
Verified current primary and standby node roles (run crm status before every procedure)
Familiarity with SAP HANA administration commands (HDB stop, hdbnsutil)

Choose the right maintenance mode

Select the appropriate Pacemaker maintenance mode:

Scenario	Recommended mode	Effect
Maintain all cluster resources (both nodes)	Cluster maintenance mode	Stops Pacemaker resource monitoring for all resources on all nodes.
Maintain resources on a single node, keep services running on the other	Node standby mode	Moves all resources off the node and stops monitoring. The node can no longer run resources.
Maintain a single node without moving resources	Node maintenance mode	Stops monitoring resources on that node but does not move them.
Manually adjust a single resource without monitoring	Resource maintenance mode	No monitoring operations are triggered for the resource.
Manually adjust a single resource while keeping monitoring active	Resource unmanaged state	Monitoring continues and reports errors, but the cluster takes no corrective action.

Important

Maintenance mode only disables Pacemaker resource management. Corosync and STONITH Block Device (SBD) remain active — SBD fencing can still trigger. If maintenance involves storage or networking changes, consider stopping cluster services entirely instead of using maintenance mode.

Environment used in examples

Parameter	Value
Node names	`saphana-01`, `saphana-02`
SAP HANA System Identifier (SID)	`H01`
SAP HANA instance number	`00`
SAP HANA instance user	`h01adm`
HSR replication mode	`syncmem`
HSR operation mode	`logreplay`

Scenario 1: Recover from a primary node failure

When the primary node fails, the cluster automatically promotes the standby node. After the original primary recovers, you must re-register it as the secondary node in HSR before starting Pacemaker. The recovered node still holds a primary role assignment after crash recovery.

Warning

Confirm which node is the current primary before reconfiguring HSR. Incorrect configuration can overwrite or destroy data.

In this example, saphana-01 is the initial primary node and saphana-02 is the standby node.

Step 1: Verify the initial cluster status

Run crm status on either node to confirm the normal HA cluster state.

# crm status
Stack: corosync
Current DC: saphana-01 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 14:33:22 2019
Last change: Mon Apr 15 14:33:19 2019 by root via crm_attribute on saphana-01

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-01
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-01
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-01 ]
     Slaves: [ saphana-02 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

After the primary node fails, the cluster promotes the standby node. The cluster status shows the original primary as OFFLINE:

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 14:40:43 2019
Last change: Mon Apr 15 14:40:41 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-02 ]
OFFLINE: [ saphana-01 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Stopped: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-02 ]
     Stopped: [ saphana-01 ]

Step 2: Re-register HSR on the recovered node

Log on to saphana-01 as the SAP HANA instance user and register it as the secondary:

h01adm@saphana-01:/usr/sap/H01/HDB00> hdbnsutil -sr_register --remoteHost=saphana-02 --remoteInstance=00 --replicationMode=syncmem --name=saphana-01 --operationMode=logreplay
adding site ...
checking for inactive nameserver ...
nameserver saphana-01:30001 not responding.
collecting information ...
updating local ini files ...
done.

Step 3: Clear SBD slots

Check the SBD slot status. If any slot is not clear, set it to clear before proceeding.

Check SBD status:

# sbd -d /dev/vdc list
0       saphana-01      reset   saphana-02
1       saphana-02      reset   saphana-01

Clear the slots:

# sbd -d /dev/vdc message saphana-01 clear
# sbd -d /dev/vdc message saphana-02 clear

Verify both slots are clear:

# sbd -d /dev/vdc list
0       saphana-01      clear   saphana-01
1       saphana-02      clear   saphana-01

Step 4: Start Pacemaker

Start Pacemaker on the recovered node. It automatically starts SAP HANA.

# systemctl start pacemaker

saphana-02 is now the primary. Verify the cluster status:

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 15:10:58 2019
Last change: Mon Apr 15 15:09:56 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Slaves: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 5: Verify HSR replication

Follow the procedure in Verify HSR replication status.

Step 6: Reset the fail count (optional)

Follow the procedure in Reset the resource fail count.

Scenario 2: Recover from a standby node failure

When the standby node fails, the primary continues normally — no failover occurs. After the standby recovers, starting Pacemaker automatically restarts SAP HANA and restores the original roles. No HSR re-registration is required.

In this example, saphana-02 is the primary node and saphana-01 is the standby node.

Step 1: Verify the initial cluster status

Run crm status on either node to confirm the cluster state.

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 15:34:52 2019
Last change: Mon Apr 15 15:33:50 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Slaves: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 2: Clear SBD slots and start Pacemaker

After the standby node recovers, check SBD status (see Scenario 1, Step 3) and clear any non-clear slots. Then start Pacemaker:

# systemctl start pacemaker

HSR maintains the original roles. Verify the cluster status:

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 15:43:28 2019
Last change: Mon Apr 15 15:43:25 2019 by root via crm_attribute on saphana-01

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Slaves: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 3: Verify HSR replication

Follow the procedure in Verify HSR replication status.

Step 4: Reset the fail count (optional)

Follow the procedure in Reset the resource fail count.

Scenario 3: Perform planned shutdown maintenance on both nodes

Shut down both nodes for maintenance such as ECS instance type changes or infrastructure upgrades. Place the cluster in maintenance mode, stop SAP HANA on both nodes, then shut down the instances.

Important

Shut down the standby node first, then the primary. This prevents an unnecessary failover.

In this example, saphana-02 is the primary node and saphana-01 is the standby node.

Step 1: Verify the initial cluster status

Run crm status on either node to confirm the cluster state.

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 15:34:52 2019
Last change: Mon Apr 15 15:33:50 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Slaves: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 2: Enable cluster and resource maintenance mode

On the primary node, enable cluster maintenance mode:

    # crm configure property maintenance-mode=true

Set the SAP HANA resource sets to maintenance mode. In this example, the resources are rsc_SAPHana_HDB and rsc_SAPHanaTopology_HDB:

    # crm resource maintenance rsc_SAPHana_HDB true
    Performing update of 'maintenance' on 'msl_SAPHana_HDB', the parent of 'rsc_SAPHana_HDB'
    Set 'msl_SAPHana_HDB' option: id=msl_SAPHana_HDB-meta_attributes-maintenance name=maintenance=true

    # crm resource maintenance rsc_SAPHanaTopology_HDB true
    Performing update of 'maintenance' on 'cln_SAPHanaTopology_HDB', the parent of 'rsc_SAPHanaTopology_HDB'
    Set 'cln_SAPHanaTopology_HDB' option: id=cln_SAPHanaTopology_HDB-meta_attributes-maintenance name=maintenance=true

Verify that the cluster shows Resource management is DISABLED and all resources are (unmanaged):

    # crm status
    Stack: corosync
    Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
    Last updated: Mon Apr 15 16:02:13 2019
    Last change: Mon Apr 15 16:02:11 2019 by root via crm_resource on saphana-02

    2 nodes configured
    6 resources configured

                  *** Resource management is DISABLED ***
      The cluster will not attempt to start, stop or recover services

    Online: [ saphana-01 saphana-02 ]

    Full list of resources:

    rsc_sbd (stonith:external/sbd): Started saphana-02 (unmanaged)
    rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02 (unmanaged)
     Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB] (unmanaged)
         rsc_SAPHana_HDB    (ocf::suse:SAPHana):    Slave saphana-01 (unmanaged)
         rsc_SAPHana_HDB    (ocf::suse:SAPHana):    Master saphana-02 (unmanaged)
     Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB] (unmanaged)
         rsc_SAPHanaTopology_HDB    (ocf::suse:SAPHanaTopology):    Started saphana-01 (unmanaged)
         rsc_SAPHanaTopology_HDB    (ocf::suse:SAPHanaTopology):    Started saphana-02 (unmanaged)

Step 3: Stop SAP HANA and shut down the ECS instances

Stop SAP HANA on the standby node first, then on the primary node.

Log on to the standby node (saphana-01) as the SAP HANA instance user and stop SAP HANA:

    saphana-01:~ # su - h01adm
    h01adm@saphana-01:/usr/sap/H01/HDB00> HDB stop
    hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
    Stopping instance using: /usr/sap/H01/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

    15.04.2019 16:46:42
    Stop
    OK
    Waiting for stopped instance using: /usr/sap/H01/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


    15.04.2019 16:46:54
    WaitforStopped
    OK
    hdbdaemon is stopped.

Log on to the primary node (saphana-02) as the SAP HANA instance user and stop SAP HANA:

    saphana-02:~ # su - h01adm
    h01adm@saphana-02:/usr/sap/H01/HDB00> HDB stop
    hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
    Stopping instance using: /usr/sap/H01/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

    15.04.2019 16:47:05
    Stop
    OK
    Waiting for stopped instance using: /usr/sap/H01/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


    15.04.2019 16:47:35
    WaitforStopped
    OK
    hdbdaemon is stopped.

Shut down both ECS instances and perform the required maintenance tasks.

Step 4: Start the nodes and restore normal mode

After maintenance is complete:

Start both ECS instances. Start Pacemaker on the primary node first, then the standby:
```
    # systemctl start pacemaker
```

Restore the cluster and resources to normal mode:

    # crm configure property maintenance-mode=false
    # crm resource maintenance rsc_SAPHana_HDB false
    Performing update of 'maintenance' on 'msl_SAPHana_HDB', the parent of 'rsc_SAPHana_HDB'
    Set 'msl_SAPHana_HDB' option: id=msl_SAPHana_HDB-meta_attributes-maintenance name=maintenance=false
    # crm resource maintenance rsc_SAPHanaTopology_HDB false
    Performing update of 'maintenance' on 'cln_SAPHanaTopology_HDB', the parent of 'rsc_SAPHanaTopology_HDB'
    Set 'cln_SAPHanaTopology_HDB' option: id=cln_SAPHanaTopology_HDB-meta_attributes-maintenance name=maintenance=false

The cluster automatically starts SAP HANA and maintains the original roles.

Verify the cluster status:

    # crm status
    Stack: corosync
    Current DC: saphana-01 (version 1.1.16-4.8-77ea74d) - partition with quorum
    Last updated: Mon Apr 15 16:56:49 2019
    Last change: Mon Apr 15 16:56:43 2019 by root via crm_attribute on saphana-01

    2 nodes configured
    6 resources configured

    Online: [ saphana-01 saphana-02 ]

    Full list of resources:

    rsc_sbd (stonith:external/sbd): Started saphana-01
    rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
     Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
         Masters: [ saphana-02 ]
         Slaves: [ saphana-01 ]
     Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
         Started: [ saphana-01 saphana-02 ]

Step 5: Verify HSR replication

Follow the procedure in Verify HSR replication status.

Step 6: Reset the fail count (optional)

Follow the procedure in Reset the resource fail count.

Scenario 4: Perform planned shutdown maintenance on the primary node only

When only the primary node requires shutdown maintenance, set it to standby mode to trigger a failover. After maintenance, re-register HSR and bring the node back online.

Important

Setting the primary node to standby triggers an automatic failover to the standby node.

In this example, saphana-02 is the initial primary node and saphana-01 is the standby node.

Step 1: Verify the initial cluster status

Run crm status on either node to confirm the cluster state.

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 15:34:52 2019
Last change: Mon Apr 15 15:33:50 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Slaves: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 2: Set the primary node to standby mode

Place the primary node in standby mode. The cluster stops SAP HANA and promotes the standby node.

# crm node standby saphana-02

saphana-01 is now the primary:

# crm status
Stack: corosync
Current DC: saphana-01 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 17:07:56 2019
Last change: Mon Apr 15 17:07:38 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Node saphana-02: standby
Online: [ saphana-01 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-01
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-01
 Clone Set: msl_SAPHana_HDB [rsc_SAPHana_HDB] (promotable)
     Masters: [ saphana-01 ]
     Stopped: [ saphana-02 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 ]
     Stopped: [ saphana-02 ]

Step 3: Shut down the ECS instance and perform maintenance

Shut down the ECS instance for the primary node (saphana-02) and perform the required maintenance tasks.

Step 4: Start the node and re-register HSR

After maintenance, start the ECS instance. Log on as the SAP HANA instance user and re-register the node as secondary:

saphana-02:~ # su - h01adm
h01adm@saphana-02:/usr/sap/H01/HDB00> hdbnsutil -sr_register --remoteHost=saphana-01 --remoteInstance=00 --replicationMode=syncmem --name=saphana-02 --operationMode=logreplay

Step 5: Start Pacemaker and bring the node online

Start Pacemaker and restore the node from standby to online:

# systemctl start pacemaker
# crm node online saphana-02

The cluster starts SAP HANA on the restored node as standby. Verify:

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 18:02:33 2019
Last change: Mon Apr 15 18:01:31 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-01
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-01
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-01 ]
     Slaves: [ saphana-02 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 6: Verify HSR replication

Follow the procedure in Verify HSR replication status.

Step 7: Reset the fail count (optional)

Follow the procedure in Reset the resource fail count.

Scenario 5: Perform planned shutdown maintenance on the standby node only

When only the standby node requires shutdown maintenance, place it in maintenance mode, stop SAP HANA, and shut down the instance. The primary continues serving requests throughout.

Important

Place the standby node in maintenance mode before stopping SAP HANA to prevent corrective cluster actions.

In this example, saphana-02 is the primary node and saphana-01 is the standby node.

Step 1: Verify the initial cluster status

Run crm status on either node to confirm the cluster state.

# crm status
Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 15:34:52 2019
Last change: Mon Apr 15 15:33:50 2019 by root via crm_attribute on saphana-02

2 nodes configured
6 resources configured

Online: [ saphana-01 saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     Masters: [ saphana-02 ]
     Slaves: [ saphana-01 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     Started: [ saphana-01 saphana-02 ]

Step 2: Set the standby node to maintenance mode

# crm node maintenance saphana-01

The standby node appears in maintenance mode with unmanaged resources:

Stack: corosync
Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Apr 15 18:18:10 2019
Last change: Mon Apr 15 18:17:49 2019 by root via crm_attribute on saphana-01

2 nodes configured
6 resources configured

Node saphana-01: maintenance
Online: [ saphana-02 ]

Full list of resources:

rsc_sbd (stonith:external/sbd): Started saphana-02
rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
 Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
     rsc_SAPHana_HDB    (ocf::suse:SAPHana):    Slave saphana-01 (unmanaged)
     Masters: [ saphana-02 ]
 Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
     rsc_SAPHanaTopology_HDB    (ocf::suse:SAPHanaTopology):    Started saphana-01 (unmanaged)
     Started: [ saphana-02 ]

Step 3: Stop SAP HANA and shut down the ECS instance

On saphana-01, switch to the SAP HANA instance user and stop SAP HANA:

saphana-01:~ # su - h01adm
h01adm@saphana-01:/usr/sap/H01/HDB00> HDB stop
hdbdaemon will wait maximal 300 seconds for NewDB services finishing.
Stopping instance using: /usr/sap/H01/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function Stop 400

15.04.2019 16:47:05
Stop
OK
Waiting for stopped instance using: /usr/sap/H01/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 00 -function WaitforStopped 600 2


15.04.2019 16:47:35
WaitforStopped
OK
hdbdaemon is stopped.

Shut down the ECS instance and perform the required maintenance tasks.

Step 4: Start the node and restore to normal mode

After maintenance:

Start the ECS instance and start Pacemaker:
```
    # systemctl start pacemaker
```

From the primary node, restore the standby node:

    saphana-02:~ # crm node ready saphana-01

The cluster starts SAP HANA and maintains the original roles.

Verify the cluster status:

    # crm status
    Stack: corosync
    Current DC: saphana-02 (version 1.1.16-4.8-77ea74d) - partition with quorum
    Last updated: Mon Apr 15 18:02:33 2019
    Last change: Mon Apr 15 18:01:31 2019 by root via crm_attribute on saphana-02

    2 nodes configured
    6 resources configured

    Online: [ saphana-01 saphana-02 ]

    Full list of resources:

    rsc_sbd (stonith:external/sbd): Started saphana-02
    rsc_vip (ocf::heartbeat:IPaddr2):       Started saphana-02
     Master/Slave Set: msl_SAPHana_HDB [rsc_SAPHana_HDB]
         Masters: [ saphana-02 ]
         Slaves: [ saphana-01 ]
     Clone Set: cln_SAPHanaTopology_HDB [rsc_SAPHanaTopology_HDB]
         Started: [ saphana-01 saphana-02 ]

Step 5: Verify HSR replication

Follow the procedure in Verify HSR replication status.

Step 6: Reset the fail count (optional)

Follow the procedure in Reset the resource fail count.

Common procedures

The following procedures are referenced by multiple scenarios.

Verify HSR replication status

Use both methods to confirm replication is healthy.

Method 1: SAP HANA Python script

On the current primary node, run systemReplicationStatus.py as the instance user. Verify that Replication Status is ACTIVE for all services.

saphana-02:~ # su - h01adm
h01adm@saphana-02:/usr/sap/H01/HDB00> cdpy
h01adm@saphana-02:/usr/sap/H01/HDB00/exe/python_support> python systemReplicationStatus.py
| Database | Host       | Port  | Service Name | Volume ID | Site ID | Site Name  | Secondary  | Secondary | Secondary | Secondary  | Secondary     | Replication | Replication | Replication    |
|          |            |       |              |           |         |            | Host       | Port      | Site ID   | Site Name  | Active Status | Mode        | Status      | Status Details |
| -------- | ---------- | ----- | ------------ | --------- | ------- | ---------- | ---------- | --------- | --------- | ---------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | saphana-02 | 30001 | nameserver   |         1 |       2 | saphana-02 | saphana-01 |     30001 |         1 | saphana-01 | YES           | SYNCMEM     | ACTIVE      |                |
| H01      | saphana-02 | 30007 | xsengine     |         3 |       2 | saphana-02 | saphana-01 |     30007 |         1 | saphana-01 | YES           | SYNCMEM     | ACTIVE      |                |
| H01      | saphana-02 | 30003 | indexserver  |         2 |       2 | saphana-02 | saphana-01 |     30003 |         1 | saphana-01 | YES           | SYNCMEM     | ACTIVE      |                |

status system replication site "1": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 2
site name: saphana-02

Method 2: SUSE SAPHanaSR tool

Run SAPHanaSR-showAttr and verify that the sync_state of the standby node is SOK.

saphana-02:~ # SAPHanaSR-showAttr
Global cib-time
--------------------------------
global Mon Apr 15 15:17:12 2019


Hosts      clone_state lpa_h01_lpt node_state op_mode   remoteHost roles                            site       srmode  standby sync_state version                vhost
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
saphana-01 DEMOTED     30          online     logreplay saphana-02 4:S:master1:master:worker:master saphana-01 syncmem         SOK        2.00.020.00.1500920972 saphana-01
saphana-02 PROMOTED    1555312632  online     logreplay saphana-01 4:P:master1:master:worker:master saphana-02 syncmem off     PRIM       2.00.020.00.1500920972 saphana-02

Reset the resource fail count

When a resource fails, Pacemaker increments its fail count. If a migration-threshold is configured, the cluster stops running the resource on a node once the threshold is reached. After resolving the failure, clean up the fail count.

Clean up the fail count for a specific resource on a node:

# crm resource cleanup [resource name] [node]

For example, to clean up the fail count for rsc_SAPHana_HDB on saphana-01:

crm resource cleanup rsc_SAPHana_HDB saphana-01

References

This guide does not replace the standard SUSE and SAP documentation. For comprehensive guidance:

Version history

Version	Revision date	Changes	Effective date
1.0	-	Initial release	2019/4/15
1.1	2019/7/30	1. Updated fail count description. 2. Updated start and stop sequence instructions.	2019/7/30