ScaleArc behavior in MySQL auto-failover
ScaleArc Auto-Failover for MySQL clusters explained.
During normal operation, ScaleArc continually monitors the health and availability of each server defined within a cluster. When one member of the cluster is detected as unavailable, the server is marked down and offline. ScaleArc issues a mysql RESET SLAVE command to the next Standby+Read server defined in the cluster. This action instructs the mysql server to forget its replication position in the master's binary log. It also deletes the master.info and relay-log.info files, all the relay log files, and starts a new relay log file on the mysql server.
Once the failed server becomes available again, ScaleArc will mark the server as healthy (green dot next to the server name/IP address in the Database Servers colum of the main Clusters Control Panel screen) with a strike through line which indicates that the server is offline from a ScaleArc traffic perspecitve.
Before the previously failed server can be returned to operation within ScaleArc, manual action is required by a DBA or systems administrator. In order to ensure that all data is active and available on each node in the cluster, a new backup must be taken from the current mysql master. The backup must then be restored to the recently recovered mysql cluster node. Then replication can be reconfigured with the recently recovered node now becoming the secondary to the current mysql master node. Once replication has been restored and all data verified, the DBA or systems administrator can mark the recovered node "online" in the ScaleArc UI. The action to return the recently recovered server to an "online" state is performed in the individual server settings section of the ScaleArc Cluster Control Panel UI. The individual database server settings can be accessed by selecting the gear icon to the right of the database server name/IP.
Select the ONLINE button in the Server Status section and then click on the Update Server button at the lower right of the window. This will make the server available to ScaleArc for auto-failover operations as well as directing client connections (assuming read/write split has been enabled).
In the case where there are multiple SLAVE servers to a single MASTER, ScaleArc will attempt to establish replication betweeen the newly promoted MASTER and the remaining active SLAVE servers.
This same behavior is followed for MySQL Master-Master replication configurations as well. In the case of a failue of one master node in a multi-master configuration, there may not be a need for a full backup of the database on the active master and restore on the failed master if the replication is able to recover once the failed node is online again. However the replication status and data integrity must be verified before the previously failed server is marked online again in the ScaleArc UI. Also, since ScaleArc executes the RESET SLAVE mysql command on the Standby + Read server when it is promoted, the slave configuration will need to be applied again in order to retore the master-master replication.
If ScaleArc has been configured with both MySQL master nodes as Read + Write servers then no action is required to restore the slave replication status since the RESET SLAVE command would not be executed during the auto-failover operation. Having mutlitple servers defined as Read + Write is not a recommended deployment design as it can lead to data collisions and inconsistencies that would require manual intervention to correct before returning to normal operation. In a MySQL Multi-Master environment it is recommended to have a single Master server defined as Read + Write in ScaleArc with the other Master servers configured as Read + Standby.
If you are experiencing issues with ScaleArc or with any of it's features, please contact ScaleArc Support. We are available 24x7 by phone at 855 800 7225 or +1 408 412 7315. For general support inquiries, you can also e-mail us at firstname.lastname@example.org.
2901 Tasman Drive Santa Clara, CA 95054 | Email: email@example.com