Version: 8.1

Setting Up Redundancy

In redundancy, both nodes will share the exact same configuration state. When a Backup node connects to a Master node, the Backup will attempt to synchronize itself with the Master. Therefore, before you set up for redundancy the following should be considered:

Start with a fresh install for the Backup node.
Because the current configuration of the Backup node will be overwritten, make sure that it does not contain anything valuable. It is a good idea to export any projects that are unique to the Backup before enabling redundancy.
All system configurations relative to the Master node must also resolve on the Backup node.
For example, OPC UA connections and database connections must use addresses that resolve from both nodes, or any OPC-COM servers must be installed and configured identically on both nodes. This means using "localhost" in any of the database connections won't work. You should use the IP address of the computer instead.
Configure firewalls between the redundancy nodes.
Redundant systems need TCP connectivity between each other on the default Gateway network ports. Turning off software firewalls or adding special exception rules for each others' addresses is required. The default Gateway Network port is port 8088 (without SSL), and port 8060 (with SSL), and the Backup node must be able to send outgoing data on that port. The port can be changed from Gateway Network settings.

note

Two Edge Gateways can be set up with redundancy. An Edge Gateway can only failover to another Edge Gateway (not a standard Ignition Gateway). Also, an Edge Gateway cannot be used as backup to a Standard Ignition Gateway.

note

While the OS platform (i.e., Windows, OS X, Linux) for the Master and Backup can differ, it is recommended to have similar OS platforms. If the OS platforms do differ, the Windows machine should be the master system or else the Force Failover option will not work.

However, different versions of the same operating system such as Windows 10 and Windows 8 or OSX 10 and OSX 11 have full functionality.

On the Master Gateway

Go to the Config section of the Master Gateway Webpage.
Select System > Redundancy. The Redundancy and Network Configuration page is displayed showing different sections and settings. See the table below for a description of all settings.
Change the following settings:
- Under Redundancy Settings, set Mode to Master.
- Optionally, configure any desired settings under Master Node Settings.
Click Save Changes. The Confirm change to Redundancy Settings page is displayed.
Click Confirm to apply your settings.
Go to the Config tab and select System > Redundancy to ensure the redundancy mode and state is properly set.

On the Backup Gateway

Do the exact same steps 1-6 above on the Backup Gateway Webpage, except replace step 3 with the following:

Under Redundancy Settings, set Mode to Backup.
Under Backup Node Settings, configure the Master Node Address and Port to point to the Master Gateway. The Master Node address should be a hostname or IP address. The Port setting (assuming default configurations) should be 8060 if using SSL, otherwise 8088.

On the Master Gateway

Return to the Config section of the Master Gateway Webpage.
Select Networking > Gateway Network.
Navigate to the Incoming Connections tab. You should see a new incoming connection from the Backup Gateway. Find the connection, select More.
caution
If you are not using SSL and your connection isn't displayed, make sure the Require SSL box is unchecked under the General Settings tab. Clearing the checkmark and returning to the Incoming Connections tab will now populate your new incoming connection.
Select approve.
To verify the redundancy setup, that is, to ensure the Master and the Backup Gateways are connected, go the Status tab of the Gateway Webpage and click on System > Redundancy. The Redundancy page will show the connected nodes and their current states.

After approving the connection, the Backup connects to the Master and downloads a system backup, then restarts. Once the restart is complete, the Backup node is synchronized and in communication with the Master.

Redundancy Settings

All redundancy settings are configured in the Gateway Webpage under the Config tab, Systems > Redundancy. Most settings are used by both the Master and Backup nodes, with their individual settings broken out into separate categories.

It is important to know that while the full system configuration is shared between nodes, redundancy settings are not shared between nodes. Therefore, it is perfectly acceptable to have different values for the same settings on the two nodes. For example, it is possible to have a different Standby Activity Level on both nodes, and, of course, the network settings will often be different.

note

The Master node shares all configuration with the Backup node, and this means that changes cannot be made to your project from the Backup. In fact, the Designer can never be opened from a Backup node, even if the Master is currently offline.

Redundancy Settings

Property	Description
Mode	Enable or disable redundancy, and specify this node's role. There should be one master and one backup node per redundant pair. Independent turns off redundancy. Independent - Redundancy is not enabled and this Ignition system runs as an independent node. Master - This is the Master node, who listens for a connection from the Backup node, and is in charge of managing system synchronization. Backup - This is the Backup node, who will connect to the Master and receive system updates.
Standby Activity Level	How the node should run when it is not currently the Active node. Cold - The system connects to all OPC servers but does not subscribe to Tag values. The Ignition OPC UA server does not communicate with any device, but third party OPC UA servers may still have device connections. This allows the system to standby without putting additional load on the devices and network. Failover takes slightly longer, as Tags must be subscribed and initialized. Warm - The system runs as if it were active, with the exception of logging data or writing to devices, allowing for faster fail-over.
Failover Timeout	This feature was removed from Ignition in version 8.1.37 The time of inactivity, in milliseconds, before the backup assumes responsibility. Default is 10000 milliseconds.
Startup Connection Allowance	The time in milliseconds that the system will wait at startup for a connection before making a decision on the node's responsibility level. This is used to prevent unnecessary switch over caused by a node starting as active, only to connect and find that the other node is active, resulting in one of the nodes being deactivated. Default is 30000 milliseconds. Note: It is important to notice that this setting can interfere with the Master Recovery mode: If the Master is active, it will always request the Backup to de-activate. If this setting is low, or 0, the Master will always become active before connecting to the Backup, and thus "manual recovery" will not be possible.
Sync Timeout	New in 8.1.22 The maximum time in seconds allowed for a redundancy sync operation. Sync operations that exceed this value will time out. Default is 60 seconds

Network Settings

Property	Description
Auto Detect Network Interface	If true, the system will automatically select which network interface to use. Most commonly disabled on systems with multiple network cards, in order to explicitly specify which interface to use. If false, the system will bind itself to the interface of the specified address.
Network Bind Interface	The IP address of the network interface to use for redundancy. Only used if "Auto Detect" is false.

Master Node Settings

Property	Description
Recovery Mode	How the Master node resumes responsibility after starting again. Automatic - The Master automatically takes back responsibility, and becomes active. The Backup node goes to standby. Manual - The Backup node is allowed to stay active. The Master will become active if the Backup node fails, or if the user requests a switchover from the Gateway configuration page.
Runtime Update Buffer Size	This feature was removed from Ignition in version 8.1.21 How many "runtime state" updates can be queued in memory before the system stops tracking and a full transfer is performed. These updates represent information that the other node should have in order to have the same running state as the Master when it's forced to take over. This is most often the values of static Tags and the current alarm state. Given that the update buffer is only used once the nodes are connected, the default value is usually fine, and only needs to be increased on systems that may have many alarms that change together, or many static Tag writes.
Config Update Queue Size	The maximum size (in megabytes) of config updates allowed before a full transfer is performed.

Backup Node Settings

Property	Description
Master Node Address	The address of the Master Ignition system.
Port	The Gateway Network port used by the Master to listen on. For the Backup, the port to connect to on the Master.
Use SSL	Use SSL to connect to the remote machine.
Ping Rate	How often, in milliseconds, to send a message from the Backup to the Master.
Ping Timeout	The maximum time, in milliseconds, allowed for a ping response. Pings that time out are counted as missed pings.
Ping Max Missed	The amount of missed pings that will force the connection to the master to be considered faulted.
Websocket Timeout	The maximum time, in milliseconds, allowed for a new web socket to connect to the Master.
HTTP Connect Timeout	The maximum time, in milliseconds, allowed to establish an HTTP connection to the Master.
HTTP Read Timeout	The maximum time, in milliseconds, allowed to read or send HTTP data to the Master.
History Mode	How history is treated by the Backup system. If Full, history will be stored normally, as it would be on the Master system. If Partial, history will be cached until the Master is available again and the Backup node is able to determine the exact time that the Master was down.
Use Active Uptime to Resolve Conflicts	New in 8.1.31 When enabled, the system will resolve data conflicts by examining if the Master node or Backup node have been active for longer. The redundancy data from the longer running active node will be selected, and will overwrite the data on the other node.

Troubleshooting

Redundancy Connectivity

When the two redundant nodes are connected, you will be able to see their state details in the Status section of the Gateway Webpage. There are also various other places where the redundancy state is shown as connected.

If the two nodes cannot connect, check the following:

Verify that the Master address is correct in the Backup. Try to ping the Master machine from the Backup machine, and verify that you're using the correct address for the network card that the Master is connected through.
If using system names (or domain names), verify that the name is resolving to the correct address by performing a ping.
Verify that the firewall on the Master is set to allow TCP traffic to the designated port.
Verify that the Backup is not connecting and then immediately disconnected for some reason.
Viewing the error log in the Gateway console section should show this. If errors are occurring at regular intervals, look at the message for an indication of what is happening. An example of a potential problem is when the failover time is set too low for the given network, which results in many socket read timeout exceptions, which in turn leads to many disconnect/reconnect attempts.
If errors are occurring, but the cause isn't clear, contact Inductive Automation Support.

Advanced Troubleshooting

A variety of loggers can be found under the Gateway console section by going to "Levels" and searching for "Redundancy". By setting these loggers to a finer level, more information will be logged to the console. This is generally only useful under the guidance of Inductive Automation support personnel, though more advanced users may find the additional logged information helpful.