Scale-Out Architecture Guide

As stated on the Scale Out Architecture page, a scale-out architecture can help ease the load between Front-End and Back-End tasks by separating them into their own Ignition installations. Since Ignition provides various options in regards to Gateway to Gateway subsystem communication, some configuration routes will be more optimal than others. If you’re planning to scale out an Ignition server or would like to get started, review these guidelines to determine the route that works best for you.

Migrating Resources

It is strongly recommended to first create a test environment for migrating your resources before making changes on your production environment. Doing so will help you take note of any special scenarios that need to be addressed in the Considerations for Each Resource section.

1. Take a Gateway Backup and store it in a secure location - not on the server’s drive!
   1. You may want to download a copy of the installer for the existing Ignition instance for disaster recovery preparedness.
2. Determine the new purpose of the existing Gateway. For this guide, it is assumed that the existing Gateway will be repurposed as the Back-End server. However, there are instances where this may not be the optimal solution. For example, if you have 100 clients all pointing to the existing server to provide visualization, it may be easier to create a new Back-End Gateway.
3. Export all visualization/Front-End elements from projects in the Back-End.
   1. Perspective Views / Vision Windows.
   2. Images & Icons
   3. Reporting resources
   4. If you are restoring a full gateway backup instead of importing individual resources, ensure that the backup is restored with Restore Disabled set to True. This will keep all duplicate Back-End resources inactive so that they can be deleted later.
4. If you’re planning to scale out using Alarming, Auditing, Reporting, SQL Bridge, or Tag History, consider the following on the new Front-End server:
   1. Create a matching database connection by profile name and connect URL.
   2. Create matching Alarm Journal or Audit profiles by both profile name, database connection name, and database table name(s) if needed.
5. On the Front-End server, create the appropriate user sources or identity providers by both profile name and domain or endpoint URLs.
   1. Ensure that all custom security levels or user mappings are properly redefined.
   2. If your original server was exposed to any external authorization services (Active Directory, OpenID Connect, etc. ), consider the different options for both Front-End and Back-End login configurations.
6. On the Back-End server, establish a Gateway Network connection to the Front-End.
   1. It is recommended to establish an outgoing connection from the Back-End server
7. On the Front-End server, create remote tag providers that correspond with the Back-End’s realtime tag providers.
   1. Match the tags by name so that migrated projects can continue to work without changing any tag provider references.
   2. Set Alarm Mode to Subscribe.
   3. Depending on your database exposure and Tag History usage, ensure the History Settings are configured appropriately.
   4. See Step 9 regarding Service Security if tags are not visible / writeable.
8. On the Front-End server, import the exported visualization/Front-End resources from step 2.
9. On the Back-End server, create a new security zone profile and specify under Identifiers > IP Addresses the IP address of your Front-End server.
   1. This will provide a basic security feature that enforces exclusive access between your Front-End and Back-End servers.
10. On the Back-End server, configure service security and to allow read access of all necessary subsystems.
    1. Allowing write access on your service security will depend on your requirements for Alarm Journals, Auditing, Tags, and Tag History on the Front-End server.
11. If all previous steps are successful, implement any Back-End message handlers and Front-End sendRequest scripts that may be required for Reporting, SFC, System functions with Gateway Scope access, Web Dev, etc.
12. If you are creating multiple Front-End servers behind a load balancer, configure a DNS entry to hit the load balancer.

Considerations for Each Resource

Before you make any changes to your production Gateway, take a moment to review your projects and configurations on every module that is currently in use, especially after you performed a test migration. This is an important step since you will need to take note of which modules, project components, and configuration profiles should be considered an exclusive Back-End resource or Front-End resource.

Alarming

What to keep on the Back-End:

• Alarms configured on various tags
• Alarm Pipelines
• Notification Profiles

What to move to Front-End:

• Perspective views and Vision windows monitoring alarm status table events.

• Perspective views and Vision windows monitoring alarm journal table events.

  • This will require your Front-End server to have access to the same database tables, which can be accomplished using one of two methods.

    • The more optimal approach is to allow the Front-End Gateway to create a direct connection to the Back-End’s same database.
    • A Remote Alarm Journal can be created which will query alarm journal events over the Gateway Network.

  • When creating an Alarm Journal profile on the Front-End, ensure the profile name is the same as the Back-End.

  • When creating an Alarm Journal profile on the Front-End that will access the Back-End’s database directly, ensure both Table Name and Event Data Table Name defined in the advanced properties match the name of any existing alarm journal tables.

    

• Create a Remote Gateway Notification profile if you require alarms to be configured on the Front-End.

• Keep in mind, this will also require alarm notification modules to be present on both servers instead of only one.

Auditing

What to keep on the Back-End:

• Existing Auditing profiles and their database tables.

What to move to Front-End:

• Perspective views and Vision windows monitoring audit events.

  • This will require your Front-End server to have access to the same database tables but it can be accomplished using one of two methods

    • The optimal approach is to allow the Front-End gateway to create a direct connection to the Back-End’s same database.
    • A second option is to create a Remote Audit Profile which will query audit events over the gateway network.

  • When creating an Audit profile on the Front-End, ensure the profile name is the same as the Back-End.

  • When creating an Audit profile on the Front-End that will access the Back-End’s database directly, set the Auto Create setting to false and ensure the Table Name property defined in the Database Settings matches the name of an existing audit table.

    

Enterprise Administration Module (EAM)

EAM does not have any special requirements. Keep the EAM controller on the Back-End and distribute the agent configurations to other remote gateways as usual.

Identity Providers (IDP) & User Sources

What to keep on the Back-End:

• A primary, internal-type user source.
  • Retaining a local authentication strategy ensures accessibility in the event of production issues since the Back-End server is primarily communicating with local, on-premise devices.
• Existing user source and IDP profiles that communicate with external authorization services ( Active Directory, OpenID Connect, etc. ) will depend on your requirements for accessibility and security.
  • If the Back-End server is required to avoid any direct requests from public networks, keep in mind that these types of user sources and IDP strategies require Back-End networks to be open to the public as well.
  • Also keep in mind that Back-End servers will typically limit access to administrative accounts and not operators which reduces the pool of users that need to be configured.

What to move to Front-End:

• Any user sources and IdP profiles that are utilized in Vision or Perspective sessions.
• Existing user source and IDP profiles that communicate with external authorization services ( Active Directory, OpenID Connect, etc. ).
  • If your original server already utilized these strategies, your Front-End clients will benefit from using them. Especially if the server will be public-facing.

Images & Icons

Custom images in the Back-End server’s Image Management will need to be manually imported into the new Front-End gateway. The Image Management tool allows the option to Save image(s) to local disk which includes individual files or nested files within folders.

OPC Drivers

Device connections can be left alone on the Back-End.

Reporting

If only one Front-End Gateway exists, reporting resources can reside on either Gateway.

• The more optimal approach will depend on the requirements of your Front-End clients:
  • Do you need to display these reports for clients/sessions?
  • Do you prefer having reports stored on a shared drive? Only Back-End or Front-End storage?
  • Which server would you prefer for executing report construction?
• Reporting resources to consider:
  • Report profiles in your projects
  • Perspective views and Vision windows using the Report Viewer.
  • Database tables uses for reports
  • Reporting modules

What to keep on the Back-End:

• If you plan to have more than one Front-End Gateway, keep scheduled reports in the Back-End to ensure single executions without any duplicates.
  • In this scenario, migrating schedule reports to Front-End would require redundant report configurations, resulting in unnecessary report duplicates and modules.
• If reports will be stored on the Back-End or on a shared drive, all reporting resources can be left alone.
• If reports are not meant to be viewed by the client, all reporting resources can be left alone.

What to move to Front-End:

• If reports need to be displayed in clients/sessions:
  • Perspective views and Vision windows using the Report Viewer.
    • If Front-End reporting requires Tag History data, review the Tag History section.
  • If Front-End will handle creation, execution, and storage of reports:
    • Reporting Modules
    • Report profiles in your projects
    • Database tables used for reports

Sequential Function Charts (SFC)

What to keep on the Back-End:

• If your Front-End server and clients require interaction with an SFC, consider implementing a message handler.

What to move to Front-End:

• To interact with a Back-End server’s message handler, consider using the scripting function system.util.sendMessage wherever you see fit.

SQL Bridge

What to keep on the Back-End:

Existing transaction groups and their corresponding database connections. What to move to Front-End:

Perspective views and Vision windows querying from transaction group-stored data. This will require your Front-End server to have direct access to the same database.

Miscellaneous

Third-party modules are not developed by Inductive Automation and should be reviewed on an individual basis to determine the appropriate Gateway for them.

What to keep on the Back-End:

• Scripting functions such as system.eam, system.device, system.opc, system.report, system.sfc, etc. possess Gateway Scope access but cannot reach across remote Ignition gateways. Any use of these functions within scripts will need to be left on the Back-End.
  • If your Front-End server and clients require interaction with these scripts defined in the Back-End, consider implementing a message handler.

What to move to Front-End:

• To interact with a Back-End server’s message handler, consider using the scripting function system.util.sendMessage wherever you see fit.
• Third-party Python libraries are typically imported for constructing complex data structures and objects that go beyond the capabilities of our system functions.
  • These types of libraries and dependent, user-defined scripts should be imported to the Front-End server. If left on the Back-End, the scripts’ execution could potentially interfere with other Back-End processes since they share the same resources.

Scripting Examples

Message Handlers

A useful application of a message handler could involve execution of reports on a Back-End server triggered by a Front-End server request. Below is a button script defined for a project that could be modified to use a message handler:

# Get the date range entered by the user.
startDate = self.parent.getChild("Start Date").getChild("DateTimeInput").props.value
endDate = self.parent.getChild("End Date").getChild("DateTimeInput").props.value
     
# Define the settings and parameters for the report.
settings = {"path": "C:\\Reports", "fileName": "Test Report %s.pdf" % system.date.format(system.date.now(),"yyyy_MM_dd_HH_mm_ss"), "format": "pdf"}
parameters={"StartDate": startDate,"EndDate": endDate}
     
# Execute the report and save it in the specified path.
system.report.executeAndDistribute(path="Test Report", project="Test_Project", parameters=parameters, action="save", actionSettings=settings)

On the Back-End server’s Gateway-scoped message handler, the script could be modified to:

# Get the report settings and parameters from the payload.
settings = payload['settings']
parameters = payload['parameters']
     
# Execute the report and distribute it.
system.report.executeAndDistribute(path="Test Report", project="Test_Project", parameters=parameters, action="save", actionSettings=settings)

On the Front-End server, the project button script could be migrated to this location and be modified to interact with the Back-End server’s Gateway message handler:

# Get the date range entered by the user.
startDate = self.parent.getChild("Start Date").getChild("DateTimeInput").props.value
endDate = self.parent.getChild("End Date").getChild("DateTimeInput").props.value
     
# Define the settings and parameters for the report.
settings = {"path": "C:\\Reports", "fileName": "Test Report %s.pdf" % system.date.format(system.date.now(),"yyyy_MM_dd_HH_mm_ss"), "format": "pdf"}
parameters={"StartDate": startDate,"EndDate": endDate}
     
# Send a message to the backend gateway so that it can
# execute and distribute the report.
system.util.sendMessage(project='Test_Project', messageHandler='Execute And Distribute Report', payload={'settings':settings,'parameters':parameters},
remoteServers=['BackendGateway'])

When using system.util.sendMessage to send a message to a Gateway message handler, errors will be logged in the Gateway (accessible on the Gateway Webpage). If there is a need to return a value from the message handler to the Front-End Gateway, then system.util.sendRequest can be used rather than system.util.sendMessage.

Third Party Libraries

A useful application of third-party Python libraries could involve the croniter library. This library provides iteration for datetime objects with cron-like format. Below is an example that utilizes the croniter library to figure out the next scheduled datetime based on a schedule in the cron format:

def getNextScheduledDatetime(cronSchedule):
    """
    Reads a schedule in the Cron format and
        returns the next scheduled datetime
        based on the current time.
         
    Agrs:
        cronSchedule: A string that contains
        the schedule in the Cron format 
        (example: 10 14 * * 1).
         
    Returns:
        The next scheduled datetime according
        to the schedule.
    """
    from croniter import croniter
    from datetime import datetime
     
    cron = croniter(cronSchedule)
     
    # Get the next scheduled time in unixtime.
    nextScheduledTime = cron.get_next()
     
    # Convert the unixtime into a datetime and return it.
    return datetime.utcfromtimestamp(nextScheduledTime)# Current datetime is 11/11/22 4:55pm (Friday)
# The following Cron schedule represents 2:10pm every Monday: '10 14 * * 1'
# So, when calling getNextScheduledDate('10 14 * * 1'), we get 11/14/22 2:10pm (Monday).print getNextScheduledDatetime('10 14 * * 1')

A possible scenario that requires the script and its dependencies above to be placed in the Front-End server could involve large sets of Cron time values being queried from a database and then being converted to datetime for displaying on a client/session.

• If a Back-End server is already responsible for retrieving very large datasets containing a number of Cron values, consider the time and resources needed to perform both an expensive query and execution of getNextScheduledDatetime() per iteration of the returned query’s dataset.
• Separating these types of tasks to individual Gateways ensures dedication of resources.

Tag History

What to keep on the Back-End:

• Existing historical tags and their corresponding database connections.

What to move to Front-End:

• Create a Remote Tag provider with the same name as the Back-End’s Realtime Tag provider hosting the historical tags.
• All Perspective views and Vision windows for tag history.
  • To display tag history, your Front-End server needs access to the data. This is accomplished through the remote tag provider's History Access Mode. History Access Mode is set to Gateway Network or Database.

    • Gateway Network: Tag History Bindings will request data over the Gateway Network connection.

    • Database: Tag History Bindings will request data directly from a database. The Front-End server will need a Database Connection created to the same database being utilized for the Back-End server’s Tag Historian for this Tag Provider. Because only a single History Datasource can be specified for the Remote Tag Provider, the Back-End server’s Tag Provider should not have tags using various History Storage Providers. If multiple Storage Providers are used, the tags should be grouped and moved to separate Tag Providers on the Back-End server. Corresponding Front-End Remote Tag Providers can then be created and configured with corresponding History Datasources.

      

  • Setting the Historical Access Mode to Database provides a more direct line of communication for sending queries compared to Gateway Network; however, all historical tags being referenced on Front-End component bindings may need to be changed to historical tagpaths since realtime tagpaths are used by default.

    • Historical data will not appear in the result of your queries if tag history bindings are using realtime tagpaths against remote tag providers. When remote tag providers are configured to query from a database connection directly, only full historical tag paths provide the information they need to properly locate tag history.

    • In order to check for realtime tagpaths in your tag history queries, check the tag path properties for any tag history configurations. If the tag path is structured to resemble a path within a tag provider, it is a realtime tag path. For example, [Sample_Tags]Realistic/Realistic0].

      

  • Historical binding configurations provide browsing for historical tags. If a tag has been storing historical data, expand the browsing folders to locate the proper historical tagpath equivalent. You will be able to see this by searching for paths starting with:

    • A database connection name

    • The original Gateway the tag resides in

    • The original tag provider the tag resides in. For example, [PostgreSQL/originalignition:sample_tags]realistic/realistic0] is a historical tagpath.

      

  • If Historical Access Mode is set to Gateway Network, realtime tag paths are allowed and bindings will not require any modification, however, tag history queries will need to share resources with other requests being sent through a Gateway Network connection.

Web Dev

• Leaving resources on the Back-End could require exposing the Back-End to http requests by Front-End users.
  • Take time to review which resources will be safe enough to expose to the Front-End Gateway. This depends on what’s being hosted and the intended use case. For example, Tag History might be more performant from the Back-End gateway.