Reducing data loss during MQTT Transmission tag tree re-discovery

Ignition Modules
1

When starting MQTT Transmission, changing one MQTT server definition, refreshing the whole MQTT transmission (under “transmission controll”), or failing over to a redundant Ignition gateway, it appears that the MQTT Transmission module must re-discover its whole tag tree.

The status field in the MQTT transmission server definitions progresses through the following states:

  1. 0 of 0
  2. 0 of X
  3. X of X

Based on my understanding, Store & Forward begins buffering data once the server connection reaches the 0 of X state. If that understanding is correct, data generated during the initial 0 of 0 period is lost, even with S&F.

My main question is:

Is there any way to reduce or eliminate data loss during the 0 of 0 period, or is the only option to reduce the duration of that state?

My Environment

  • Approximately 50,000 tags are published through MQTT Transmission.
  • Initially, the 0 of 0 period lasted about 50 seconds before all MQTT server connections completed discovery. The step between 0 of X and X of X always take a negligible amount of time.
  • During this process, Ignition reports CPU and memory utilization well below 100%.

What I’ve tried

  1. Split the data across 60 MQTT clients (60 Edge nodes) within a single MQTT Server definition.

    • No noticeable reduction in discovery time during failover to a redundant GW.

  2. Removed custom metadata properties on the tag field.

    • No significant improvement.

  3. Kept the same number of MQTT clients but split them across multiple MQTT Server definitions in the transmission module.

    • Reduced the discovery time by approximately 50%, but it is still around 35 seconds.

Questions

  • Is there any way to reduce or eliminate data loss during the 0 of 0 period, or is the only option to reduce the duration of that state?
  • Are there any other settings or architectural changes that can reduce the 0 of 0 discovery period?
  • We have a redundant Ignition Gateway configured with a Warm redundancy activity level. While the gateway is running as the backup, the MQTT Transmission server connection status remains at 0 of 0. Is there a configuration option that allows the backup gateway to pre-discover the tag tree while it is on standby? In other words, can it remain in a state equivalent to 0 of X
2

There isn’t really much that can be done during this period. During this period, Transmission is discovering the Edge Nodes. It first has to find out what Group, Edge Nodes, and Devices exist. Then it has to iterate over the tag tree(s) to find all of the tags under the scope of each Sparkplug descriptor. Transmission can’t store history if it doesn’t know what tags are even under its purview.

The process can be sped up by spreading the load out across multiple different Transmitters. Each Transmitter is allocated a number of resources (threads, memory, etc) to handle this discovery and also change events as they occur. More Transmitters will increase parallel operations reducing the overall discovery time.

1 Like
3

Thanks, that matches my experience.

One follow-up question: in a redundant Ignition setup running in Warm mode, is there a reason the backup gateway cannot pre-discover the Edge Nodes/tag tree while on standby? meaning that the standby ignition reports 0 of X.

4

This is an issue in our product backlog. However, it will likely be fairly difficult to implement due to sync events and potential structure change events that may occur on an active node that need to be reflected on the inactive node.

5

Thanks for the quick feedback. This forum is useful and saves us a lot of time as customers.