I am hoping that someone can help me with some additional details around the implementation details and limitations with the “Store and Forward” functionality integrated into the Ignition MQTT Transmission module.
Is there a limit to the size of the Store & Forward data when using the buffer “Disk-Backed” option for History (aside from the practical limit of hard drive space)?
Is there a duration limit to the “Store & Forward” buffer (i.e. Max duration of time that the specification can handle for Old data or is it just file size limits)?
Where can I see this buffer file when “Enable History Storage by Default” is selected and History Store is set to “Disk-Backed”?
If there are more than one MQTT Engines receiving data from an MQTT Transmission module, if one of the Engines disconnects temporarily, does the MQTT Transmission module continue transmitting to the Engine that is still Online while “buffering” (Store & Forward) for the Engine that is Offline?
Is there a limit to the size of the Store & Forward data when using the buffer “Disk-Backed” option for History (aside from the practical limit of hard drive space)?
There is no limit. However, as size increases performance and access to the data slows. So, it is possible to run into performance issues related to read/write before you actually run into a hard drive space issue.
Is there a duration limit to the “Store & Forward” buffer (i.e. Max duration of time that the specification can handle for Old data or is it just file size limits)?
There is no limit here either. Data is not dropped unless the size of the buffer is exceeded. When/if it is, the oldest data is dropped first.
Where can I see this buffer file when “Enable History Storage by Default” is selected and History Store is set to “Disk-Backed”?
This will be stored in [ignition-install-dir]/user-lib/cls/data/h2/. It will be a ‘.db’ file with the name you’ve specified in your history store config. I should also note there are ‘MQTT Transmission info tags’ under the MQTT Transmission tag provider showing how much space the history store is using.
If there are more than one MQTT Engines receiving data from an MQTT Transmission module, if one of the Engines disconnects temporarily, does the MQTT Transmission module continue transmitting to the Engine that is still Online while “buffering” (Store & Forward) for the Engine that is Offline?
Sparkplug (and MQTT Transmission) only support a single ‘primary host’ application. So, the above example will only honor one Engine that is the primary host. As a result it will also only store and forward based on the primary host’s connected status. This is something we’d like to fix/improve via the Sparkplug Working Group but it will likely be some time before we get this done.
Thank you for your great explanation. Could you please also clarify the following?
1- Edge Node Tag Capacity
2- Device Tag Capacity
3- Flush Quantity
Let us say I set the “Edge Node Tag Capacity” to 10000 and I have 10 tags:
1- Does that mean for each tag, 10000 values will be stored (meaning 10000 x 10)?
2- If a tag exceeds storing 10000 values, then the oldest values for that specific tag would be dropped?
Based on your Transmitter and Tag Tree setup, you will have Edge Node and/or Device level tags. In other words, tags will be associated with either Edge Node or Device messages.
Then that leaves the following:
Edge Node Tag Capacity is the maximum number of tag change events to store for each Edge Node. This only applies to the ‘Edge Node level tags’ for this Edge Node. Every Device under this Edge Node has it’s own storage container and is capped by the ‘Device Tag Capacity’.
Device Tag Capacity is the maximum number of tag change events to store for each Device
Flush Quantity is the maximum number of tag change events to flush on a per message basis
Flush Rate is how frequently to flush historical data (where each message would contain a maximum number of tag change events defined by the Flush Quantity).
Based on this - I can’t really answer the first question from your example because it depends on ‘where’ those tags live (i.e. under a Device or an Edge Node or a combination of them). On the dropping of data - yes, the oldest data is dropped once the capacity limit is hit.