We have a bizarre issue since we just started publishing NDATA messages in conjunction with our normal flow telemetry from our devices.
The issue is we send an NBIRTH, followed by some NDATA and then a DBIRTH and then ignition immediately requests a NCMD for rebirth of the node. The logs look strange suggesting that a DBIRTH was received for an offline node which is not the case as we sent a NBIRTH prior to the DBIRTH.
Here’s the log:
I’m not sure if the message sequence errors we see are related or not, but it is possible with our app being asynchronous that messages may be generated with sequences which are received out of order.
In this example we have a NDATA log that says a birth is happening received and sequenced as seq: 3 but is received prior to the actual DBIRTH which is seq: 2. Ignition barks about it and you can see from the message flow that the DCMD for rebirth is received right after it receives the birth.
Here’s the related messages to go with the ignition log.
NBIRTH
NDATA
NDATA
DBIRTH
NCMD from ignition requesting a node rebirth
So I guess the questions:
- why does ignition think the node is offline when we have a DBIRTH happening after an NBIRTH?
- if messages are sent and/or received out of sequence, does this cause any issue?
Out of order messages will cause this behavior from Engine. Take a look at the bottom of this doc: Connecting to AWS IoT Core - MQTT Modules for Ignition 8.x - Confluence. MQTT Engine supports a ‘re-ordering timeout’ which will buffer and re-order messages within the timeout window. This may get you over this issue.
1 Like
Wes thanks for the quick response and explanation. I will give the setting a try, although I would think that this scenario of out of order messages would be a common occurrence with mqtt so am not sure i understand the justification for a rebirth request!
Affirmative, adding the custom property to introduce the delay worked like a charm, ignition no longer requesting continuous rebirths for out of order messages. Thanks for the tip!
On the rebirth request - many SCADA systems including Ignition expect data to show up in order. For example, if a value shows up with an older timestamp than the current value for a tag, Ignition will throw it out. As a result, Sparkplug enforces order of data.
Got it, we’re actually going to be tweaking our process which is sequencing the messages so that they are submitted in order.
Having said this, I was under the impression that even though we can enforce a publisher to transmit messages in proper sequence to a broker, there really wasn’t any guaranteed way on the subscribing side for an MQTT client subscribing to messages to ensure that they are received in the correct order, is this correct?
MQTT requires that QoS 0 messages on the same topic must be delivered in order. But, Sparkplug does use different topics from the same Edge Node. So, there really isn’t a built in/guaranteed order of messages. This is why the reordering option exists in Engine. This is also why the new version of the Sparkplug spec makes this a requirement for Sparkplug Host Applications.
With that said, many MQTT servers will deliver messages in order even across different topics. Cirrus Link’s Chariot and MQTT Distributor do deliver in order even across topics. This is why the early versions of Sparkplug and MQTT Engine did not have this feature.
Got it, this all makes sense! Thanks for all the insights.
