MQTT Engine INTERNAL_ERROR crashes Ignition Project

christiankyte-bright · October 20, 2023, 8:03pm

MQTT Engine 4.0.14
Ignition Version 8.1.23

Problem Statement

Ignition SFCs are being aborted due to an exception from the MQTT Engine plugin, this happens when the MQTT (RMQ) broker suddenly crashes (hosted on a separate server). The simple answer is to ensure the RMQ Broker doesn’t crash, but I would like a solution that doesn’t include our broker in the dependency chain for Ignition. How can I trap the following errors to prevent Ignition Projects from crashing entirely?

Screen shot of Ignition Log showing the order of events

Code to send MQTT Messages

def _publishEvent(topic, payload):
	try:
		#Server (see config webui), topic, payload, qos, retain
		system.cirruslink.engine.publish("ELK", topic, str(payload).encode(encoding="UTF-8"), 0, 0)
		print("publish MQTT for event: {}".format(payload))
	except Exception, e:
		msg = "Exception thrown publishing MQTT Msg. Topic [{}], Payload [{}]".format(topic, payload)
		print(msg)
		logger.error(msg)
		logger.error(str(e))

Note: no log messages are collected with that exception, which tells me the following exceptions are happening in a different call stack. Presumably whatever thread is handing MQTT message sending

MQTT Engine Plugin seems to be crashing with the following stacktrace

EngineCallback - Connection lost due to - Connection Lost

org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:197)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.base/java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:137)
... 1 common frames omitted

Which then causes an exception on each of the running SFC instances, with the following stacktrace… from what I see the meaningful part is:

Chart Instance Aborted

Caused by: org.eclipse.tahu.exception.TahuException: INTERNAL_ERROR
at org.eclipse.tahu.mqtt.TahuClient.publish(TahuClient.java:437)

Complete stacktrace

com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last): File "
at org.python.core.Py.JavaError(Py.java:547)
at org.python.core.Py.JavaError(Py.java:538)
at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:192)
at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.call(ScriptManager.java:552)
at org.python.core.PyObject.call(PyObject.java:433)
at org.python.core.PyObject.call(PyObject.java:437)
at org.python.pycode._pyx290._publishEvent$5(:127)
at org.python.pycode._pyx290.call_function()
at org.python.core.PyTableCode.call(PyTableCode.java:173)
at org.python.core.PyBaseCode.call(PyBaseCode.java:150)
at org.python.core.PyFunction.call(PyFunction.java:426)
at org.python.pycode._pyx290.publish_conv_event$2(:25)
at org.python.pycode._pyx290.call_function()
at org.python.core.PyTableCode.call(PyTableCode.java:173)
at org.python.core.PyBaseCode.call(PyBaseCode.java:306)
at org.python.core.PyBaseCode.call(PyBaseCode.java:158)
at org.python.core.PyFunction.call(PyFunction.java:437)
at org.python.pycode._pyx316.main$1(:111)
at org.python.pycode._pyx316.call_function()
at org.python.core.PyTableCode.call(PyTableCode.java:173)
at org.python.core.PyBaseCode.call(PyBaseCode.java:168)
at org.python.core.PyFunction.call(PyFunction.java:437)
at org.python.pycode._pyx420.onStart$1(
at org.python.pycode._pyx420.call_function(
at org.python.core.PyTableCode.call(PyTableCode.java:173)
at org.python.core.PyBaseCode.call(PyBaseCode.java:306)
at org.python.core.PyFunction.function___call__(PyFunction.java:474)
at org.python.core.PyFunction.call(PyFunction.java:469)
at org.python.core.PyFunction.call(PyFunction.java:464)
at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:846)
at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:828)
at com.inductiveautomation.ignition.gateway.project.ProjectScriptLifecycle$TrackingProjectScriptManager.runFunction(ProjectScriptLifecycle.java:832)
at com.inductiveautomation.sfc.elements.steps.StepScriptRunner.invoke(StepScriptRunner.java:62)
at com.inductiveautomation.sfc.elements.steps.StepScriptRunner.invoke(StepScriptRunner.java:40)
at com.inductiveautomation.sfc.elements.steps.action.ActionStep.activateStep(ActionStep.java:50)
at com.inductiveautomation.sfc.elements.StepContainer.onActivateRequested(StepContainer.java:145)
at com.inductiveautomation.sfc.fsm.element.ElementInactive.action(ElementInactive.java:35)
at com.inductiveautomation.sfc.fsm.element.ElementInactive.action(ElementInactive.java:8)
at com.inductiveautomation.sfc.fsm.StateContext.lambda$handleEvent$0(StateContext.java:49)
at com.inductiveautomation.sfc.api.ExecutionQueue$ThrowableCatchingRunnable.run(ExecutionQueue.java:146)
at com.inductiveautomation.sfc.api.ExecutionQueue$PollAndExecute.run(ExecutionQueue.java:120)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.python.core.PyException: org.eclipse.tahu.exception.TahuException: org.eclipse.tahu.exception.TahuException: INTERNAL_ERROR
... 46 common frames omitted
Caused by: org.eclipse.tahu.exception.TahuException: INTERNAL_ERROR
at org.eclipse.tahu.mqtt.TahuClient.publish(TahuClient.java:437)
at com.cirruslink.mqtt.engine.gateway.EngineRPCImpl.publishImpl(EngineRPCImpl.java:60)
at com.cirruslink.mqtt.engine.common.EngineAbstractScriptModule.publish(EngineAbstractScriptModule.java:30)
at jdk.internal.reflect.GeneratedMethodAccessor495.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:190)
... 43 common frames omitted
Caused by: org.eclipse.tahu.exception.TahuException: MQTT client: ME-5383d9f5-8085-485d is not connected
at org.eclipse.tahu.mqtt.TahuClient.publish(TahuClient.java:434)

And finally, the TahuClient warning that is thrown after the SFC are aborted

org.eclipse.paho.client.mqttv3.MqttException: Unable to connect to server
at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:80)
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:724)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:74)
... 2 common frames omitted

.
.

Any support on how to trap this exception to prevent upsteam MQTT (RMQ) broker crashes from propagating to Ignition SFCs would be extremely helpful!

prince.cyril · October 25, 2023, 6:19pm

We have projects which has similar configuration as you described and I am eager to know what would be the solution to this matter.
Do you think this problem is something Inductive Automation would be better off at handling since it has to do with SFC depenencies ?

wes.johnson · October 31, 2023, 2:14pm

What happens if you just try to catch all Exceptions and not just java.lang.Exception? i.e. just use ‘except’.

Topic		Replies	Views
Ignition MQTT Issue: Increasing Threads Leading to Server Instability MQTT Modules mqtt-engine	1	72	June 18, 2024
Getting a SequenceNumberException Ignition Modules mqtt-transmission , mqtt-distributor , mqtt-engine	2	338	April 11, 2022
MQTT Transmission Module stops sending and will not reconnect after network hit Ignition Modules	1	84	April 24, 2024
Failed to handle Sparkplug B message MQTT Modules mqtt-engine	8	471	November 6, 2023
Ignition can't write to binary data, byte_array tags in 4.0.17 MQTT Modules	2	227	August 16, 2023

MQTT Engine INTERNAL_ERROR crashes Ignition Project

Problem Statement

Code to send MQTT Messages

MQTT Engine Plugin seems to be crashing with the following stacktrace

Related topics