Ignition MQTT Issue: Increasing Threads Leading to Server Instability

Ignition Modules MQTT Modules
1

Hi everyone,

I’m currently facing a significant issue with Ignition related to MQTT threads that are steadily increasing until my server becomes unstable. The issue seems to be tied to the MQTT engine creating more and more threads, and I’m unsure of the root cause or how to resolve it. Below is an example of the thread messages I’m seeing:

Thread [MQTTEngineCust-45-f7d8386b] id=481, (WAITING for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b320bbe)
  java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method)
  java.base@17.0.10/java.util.concurrent.locks.LockSupport.park(Unknown Source)
  java.base@17.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(Unknown Source)
  java.base@17.0.10/java.util.concurrent.ForkJoinPool.unmanagedBlock(Unknown Source)
  java.base@17.0.10/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
  java.base@17.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
  java.base@17.0.10/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
  java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
  java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.base@17.0.10/java.lang.Thread.run(Unknown Source)

Additionally, I am receiving regular log entries indicating null messages over MQTT from some of our devices. Here is an example log:

JSON value for 'swisscom/mqtt/things/A81758FFFE08C4C8/uplink/DevEUI_uplink/ModelCfg' is NULL

Here is a screenshot showing a list of these threads for further context.

Screenshot 2024-06-18 at 08.50.28
Screenshot 2024-06-18 at 08.50.281253×1399 196 KB

Setup Details:

  • Ignition version: 8.1.41 (just upgraded to see if it would solve the issue)
  • MQTT module version: 4.0.21
  • MQTT Broker: Also cirrus link

Steps I’ve Taken So Far:

  1. Restarting the MQTT Engine and Ignition server.
  2. Updated Ignition and the modules

Questions:

  1. Has anyone experienced a similar issue with MQTT threads in Ignition?
  2. Could the null messages over MQTT be contributing to this problem?
  3. What steps can I take to debug and resolve this issue?

Any help or suggestions would be greatly appreciated!

Jasper

2

I don’t think this is something we’ve seen before. Can you open a ticket at support@cirrus-link.com and send full logs along with a full thread dump?

1 Like