An issue occurred affecting approximately 5% of all online devices, where devices were displayed as offline even though they were actually connected, and new connections from obniz.js could not be established.
The issue was caused by a specific server becoming isolated and removed from the routing pool while still maintaining connections to some devices.
Existing connections remained active during this period, and there was no impact when a device itself reconnected, as it would establish a new connection to another server. However, if an obniz.js connection was disconnected and then re-established while the device was still connected, the reconnection attempt could fail.
Additionally, connection logs during the affected period incorrectly recorded the devices as offline.
The issue was caused by a problem with inter-server communication. We will address this by updating our server health monitoring system.