Incident Date: May 19, 2020

On May 19, 2020, at 11:32 PM ET, We experienced an event with Tucows’ Domains platform impacting Domain lookup services and accessibility of enom.com. Tucows Engineers were engaged and started investigating the issue.

At 11:58 PM ET, Tucows Engineers identified the cause to be related to excessive logging resulting in capacity constraints.

On May 20, 2020, at 12:15 AM ET, Tucows Engineers increased the resources on the underlying nodes which reduced the impact of the incident.

At 12:45 AM ET, Tucows Engineers deployed a hotfix to address the logging issue to reduce further impact. At 1:50 AM ET, the deployment was successfully completed and both the underlying nodes synchronized completely.

Preventive Measures:

• Tucows to set up additional monitors and update the threshold on the existing monitors.
• Tucows to update the retention policy on the logged messages to prevent further capacity issues.

--

Thank you,

Tucows Engineering Team