Ticketing Delays: Third Party Incident
Incident Report for Electric
Resolved
This incident has been resolved.
Posted Jul 28, 2023 - 09:20 EDT
Update
Hello,

As of 9:02 PM ET, Kustomer has recovered from their platform outage that impacted chat and ticketing for Electric. You can now use the regular support channels to contact our service desk for assistance.

The temporary email address we set up during the outage has been disabled, and an auto-responder has been set to ask you and your team to use Slack / Microsoft Teams or the End User Hub to submit support requests. Please refrain from using this email address going forward.

In the coming days, we will work closely with Kustomer to fully understand this platform outage, and we will provide you with a summary of the root cause analysis.

Thank you again for your patience and understanding during this Kustomer outage. We sincerely appreciate your business.

Sincerely,

Todd Mitchell
Chief Operating Officer
Electric AI, Inc.
Posted Jul 27, 2023 - 21:16 EDT
Update
As of 8:00 PM ET, Kustomer continues to recover from their outage. A portion of their platform has returned to normal operation and working as expected. We continue to monitor the performance and stability of their platform very closely.

As soon as Kustomer informs us that their platform is fully operational, we will run several acceptance tests internally before marking this issue resolved. We will send a follow-up communication with additional information at that time.

In the meantime, if you have an urgent request that requires immediate assistance, please contact outage-support@electric.ai. We have our service desk team monitoring this address, and they can provide you with immediate assistance.

Thank you again for your patience.

Sincerely,

Todd Mitchell
Chief Operating Officer
Electric AI, Inc.
Posted Jul 27, 2023 - 20:12 EDT
Update
At approximately 11:30 AM ET today, we began to experience instability with the vendor we leverage as a chat and ticketing platform. The platform that supplies chat and ticketing capabilities, Kustomer, notified its customers just before noon that they were experiencing a material disruption to their service, affecting a substantial portion of their customer base.

We have been in close contact with senior management at Kustomer as they work diligently to resolve the challenges with their support platform. As of 5 PM ET, a portion of their platform has been restored to production, and they are now scaling the necessary web application resources to manage the inbound traffic to their platform.

Kustomer has not provided an Estimated Time to Repair (ETR) for their platform. They continue to post frequent updates on their status page located here: https://status.kustomer.com/

While Kustomer is not accessible, the requests you and your team members have submitted through the End User Hub (web interface) are currently queued. Once Kustomer is back online and stable, your requests will receive a ticket number, and our team will be staffed and ready to work on your requests as soon as the Kustomer system allows us to.

If you or your team members submitted any requests via chat (Slack / Microsoft Teams), those real-time messages are not queued. In this case, those requests must be resubmitted once Kustomer returns to normal operations.

We will send another communication once the Kustomer platform has stabilized. This will allow you and your teams to submit support requests. We will also post additional notifications and updates on our official status page located here: https://status.electric.ai/

We sincerely apologize for the inconvenience this disruption has caused you and your team members. Please rest assured that we continue to work closely with Kustomer to restore service. And we will follow up with a Root Cause Analysis (RCA) once the platform has stabilized and we receive additional information from Kustomer.

Thank you for your patience as we work through this disruption.

Sincerely,

Todd Mitchell
Posted Jul 27, 2023 - 17:39 EDT
Monitoring
The issue has been identified as a problem with third-party (Kustomer's) API. We will continue to monitor the issue.
https://status.kustomer.com/incidents/ynpzl8z0r8wc
Posted Jul 27, 2023 - 17:15 EDT
Identified
The issue has been identified as a problem with a third-party (Kustomer's) API.

https://status.kustomer.com/incidents/ynpzl8z0r8wc
Posted Jul 27, 2023 - 11:56 EDT
Investigating
Electric is aware of an event reported by one of our third-party vendors affecting communication through our ticketing system. This may prevent customers from sending or receiving messages through messaging apps, email, or the Electric platform.

Our team will continue to take action from our side and will proactively monitor the status with our vendor. Please expect further updates within the next hour.
Posted Jul 27, 2023 - 11:53 EDT
This incident affected: Electric Services (Service Desk Correspondence).