We are continuing to monitor for any further issues.
Posted Jan 05, 2023 - 20:24 EST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 05, 2023 - 17:40 EST
Update
We are within a few minutes of having a full resolution to the Greensboro, NC Voice Services platform. The engineering team is still working to determine the root cause that caused this issue. They are suspicious of a wildcard SSL issue on the Greensboro, NC voice service server and a possible corrupt SSL certificate. Not having a valid and working SSL makes the geo-redundancy voice platform vulnerable to being compromised and we do not want this issue to put calls at risk. This SSL configuration has now been resolved and our team has plans to continue testing tonight during a maintenance window. We plan to keep the Greensboro, NC voice server in 'Degraded Performance' mode until a final tested resolution is determined. If you are having issues with your mobile application, just log out of the app and log back in and it will work fine.
We have stabilized the call audio issue now and two of the three voice servers are online and fully operational. Call volume has also stabilized and you should not be experiencing any in our outbound call issues at this time. We are still investigating the issue with our voice services platform in our Greensboro, NC data center and will update you when services are fully restored. As stated earlier the issue we experienced was that calls that were routed to our Greensboro voice services platform made a correct connection but the audio channel did not connect correctly. The Greensboro, NC server is still in a "Partial Outage" situation but all voice traffic is being balanced between our Dallas TX, and Research Triangle Park NC voice services servers.
Posted Jan 05, 2023 - 16:29 EST
Update
The "degraded performance" of voice services located in Research Triangle Park, NC (at our data center at https://mcnc.org) is NOT due to any outage in that data center but the massive volume of calls, both inbound and outbound that are failing over to this voice server. This Research Triangle Park, NC voice services server should stabilize in a few minutes. Thank you for your patience. We will continue to update this status page as all services are stable in all three data centers.
Posted Jan 05, 2023 - 15:52 EST
Update
We are continuing to work on this outage in our Greensboro, NC data center and have all voice engineers working to resolve the issue. What we are seeing currently is calls hitting our switch with no audio traffic passing to or from us. Because the call is not having a hard failure, it thinks the call is good and does not failover to another geo-redundant server in another data center. We will continue to update you via this page as we find the root cause and resolve this issue.
Posted Jan 05, 2023 - 15:35 EST
Identified
We are experiencing issues with our Greensboro Datacenter but all calls are failing over to our Research Triangle Park, NC data center. The problems have been identified and we are hopeful all services will be restored quickly. Because our three data centers operate in a fully geo-redundant environment you should have no issues whatsoever and will not experience any downtime. Your desktop application and the mobile smartphone will continue to work as they do now. All in and outbound calls will flow normally on the other two server clusters. If you do experience an issue please send an email to support@carolinadigitalphone.com with the details of any issues you might experience. Our voice engineers will look at your issue quickly and resolve it within a few minutes.