Resolved -
Attention: Greensboro Data Center Clients Only
I am pleased to report that all services for our Greensboro Data Center server and hosting clients have now been restored.
At this time, we are closing this outage incident as resolved. We still have work to do with updating our firewall, monitoring, and BGP routing to our upstream Tier 1 carriers.
This incident was limited to customers with servers and hosting services located in our Greensboro Data Center. Our voice and messaging services were not impacted. Those services remained operational throughout the incident because of the geo-redundant design of our voice and messaging platform, with voice services also located in Raleigh, NC, and Dallas, TX,
If you are a Greensboro Data Center client and are still experiencing any connectivity or service issues, please let us know immediately. You may update your existing support ticket or open a new support ticket by emailing support@carolinadigitalphone.com.
I know this outage was disruptive, and I sincerely appreciate your patience while our team worked to restore every connection.
While I am closing the active outage incident, my personal follow-up does not end here. I will be reaching out individually to our long-term data center clients to discuss this incident, answer questions, and talk about the steps we are taking going forward.
Thank you for your patience, your understanding, and most importantly, the trust you have placed in Carolina Digital Phone over the past 25+ years.
Nicky Smith, Founder/CEO
Carolina Digital Phone
Aug 22, 18:06 EDT
Monitoring -
Attention Greensboro Data Center Clients Only:
We know how disruptive today's outage has been, and we sincerely apologize for the impact on your business. Connectivity in our Greensboro Data Center has now been restored for approximately 90% of our hosting and server clients. Our team will continue working through the night, without pause, until every client is back to 100%.
If your servers are in our Greensboro Data Center and you are still without connectivity, please let us know right away. With hundreds of connections across servers, switches, and customer equipment, it is possible we have missed something, and we would rather hear from you than leave you waiting. Simply update your existing support ticket to let us know you are still down, and we will escalate your connection as a top priority.
Please note: our voice and messaging clients experienced no interruption of service at any time. Thanks to the geo-redundant design of our platform, all calling and texting services remained fully operational throughout this event. If you are a voice or messaging client only, you may disregard this message.
Thank you for your patience and your trust. We do not take either for granted.
Nicky Smith
Aug 21, 19:00 EDT
Update -
To our Greensboro Data Center colocation clients:
We want to give you a direct update on the ongoing issue affecting our Greensboro data center.
The replacement router is on a truck this morning, en route to the data center, and is expected to arrive this morning. Once it's on site, our team will install and test it, with the goal of having it back online today.
Please note: this issue is isolated to data center services (hosted infrastructure, backup, and disaster recovery environments) at our Greensboro facility. It has had no impact on our voice and messaging customers. Our geo-redundant platform has kept all voice and messaging services fully operational throughout this event.
We will follow up again as we get closer to having all data center services fully restored and validated, and once everything is back online, we will provide a detailed report covering the actions taken and the actions we will have in place to ensure this never happens again.
We appreciate your patience and apologize for any inconvenience this has caused.
Regards,
Nicky Smith
Founder, Carolina Digital Phone
Aug 20, 08:18 EDT
Update -
Update - Dear Greensboro Data Center Clients,
We are pleased to report that some services have now been restored using temporary routing appliances while our team continues working toward full restoration. Based on our current progress, we expect all services to be restored within the next two days.
This outage is limited to colocation and data center clients within our Greensboro Data Center. Carolina Digital Phone voice and messaging clients should have experienced no service interruption. Our geo-redundant platform, with additional data centers in Raleigh, North Carolina, and Dallas, Texas, continued carrying voice and messaging traffic as designed.
Our entire technical team remains focused on restoring the remaining Greensboro Data Center services. We sincerely appreciate your patience and will continue updating this status page as we make progress.
Nicky Smith, Founder
Carolina Digital Phone
Aug 17, 19:43 EDT
Update -
Dear Greensboro Data Center Clients,
I want to speak with you directly and honestly about where things stand.
This morning we received what we believed was the replacement part needed to bring our core router back online. FedEx delivered it on schedule, our team installed it immediately, and we tested it with real hope that this message would be the one telling you service was restored. It was not. The part did not resolve the issue, and I know how frustrating that is to read, because it was frustrating for us to discover.
We escalated at once. A complete replacement router has been shipped and is scheduled to arrive Monday. All router configurations are saved, so once the hardware is in our hands, the cutover should be quick.
I also want to be straightforward about why this has taken longer than any of us would have wanted. High-end data center routers are not equipment anyone keeps sitting on a shelf, and supply chains for this class of hardware are slower today than they were a few years ago. That is an explanation, not an excuse. The responsibility for getting you back online is ours, and we accept it fully.
If there is one piece of reassurance I can offer, it is this: our hosted voice and messaging clients have not missed a single call throughout this event. Our geo-redundant voice platform design, with additional data centers in Raleigh, NC and Dallas, TX, carried that traffic exactly as it was engineered to do.
Over more than 25 years, across three generations of router platforms, we have never had an outage last longer than a few minutes within a planned maintenance window. That record makes this one sting all the more. You trusted us with your operations, and this week we have fallen short of the standard you deserve and the one we hold ourselves to. I am sorry. Our entire team is working around the clock, and when this is behind us, we will take a hard look at what must change so it never happens again.
You will hear from me the moment the new router arrives and again when service is fully restored. If you have questions or concerns before then, please call us at (336) 346-6000. A real person on our team will answer.
Thank you for your patience, and for the trust you have placed in us for so many years. We intend to keep earning it.
With sincere apologies,
Nicky Smith
Founder, Carolina Digital Phone
Aug 14, 20:37 EDT
Update -
We are continuing to work on a fix for this issue.
Aug 14, 20:23 EDT
Update -
Update: We have identified the root cause of the outage as a hardware failure within our core router.
Our entire technical team is actively engaged, and we have all available resources focused on restoring service as quickly and safely as possible. We are working with senior network engineers and our vendors to determine whether the best course is to replace the failed component or accelerate the major router upgrade that was already planned for this year.
High-end data center routing equipment is specialized and not typically available for immediate delivery. We are actively sourcing the necessary upgrade equipment today and evaluating every available option to reduce the restoration time.
We understand how important these services are to our affected customers, and resolving this outage is our highest priority. We sincerely apologize for the disruption and appreciate your patience while our team works through this hardware failure.
This outage is limited to customers with equipment colocated in our Greensboro data center. Carolina Digital Phone voice and messaging customers remain fully operational and have experienced no service impact from this outage.
Our team will remain actively engaged until service is restored. We will continue posting updates to this status page as additional information becomes available and as we make progress toward restoration.
Aug 13, 16:14 EDT
Update -
We sincerely apologize for this outage and the disruption it is causing.
We have identified the source of the outage as an SSD failure in our core router, which resulted in corrupted system files. Unfortunately, the failover routing engine was also affected because it contained copies of the corrupted files. We do not currently believe this is a cyber attack on our Greensboro Data Center network.
We are working directly with senior router engineers to determine the safest and most effective recovery procedure. We want to be candid that we do not expect this to be a quick restoration. Please know that we are treating this as our highest priority and working to restore service as quickly as possible.
This outage affects only data center customers who host servers in our Greensboro Data Center. It should not affect our voice or messaging customers. Our geographically redundant platform continues to carry voice and messaging traffic through our Raleigh and Dallas, Texas, data centers.
We are deeply sorry for the inconvenience. I will continue to provide updates on this status page as we make progress and learn more.
Aug 13, 01:43 EDT
Identified -
We are experiencing an outage in our Greensboro, NC data center. All voice traffic is easily handled by our geo-redundant network in Raleigh, NC, and Dallas, TX. More updates will be posted as the root cause is identified cause is discovered service restored.
Aug 13, 00:19 EDT
