Q: What’s happening?

We are aware of an issue on pl3.projectlocker.com. This affects all users whose server name ends in 3. We are researching the issue and attempting to bring the server back online. Updates are included in this blog in reverse chronological order.

29 June 06:44 GMT-5

The RAID battery that was faulty has been replaced and the server is back online. We will be reviewing the diagnostic results as soon as we receive them to see if another maintenance window is required. You should have access to services at this time.

We are aware of an issue on pl1.projectlocker.com. This affects all users whose server name ends in 1. We are researching the issue and attempting to bring the server back online. We will update this ticket as we learn more.

UPDATE: 20 Jan 21:49 GMT-5

There appears to have been a routing issue in our provider's data center. There was a bad connection between two routers outside of the ProjectLocker-specific portion of the network but inside our data center, and traffic through those routers had problems. The resulting packet loss was significant enough to impede network speeds.

Thank you for your patience. At this time, we believe the issue is resolved but will continue to monitor. Please try your connections now and let us know via support@ if you still see performance problems.

UPDATE: 20 Jan 20:59 GMT-5

The network team has identified and tuned a possible culprit. We are testing to determine the status.

UPDATE: 20 Jan 19:05 GMT-5 

Diagnostics on the network have proved inconclusive. We're doing further diagnostics on the server to determine if there's any hardware issue.

UPDATE: 20 Jan 17:29 GMT-5

The network team is performing a more detailed physical investigation of the network to determine where there might be problems.

UPDATE: 20 Jan 17:00 GMT-5

The network team has performed an initial investigation, and we're pushing for additional checking on a packet loss issue we're seeing on some of the internal switches inside the data center and outside of our specific network.

We've been tracking a networking issue affecting some of our customers on some of our servers today (see the list of affected servers below). We are aware of this issue and are actively working to resolve it. 

4:52 ET

The server appears to be back online. Customers should again have access.

4:08 ET

The RAID Arrays are 93% rebuilt.

3:21 ET

Tthe RAID Arrays are 67% rebuilt. 

2:21 ET

The faulty drives have been replaced and the RAID Arrays are 45% rebuilt. 

12:32 ET

The data center has reported two drives with warnings, which caused degradation of the array. We have requested that they replace the drive that is in worse shape, rebuild the array, and get customers back online ASAP. We will then take an extra backup of the server and schedule an outage to replace the second drive during off-hours, where there will be less customer impact.

11:22 ET