Ventrilo Server Host Announcement - Resolved

Finally our servers have been upgraded but unfortunately our hosts are experiencing some issues, there is a back up ventrilo server kindly offered by Officer Quaint, please contact one of your Legion officers for the ventrilo information, we don't expect the outage to last long.

Thank You Quaint :)

London, Paris, and Ireland Issues Ongoing

Updated: 6:34 PM BST on September 25th, 2009.

Rapidswitch (our provider in London -- we are at the mercy of them at the moment) continues to make final adjustments to their new router configuration this afternoon as advised by Cisco. This could mean a couple more small down time problems this evening, but they promise to have everything resolved in relation to their routing gear today, and are working as fast as they can and in conjunction with Cisco systems. Please see http://universe.darkstarllc.com/showthread.php?t=57 for more details on the maintenance window (it was an emergency according to the provider). Thank you!

Temporary Servers For Customers

Because of such a serious outage, we are giving EVERY customer the opportunity to create a temporary server in another location. Please submit a ticket if you wish to take this approach. We've got extra staff on site right now helping answer tickets and to accommodate people. Thank you.

Alternately, if you would like we have set up a massive server located at v44.darkstarllc.com:27766 . Please feel free to log into this and ask one of us in there for your own private channel

Outage Report & Summary

• 09/24/2009 | 11:00 PM - 3:30 AM | 4.30 hours of emergency maintenance and thus service outage.
• 09/25/2009 | 2:30 PM - 3:00 PM | 30 minutes of emergency maintenance and thus service outage.
• 09/25/2009 | 5:00 PM - (now) | 1 hour+ of emergency maintenance and thus service outage.

Latest Statement From RapidSwitch

Below is a summary update of the issues experienced by clients in RSH-North today and yesterday, 25th and 24th September 2009.

Fundamentally we have experienced serious issues that have affected all clients in RSH-North. This has been due to the Cisco equipment at the core of our Spectrum House services not responding as per specification and documentation.

As has already been explained, we have been in direct contact with Cisco, working on a Level 1 priority request to solve the issues that have affected our clients today. We take full responsibility for our vendor selection and do not wish to appear to be passing blame "conveniently". We pride ourselves in the level of service that we provide and also in the quality of communication that we send to clients. We accept that neither have been anywhere close to our usual standard during this prolonged incident. However, we would like to take this opportunity to clarify a few points that we are aware have been questioned and discussed by our clients and competitors:

- RapidSwitch's Spectrum House routers were configured in a redundant VSS cluster
- CPU usage was recorded as being very high for normal usage
- Cisco have offered two possible solutions to the issue, including a firmware update provided to RapidSwitch yesterday. These solutions have failed to resolve the issues experienced
- RapidSwitch's priority has, at all times, been to provide stable service for as much time as possible, to as many clients as possible. This has been the endeavour and sometimes this has not been possible. We do not shirk the responsibility for this matter and recognise the impact that it has on our clients' businesses and both our and their reputations. - We will be looking at the issue of compensation next week. Obviously all of our clients will have been financially affected by the issues this week and we will be taking a detailed look at what we can do for our clients, but the primary issue will always be the continuity of service.


Our network team are continuing to investigate why some of our clients are still experiencing outages, interrupted service and packet-loss and are doing so with Cisco.

We will endeavour to provide further updates and detail to all clients affected but we will ultimately also provide a full explanation next week.

Regards,

Paul Tacey-Green on behalf of the RapidSwitch Team