Hello and welcome to our community! Is this your first visit?
Register
Results 1 to 3 of 3
  1. #1
    Obsessed Veteran Adam's Avatar



    Join Date
    Nov 2004
    Location
    UK / England
    Posts
    17,271

    Network problems

    SoftLayer Emergency Maintenance - GLOBAL [12/30/2010]
    Date: Thursday, December 30, 2010 (12/30/2010)
    Start Time: 05:00 PM CST
    End Time: 07:00 PM CST
    Services affected: Public / Private Network
    Location: Global
    Duration: 2 hours

    ==================================================
    SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded.

    Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload.

    Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted).
    ==================================================

    If you have any problems after this time frame with regard to connectivity, or if you have any questions regarding the maintenance at any point, please open a ticket in the customer portal or contact our support via phone.

    We appreciate your patience during this work and welcome any feedback.

    Thank you,
    Network Engineering

  2. #2
    Obsessed Veteran Adam's Avatar



    Join Date
    Nov 2004
    Location
    UK / England
    Posts
    17,271

    Re: Network problems

    Engineers have completed the reload of BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01. This has resolved the immediate issues with customer connectivity.

    During work on these routers, it was determined the same issue is occurring at a much smaller scale at a number of network POPs around the SoftLayer backbone. Engineers will be reloading each of those routers later tonight. Prior to each reload, traffic will be shifted off to redundant paths. Engineers expect no interruption to customer services during these reboots; however, some customers may experience higher latency as their routes shift to other paths.

    Thank you,
    Network Engineering

  3. #3
    Obsessed Veteran Adam's Avatar



    Join Date
    Nov 2004
    Location
    UK / England
    Posts
    17,271

    Re: Network problems

    SoftLayer Engineers are aware of a routing issue impacting a subset of customers globally on the SoftLayer backbone. Engineers have traced this issue back to a hardware forwarding limitation on a number of Cisco routers in the backbone. Engineers are currently in contact with Cisco to determine the best course of action to resolve this issue.

    We apologize for the inconvenience and appreciate your patience during this issue.

    Sincerely,
    SoftLayer Network Team


 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •