Onderwerp: Grote storing EVOswitch / Easynet / Telecity 2 Leaseweb netwerk

Oorspronkelijk geplaatst door Bakker ICT

Het lijkt me onzin dat ze dat zo zeggen, van het kan max 96 uur duren. Schandalig als dat zo zou zijn, anders bel je ze gewoon even op. LSW is zich er zeker van bewust dat ze zoiets niet kunnen maken.

Mijn laatste dedi afgelopen maand gestopt, maar altijd bij problemen super snel en correct geholpen. En met colo ben ik ook tevreden, helaas van gisteren natuurlijk

Tuurlijk is het schandalig als dat zo is, maar wat wil je er tegen doen?
Je hebt geen SLA en in principe dus geen poot om op te staan.

Oorspronkelijk geplaatst door pierce

Tuurlijk is het schandalig als dat zo is, maar wat wil je er tegen doen?
Je hebt geen SLA en in principe dus geen poot om op te staan.

Het is je eigen risico, maar ik kan me nauwelijks voorstellen dat LSW zoiets doet en zegt. Daarom zeg ik, bel ze gewoon op. Geen enkel bedrijf kan zich zoiets permiteren dat snappen ze zelf ook wel. En blijf gewoon hangen / eisen dat ze direct helpen. Overigens bij een ded en helemaal bij meerdere deds heb je een stukje gratis support. Netwerksupport, reboots etc. Deze kan je gewoon eisen.

Maar goed, volgens mij ga ik lekker OT, ik heb alleen tussen half 3 en half 4 wat downtime gehad.

Aangekondigd onderhoud bij LSW:

http://89.238.146.98/maintenance.php?i=148

---
Emergency maintenance Ring3
Start: Fri 26 Sep 2008 03:30
End: Fri 26 Sep 2008 06:00

Dear customer,

As a precaution we will preform some emergency maintenances. This maintenance is needed to update the software running on our ring 3 routers. This ring is used mainly for bulk traffic between our hops. Following the problems we had yesterday we will upgrade to a new version of IOS. Impact to customers should be minimal as traffic will directed to our other rings. Customers can see packetloss at the moment of reboot and after reboot has finished. The reboot is needed on the following locations:
Easynet
Schuberg Philis
EvoSwitch
Telecity Redbus
Telecity 2
Nikhef

The software update is also needed so we can migrate ring 3 to mpls. This migration is scheduled for next week and details will be posted when we have the schedule ready.

We are currently still investigating all issues we had yesterday.

Leaseweb Update

Dear customer,

as you might or might not have noticed, there were some network issues at Leaseweb on Wednesday, September 24. Unfortunately, this affected our phone system as well, so for a period of time you could not reach our support.

We apologize sincerely for any inconvenience you might have experienced from this.

Currently we are still investigating the causes in detail. With this newsletter we would like to inform you about the current state of these investigations:

Starting 13:20 CET and continuing through the afternoon, we experienced some severe DDoS attacks on several IPs. The heavy load continued even after null routing the affected IPs. This was caused by multiple blade failures in one of our datacenters in combination with software errors.

At around 16:00 CET the network was stable again but got under a new stress attack right away. The connection between EVO and other parts of our network still experienced some package losses. We could keep the effect mostly local, so the overall network remained stable. At 19:00 everything was finally totally contained.

In summary, we can only say for sure at this stage that the instability was due to some combination of the DDoS attacks and hardware and software issues which were most likely caused or at least accelerated by the DDoS.

As the problems fluctuated and we took countermeasures right away, not all servers were affected and even the ones that were mostly only experienced very limited outages.

Nevertheless and again, we apologize for the trouble you have had with connectivity, your servers, websites and customers.

Please note that we will update the software on Ring 3 tonight as a precautionary measure. (This will also be announced separately on the NOC site.)

For the future, we already started taking precautions for similar issues several weeks ago: we have been working on redesigning our network setup for two new Cisco CRS-1 Backbone Core routers. This top notch technology will reduce the potential points of failure and enable us to react even faster to network issues. Furthermore, we have been redesigning our telephone system with redundancy capacities. It will go online in a few days, so you will be able to reach us in such cases of emergency and we can deliver the usual good Leaseweb service.

On this as well as the current issues we will keep you updated. Especially for the latter, please also check for news on our NOC (Network Operations Center) site at http://noc.leaseweb.com/ where we always announce important network issues.

We would also like to remind you on this occasion on the ways to contact our support. For any issues, please feel free to contact us at +31 20 3162880 via phone or support@leaseweb.com via email. Please do so especially if you are still experiencing problems with relation to Wednesday’s issues.

Best regards
The Leaseweb B.V. Management

Nette berichtgeving, overigens krijg ik geen mails meer vanuit het NOC ? Meer mensen hier last van?

Komt hierdoor Bakker ICT:
Content analysis details: (5.9 points, 4.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
3.9 HELO_LOCALHOST HELO_LOCALHOST
0.0 BAYES_50 BODY: Bayesian spam probability is 40 to 60%
[score: 0.4537]
3.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
[89.238.146.98 listed in zen.spamhaus.org]
-1.1 AWL AWL: From: address is in the auto white-list

Komt ook boven mijn trashold uit.

Dit zou niet meer voor moeten komen, volgens Spamhaus staat het IP niet in de lijsten.

Volgens mij zijn er weer problemen. Kon eerst mijn eigen sites niet goed benaderen en nu ook leaseweb.com niet meer (vanaf zowel upc als reasonnet netwerk)

Hier nog niets van te merken... laten we het zo houden

http://www.webhostingtalk.nl/quickte...=169226&r=1183

Vaak gaan ze allemaal goed, soms allemaal fout en vaak een paar fout.

[update 03:13]
We will start with at 3:30 with rebooting easynet.

[update 03:58]
reboot of easynet has finished schuberg philis is rebooting now.

[update 04:09]
we will now reboot telecity 2.

[update 04:20]
we will now reboot nikhef.

[update 04:26]
we will now reboot telecity redbus.

[update 04:34]
we will reboot evoswitch next. this will be done within the next 1/2 hour.

[update 04:54]
all reboots are done maintenance is closed.

Bron: http://noc.leaseweb.com/maintenance.php?i=155

Zo en dat is ook weer gebeurd.. Hier geen meldingen binnengehad dus het is zo te zien zonder downtime gegaan...

Oorspronkelijk geplaatst door TCM

Zo en dat is ook weer gebeurd.. Hier geen meldingen binnengehad dus het is zo te zien zonder downtime gegaan...

Ze hebben lang genoeg gehad om te oefenen

Storing opgelost, topic dicht.