Re: lua.org down

[Lorenzo Donati <lorenzodonatibz@...>]

Thank you very much for the prompt reply!

On 13/11/2016 12:10, Daniel Silverstone wrote:
> On Sun, Nov 13, 2016 at 11:17:58 +0100, Lorenzo Donati wrote:
> > Just curious, could you explain in a bit more detail what happened? I was
> > talking about catastrophic system failures to my students [1] last week and
> > maybe this could make a nice case study? Of course feel free to ignore my
> > request if you are too busy or if you cannot disclose the details.
>
> The machine in question is a virtual machine which means we actually got
> to look at its console.  The console was full of messages along the lines
> of:
>
> INFO: task <process>:<pid> blocked for more than 120 seconds
>
> In this instance, it was pretty much all the apps which suggests that the
> IO subsystem for the VM had a hiccough.  The host system was fine so we
> are basically only going to blame bogons for the fault.

So no dead rat entangled in mains wiring :-D, but still an interesting case.

> As for not spotting it in time; that was simply human error combined with
> bad error reporting design.  The box in question is our primary web server
> which means the monitoring apps present their reports there; and as a human
> I simply didn't try to look at anything for about 12 hours.
>
> In mitigation, I have in the past mentioned ways to contact me out of band
> (since yes, that server is also the mail delivery box) but in the end it
> was simply the sort of thing that a non-professional small-time hosting
> provider hits.

Don't worry, I think we all appreciate your efforts here.

BTW, it is a particularly interesting case for my students because the 
local business situation here is made of primarily small firms, 
sometimes with highly varied and specialized needs (touristic 
facilities, high-tech niche mechanics, civil engineering design teams, 
etc.). Oftentimes a sysadmin will end working in a very resource-limited 
environment, with few personnel available for ICT things.


> Sorry again for the inconvenience.
>
> D.

Cheers!

-- Lorenzo