http://oldcgi.distributed...t-finger.cgi?user=decibel
The RC5-72 statsrun bombed; I'm going to have to re-load today's data, so
expect stats to be a bit late.
The problems with stats are more serious than I thought. The table that stores
how much work each email address did on each day no longer matches the other
tables. I know that sounds rather serious, but that information can always
be re-created from the log files if it comes to that. I'm in the process of
loading in a backup copy of that table; hopefully it will allow me to fix this
with a minimum of downtime.
More info when available...
RC5 stats are still wrong, but I've turned access back on. Hopefully I'll have
it fixed tomorrow.
As I feared, the Sybase stats database is corrupt. I have copies of all
participant data (team membership, retires, team and participant settings) as
of ~8/29/03 0:00 UTC. If I have to restore using this data, any changes people
made last night will be lost. I'm still trying to find a way to at least read
from the corrupt database, since only one table is corrupt. If I can do this,
no data will be lost during the restore.
Unfortunately, no matter which route I take, stats will be down for most of
today at a minimum.
More info as available.
Looks like all the trouble is being caused by hardware issues. Moose is working
on getting blower up and running again, at which point I'll have a better idea
what's left to be done.
Blower is up and running again; apparently it died because the raid controller
freaked out after a disk failure. I'm still hoping to recover any data that was
modified last night, but if that doesn't happen soon I'll just go with what we
have.
MattR was able to get the database online again, so I now have copies of the 3
tables. This means that no data should end up lost out of all of this.
There are 3 options we have right now. First, we can attempt to repair the
existing stats database. MattR's attempting this right now. The second option
is to drop the existing database, restore from the July 12th backup, and bring
in the updated information. The third option is for us to cut-over to stats
running on PostgreSQL, which is 95% done right now.
I'm playing it a bit by ear before deciding which way to go. Going to
PostgreSQL is very tempting, since we'll need to do it in the near future
anyway, but I don't like the idea of going to production when it's not complete
and hasn't been beta-tested.
Whatever happens, stats definitely won't be up until tomorrow afternoon at the
very earliest.
I just talked with MattR; he's not going to be able to recover the existing
database. We'll be restoring from a backup as soon as it's done bunzipping.
[ Voor 153% gewijzigd door [eNeRGy] op 30-08-2003 09:11 ]