Unable to start slurmdbd after system crash

[Trey Dockendorf]

Yesterday the VMs running slurmctld, slurmdbd and MySQL all crashed due to
issues with storage networks. I am now unable to start slurmdbd. The
error I get is below. The part about "is marked as crashed and should be
repaired", is that coming from MySQL or from SLURM? Any advice on how to
proceed is appreciated.

Thanks,
- Trey

[2014-10-21T22:12:18.849] error: mysql_query failed: 145 Table
'./mysql/proc' is marked as crashed and should be repaired
drop procedure if exists get_parent_limits; create procedure
get_parent_limits(my_table text, acct text, cluster text, without_limits
int) begin set @par_id = NULL; set @mj = NULL; set @msj = NULL; set @mcpj =
NULL; set @mnpj = NULL; set @mwpj = NULL; set @mcmpj = NULL; set @mcrm =
NULL; set @def_qos_id = NULL; set @qos = ''; set @delta_qos = ''; set
@my_acct = acct; if without_limits then set @mj = 0; set @msj = 0; set
@mcpj = 0; set @mnpj = 0; set @mwpj = 0; set @mcmpj = 0; set @mcrm = 0; set
@def_qos_id = 0; set @qos = 0; set @delta_qos = 0; end if; REPEAT set @s =
'select '; if @par_id is NULL then set @s = CONCAT(@s, '@par_id :=
id_assoc, '); end if; if @mj is NULL then set @s = CONCAT(@s, '@mj :=
max_jobs, '); end if; if @msj is NULL then set @s = CONCAT(@s, '@msj :=
max_submit_jobs, '); end if; if @mcpj is NULL then set @s = CONCAT(@s,
'@mcpj := max_cpus_pj, ') ;end if; if @mnpj is NULL then set @s =
CONCAT(@s, '@mnpj := max_nodes_pj, ') ;end if; if @mwpj is NULL then set @s
= CONCAT(@s, '@mwpj := max_wall_pj, '); end if; if @mcmpj is NULL then set
@s = CONCAT(@s, '@mcmpj := max_cpu_mins_pj, '); end if; if @mcrm is NULL
then set @s = CONCAT(@s, '@mcrm := max_cpu_run_mins, '); end if; if
@def_qos_id is NULL then set @s = CONCAT(@s, '@def_qos_id := def_qos_id,
'); end if; if @qos = '' then set @s = CONCAT(@s, '@qos := qos, @delta_qos
:= REPLACE(CONCAT(delta_qos, @delta_qos), \',,\', \',\'), '); end if; set
@s = concat(@s, '@my_acct := parent_acct from "', cluster, '_', my_table,
'" where acct = \'', @my_acct, '\' && user=\'\''); prepare query from @s;
execute query; deallocate prepare query; UNTIL (@mj != -1 && @msj != -1 &&
@mcpj != -1 && @mnpj != -1 && @mwpj != -1 && @mcmpj != -1 && @mcrm != -1 &&
@def_qos_id != -1 && @qos != '') || @my_acct = '' END REPEAT; END;
[2014-10-21T22:12:18.850] error: Couldn't load specified plugin name for
accounting_storage/mysql: Plugin init() callback failed
[2014-10-21T22:12:18.850] error: cannot create accounting_storage context
for accounting_storage/mysql
[2014-10-21T22:12:18.850] fatal: Unable to initialize
accounting_storage/mysql accounting storage plugin



=============================

Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: [phone]
Email: treydock-mRW4Vj+***@public.gmane.org
Jabber: treydock-mRW4Vj+***@public.gmane.org

[Trey Dockendorf]

Sorry for the noise, I had never seen this error before so figured it was
SLURM specific, but it's not. Been long day and should have googled it
first :)

mysql mysql -e "REPAIR TABLE proc" resolved the issue.

- Trey

=============================

Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: [phone]

Post by Trey Dockendorf
Yesterday the VMs running slurmctld, slurmdbd and MySQL all crashed due to
issues with storage networks. I am now unable to start slurmdbd. The
error I get is below. The part about "is marked as crashed and should be
repaired", is that coming from MySQL or from SLURM? Any advice on how to
proceed is appreciated.
Thanks,
- Trey
[2014-10-21T22:12:18.849] error: mysql_query failed: 145 Table
'./mysql/proc' is marked as crashed and should be repaired
drop procedure if exists get_parent_limits; create procedure
get_parent_limits(my_table text, acct text, cluster text, without_limits
@my_acct = acct; if without_limits then set @mj = 0; set @msj = 0; set
@mcpj = 0; set @mnpj = 0; set @mwpj = 0; set @mcmpj = 0; set @mcrm = 0; set
@def_qos_id = 0; set @qos = 0; set @delta_qos = 0; end if; REPEAT set @s =
@s = CONCAT(@s, '@mcmpj := max_cpu_mins_pj, '); end if; if @mcrm is NULL
@def_qos_id is NULL then set @s = CONCAT(@s, '@def_qos_id := def_qos_id,
@s = concat(@s, '@my_acct := parent_acct from "', cluster, '_', my_table,
@mcpj != -1 && @mnpj != -1 && @mwpj != -1 && @mcmpj != -1 && @mcrm != -1 &&
@def_qos_id != -1 && @qos != '') || @my_acct = '' END REPEAT; END;
[2014-10-21T22:12:18.850] error: Couldn't load specified plugin name for
accounting_storage/mysql: Plugin init() callback failed
[2014-10-21T22:12:18.850] error: cannot create accounting_storage context
for accounting_storage/mysql
[2014-10-21T22:12:18.850] fatal: Unable to initialize
accounting_storage/mysql accounting storage plugin
=============================
Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: [phone]