Hi Jan, thanks for your notes. I have gone through the instructions in that doc link, but they do not help. In the prior emails I have included stack dumps and straces of what happens when I follow those instructions.
There is something else going on, or perhaps a regression that prevents the procedure you mention from working right. We are definitely NOT hitting ENOSPC, nor any FS or storage side issue. I have double checked on that. cheers, m On Tue, Jan 27, 2015 at 12:37 AM, Jan Lindström <jan.lindst...@mariadb.com> wrote: > Hi, > > Failure you see on > > InnoDB: Error: tablespace id is 47073 in the data dictionary InnoDB: but in > file ./moodle_sand/mdl_tag_correlation.ibd it is 48463! > 150119 17:14:45 InnoDB: Assertion failure in thread 140045886629632 in file > fil0fil.c line 778 > > indicates database corruption that is caused most likely disk failure or a > bug on a software. With this information not easy to say which one. If drop > table crashes while the operation is done, deleting .ibd file is not enough > to re-create the table because table definition is on innodb data dictionary > while it is not on MySQL/MariaDB data dictionary. Please refer to > http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html > how to resolve the issue. > > R: Jan > > On Tue, Jan 20, 2015 at 5:00 PM, Martin Langhoff <martin.langh...@gmail.com> > wrote: >> >> Hi Folks, >> >> We run a very large infra on MariaDB/RHEL6.x/RHEV/NetApp; thousands of >> VMs, each running MariaDB and very busy. MariaDB is 5.5.32. >> >> We recently had some NFS problems which led to minor DB corruption. >> Luckily, the corruption was isolated to a table holding cache data, >> and in fact one that is empty in most of our instances. That's the >> good news. >> >> The bad news is that the data dict is really unhappy about that table. >> >> - Cannot drop it, we get a nasty stack trace at [1] >> >> - Stopping MariaDB, removing the table by hand, restarting MariaDB, >> attempting to re-create the table hits a data dict error [2]. On disk >> I can see it created an ibd file, but no frm file. Following the >> instructions at >> >> http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting-datadict.html >> does not help. >> >> I have straced the process, and it does not seem to even try to create >> the frm file, so it really looks like it's internal state (data dict), >> not errors from further down the stack. OS / kernel logs are clean. >> >> Is there any reasonable method to cure this "in place" with minimal >> disruption? (These are large DBs, dump restore is more than awkward.) >> >> Oh and careful with NFSv4 :-/ >> >> thank you! >> >> >> martin >> >> 1 - Stack trace when trying to drop table >> >> Version: '5.5.32-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' >> port: 3306 MariaDB Server >> InnoDB: Error: tablespace id is 47073 in the data dictionary >> InnoDB: but in file ./moodle_sand/mdl_tag_correlation.ibd it is 48463! >> 150119 17:14:45 InnoDB: Assertion failure in thread 140045886629632 >> in file fil0fil.c line 778 >> InnoDB: We intentionally generate a memory trap. >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. >> InnoDB: If you get repeated assertion failures or crashes, even >> InnoDB: immediately after the mysqld startup, there may be >> InnoDB: corruption in the InnoDB tablespace. Please refer to >> InnoDB: >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html >> InnoDB: about forcing recovery. >> 150119 17:14:45 [ERROR] mysqld got signal 6 ; >> This could be because you hit a bug. It is also possible that this binary >> or one of the libraries it was linked against is corrupt, improperly >> built, >> or misconfigured. This error can also be caused by malfunctioning >> hardware. >> >> To report this bug, see http://kb.askmonty.org/en/reporting-bugs >> >> We will try our best to scrape up some info that will hopefully help >> diagnose the problem, but since we have already crashed, >> something is definitely wrong and this may fail. >> >> Server version: 5.5.32-MariaDB-log >> key_buffer_size=134217728 >> read_buffer_size=131072 >> max_used_connections=1 >> max_threads=187 >> thread_count=1 >> It is possible that mysqld could use up to >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = >> 182265 K bytes of memory >> Hope that's ok; if not, decrease some variables in the equation. >> >> Thread pointer: 0x0xa92ee60 >> Attempting backtrace. You can use the following information to find out >> where mysqld died. If you see no messages after this, something went >> terribly wrong... >> stack_bottom = 0x7f5ef9531d78 thread_stack 0x48000 >> ??:0(my_print_stacktrace)[0xa8631e] >> ??:0(handle_fatal_signal)[0x6cdd0b] >> ??:0(??)[0x30a7c0f500] >> ??:0(??)[0x30a78328a5] >> ??:0(??)[0x30a7834085] >> ??:0(fil_node_open_file)[0x8f3add] >> ??:0(fil_node_prepare_for_io)[0x8f3be5] >> ??:0(fil_space_get_flags)[0x8f7420] >> ??:0(fil_space_get_zip_size)[0x8f7529] >> ??:0(dict_drop_index_tree)[0x8d64e6] >> ??:0(row_upd_clust_step)[0x86d84d] >> ??:0(row_upd_step)[0x86e22e] >> ??:0(que_run_threads)[0x9577a8] >> ??:0(que_eval_sql)[0x957ea6] >> ??:0(row_drop_table_for_mysql)[0x85ae6a] >> ??:0(ha_innobase::delete_table(char const*))[0x8424e3] >> ??:0(ha_delete_table(THD*, handlerton*, char const*, char const*, char >> const*, bool))[0x6d5233] >> ??:0(mysql_rm_table_no_locks(THD*, TABLE_LIST*, bool, bool, bool, >> bool))[0x5ff928] >> ??:0(mysql_rm_table(THD*, TABLE_LIST*, char, char))[0x60001d] >> ??:0(mysql_execute_command(THD*))[0x58ad90] >> ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x58e369] >> ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned >> int))[0x58f790] >> ??:0(do_handle_one_connection(THD*))[0x647cdf] >> ??:0(handle_one_connection)[0x647d7c] >> ??:0(??)[0x30a7c07851] >> ??:0(??)[0x30a78e811d] >> >> Trying to get some variables. >> Some pointers may be invalid and cause the dump to abort. >> Query (0x7f5db800afa8): is an invalid pointer >> Connection ID (thread ID): 2 >> Status: NOT_KILLED >> >> Optimizer switch: >> >> index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off >> >> >> >> 2 - Error when attempting to recreate table >> 150119 17:15:22 InnoDB: Operating system error number 2 in a file >> operation. >> InnoDB: The error means the system cannot find the path specified. >> InnoDB: If you are installing InnoDB, remember that you must create >> InnoDB: directories yourself, InnoDB does not create them. >> 150119 17:15:22 InnoDB: Error: trying to open a table, but could not >> InnoDB: open the tablespace file './moodle_sand/mdl_tag_correlation.ibd'! >> InnoDB: Have you moved InnoDB .ibd files around without using the >> InnoDB: commands DISCARD TABLESPACE and IMPORT TABLESPACE? >> InnoDB: It is also possible that this is a temporary table #sql..., >> InnoDB: and MySQL removed the .ibd file for this. >> InnoDB: Please refer to >> InnoDB: >> http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html >> InnoDB: for how to resolve the issue. >> >> >> -- >> martin.langh...@gmail.com >> - ask interesting questions >> - don't get distracted with shiny stuff - working code first >> ~ http://docs.moodle.org/en/User:Martin_Langhoff >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-discuss >> Post to : maria-discuss@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-discuss >> More help : https://help.launchpad.net/ListHelp > > -- martin.langh...@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
