With Oracle’s stock MySQL you cannot move your ibd files freely from one server to another or from one database to another. The reason is that the table definition is stored in the InnoDB shared tablespace (ibdata) and the transaction IDs and log sequence numbers that are stored in the tablespace files also differ between servers. Therefore our example will be very straightforward: we’ll delete some rows from a table in order to recover the table later.

Most of these limitations are solved on Percona Server . More info about this in the conclusion section of this post. This post will be focus on how to recover a single tablespace using stock MySQL server.

First, you must meet certain prerequisites to be able to restore a ibd tablespace:

• The ibd file must be from a consistent backup with all insert buffer entries merged and have no uncommitted transactions in order to not be dependent of the shared tablespace ibdata. That is, shutting down with innodb_fast_shutdown=0. We’ll use XtraBackup to avoid the server shutdown.
• You must not drop, truncate or alter the schema of the table after the backup has been taken.
• The variable innodb_file_per_table must be enabled.

Then, our first step is to get a consistent backup.

First we need to copy all the data to an output directory:

The –export option is the magic trick that will help us to get a consistent backup with complete independent ibd files without shutting down the service. In the second step the use of –export option runs a recovery process on the backup with innodb_fast_shutdown=0 and therefore merging all the insert buffers.

# innobackupex --defaults-file=/etc/my.cnf --export /tmp/

Then apply the logs to get a consistent backup:

# innobackupex --defaults-file=/etc/my.cnf --apply-log --export /tmp/2012-01-22_14-13-20/

Now we’re going to delete some data from one table. In this case we’re going to delete the salary information from the user 10008:

mysql> SELECT * FROM salaries WHERE emp_no=10008;
+--------+--------+------------+------------+
| emp_no | salary | from_date | to_date |
+--------+--------+------------+------------+
| 10008 | 46671 | 1998-03-11 | 1999-03-11 |
| 10008 | 48584 | 1999-03-11 | 2000-03-10 |
| 10008 | 52668 | 2000-03-10 | 2000-07-31 |
+--------+--------+------------+------------+

mysql> DELETE FROM salaries WHERE emp_no=10008;

The next step is where we are going to save a lot of time and some headaches Instead of recovering all the InnoDB data we are going to recover only the “salaries” table:

• Discard the tablespace of the salaries table:

mysql> ALTER TABLE salaries DISCARD TABLESPACE;

• Copy the salaries.ibd files from the backup to the database data directory:

# cp /tmp/2012-01-22_14-13-20/employees/salaries.ibd /var/lib/mysql/data/employees/

• Import the new tablespace:

mysql> ALTER TABLE salaries IMPORT TABLESPACE;

mysql> SELECT * FROM salaries WHERE emp_no=10008;
+--------+--------+------------+------------+
| emp_no | salary | from_date | to_date |
+--------+--------+------------+------------+
| 10008 | 46671 | 1998-03-11 | 1999-03-11 |
| 10008 | 48584 | 1999-03-11 | 2000-03-10 |
| 10008 | 52668 | 2000-03-10 | 2000-07-31 |
+--------+--------+------------+------------+

The salary history from the user is back again!

Conclusion:

As we learned , you can also recover single InnoDB table as with MyISAM but knowing in advance that there are some prerequisites to comply.

Percona Server relaxes a lot of limitations and is able to import tables from different Server instance, when table was altered or truncated in the meanwhile. Though this only works if table was
“exported” with Xtrabackup as this exports essential information from main tablespace which is not stored in .ibd file. innodb_import_table_from_xtrabackup=1 should be enabled for such advanced import process to work. You can read more about this feature in Percona Server Documentation

In the next blog post I’ll explain how to do recovery using Percona Data Recovery toolkit.

View full post on MySQL Performance Blog

View full post on MySQL Performance Blog

We have a raft of new features planned for future releases, including advanced configuration options, supersafe settings to prevent bad things from happening, and much more. In the future we plan to add more online tools to help you be more productive.

Please give it a spin and let us know what you think. Credit for this tool must go to Miguel Trias, our talented lead developer. Thanks also to the many experts inside Percona who helped test, and made suggestions for improving the tool.

View full post on MySQL Performance Blog

View full post on MySQL Performance Blog

While cleaning out the digital attic I ran into this drawing that Hugh MacLeod (aka “gapingvoid”) made for me in reponse to a storm-in-a-teacup about blogging Amazon. As usual Hugh came straight to the heart of the matter

src="http://www.allthingsdistributed.com/images/should%20amazon%20blog.jpg" width="300" height="318" class="mt-image-center" style="text-align: center; display: block; margin: 20px 20px 20px 20px;" />

BTW Hugh’s new book “Evil Plans, having Fun on the Road to World Domination” was released last week. You can order it here

View full post on All Things Distributed

• all the tables ibd files (Nearly 200 of them, 40 GB of data)
• the schema in sql format
• No remote access

Their first question was “Is it possible to recover the data?”

The answer was yes since I was sure to at least be able to recover the data with the Innodb recovery tool of Aleksandr Kuzminsky a colleague on the Percona European team of consultants. In the past, I used this tool a few times and, although it is awesome to dig for data, it is time consuming and the perspective of recovering by hand nearly 200 tables was not really exciting. I needed something faster and more automatic so, after reading this post from Chris Calendar, I decided to use the following strategy.

1. Start with a clean MySQL install so that Innodb tablespace id is set to 1
2. Import the schema in MyISAM format to avoid screwing up the tablespace id
3. Record the tablespace id of all the tables provided by the customer and but the info in a MyISAM table
4. Advance the Innodb tablespace id by creating fake Innodb tables
5. Once the Innodb tablespace id is minus one of a customer table
1. alter the table definition from MyISAM to Innodb
2. Discard the tablespace
3. Replace the table ibd file with the one provided by the customer
4. Import the tablespace
5. At this point I should have access to the table but, although MySQL was not complaining, the data was not available. An “alter table tablename engine=InnoDB;” would solve the problem but then, the tablespace id would increase by one, annoying. I decided, also for portability, to do “alter table tablename engine=MyISAM;”
6. Script all that

In order to get a decent disk space and good internet connectivity, I fired up an Ubuntu 10.04 EC2 instance, installed MySQL and provided the customer the credentials to upload their files. Then, I proceed.

Loading the schema

root@domU:/mnt/tables# perl -p -i -e 's/ENGINE=InnoDB/ENGINE=MyISAM/g' schema.sql
root@domU:/mnt/tables# cat schema.sql | mysql -u root test

The MyISAM table recording the space ids of the tables

mysql> show create table test.tables\G
*************************** 1. row ***************************
       Table: tables
Create Table: CREATE TABLE `tables` (
  `name` varchar(100) DEFAULT NULL,
  `spaceid` int(11) NOT NULL,
  PRIMARY KEY (`spaceid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

Filling up the table with table names and space ids
The challenge was now to fill up the table. It is fairly easy to find the space id in a .ibd file.

#root@domU:/mnt/tables# hexdump profile_data.ibd -C | head -4
#00000000  62 bd c3 19 00 00 00 00  ff ff ff ff ff ff ff ff  |b...............|
#00000010  00 00 00 54 86 9a 91 93  45 bf 00 00 00 00 00 00  |...T....E.......|
#00000020  00 00 00 00 00 11 00 00  00 11 80 39 00 00 00 02  |...........9....|
#00000030  4f 00 00 02 47 c0 00 00  00 00 00 00 00 13 00 00  |O...G...........|

In the above example, the table space id is “00 11″ as stored in bytes 0×24 and 0×25 (actually, 0×22 and 0×23 are also for tablespace id but always 00 in this case). To load the data was just some Bash scripting:

# for t in `ls *.ibd`; do tname=`echo $t | cut -d'.' -f1`; spaceid=`hexdump -C $t | head -3 | tail -1 | awk '{print $6$7}'`; mysql -u root -e "insert into tables (name,spaceid) values ('$tname',conv('$spaceid',16,10))" test; done

Recovering the tables
With all this in place, recovery the data was just another Bash script:

#!/bin/bash

CURRENT_ID=1

mysql -u root "create database if not exists filler;"

while [ 1 ]
do
        CURRENT_SPACEID=`mysql -u root -B -N -e "select spaceid from test.tables where spaceid >= $CURRENT_ID order by spaceid limit 1;"`

        if [ "a$CURRENT_SPACEID" == "a" ]; then
                exit
        fi

        CURRENT_TABLENAME=`mysql -u root -B -N -e "select name from test.tables where spaceid =  $CURRENT_SPACEID;"`

        echo "doing $CURRENT_TABLENAME $CURRENT_SPACEID"

        # DO WE NEED TO SKIP SOME ID
        while [ "$CURRENT_ID" -lt "$CURRENT_SPACEID" ]
        do
                echo "skipping spaceID $CURRENT_ID"
                mysql -u root -e "create table space_$CURRENT_ID (id int) engine=Innodb;" filler
                let CURRENT_ID+=1
        done

        exit
        echo "creating the table"
        mysql -u root -e "create table recover.$CURRENT_TABLENAME like test.$CURRENT_TABLENAME;"
        sleep 1

        echo "switching to Innodb "
        mysql -u root -e "alter table recover.$CURRENT_TABLENAME engine=Innodb;"
        sleep 1

        echo "discarding tablespace "
        mysql -u root -e "alter table recover.$CURRENT_TABLENAME discard tablespace;"
        sleep 1

        echo "moving the datafile in place"
        cp /mnt/tables/$CURRENT_TABLENAME.ibd /var/lib/mysql/recover/
        chown mysql.mysql /var/lib/mysql/recover/$CURRENT_TABLENAME.ibd
        sleep 1

        echo "importing tablespace"
        mysql -u root -e "alter table recover.$CURRENT_TABLENAME import tablespace;"
        sleep 1

        echo "switching to MyISAM"
        mysql -u root -e "alter table recover.$CURRENT_TABLENAME engine=MyISAM;flush tables;"
        sleep 1

        let CURRENT_ID+=1
done

I did a backup of /var/lib/mysql before the first run to make sure I could restart if needed, which of course, I did a few times. And that, recovered all except one table. For that one, I had to use the Innodb recovery tools but even then, I had a hard time. I pulled in Aleksandr and he basically reversed engineer the table structure and found that the schema the customer provided (likely from dev or staging) contained an extra column. With a modified schema, the recovery completed. The customer was able to download the tables in MyISAM format and then he just convert them back to InnoDB.

View full post on MySQL Performance Blog

Were you excited by some other talk on MySQL Conference and Expo ? Feel free to leave it as a comment to this post.

View full post on MySQL Performance Blog

I used to use MacPorts on my Macs to install the necessary libraries that are needed by Drizzle. I’ve recently dumped that because I didn’t like all of the extra stuff that was installed (do you really need to install a separate Perl installation?). And a recent b0rk of their Perl installations was the final straw.

It turns out that all you really need are just a few extra packages to build Drizzle on your Mac. Here are the packages that I currently have installed on my machines:

• autoconf
• automake
• libtool
• pcre
• boost
• protobuf

export PATH=/usr/local/bin:$PATH

With all that in place, to build Drizzle is just:

• ./config/autorun.sh
• ./configure

View full post on Planet Drizzle

I used to use MacPorts on my Macs to install the necessary libraries that are needed by Drizzle. I’ve recently dumped that because I didn’t like all of the extra stuff that was installed (do you really need to install a separate Perl installation?). And a recent b0rk of their Perl installations was the final straw.

It turns out that all you really need are just a few extra packages to build Drizzle on your Mac. Here are the packages that I currently have installed on my machines:

• autoconf
• automake
• libtool
• pcre
• boost
• protobuf

export PATH=/usr/local/bin:$PATH

With all that in place, to build Drizzle is just:

• ./config/autorun.sh
• ./configure

View full post on Planet Drizzle

I used to use MacPorts on my Macs to install the necessary libraries that are needed by Drizzle. I’ve recently dumped that because I didn’t like all of the extra stuff that was installed (do you really need to install a separate Perl installation?). And a recent b0rk of their Perl installations was the final straw.

It turns out that all you really need are just a few extra packages to build Drizzle on your Mac. Here are the packages that I currently have installed on my machines:

• autoconf
• automake
• libtool
• pcre
• boost
• protobuf

export PATH=/usr/local/bin:$PATH

With all that in place, to build Drizzle is just:

• ./config/autorun.sh
• ./configure

View full post on Planet Drizzle