AWS Import/Export transfers data off of storage devices using Amazon’s high-speed internal network and bypassing the Internet. With this new functionality AWS Import/Export now supports importing data directly into Amazon EBS snapshots. Once loaded into an Amazon EBS snapshot, The customer can create a volume based on that snapshot and attach it to an Amazon EC2 instance, or they can share that snapshot with others.

Amazon Import/Export is an important tool for customers to accelerate moving large amounts of data into the AWS storage systems. It is the proverbial “Do not underestimate the bandwidth of a Fedex box” but it are not only bandwidth constrained customers that are using the service, also those who have complex object collection layouts and who feel more comfortable writing it to a disk than transferring it over the network. With Import into EBS customers can now develop arbitrary complex layouts as the import service is doing a full binary copy of the disk into Amazon EBS and is not interpreting file system layouts, etc.

More information on AWS Import/Export can be found at their detail page.

View full post on All Things Distributed

For a chance to win a free ticket, check out the guest blog post I wrote for Engine Yard on how to make Rails migrations faster and safer.

In other news, Percona Live now has an open-bar evening event, sponsored by Clustrix. It is open to all, not just registered attendees of Percona Live. You can come to Mickey Mantle’s legendary Sports Bar and Restaurant for free drinks and food, networking, prizes, and a chance to win an Unlimited Gold MySQL Support contract from Percona, worth over $15,000!

In case you don’t know, Percona Live is a one-day intensive MySQL conference, with four tracks of 100% technical content from world-famous MySQL experts. Tickets are still available at http://www.percona.com/live/. Registration is not required to attend the evening event, so if you can’t attend Percona Live, come meet us afterwards anyway! We’ve arranged for a substantial amount of free food, so you won’t have to leave to find dinner.

The evening event begins at 6pm this Thursday the 26th, and is open-bar (free drinks) until 8pm. Mickey Mantle’s is at 42 Central Park South. From the conference center, just walk west on 59th street. For more information and directions, visit the Percona Live page for venue details.

View full post on MySQL Performance Blog

It is available from launchpad source tree lp:sysbench .

This is set of LUA scripts for sysbench 0.5 ( it supports scripting), and it works following way:

- you should use --test=tests/db/oltp.lua to run OLTP test
i.e. prepare

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=25 prepare

run:

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=25 --num-threads=5 run

oltp.lua should understand most options that available for regular sysbench –test=oltp

there are couple other scripts, like
oltp_simple
select
insert
delete
update_index
update_non_index

to support different OLTP modes.

Most interesting: all scripts support --oltp-tables-count=N ( default 1), in this case N sbtest tables will be used.

Additional parameter --oltp-secondary is taken from Mark Callaghan’s sysbench tree , in this case
table sbtest will be created with KEY xid (ID) , instead of PRIMARY KEY (ID)

Now, as we have N tables, we may want to prepare them in parallel, to save some time for preparation
I implemented that in script parallel_prepare.lua

so instead of

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=25 prepare

you may use

./sysbench --test=tests/db/parallel_prepare.lua --oltp-tables-count=64 --num-threads=8 run

( please note that oltp-tables-count should be multiplier of num-threads)

View full post on MySQL Performance Blog

• SSL support – add ?useSSL=true in the connection string (and do the usual ssl-in-java magic to make it work. Check the MySQL documentation for more information, it is set up exactly the same way here)
• Multi queries – add ?allowMultiQueries=true to your connection string to be able to send several queries in one round trip to the server.

Note that these features are only supported when using the driver against a MySQL server – Drizzle does not have these features (yet).

Download it from the central maven repo (should be synced within an hour or so)

View full post on Planet Drizzle

We’ve been advocates of the Drizzle project for a long time. I’ve blogged several times about the investments that Drizzle developers are making in the project’s technical and social assets: a clean codebase and an engaged community. I think Drizzle has a lot of things right to make a long-term play at producing a really good database server.

Now that Drizzle is GA, it would be downright strange if we didn’t offer commercial support for it. We welcome current and prospective Drizzle users to contact us about Drizzle support (and consulting) contracts. As always, we will do our best to give objective advice about its merits. And you can watch this blog for much more technical content on Drizzle.

View full post on Planet Drizzle

We’ve been advocates of the Drizzle project for a long time. I’ve blogged several times about the investments that Drizzle developers are making in the project’s technical and social assets: a clean codebase and an engaged community. I think Drizzle has a lot of things right to make a long-term play at producing a really good database server.

Now that Drizzle is GA, it would be downright strange if we didn’t offer commercial support for it. We welcome current and prospective Drizzle users to contact us about Drizzle support (and consulting) contracts. As always, we will do our best to give objective advice about its merits. And you can watch this blog for much more technical content on Drizzle.

View full post on MySQL Performance Blog

If someone in the community wants to take over maintenance of Drizzle on FreeBSD, we would be delighted! It is simply a matter of it taking too much of our limited time to keep things running on this system.

UPDATE:  As you can see below, we’ve had the FreeBSD port maintainer offer to help us keep things going.  Go open source! : )

View full post on Planet Drizzle

If someone in the community wants to take over maintenance of Drizzle on FreeBSD, we would be delighted! It is simply a matter of it taking too much of our limited time to keep things running on this system.

UPDATE:  As you can see below, we’ve had the FreeBSD port maintainer offer to help us keep things going.  Go open source! : )

View full post on DrizzleBlog

Setup for Server A

drizzled –datadir=/home/jdaly/drizzle/repos/inno_repl/tests/serverA –innodb.replication-log –-server-id=1 –drizzle-protocol.port=9307 –mysql-protocol.port=3307

Setup for Server B

drizzled –datadir=/home/jdaly/drizzle/repos/inno_repl/tests/serverB –server-id=2 –innodb.replication-log –plugin-add=slave –slave.config-file=/home/jdaly/drizzle/repos/inno_repl/tests/serverB/slave.config –drizzle-protocol.port=9308 –mysql-protocol.port=3308

slave.config

master-host=127.0.0.1
master-port=9307
master-user=test
master-pass=me
max-reconnects=1000
seconds-between-reconnects=1

Setup for Server C

drizzled –datadir=/home/jdaly/drizzle/repos/inno_repl/tests/serverC –server-id=3 –innodb.replication-log –plugin-add=slave –slave.config-file=/home/jdaly/drizzle/repos/inno_repl/tests/serverC/slave.config –drizzle-protocol.port=9309 –mysql-protocol.port=3309

slave.config

master-host=127.0.0.1
master-port=9308
master-user=test
master-pass=me
max-reconnects=1000
seconds-between-reconnects=1

Pretty easy to setup, now lets look at the data on Server A compared to Server C in the replication tables.

Server A

drizzle> create database serverA;

drizzle> select ID,  SEGID, COMMIT_ID, END_TIMESTAMP from DATA_DICTIONARY.SYS_REPLICATION_LOG;

+------+-------+-----------+------------------+

| ID   | SEGID | COMMIT_ID | END_TIMESTAMP    |

+------+-------+-----------+------------------+

|  772 |     1 |         1 | 1300842502563167 |

|  818 |     1 |         2 | 1300842824136908 |

+------+-------+-----------+------------------+

Server C

drizzle> select ID,  SEGID, COMMIT_ID, END_TIMESTAMP from DATA_DICTIONARY.SYS_REPLICATION_LOG;

+------+-------+-----------+------------------+

| ID   | SEGID | COMMIT_ID | END_TIMESTAMP    |

+------+-------+-----------+------------------+

|  772 |     1 |         1 | 1300842759914337 |

|  819 |     1 |         2 | 1300842830416566 |

+------+-------+-----------+------------------+

This looks pretty straightforward if Server B were to fail, to point Server C at Server A. They both have two entries and both have a commit_id of 2. However this only works in this very simple example. If a backup was restored and replication enabled, or if Server C was stopped and restarted the entries would not be ordered so nicely. Both these events would throw the two tables out of sync. So you would be back to looking at the actual SQL statement that was last to execute on Server C to find where it matched in Server A.

To remedy this problem I added two new columns to the SYS_REPLICATION_LOG table, ORIGINATING_SERVER_ID and ORIGINATING_ID. These are the ID and SERVER_ID for the server that originally applied the changes. The table now looks like:

drizzle> describe DATA_DICTIONARY.SYS_REPLICATION_LOG;

+-----------------------+---------+------+---------+-----------------+-----------+

| Field                 | Type    | Null | Default | Default_is_NULL | On_Update |

+-----------------------+---------+------+---------+-----------------+-----------+

| ID                    | BIGINT  | YES  |         | NO              |           |

| SEGID                 | INTEGER | YES  |         | NO              |           |

| COMMIT_ID             | BIGINT  | YES  |         | NO              |           |

| END_TIMESTAMP         | BIGINT  | YES  |         | NO              |           |

| ORIGINATING_SERVER_ID | INTEGER | YES  |         | NO              |           |

| ORIGINATING_ID        | BIGINT  | YES  |         | NO              |           |

| MESSAGE_LEN           | INTEGER | YES  |         | NO              |           |

| MESSAGE               | BLOB    | YES  |         | NO              |           |

+-----------------------+---------+------+---------+-----------------+-----------+

On Server A this looks like:

drizzle> create database serverA;

drizzle> select ID,  SEGID, COMMIT_ID, END_TIMESTAMP, ORIGINATING_SERVER_ID,  ORIGINATING_ID from DATA_DICTIONARY.SYS_REPLICATION_LOG;

+------+-------+-----------+------------------+-----------------------+----------------+

| ID   | SEGID | COMMIT_ID | END_TIMESTAMP    | ORIGINATING_SERVER_ID | ORIGINATING_ID |

+------+-------+-----------+------------------+-----------------------+----------------+

|  772 |     1 |         1 | 1300839966870903 |                     1 |            772 |

|  861 |     1 |         2 | 1300840709527015 |                     1 |            861 |

+------+-------+-----------+------------------+-----------------------+----------------+

On Server B this looks like:

drizzle> select ID,  SEGID, COMMIT_ID, END_TIMESTAMP, ORIGINATING_SERVER_ID,  ORIGINATING_ID from DATA_DICTIONARY.SYS_REPLICATION_LOG;

+------+-------+-----------+------------------+-----------------------+----------------+

| ID   | SEGID | COMMIT_ID | END_TIMESTAMP    | ORIGINATING_SERVER_ID | ORIGINATING_ID |

+------+-------+-----------+------------------+-----------------------+----------------+

|  772 |     1 |         1 | 1300840469433946 |                     2 |            772 |

|  855 |     1 |         2 | 1300840715001766 |                     1 |            861 |

+------+-------+-----------+------------------+-----------------------+----------------+

On Server C this looks like:

drizzle>  select ID,  SEGID, COMMIT_ID, END_TIMESTAMP, ORIGINATING_SERVER_ID,  ORIGINATING_ID from DATA_DICTIONARY.SYS_REPLICATION_LOG;

+------+-------+-----------+------------------+-----------------------+----------------+

| ID   | SEGID | COMMIT_ID | END_TIMESTAMP    | ORIGINATING_SERVER_ID | ORIGINATING_ID |

+------+-------+-----------+------------------+-----------------------+----------------+

|  772 |     1 |         1 | 1300840574916708 |                     3 |            772 |

|  833 |     1 |         2 | 1300840715511684 |                     1 |            861 |

+------+-------+-----------+------------------+-----------------------+----------------+

The last entry in each SYS_REPLICATION_LOG on all three servers has the same values for ORIGINATING_SERVER_ID and ORIGINATING_ID. Note: The first entry is a startup event and is unique to each server. How is this helpful? If Server B were to fail and Server C needed to be pointed to a new master it would be as simple as running this query on Server A:

select commit_id FROM DATA_DICTIONARY.SYS_REPLICATION_LOG WHERE originating_id= 861;

Then restarting Server C with that COMMIT_ID position.

This is still in design heavily, walking through this example it may be better to have the ORIGINATING_COMMIT_ID rather then ORIGINATING_ID in the table. I think that would then allow just reconfiguring the slave to point at the new master. The example also illustrates the need for more on the fly configuration such as configuring a slave to point to a different master without restarting it. The code for this is at:

lp:~skinny.moey/drizzle/hierarch_repl

Comments and design suggestions are more then welcome!

View full post on Planet Drizzle

To get a better understanding of what exactly would be involved in supporting multiple masters, I decided to just start hacking it up. I did this mainly to get a sense of what would need to be changed, since my original design didn’t allow for this at all. (Shortsightedness on my part I suppose.)

So I have a beta version of my results available in this Launchpad branch:

lp:~dshrews/drizzle/beta-multi-master

From my simple tests, it seems to work. I’m not real happy with the code (like I said, this was a hack), but functionality is there. I’m not promising this will go into Drizzle trunk just yet. I would like to make some improvements on it, and I’d really like to get some feedback from people on it.

To use it, you’ll first need to create a modified slave configuration file. Here is a sample one:

ignore-errors

[master1]master-host = foo.my.domainmaster-port = 3306master-user = user1master-pass = password

[master2]master-host = bar.my.domainmaster-port = 3306master-user = user2master-pass = password

Currently, a total of 10 masters are supported. This was an arbitrary number. It was simplest to just predetermine a set number of masters due to some complications with config file parsing which I wasn’t prepared to solve (this is one of the things I want to see fixed). One IO thread per master will be started, though we still use a single applier thread for the time being.

You’ll notice in the sample config a new option, ignore-errors. If this option is present, the slave ignores any errors from replication events received from the masters that it executes locally. I highly recommend you have this option enabled. Also note the addition of the [master1] and [master2] sections that define options for each master. You can go all the way to a [master10] section.

Nothing changes with how you start your slave or masters (see my post on setting up a simple replication example).

Give it a try and let me know how it works for you. Again, this is bleeding edge stuff (does any other database support this?  :) ), so be prepared for bugs.

View full post on Planet Drizzle