Keep in mind this solution is hot from the press, consider it alpha. Like I said above, it survived testing in a very controlled environment but it is young and many issues/bugs are likely to be found. Also, it is different from Yoshinori Matsunobu’s MHA solution and in fact it is quite a complement to it. One of my near term goal is to integrate with MHA for master promotion.

The solution is basically made of 3 pieces:

• The Pacemaker cluster manager
• A Pacemaker configuration
• A MySQL resource agent

Here I will not cover the Pacemaker installation since this is fairly straightforward and covered elsewhere. I’ll discuss the MySQL resource agent and the supporting configuration while assuming basic knowledge of Pacemaker.

But, before we start, what does this solution offers.

• Reader and writer VIPs behaviors similar to MMM
• If the master fails, a new master is promoted from the slaves, no master to master setup needed. Selection of master is based on scores published by the slaves, the more up to date slaves have higher scores for promotion
• Some nodes can be dedicated to be only slaves or less likely to become master
• A node can be the preferred master
• If replication on a slave breaks or lags beyond a defined threshold, the reader VIP(s) is removed. MySQL is not restarted.
• If no slaves are ok, all VIPs, readers and writer, will be located on the master
• During a master switch, connections are killed on the demoted master to avoid replication conflicts
• All slaves are in read_only mode
• Simple administrative commands can remove master role from a node
• Pacemaker stonith devices are supported
• No logical limits in term of number of nodes
• Easy to add nodes

In order to setup the solution you’ll need my version of the MySQL resource agent, it is not yet pushed to the main Pacemaker resource agents branch. More testing and cleaning will be needed before that happen. You can get the resource agent from here:

https://github.com/y-trudeau/resource-agents/raw/master/heartbeat/mysql

You can also the whole branch from here:

https://github.com/y-trudeau/resource-agents/zipball/master

On my Ubuntu Lucid VM, this file goes in /usr/lib/ocf/resource.d/heartbeat/ directory.

To use this agent, you’ll need a Pacemaker configuration. As a starting point, I’ll discuss the configuration I use during my tests.

node testvirtbox1 \
        attributes IP="10.2.2.160"
node testvirtbox2 \
        attributes IP="10.2.2.161"
node testvirtbox3 \
        attributes IP="10.2.2.162"
primitive p_mysql ocf:heartbeat:mysql \
        params config="/etc/mysql/my.cnf" pid="/var/run/mysqld/mysqld.pid" \
               socket="/var/run/mysqld/mysqld.sock" replication_user="root" \
               replication_passwd="rootpass" max_slave_lag="15" evict_outdated_slaves="false" \
               binary="/usr/bin/mysqld_safe" test_user="root" \
               test_passwd="rootpass" \
        op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1" \
        op monitor interval="2s" role="Slave" OCF_CHECK_LEVEL="1"
primitive reader_vip_1 ocf:heartbeat:IPaddr2 \
        params ip="10.2.2.171" nic="eth0"
primitive reader_vip_2 ocf:heartbeat:IPaddr2 \
        params ip="10.2.2.172" nic="eth0"
primitive reader_vip_3 ocf:heartbeat:IPaddr2 \
        params ip="10.2.2.173" nic="eth0"
primitive writer_vip ocf:heartbeat:IPaddr2 \
        params ip="10.2.2.170" nic="eth0" \
        meta target-role="Started"
ms ms_MySQL p_mysql \
        meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Master" is-managed="true"
location No-reader-vip-1-loc reader_vip_1 \
        rule $id="No-reader-vip-1-rule" -inf: readerOK eq 0
location No-reader-vip-2-loc reader_vip_2 \
        rule $id="No-reader-vip-2-rule" -inf: readerOK eq 0
location No-reader-vip-3-loc reader_vip_3 \
        rule $id="No-reader-vip-3-rule" -inf: readerOK eq 0
location No-writer-vip-loc writer_vip \
        rule $id="No-writer-vip-rule" -inf: writerOK eq 0
colocation reader_vip_1_dislike_reader_vip_2 -200: reader_vip_1 reader_vip_2
colocation reader_vip_1_dislike_reader_vip_3 -200: reader_vip_1 reader_vip_3
colocation reader_vip_2_dislike_reader_vip_3 -200: reader_vip_2 reader_vip_3
property $id="cib-bootstrap-options" \
        dc-version="1.0.11-a15ead49e20f047e129882619ed075a65c1ebdfe" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="3" \
        stonith-enabled="false" \
        no-quorum-policy="ignore" \
        last-lrm-refresh="1322236006"
property $id="mysql_replication" \
        replication_info="10.2.2.162|mysql-bin.000090|106"
rsc_defaults $id="rsc-options" \
        resource-stickiness="100"

Let’s review the configuration. First it begins by 3 node entries defining the 3 nodes I have in my cluster. One attribute is required to each node, the IP address that will be used for replication. This is a real IP address not a reader or writer VIP. This attribute allows the use of a private network for replication if needed.

Next is the mysql primitive resource declaration. This primitive defines the mysql resource on each node and has many parameters, here’s the ones I had to define:

• config: The path of the my.cnf file. Remember that Pacemaker will start MySQL, not the regular init.d script
• pid: The pid file. This is use by Pacemaker to know if MySQL is already running. It should match the my.cnf pid_file setting.
• socket: The MySQL unix socket file
• replication_user: The user to use when setting up replication. It is also currently used for the ‘CHANGE MASTER TO’ command, something that should/will change in the future
• replication_passwd: The password for the above user
• max_slave_lag: The maximum allowed slave lag in seconds, if a slave lags by more than that value, it will lose its reader VIP(s)
• evict_outdated_slaves: A mandatory to set this to false otherwise Pacemaker will stop MySQL on a slave that lags behind. This will absolutely not help its recovery.
• test_user and test_passwd: The credentials to test MySQL. Default is to run select count(*) on mysql.user table, so the user given should at least have select on that table.
• op monitor: An entry is needed for each role, Master and Slave. Intervals must not be the same.

Following the mysql primitive declaration, the primitives for 3 reader vips and one writer vip are defined. Those are straightforward so I’ll skip detailed description. The next interesting element is the master-slave “ms” declaration. This is how Pacemaker defines an asymmetrical resource having a master and slaves. The only thing that may change here is clone-max=”3″ which should match the number of database nodes you have.

The handling of the VIPs is the truly new thing in the resource agent. I am grateful to Florian Haas who told me to use node attributes to avoid Pacemaker from over reacting. The availability of a reader or writer VIPs on a node are controlled by the attributes readerOK and writerOK and the location rules. An infinite negative weight is given when a VIP should not be on a host. I also added a few colocation rules to help spread the reader VIPs on all the nodes.

As a final thought on the Pacemaker configuration, remember that in order for a pacemaker cluster to run correctly on a 2 nodes cluster, you should set the quorum policy to ignore. Also, this example configuration has no stonith devices defined so stonith is disable. At the end of the configuration, you’ll notice the replication_info cluster attribute. You don’t have to define this, the mysql RA will add it automatically when the first a node will promoted to master.

There are not many requirements regarding the MySQL configuration, Pacemaker will automatically add “skip-start-slave” for a saner behavior. One of the important setting is “log_slave_updates = OFF” (default value). In some cases, if slaves are logging replication updates, it may cause failover issues. Also, the solution relies on the read_only setting on the slave so, make sure the application database use doesn’t have the SUPER privilege which overrides read_only.

Like I mentioned above, this project is young. In the future, I’d like to integrate MHA to benefit for its capacity of bringing all the nodes to a consistent level. Also, the security around the solution should be improved, a fairly easy task I believe. Of course, I’ll work with the maintainers of the Pacemaker resources agents to include it in the main branch once it matured a bit.

Finally, if you are interested by this solution but have problems setting it up, just contact us at Percona, we’ll be pleased to help.

View full post on MySQL Performance Blog

In all seriousness, it’s kind of neat that we’re using dbqp to run some of our staging tests and we gain a few neat things:

Speed

Here are the trend charts for randgen and crashme.  While it doesn’t look like randgen is showing much of an improvement, it is worth mentioning that this job now runs both the standard and the transaction log tests in a single run >: )  Previously, we had a separate drizzle-automation job for the transaction log.  Just the trx_log tests took ~30 minutes to run (plus build time).  Long story short, we’re saving about 30-40 minutes on randgen testing per staging run and only needing to build once!

Maintainability

The jobs we run are in the tree and anyone can easily repeat them.  While Drizzle-automation kicks major butt (and I have taken many ideas from it), it is a separate piece of software that requires setup and maintenance.  Basing things around an in-tree setup means that you only need the code and any required bits and pieces.  Now if we need to set up a new randgen machine, we only need the randgen and dbd::drizzle installed (and we plan on including randgen in-tree soon, so you won’t even need that!).  If we need to set up a new crash-me machine, we only need dbd::drizzle – and everyone should have dbd::drizzle installed! ; )

Ease of use

Pretty much all tests provide the same standard output:

dtr mode

From the command:

./dbqp

Our default mode is dtr (aka using drizzletest.cc to execute standard .test files). To run all available tests, use the make target – make test-dbqp

20110621-081404  trigger_dictionary.loaded                  [ pass ]       43
20110621-081408  logging_stats.cumulative                   [ pass ]     1045
20110621-081412  errmsg_stderr.stderr                       [ pass ]       36
20110621-081412  ===============================================================
20110621-081412 INFO Test execution complete in 496 seconds
20110621-081412 INFO Summary report:
20110621-081412 INFO Executed 566/566 test cases, 100.00 percent
20110621-081412 INFO STATUS: PASS, 566/566 test cases, 100.00 percent executed
20110621-081412 INFO Spent 254 / 496 seconds on: TEST(s)
20110621-081412 INFO Test execution complete
20110621-081412 INFO Stopping all running servers...


randgen mode

From the command:

./dbqp --mode=randgen --randgen-path=/path/to/your/randgen


20110621-170141  main.subquery                              [ pass ]     3780
20110621-170148  main.subquery_semijoin                     [ pass ]     3016
20110621-170156  main.subquery_semijoin_nested              [ pass ]     3750
20110621-170202  main.varchar                               [ pass ]     2658
20110621-170202  ===============================================================
20110621-170202 INFO Test execution complete in 147 seconds
20110621-170202 INFO Summary report:
20110621-170202 INFO Executed 19/19 test cases, 100.00 percent
20110621-170202 INFO STATUS: PASS, 19/19 test cases, 100.00 percent executed
20110621-170202 INFO Spent 77 / 147 seconds on: TEST(s)
20110621-170202 INFO Test execution complete
20110621-170202 INFO Stopping all running servers...


crashme mode

From the command:

./dbqp --mode=crashme


20110621-181515  main.crashme                               [ fail ]   149840
20110621-181515  func_extra_to_days=error        # Function TO_DAYS
20110621-181515  ###
20110621-181515  ###<select to_days('1996-01-01') from crash_me_d
20110621-181515  ###>2450084
20110621-181515  ###We expected '729024' but got '2450084'
20110621-181515  func_odbc_timestampadd=error        # Function TIMESTAMPADD
20110621-181515  ###
20110621-181515  ###<select timestampadd(SQL_TSI_SECOND,1,'1997-01-01 00:00:00')
20110621-181515  ###>1997-01-01 00:00:01.000000
20110621-181515  ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
20110621-181515  ###
20110621-181515  ###<select {fn timestampadd(SQL_TSI_SECOND,1,{ts '1997-01-01 00:00:00'}) }
20110621-181515  ###>1997-01-01 00:00:01.000000
20110621-181515  ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
20110621-181515
20110621-181515 ERROR Failed test.  Use --force to execute beyond the first test failure
20110621-181515  ===============================================================
20110621-181515 INFO Test execution complete in 153 seconds
20110621-181515 INFO Summary report:
20110621-181515 INFO Executed 1/1 test cases, 100.00 percent
20110621-181515 INFO STATUS: FAIL, 1/1 test cases, 100.00 percent executed
20110621-181515 INFO FAIL tests: main.crashme
20110621-181515 INFO Spent 149 / 153 seconds on: TEST(s)
20110621-181515 INFO Test execution complete
20110621-181515 INFO Stopping all running servers...


While this isn’t a huge feature, it is nice to have a standardized report for knowing if something failed, what failed and how (we always dump test tool output on test failures).  Why is this nice?  Well, the world is a busy place and only needing to know one way of reading test output simplifies things just a teensy little bit.  This small improvement becomes a huge benefit over time if you happen to spend good chunks of your day looking at test output like me : )

Other than that, I’m still working on teaching dbqp interesting new tricks that will help me in testing SkySQL‘s Reference Architecture – expect to hear more about that next month!

View full post on Planet Drizzle

Here is a generic method for determining if any subset of numbers in a set adds up to any other number. By naturally compressing the data in a computational friendly way, we can change the way we work with sets of numbers.

Lets find out if any of the numbers in our list add up to any other number. First, the set is contained in a naturally compressed (rle encoded) table:

CREATE TABLE `ex2` (
  `val` bigint(20) NOT NULL DEFAULT '0',
  `cnt` bigint(21) NOT NULL DEFAULT '0'
);

mysql> select * from test.ex2;
+-----+----------+
| val | cnt      |
+-----+----------+
| -10 |        1 |
|  -3 |        1 |
|  -2 |        1 |
|   0 |   250000 |
|   4 |  8000000 |
|   7 | 14680064 |
|  14 | 14680064 |
|  15 | 35417088 |
|  16 |        1 |
+-----+----------+
9 rows in set (0.00 sec)

mysql> select * from sum_to_check;
+------+------------------+
| id   | sum_to_check_for |
+------+------------------+
|    1 |                0 |
|    2 |                1 |
|    3 |                2 |
|    4 |                5 |
|    5 |               -5 |
+------+------------------+
5 rows in set (0.00 sec)

Here, we can test for any N items to see if they add up:

mysql> select sum_to_check_for,
group_concat(a.val),
group_concat(sum_to_check_for),
sum(a.val) = sum_to_check_for matched
from ex2 a
join sum_to_check
where val in(-2,-3)
group by 1
having sum(a.val) = sum_to_check_for ;
+------------------+---------------------+--------------------------------+---------+
| sum_to_check_for | group_concat(a.val) | group_concat(sum_to_check_for) | matched |
+------------------+---------------------+--------------------------------+---------+
|               -5 | -3,-2               | -5,-5                          |       1 |
+------------------+---------------------+--------------------------------+---------+
1 row in set (0.00 sec)

Now we can check the entire set in a reasonable amount of time, even though it is millions of entries long:

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10,-3) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10,-3,-2) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10,-3,-2) group by 1 having sum(a.val) = sum_to_check_for ;

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10,-3) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10,-3,-2) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

...

Note that here we search the entire set of millions of items 5 times over (one for each sum_to_check_for):

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-10,-3,-2,0,4,7,14,15,16) group by 1 having sum(a.val) = sum_to_check_for ;
Empty set (0.00 sec)

Then move along:

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-3) group by 1 having sum(a.val) = sum_to_check_for ;Empty set (0.00 sec)

mysql> select sum_to_check_for, group_concat(a.val), group_concat(sum_to_check_for), sum(a.val) = sum_to_check_for matched from ex2 a join sum_to_check where val in(-3,-2) group by 1 having sum(a.val) = sum_to_check_for ;
+------------------+---------------------+--------------------------------+---------+
| sum_to_check_for | group_concat(a.val) | group_concat(sum_to_check_for) | matched |
+------------------+---------------------+--------------------------------+---------+
|               -5 | -3,-2               | -5,-5                          |       1 |
+------------------+---------------------+--------------------------------+---------+
1 row in set (0.00 sec)

Keep in mind, that there are a lot of items in my list:

mysql> select sum(cnt) from ex2;
+----------+
| sum(cnt) |
+----------+
| 73027220 |
+----------+
1 row in set (0.00 sec)

mysql> select * from ex2;
+-----+----------+
| val | cnt      |
+-----+----------+
| -10 |        1 |
|  -3 |        1 |
|  -2 |        1 |
|   0 |   250000 |
|   4 |  8000000 |
|   7 | 14680064 |
|  14 | 14680064 |
|  15 | 35417088 |
|  16 |        1 |
+-----+----------+
9 rows in set (0.00 sec)

This works for two reasons. First, I know the contents of the set, and the distribution characteristics from the from structure of the compressed version of the table itself. The data is stored in a histogram. This allows me to collapse the set of items to search to only the items in the set, and then do a linear search. You can do all of the searches in parallel in a parallel database.

If we keep the counts with the items (maintained with changes to the set) then this allows us to create a finite and easily computable and easily checkable result. We know not to check to see if 17 adds up to something because we know it is not in our set.

View full post on MySQL Performance Blog

Shard-Query introduces set based processing, which on the surface appears to be similar to other technologies on the market today. However, the scaling features of Shard-Query are just a side effect of the fact that it operates on sets in parallel. Any set can be operated on to any arbitrary degree of parallelism up to, and including, the cardinality of the set.
This is because:

1. It is often possible to arbitrarily transform one type of expression into a different, but compatible type for computational purposes as long as the conversion is bidirectional
2. An range operation over a set of integers or dates can be transformed into one or more discrete sub-ranges
3. Any operation on an entire set is the same as running that operation on each item individually.
4. After expanding a set operation into N discrete dimensions, it can always be collapsed back into a one dimensional set.
5. Arrays are sets

Treating a general purpose computer as an SIMD computer is possible because in a set, you can perform operations on all of the items independently. The SIMD processor simply needs to wait for all parallel operations on its input to complete. Parallelism is embarrassing and the maximum degree of parallelism is easily enforced with queue.

Today I am going to show you how to take almost any function, and treat any size cluster of Turing computers as a specialized purpose SIMD computer with respect to your function. The SQL interface to Shard-Query imposes a wait for all the workers to complete, but you can register a callback function to handle the output of each input asynchronously, if you like.

Right now I believe this only works on finite sets. I’ve decided to show how to count the number of unique words, an how many times those words appear in a document. Set based processing of course works on sets. A document is a set of words.

Before you read further, I want to tell you why I’ve decided to use the words of the Constitution of the United States of America as an example. It is my favourite document in the world. It speaks of honesty, and integrity, truth and openness. I believe in all of these things. I believe, that with cheap computation, our world can become an amazing place. Please use this technology constructively and for peaceful purposes. Love one another and let’s solve all the complex problems in the world together in peace and harmony.

The following is a somewhat naive example, since grammar will not be taken into account. I start by splitting our document into a list of words:
cat /tmp/constitution.txt | sed -e’s/ /\n/g’ > words

I am going to perform the following operations on every word in the constitution:
1) compute the md5 of every item
2) compute the md5 on the reverse of every item
3) count the total number of words
4) count the frequency of words
5) order by the frequency of words, then by the md5 of the word, then by the md5 of the reverse of the word.
6) determine the number of unique words. This is not projected, but you can infer it from the number of items in the output set.

The US Constitution is not very large. I inflated the document size significantly to over 3 million “words” by duplicating the entire set multiple times.
mysql> load data infile ‘/tmp/words’ into table words (chars);
Query OK, 6033 rows affected (0.01 sec)
Records: 6033 Deleted: 0 Skipped: 0 Warnings: 0

I blow up the size of the words table and I create words2. This is the data upon which we will operate:

create table words2 partition by hash(bucket) partitions 12 as select id % 6 bucket, chars words from words;
Query OK, 3088896 rows affected (2.41 sec)
Records: 3088896  Duplicates: 0  Warnings: 0

Here is the serial version as run by the native database interface (MySQL):

mysql> select word, md5(word), md5(reverse(word)), count(*)
from words2 group by 1,2,3 order by 3,1 desc;
...
| Legislature      | 4380f755e4150b1c11f0ae9ca1910bcb | fecd2758f3c64c8176ce60c4ff7c1cf3 |     3072 |
| consent          | 9d721d9a89406a2a6861efaae44a785f | fede6baff4c3716c37a3c60bf4051b3f |      512 |
| admit,           | d803450bb41af1f7372af6ddc8e42d14 | fee1e6f166edfccd849fe4438eb1924f |      512 |
| Affirmation.     | 9568b7e19ee3da70d3e486134add2743 | fee5d3a27ec5be41941b5689f70c5587 |      512 |
| may,             | 289cf5ceddb80bab96c92de0a918e122 | fee80b247ce32faca9de1a031119533c |     1024 |
| legislatures     | 0640c734a3d25eed18126c7db6a39523 | ff238c73fea4086c10cda4a46aeb9d9a |     1024 |
| Time             | a76d4ef5f3f6a672bbfab2865563e530 | ff38a346616fc8a4df42c7f6c95bf1cc |     2048 |
| Congress:        | 873c419d2c2139bc8bbc3cbaffcc3473 | ff592a4dac2aa93c8a0589898885fe48 |      512 |
| Charles          | 399423ff652ebb6a6701be7ec3202fc6 | ffac637b74c0f062904ab466d9bf9e01 |     1024 |
| impairing        | 1c718d732bc6f6805835f8be6ef6e43e | ffc86c559e06009a743d891ce1e4fc4f |      512 |
+------------------+----------------------------------+----------------------------------+----------+
1427 rows in set (4.94 sec)

1427 rows in set (5.00 sec)
1427 rows in set (5.03 sec)
1427 rows in set (5.00 sec)

Since the data fits in memory speed is near constant and the single threaded operation burns one CPU.

To help completely demonstrate how Shard-Query makes parallel set operations work, I’ll operate in only one dimension for the first example, just like the MySQL client. This will be a linear operation because Shard-Query has no idea how to add parallelism in this case. It is data set agnostic, operating only on sets, not relations. If it were smarter it would ask the data dictionary about partitioning.

Array
(
    [word] => Congress:
    [md5(word)] => 873c419d2c2139bc8bbc3cbaffcc3473
    [md5(reverse(word))] => ff592a4dac2aa93c8a0589898885fe48
    [count(*)] => 512
)
Array
(
    [word] => Charles
    [md5(word)] => 399423ff652ebb6a6701be7ec3202fc6
    [md5(reverse(word))] => ffac637b74c0f062904ab466d9bf9e01
    [count(*)] => 1024
)
Array
(
    [word] => impairing
    [md5(word)] => 1c718d732bc6f6805835f8be6ef6e43e
    [md5(reverse(word))] => ffc86c559e06009a743d891ce1e4fc4f
    [count(*)] => 512
)
1427 rows returned (5.0057470798492s, 4.9994130134583s, 0.0063340663909912s)

The set of three numbers are wall clock time (as calculated by microtime()), SQL execution time, and parse time, respectively.

Actually performance is a little worse. This is not unexpected. Since Shard-Query must add at small amount of overhead, a single threaded operation may be slower than the same operation on the native database.

That doesn’t matter because Shard-Query is a smart database proxy that can add parallelism. In this mode it will add additional six degrees of parallelism the query:

Array
(
    [word] => Congress:
    [md5(word)] => 873c419d2c2139bc8bbc3cbaffcc3473
    [md5(reverse(word))] => ff592a4dac2aa93c8a0589898885fe48
    [count(*)] => 342
)
Array
(
    [word] => Charles
    [md5(word)] => 399423ff652ebb6a6701be7ec3202fc6
    [md5(reverse(word))] => ffac637b74c0f062904ab466d9bf9e01
    [count(*)] => 598
)
Array
(
    [word] => impairing
    [md5(word)] => 1c718d732bc6f6805835f8be6ef6e43e
    [md5(reverse(word))] => ffc86c559e06009a743d891ce1e4fc4f
    [count(*)] => 512
)
1427 rows returned (0.87930011749268s, 0.87229418754578s, 0.0070059299468994s)

Why six degrees of parallelism? Because that is how many physical cores are connected to my bus, and because I chose to create six hash “buckets” in the table. This allows MySQL to set up a sequential scan over the items in this bucket, particularly since we are examining all the items. We operate on all the buckets and then use intelligent expression substitution to put the results back together, when necessary. When sorting or grouping are used, a final pass over the final result may be necessary, and this may add a small amount of serialization at the end.

How does this work?

Here is the most important part of the explain plan in the mode without parallelism. Notice that there is only one query. If your database system can not provide native parallelism, then performance will be poor.

-- SQL TO SEND TO SHARDS:
Array
(
    [0] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` GROUP BY 1,2,3 ORDER BY NULL
)

The other important optimization combines results from multiple queries together. This query is single threaded, and thus this serves no purpose. It will be much more important in a moment.

-- AGGREGATION SQL:
SELECT `word`,`md5(word)`,SUM(`count(*)`) AS `count(*)`
FROM `aggregation_tmp_39323566`  GROUP BY 1,2 ORDER BY 1 ASC
ON DUPLICATE KEY UPDATE
`word`=VALUES(`word`),
`md5(word)`=VALUES(`md5(word)`),
`count(*)`=`count(*)` +  VALUES(`count(*)`)

Now, consider the query with BETWEEN 1 and 6 added to the where clause. This creates boundary conditions for our query. Any set of integers can be broken up into as many items are the set contains, and thus it is possible to convert the BETWEEN expression into a partition elimination expression.

Here is the output from the parallel version:

-- SQL TO SEND TO SHARDS:
Array
(
    [0] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 1   GROUP BY 1,2,3 ORDER BY NULL
    [1] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 2   GROUP BY 1,2,3 ORDER BY NULL
    [2] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 3   GROUP BY 1,2,3 ORDER BY NULL
    [3] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 4   GROUP BY 1,2,3 ORDER BY NULL
    [4] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 5   GROUP BY 1,2,3 ORDER BY NULL
    [5] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 6   GROUP BY 1,2,3 ORDER BY NULL
)

This powers the UPSERT. The results from the six branches are combined with this.

-- AGGREGATION SQL:
SELECT `word`,`md5(word)`,SUM(`count(*)`) AS `count(*)`
FROM `aggregation_tmp_27656998`  GROUP BY 1,2 ORDER BY 1 ASC
ON DUPLICATE KEY UPDATE
`word`=VALUES(`word`),
`md5(word)`=VALUES(`md5(word)`),
`count(*)`=`count(*)` +  VALUES(`count(*)`)

All six branches compute fully in parallel.

Set processing also allows us to capture and effectively aggregate and manage every single change in any database. The entire collection of human intelligence can be stored an efficiently reused over and over again, conserving resources now lost through wasted re-computation. I will be sharing more information about this in the future.

Finally, I would like to note that this set has a cardinality of 3088896. This is the maximum theoretical degree of parallelism that this data set can achieve with my method. Likely current network technologies can not support such a degree.

mysql> select min(id),max(id) from words;
+---------+---------+
| min(id) | max(id) |
+---------+---------+
|       1 | 3088896 |
+---------+---------+
1 row in set (0.00 sec)

View full post on MySQL Performance Blog

For Drizzle, we’ve created some tables (via the randgen’s data generator if you are curious), saved a copy of the datadir, and then created a test case that uses said datadir for the test server.  The test executes some simple SQL queries to make sure we can read the tables properly.  This way, if we ever do something to either the server or .dfe format (data format exchange – had a most enlightening conversation with the team about this format’s history at the MySQL UC), we’ll have a broken test that cries about it.  From there, we’ll know we have to take some action.  The always-amazing Stewart Smith has also created some foreign key backwards compatibility tests, which I believe marks further progress towards the magical goodness that is catalogs!

We signal that we want to do this by using a .cnf file:


[test_servers]
servers = [[]]

[s0]
load-datadir=backwards_compat_data


Each server is named s0,s1,sN.  If a server name is contained in the .cnf file, the test-runner will do the appropriate magic to use the specified datadir for that server.  The argument to load-datadir is the name of the directory that is intended for use in the test.  All datadirs are expected to live in drizzle/tests/std_data.  Tests that do use a .cnf file, like main.backwards_compatibility and slave.basic are skipped by test-run.pl automatically (you *can’t* run them via test-run.pl).

This is something that I don’t believe could be accomplished with the old test runner, or at least not *easily* done (see Rube Goldberg) ; ).  At some point, we will switch over to dbqp entirely and remove test-run.pl.  Seeing comments like this, makes me happy and think things are on track.

dbqp was created with the idea that it should be easy to express complex testing setups (multiple servers, using a preloaded datadir, etc, etc) and it looks like the incubation is starting to pay some benefits.  In addition to allowing this voodoo to happen, the code I’ve added to the test runner will allow us to start doing proper tests of the super Mr. Shrewsbury’s multi-master replication.  Joe Daly has also been doing some very promising work for hierarchical replication based on Dave’s tree.  I’ll be creating some example tests for these badass features soon.  The moral of the story is that by rethinking our test-runner, one tiny bit of code helps us move the ball forward on testing replication, backwards compatibility, and catalogs.

It’s honestly one of the best parts of working on the Drizzle project – being encouraged to experiment and rethink problems has enabled all sorts of innovation (but one example of Monty Taylor’s computing wizardry!) and cool features.  Thanks to this freedom to experiment, we now have even more ways of making sure we are producing quality code.

My view of QA is that we do help test, but that we also help other people answer their own questions about quality (via tools, documentation, examples, etc).  Ultimately, a test is a question – “Do you return the right answer for this query?”, “Can you survive a beating from the randgen?”, etc – and asking questions should be easy and informative.  QA shouldn’t be the sole province of some obscure priest class, but everyone’s playground.  When I see developers like Stewart writing interesting test cases and even contributing to the test tool itself, I’m even happier than when I find a bug (and finding bugs is quite awesome!).

Anyway, the code is proposed for a merge to trunk and documentation is available (under testing/writing test cases).  I hope that this makes trying to break things even more fun for people >: )

View full post on Planet Drizzle

First of all, Tungsten does not exactly use schema-based parallel replication.  Tungsten is actually based on what I call the serialized shard model of replication.  We assign global transaction IDs to all transactions, which means that for any particular set of transactions we can always figure out the correct serialization and apply in the right order.  This is true even if the transactions travel across independent replication paths or if we have master failover.

Second, we assign a shard ID to all transactions.  Shards are independent streams of transactions that execute correctly when applied by themselves in serial order.  Shards are typically independent, which means transactions in different shards can execute in parallel without deadlocking or corrupting data.  This is the case when each shard contains data for a single customer in a multi-tenant application.  We also have a notion of “critical shards,” which are shards that contain global data, such as shared currency rate tables.  Updates in critical shards cause full serialization across all shards.  

You can define shards in a variety of ways, but as a practical matter identifying durable shards inside individual MySQL schemas is hard for most applications, especially if there are constraints between tables or you have large transactions.   Many SQL applications tend to make most of their updates to a small number of very large tables, which makes finding stable dividing lines even more difficult.  Schemas are therefore a natural unit of sharding, and Tungsten uses these by default.

Schema-based sharding seems pretty limiting, but for current SQL databases it is really the only approach that works.  Here are some important reasons that give you a flavor of the issues.

* Restart.  To handle failures you need to mark the exact restart point on each replication apply thread or you will either repeat or miss transactions.  This requires precise and repeatable serialization on each thread, which you get with the serialized shard model.

* Deadlocks.  If there are conflicts between updates you will quickly hit deadlocks.  This is especially true because one of the biggest single thread replication optimizations is block commit, where you commit dozens of success transactions at once–it can raise throughput by 100% in some cases.  Deadlocks on the other hand can reduce effective throughput to zero in pathological cases.   Shard-based execution avoids deadlocks.

* Ordering.  SQL gives you a lot of ways to shoot yourself in the foot through bad transaction ordering.  You can’t write to a table before creating it.  You can’t delete a row before it is inserted.  Violating these rules does not just lead to invalid data but also causes errors that stop replication.  The workarounds are either unreliable and slow (conflict resolution) or impractical for most applications (make everything an insert).  To avoid this you need to observe serialization very carefully.

* Throughput.  SQL transactions in real systems vary tremendously in duration, which tends to result in individual long transactions blocking simpler parallelization schemes that use in-memory distribution of updates.  In the Tungsten model we can solve this by letting shard progress vary (by hours potentially), something that is only possible with a well-defined serialization model that deals with dependencies between parallel update streams.  I don’t know of another approach that deals with this problem.

If you mess up the solution to any of the foregoing problems, chances are good you will irreparably corrupt data, which leads to replication going completely off the rails.  Then you reprovision your slave(s).  The databases that most need parallel replication are very large, so this is a multi-hour or even multi-day process.  It makes for unpleasant calls with customers when you tell them they need to do this.

I don’t spend a lot of time worrying that Tungsten parallel replication is not well suited to the single big schema problem.  So far, the only ways I can think of making it work scalably require major changes to the DBMS or the applications that use it.  In many cases your least costly alternative may be to use SSDs to boost slave I/O performance.

My concerns about Tungsten’s model lie in a different area.  The serialized shard model is theoretically sound–it has essentially the same semantics as causally dependent messaging in distributed systems.  However, if we fail to identify shards correctly (and don’t know we failed) we will have crashes and corrupt data.  I want Tungsten either to work properly or tell users it won’t work and degrade gracefully to full serialization.  If we can’t do one of these two for every conceivable sequence of transactions that’s a serious problem.

So, to get back to my original point, serialized shards are the best model for parallel replication in SQL databases as we find them today.  I suspect if you look at some of the other incipient designs for parallel replication on MySQL you will find that they follow this model in the end if not at first.  I would think in fact that the next step is to add MySQL features that make sharded replication more effective.  The drizzle team seems to be thinking along these lines already.

View full post on Planet Drizzle

So, the compontents involved:

• Drizzle with the rabbitmq plugin, see this blog post for more information on how to get it up and running.
• Drizzle-JDBC to generate the data in the database
• RabbitReplication
• RabbitMQ to transport the transactions from drizzle to the applier
• Your browser, both Chrome and Firefox support web sockets, but for firefox you need to enable them first

The basic flow is this:

1. 10 inserts per second are executed in the drizzle database using drizzle-jdbc, these inserts contain a random IP address
2. Drizzle does its magic and packages up each insert as a google protobuf transaction message
3. The rabbitmq plugin picks up the message and publishes it to a RabbitMQ server
4. RabbitReplication consumes the message from RabbitMQ and maps it onto an annotated java object. This gives alot of flexibility to the rabbitreplication user since you can manipulate the information any way you want, in this example i extract a longitude and latitude value from the IP address using MaxMind GeoLite City. Have a look at row the “setSomething()” method in the ExampleRepl.java file. This file is something a rabbitreplication end user would implement and provide
5. The java object is serialized to json
6. The json string is published to all connected clients
7. A javascript on the page takes the lat/lng-values and maps these onto a google map
8. The same javascript appends some of the raw data to a table

So, I’m hoping the little demo i’m linking below shows some of the extreme flexibility of the Drizzle replication. This, for me, is the unique selling point for Drizzle. Also, if anyone has an actual production use case for this, i’d be extremely happy to help implement it!

View full post on Planet Drizzle

In my first post in this series, I described materialized views (MVs). An MV is essentially a cached result set at one point in time. The contents of the MV will become incorrect (out of sync) when the underlying data changes. This loss of synchronization is sometimes called drift. This is conceptually similar to a replication slave that is behind. Until it catches up, the view of the data on the slave is “behind” the changes on the master. An important difference is that each MV could have drifted by a different length of time.

A view which has drifted out of sync must be refreshed. Since an MV drifts over time from the “base tables” (those tables on which the view was built) there must be a process to bring them up-to-date. Flexviews includes two different refresh methods. Each method is named after the way in which the contents of the view are updated. The two methods are complete refresh and incremental refresh.   The view type/refresh method are interchangeable since a refresh method is selected at the time of view creation and may not be changed. If I talk about a ‘complete MV’, I mean a ‘MV for which the complete refresh method was selected at creation’.  Also, when I say ‘view’ I mean MV.

I will fully describe each method below, but the important thing to know right now is that there are two significant differences between the methods.

• A view which uses the complete refresh method must be rebuilt from scratch each time the view is refreshed. This is similar to storing the results of a query in Memcache. When the cache “expires”, the contents must be fully recalculated..
• An incrementally refreshable view (that is, one which uses the incremental method) can be updated much more efficiently, by examining a history of the rows which have changed since the view was either created, or last refeshed.

The incremental method is somewhat conceptually similar to using mysql binary logs for point-in-time recovery after a restoring a backup. Since the backup is a essentially a ‘snapshot’ of the data at the time of the backup, upon restoration the backup may be ‘out of date’ since changes to the database may have happened since the backup was taken. Replaying the binary logs brings the database up-to-date by replaying the changes.

An MV is also like a snapshot. The incremental refresh method uses the changelogs collected by FlexCDC to update the snapshot to reflect the changes that have happened in the database, instead of replaying binary logs directly.

Selecting a refresh method

The refresh method you select is determined by the following two major factors:

1. The cost (in terms of execution time) of the query on which the view is based.
2. The SQL features used in the query on which the view is based.

The cost of the query

The complete method completely rebuilds an MV from scratch at each refresh. This means that the minimum amount of drift for this type of MV is the amount of time it takes to execute the query on which the view is based.  An incrementally refreshable view, on the other hand, can usually be refreshed very quickly, depending on the amount of data which has changed since the last refresh. This is because the incremental refresh method must examine only those rows related to the ones which changed since the last time the view was refreshed.

The incremental refresh method can be many orders of magnitude faster than the original query execution time. It may be possible, therefore, to refresh an MV based on a query that takes 45 minutes to execute with a greater frequency than 45 minutes, even as frequently as every minute, perhaps.

The refresh method you select effectively controls the minimum amount of time it takes to refresh the view, and therefore the minimum amount of drift it will encounter. Keep in mind that you can build complete refresh views on top of incrementally refreshable view, which can give you a lot of flexibility to get the results you want, but more quickly. More on this later.

SQL features used in the query

Incremental refresh doesn’t support all of the SQL features available to SELECT statements.

*All aggregate functions supported, except GROUP_CONCAT and AVG(distinct).
**non-deterministic functions like RAND(),NOW(), etc, are not supported.
***There is a script to convert SQL to the Flexviews API (see convert.php below)

Refresh methods

The incremental refresh method

The incremental refresh method uses table changelogs which are created by FlexCDC. The refresh algorithm computes changes in multiple transactions. After each transaction it must wait for FlexCDC to process changes from that transaction. In practice, this means that it takes a minimum of a few seconds to incrementally refresh a view, even when the number of changes is very small.

The Flexviews SQL_API

Unlike views which use the complete refresh method, incrementally refreshable views are not built directly from SQL. Instead, Flexviews includes a MySQL stored procedure API, called the SQL_API which is used to define the view. These stored procedures maintain the Flexviews data dictionary. The data dictionary is used by the incremental refresh algorithm to build and maintain the view. You should always use the stored procedures to modify the dictionary. Do not modify it directly.

convert.php
You are probably unfamiliar with the SQL_API. In order to make working with the Flexviews easier, it includes a script called ‘convert.php’. This script makes it easy to create incrementally refreshable views from SQL statements. It reads one or more “CREATE TABLE db.schema … AS SELECT” and/or “INSERT INTO db.schema … AS SELECT” statements from standard input, and outputs the SQL_API statements representing the original SQL statements. Each MV name will be taken from the table name specified in each statement. You should try each statement for correctness before you attempt to convert it. You can easily check that it parses by running the SELECT portion, adding LIMIT 0 to the SELECT part of the query.

convert.php
This script takes as one argument the default database. Notice in the following example, the schema is not specified.

The MV will be placed into whatever schema is specified as the first argument on the command line:

$ cat sales.sql
create table dashboard_customer_sales AS
select customer_id,
customer_name,
DATE_FORMAT(order_date,'%Y%m') as sale_when,
sum(quantity) as total_items,
sum(price * quantity) as total_price,
count(*) as total_lines
from orders o
join customers c using(customer_id)
join order_lines ol using(order_id)
group by customer_id,
customer_name, sale_when;

$ php convert.php demo < sales.sql
CALL flexviews.create('demo', 'dashboard_customer_sales', 'INCREMENTAL');
SET @mvid := LAST_INSERT_ID();
CALL flexviews.add_expr(@mvid,'GROUP','customer_id','customer_id');
CALL flexviews.add_expr(@mvid,'GROUP','customer_name','customer_name');
CALL flexviews.add_expr(@mvid,'GROUP','DATE_FORMAT(order_date,"%Y%m")','sale_when');
CALL flexviews.add_expr(@mvid,'SUM','quantity','total_items');
CALL flexviews.add_expr(@mvid,'SUM','price * quantity','total_price');
CALL flexviews.add_expr(@mvid,'COUNT','*','total_lines');

CALL flexviews.add_table(@mvid,'demo','orders','o',NULL);
CALL flexviews.add_table(@mvid,'demo','customers','c','USING (customer_id) ');
CALL flexviews.add_table(@mvid,'demo','order_lines','ol','USING (order_id) ');
CALL flexviews.enable(@mvid);

The script (convert.php) supports basic queries which use select/group by/join/where. You may not use sub-queries or any non-deterministic functions like NOW() or RAND(). HAVING clauses, ORDER BY clauses, etc, are not supported. You can work around many of these limitations. I will discuss one of the workarounds below, in the ‘complete refresh’ section. I’ll discuss others in future blog posts.

Enable the view to use it

The SQL_API call flexviews.enable() is used to actually build the contents of the view, making it available for querying:

mysql> call flexviews.enable(
-> flexviews.get_id('demo','dashboard_customer_sales'));
Query OK, 0 rows affected (41 min 52.04 sec)

Data dictionary

Here is a quick example of the list of tables used by the above view, as stored in the data dictionary:

mysql> select *
from flexviews.mview_table
where mview_id=
flexviews.get_id('demo','dashboard_customer_sales')\G
*************************** 1. row ***************************
      mview_table_id: 27
            mview_id: 21
    mview_table_name: orders
  mview_table_schema: demo
   mview_table_alias: o
mview_join_condition: NULL
    mview_join_order: 999
*************************** 2. row ***************************
      mview_table_id: 28
            mview_id: 21
    mview_table_name: customers
  mview_table_schema: demo
   mview_table_alias: c
mview_join_condition: USING (customer_id)
    mview_join_order: 999
*************************** 3. row ***************************
      mview_table_id: 29
            mview_id: 21
    mview_table_name: order_lines
  mview_table_schema: demo
   mview_table_alias: ol
mview_join_condition: USING (order_id)
    mview_join_order: 999
3 rows in set (0.00 sec)

Using the dictionary

The following SQL is dynamically generated by the flexviews.get_sql() function call. This function reads the meta-data stored in the Flexviews data dictionary. It returns the SQL which represents the data stored in the view. This is much more convenient than trying to read from the dictionary directly to determine the contents of the view.

mysql> select flexviews.get_sql(
-> flexviews.get_id('demo','dashboard_customer_sales'))
-> as 'SQL' \G
*************************** 1. row ***************************
SQL:
SELECT NULL as mview$pk,
(customer_id) as `customer_id`,
(customer_name) as `customer_name`,
(DATE_FORMAT(order_date,'%Y%m')) as `sale_when`,
SUM(price * quantity) as `total_price`,
COUNT(*) as `total_lines`
FROM  demo.orders as o
JOIN demo.customers as c USING (customer_id)
JOIN demo.order_lines as ol USING (order_id)
GROUP BY (customer_id),
 (customer_name),
(DATE_FORMAT(order_date,'%Y%m'))
1 row in set, 1 warning (0.00 sec)

The complete refresh method

This method is actually quite simple in its operation. It will completely rebuild a view based on the SQL which defines it, each time the view is refreshed.

This is not much different from building a table with CREATE TABLE .. AS SELECT. In fact, this is actually part of what the COMPLETE refresh method does during its operation. It also takes care of atomically* replacing the new contents of the view with the old contents (using RENAME TABLE) which means that the view remains available for querying, even during refresh.

A complete refresh type view can use non-deterministic functions like NOW(). In fact any valid SELECT statement can be materialized. The definition of the MV is stored internally, and represents the actual SQL that defines the view. A simple API call is made to associate the SQL definition with the view, but the definition itself is a SQL statement, similar to a regular view. If you choose to use only the complete refresh method, then FlexCDC is not required to use Flexviews.

In the following example, the total sales amounts are calculated (for all time) from a monthly summary. The monthly summary is actually another MV. This saves a significant amount of time. If this MV did not access the other MV, then the SQL necessary to compute this list would take over 40 minutes to execute, because that is how long it takes to create the monthly summary from scratch.

Here is an example of a complete refresh MV which computes the total sales for all customers. It computes this from the dashboard_top_customers MV, ordering them in descending order:

mysql> call flexviews.create(
-> 'demo','dashboard_top_customers','COMPLETE');
Query OK, 1 row affected (0.00 sec)

mysql> call flexviews.set_definition(
->flexviews.get_id('demo','dashboard_top_customers'),
    -> 'select customer_id,
    '> sum(total_price) total_price,
    '> sum(total_lines) total_lines
    '> from demo.dashboard_customer_sales dsc
    '> group by customer_id
    '> order by total_price desc;
    '> ');
Query OK, 1 row affected (0.00 sec)

When the view is ‘enabled’, the contents are created:

mysql> call flexviews.enable(
-> flexviews.get_id('demo','dashboard_top_customers'));
Query OK, 0 rows affected (5.73 sec)

It only took about six seconds to calculate the total sales for all customers from the dashboard_customer_sales table. If you were paying attention to the incremental example, the dashboard_customer_sales table just happens to be the view we created above.

I picked this example, because it is important to understand that you can build complete refresh views on top of incremental ones. This will allow you to create complete refresh views that refresh much more quickly.

If this had to actually access the orders, order_lines and customers tables directly (the tables on which dashboard_customer_sales is built), then the query would take significantly longer (40+ minutes):

mysql> SELECT
-> customer_id as `customer_id`,
-> SUM(price * quantity) as `total_price`,
-> COUNT(*) as `total_lines`
-> FROM  demo.orders as o
-> JOIN demo.customers as c USING (customer_id)
-> JOIN demo.order_lines as ol USING (order_id)
-> GROUP BY (customer_id)
-> ORDER BY total_price desc
-> LIMIT 10;
+-------------+-------------+-------------+
| customer_id | total_price | total_lines |
+-------------+-------------+-------------+
|         689 |      770793 |        3811 |
|        6543 |      754138 |        3740 |
|        5337 |      742034 |        3674 |
|        5825 |      738420 |        3593 |
|        5803 |      733495 |        3670 |
|        1579 |      732507 |        3666 |
|        9316 |      731091 |        3610 |
|        2046 |      722631 |        3531 |
|        6319 |      720100 |        3572 |
|        6019 |      718031 |        3475 |
+-------------+-------------+-------------+

10 rows in set (43m 10.11 sec)

As an added benefit, if you build more than one complete refresh view from an incrementally refreshable one, then you can keep the complete views transactionally consistent with each other, as long as you refresh all of the complete views after the incremental one.

We can use the new MV (dashboard_top_customers) is built, to list the top 10 customers nearly instantly. Note that every MV gets a special column `mview$pk` which is an auto_increment BIGINT surrogate key for the table. For complete refresh views which are ordered with ORDER BY, such as this one, this creates a ranking function automatically. In cases where this is not useful, simply ignore this column. It is used to prevent wide innodb primary keys on the MV.

mysql> select mview$pk as rank,
 customer_id,
 total_price,
 total_lines
 from demo.dashboard_top_customers
limit 10;
+------+-------------+-------------+-------------+
| rank | customer_id | total_price | total_lines |
+------+-------------+-------------+-------------+
|    1 |         689 |      770793 |        3811 |
|    2 |        6543 |      754138 |        3740 |
|    3 |        5337 |      742034 |        3674 |
|    4 |        5825 |      738420 |        3593 |
|    5 |        5803 |      733495 |        3670 |
|    6 |        1579 |      732507 |        3666 |
|    7 |        9316 |      731091 |        3610 |
|    8 |        2046 |      722631 |        3531 |
|    9 |        6319 |      720100 |        3572 |
|   10 |        6019 |      718031 |        3475 |
+------+-------------+-------------+-------------+
10 rows in set (0.00 sec)

You can also use this MV to display the total sales amount for any particular customer.

Notice how you can use this summary to calculate the number of lines in the order_lines table, three orders of magnitude more quickly than COUNT(*).

mysql> select count(*) cnt from order_lines\G
*************************** 1. row ***************************
cnt: 155187034
1 row in set (32.03 sec)

mysql> select sum(total_lines) cnt from dashboard_top_customers\G
*************************** 1. row ***************************
cnt: 155187034
1 row in set (0.03 sec)

Notice the difference in response time.

Refresh method performance comparison

For demonstration purposes, I did the following:

1. Created one view of each type. The view is the same as the one specified above.
2. Both views took about 40 minutes to build
3. Deleted 100 order lines.
4. Refreshed both views.
5. Compared the refresh performance and query results.

The view type does not usually* affect the time it takes to build the MVs the first time. Both of these views build in about the same about of time:

mysql> call flexviews.enable(
-> flexviews.get_id('demo','complete_example2'));
Query OK, 0 rows affected (42 min 42.14 sec)

mysql> call flexviews.enable(
-> flexviews.get_id('demo','dashboard_customer_sales'));
Query OK, 0 rows affected (41 min 52.04 sec)

*If you use MIN/MAX/COUNT_DISTINCT, a secondary view will be built in the flexviews schema, transparently, to manage the distinct values for each group in the view. This will increases the time it takes to build, and the refresh will be more expensive. This is an optimization which is required to efficiently refresh MV which use those aggregate functions.

Now I delete some line items from orders:

mysql> delete
-> from order_lines
-> where order_id
-> between 1 and 100
-> limit 500;
Query OK, 484 rows affected (0.27 sec)

Since the base table has changed, the views now exhibit drift. There should be 484 less lines reflected in the count in our MVs but they are out-of-date and must be refreshed:

mysql> select sum(total_lines) from complete_example2;
+------------------+
| sum(total_lines) |
+------------------+
|        155187034 | <-- too high
+------------------+
1 row in set (0.68 sec)

mysql> select sum(total_lines) from  dashboard_customer_sales ;
+------------------+
| sum(total_lines) |
+------------------+
|        155187034 | <-- too high
+------------------+
1 row in set (0.61 sec)

Refreshing the MVs

The view which uses the complete refresh method takes a very long time to refresh:

mysql> call flexviews.refresh(
-> flexviews.get_id('demo','complete_example2'),'BOTH',NULL);
Query OK, 0 rows affected (42 min 42.14 sec)

But the incrementally refreshable version does not take long to refresh:

mysql> call flexviews.refresh(
-> flexviews.get_id('demo',
-> 'dashboard_customer_sales'),'BOTH',NULL);
Query OK, 0 rows affected (7.01 sec)

The second one is 365x faster because it examines only the rows that changed. This exceptional ability to look at only what changed, even for MVs with aggregation and joins, is the value proposition for Flexviews.

The flexviews.refresh() stored procedure

The first parameter to flexviews.refresh() is the materialized view id. Each view has an identifier which can be obtained with flexviews.get_id(‘schema’,’table’).

The second parameter reflects the refresh type. This parameter can take the options: ‘BOTH’,’COMPUTE’,’APPLY’, or ‘COMPLETE’. I specified ‘BOTH’ for the second parameter in both examples. ‘BOTH’ means compute the changes to the view, and also apply them. This is a combination of the ‘COMPUTE’ and ‘APPLY’ options. When you specify ‘COMPUTE’ then Flexviews will compute the row changes for the view, but not apply them. If their are changes which have not been applied, then you can apply those changes by passing the ‘APPLY’ option.

In both examples, I pass NULL as the last parameter. Materialized views which based on the complete refresh method will always take NULL for this parameter. For incrementally refreshable views, the last parameter is a transaction id number, which is usually obtained with flexviews.get_uow_id_from_datetime() function. If you pass a NULL value, then it refreshes the view up to the latest changes which have been collected. You can use this to refresh multiple incremental refresh views to the same transactional point in time.

Also note that you can ‘COMPUTE’ or ‘APPLY’ changes to a particular transaction id. Logically, you can not apply changes past the transaction id to which you have computed them.

And then confirm they contain the same results

mysql> select sum(total_lines) from dashboard_customer_sales ;
+------------------+
| sum(total_lines) |
+------------------+
| 155186550        |
+------------------+
1 row in set (0.64 sec) 

mysql> select sum(total_lines) from complete_example2 ;
+------------------+
| sum(total_lines) |
+------------------+
| 155186550        |
+------------------+
1 row in set (0.68 sec)

Conclusion

Flexviews supports two refresh methods, the complete method and the incremental method. I think you will agree that the incremental method has significant advantages over the complete method. In this example the former method was over 350x faster than the latter.

I also showed how you can combine both types of views together. The complete method examples show how to create a complete refresh view which reads from an incrementally refreshable one. This allows the use of SQL features not available with the incremental method, like ORDER BY or use of NOW(), it but still provides improved performance during refresh by accessing summarized data.

I hope this helps you understand how Flexviews can help you ensure fast response times in your application by making access to summary data efficient.

View full post on MySQL Performance Blog

As a reminder, the first post covered the following topics:

1. What is a materialized view(MV)?
2. It explained that an MV can pre-compute joins and may aggregate and summarize data.
3. Using the aggregated data can significantly improve query response times compared to accessing the non-aggregated data.
4. Keeping MVs up-to-date (refreshing) is usually expensive.
5. A change data capture tool can be used to implement an efficient way of refreshing them.

This post begins with an introduction to change data capture technology and describes some of the ways in which it can be leveraged for your benefit. This is followed by a description of FlexCDC, the change data capture tool included with Flexviews. It continues with an overview of how to install and run FlexCDC, and concludes with a demonstration of the utility.

What is Change Data Capture (CDC)?

As the name implies, CDC software captures the changes made to database rows and makes those changes available in a convenient form which can be accessed by other programs. CDC applications exist for many commercial databases but until recently this type of software was not available for MySQL.

Change Data Capture can be used to:

1. Monitor a database table, or tables for changes.
2. Improve ETL performance by identifying the data which has changed.
3. Maintain materialized views with Flexviews (the primary purpose of FlexCDC).
4. Feed search engines like Sphinx or Solr only the rows that change.
5. Feed third party replication systems.
6. Provide data to “external triggers” such as Gearman jobs.

CDC tools usually operate in one of the following ways:

1. Timestamps (usually more than one) to identify rows that have changed
2. Triggers to capture changes synchronously
3. Database log reading to capture the changes asynchronously

The first method has serious drawbacks, such that it can’t identify deleted rows and MySQL timestamps may not be flexible enough.

The trigger method has a lot of problems. Triggers add a significant overhead. When the structure of a table is changed, the triggers must be changed. The work in the trigger is immediate and affects every transaction. Finally, MySQL has limited trigger support, some of which is the cause of the aforementioned problems. The biggest problem, at least from standpoint of how Flexviews works, is that triggers can not, under normal conditions, detect the commit order of transactions. This above all makes triggers an unacceptable CDC method.

This leaves the third method, log based capture as the best option because it imposes less overhead than triggers and change data capture may be done remotely and asynchronously.

Binary log based CDC

The CDC tool included with Flexviews is called FlexCDC. It seemed like an appropriate name. The Binary Log is the MySQL log which records changes to tables in the database. FlexCDC reads the binary log to determine what rows have changed. Because of this, FlexCDC requires that you use row-based binary logs (RBR). If you don’t have MySQL 5.1 or aren’t using RBR, then it is possible to set up a dedicated MySQL slave which has log_slave_updates=1 and binlog_format=row to process SBR changes from a MySQL master. I’ll talk more about that in another blog post.

FlexCDC does not implement a full binary log parser. Instead, it invokes the ‘mysqlbinlog’ utility and it processes the predictable output of this program. mysqlbinlog will always be able to read the binary logs of the version of mysql it ships with (and previous versions) so there is no worry about binary log format changes. FlexCDC is written in PHP so it is portable.

Setting up FlexCDC

FlexCDC has some basic requirements:

• MySQL 5.1+
• row based logging (binlog_format=1)
• unique server_id in the my.cnf
• log_slave_updates=1, if this is a MySQL slave
• transaction_isolation=READ-COMMITTED

You can get FlexCDC directly out of the Flexviews SVN. I suggest that you just grab all of Flexviews:

$ svn checkout http://flexviews.googlecode.com/svn/trunk/ flexviews

Next you have to customize the example ini file. FlexCDC is located in the flexviews/consumer/ subdirectory.

Create the settings file:

Change to the flexviews/consumer directory and copy the consumer.ini.example file to consumer.ini and edit it, making appropriate changes. The file is well commented. The example settings file should work for most MySQL installations which allow connections as root with no password from localhost. It is possible to read from and/or write to remote servers, but this example uses the local machine which is the usual configuration for Flexviews since it requires local access to the tables and the changelogs in order to maintain materialized views. Most database servers have some spare CPU for binary log parsing.

Run the setup script:

This will create the metadata tables and capture the initial binary log position.

php ./setup_flexcdc.php --ini consumer.ini
$ php setup_flexcdc.php
setup starting
setup completed

If the setup detects any problems (such as binary logging not being enabled) it will exit with an error. It will exit with a message “setup completed” otherwise.

Verify installation

The binary log stores it progress in a metadata table:

$ mysql -e 'select * from flexviews.binlog_consumer_status\G' -uroot
*************************** 1. row ***************************
server_id: 999
master_log_file: binary_log.000001
master_log_size: 214652
exec_master_log_pos: 214652

If you select from that table you won’t see anything changing, even if you are writing into your database. This isn’t anything to worry about, since the background process isn’t running yet.

Starting the background process:

FlexCDC includes a consumer_safe.sh script that will start up a copy of FlexCDC and restart it if it exits with error. You can shut down FlexCDC by sending it a HUP signal. The script will drop a .pid file so you know what process to HUP.

$ ./consumer_safe.sh --ini=consumer.ini &
[1] 959

$ ps
PID TTY          TIME CMD
959 pts/1    00:00:00 consumer_safe.sh
960 pts/1    00:00:00 php
967 pts/1    00:00:00 ps
6248 pts/1    00:00:00 bash

$ cat flexcdc.pid
960

Adding a changelog to a table

FlexCDC copies the contents of database rows which change into special tables called changelogs. Each changelog is located in the flexviews database and is named $SCHEMA_$TABLE where $SCHEMA is the schema in which the source table is located and $TABLE the name of the source table. If that is confusing it should be clear in a moment.

Lets create a table, insert some rows, add a change log, delete rows and then insert some more of them:

mysql> create table
> test.demo (
> c1 int auto_increment primary key,
> c2 int
> )
> engine=innodb;
Query OK, 0 rows affected (0.00 sec)

mysql> insert into test.demo values (1,1);
Query OK, 1 row affected (0.00 sec)

Even though FlexCDC is running in the background, it didn’t capture any changes from that insert. We need to add the table to the list of tables to changelog. There is a utility included with FlexCDC called ‘add_table.php’. This script automates the process of adding a table to the list of tables to changelog. It does this by adding an entry to the `flexviews`.`mvlogs` metadata table, and it creates the changelog table itself.

$ php add_table.php --schema=test --table=demo
success

Note that you can enable auto_changelog=true in the config file to automatically record changes for any table, starting from the first time a change is seen for that table. This is generally only useful if you have a small number of tables, and you want to track changes on all of them.

You may have also noted that I did not include –ini=consumer.ini. This is because this is the default config filename to search for. I included it in the earlier examples for illustration purposes.

Examine the changes

Now that the changelog has been added, any changes to `test`.`demo` will automatically be captured.

Insert data in one transaction (two rows):

mysql> insert into test.demo values (NULL,2),(NULL,3);
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

And delete data in a second transaction:

mysql> delete from test.demo where c1=1;
Query OK, 1 row affected (0.00 sec)

The changelog is flexviews.test_demo. This because the source table is `test`.`demo`.

mysql> select * from flexviews.test_demo\G
*************************** 1. row ***************************
dml_type: 1
uow_id: 10
fv$server_id: 999
c1: 2
c2: 2
*************************** 2. row ***************************
dml_type: 1
uow_id: 10
fv$server_id: 999
c1: 3
c2: 3
*************************** 3. row ***************************
dml_type: -1
uow_id: 11
fv$server_id: 999
c1: 1
c2: 1
3 rows in set (0.00 sec)

As you can see, there are three rows in the changelog, each representing one of the changes we made.

You will notice that the source table only has two columns, but the changelog contains five. All change logs contain three additional metadata columns: dml_type, uow_id and fv$server_id. These columns represent the type of change (insert is 1, delete -1), the transaction order and the source of the changes, respectively.

Finally, note that the two insertions happened inside of the same transaction, and that the insertions happened before the deletion. Though they are none shown here, updates would be represented by a deletion followed by an insertion.

View full post on MySQL Performance Blog

You might be asking yourself “what is an incrementally refreshable materialized view?”. If so, then keep reading. This is the first in a multi-part series describing Flexviews.

The output of any SQL query is logically an in-memory table
The output of a SQL query, and what we normally think of as a result set is really a virtual table. It has columns and rows, just like a database table, but it is temporary in nature, usually existing only in memory for a short time. This concept extends from relational algebra, upon which SQL is built. All SQL can be broken down into relational algebra, which is convenient because that means there are lots of transformations that can be done to it, all without changing the meaning or the output. The output of all relational algebraic statements is a table, so conceptually so too is the output of a SQL statement.

MySQL even includes a SQL statement that makes this perfectly clear: CREATE TABLE .. AS SELECT (CTAS). The results of the SELECT portion of the statement are stored in a table. Storing the results of a SQL statement into a table (even a temporary table) is called materializing the results.

Views are similar to regular SQL statements.
A view is a SQL statement which acts like a table. When you execute a query on a view, the result set is ephemeral, generated at run-time for consumption and then immediately discarded.

Views are not generally considered to be a performance optimization because:

• The contents of the result set are computed each time the view is accessed.
• If multiple statements access the same view repeatedly or concurrently, then this computation is likely to be expensive
• If the view accesses large quantities of data, then the computation is likely expensive.
• Views containing grouping, aggregation, sorting, distinct or other conditions must be fully computed and stored in a temporary table before they can be accessed, which is very expensive.

What is a materialized view (MV)?
A materialized view is similar to a regular view, in that it represents the result set of a query, but the contents are stored (materialized!) as a real table.  So a MV is similar to a table created with the CTAS command described above. This similarity is fairly superficial though. While both store results in a table, the MV represents the results of a SQL statement at a specific point in time, essentially a consistent snapshot of the query result. It is not possible to create multiple different tables via CTAS and have them all be transactionally consistent with one another, unless you stop all database write activity.

Materialized views can be used to enhance performance by acting as a cache. Further, the cost of a cache miss is lower because incremental refresh is faster than recomputing the contents from scratch.

• The contents of the result set are updated periodically, not each time the view is accessed.
• If multiple statements access the same view repeatedly or concurrently it is not likely to be very expensive.
• If the view is large, accessing the table will be considerably cheaper
• You can add indexes to the MV.
• Since the data is already joined together and pre-aggregated, CPU and memory usage may be reduced compared to computing the results.
• You can often generate multiple different blocks of content from the summarized data in one or more views

Materialized views must be refreshed.
Refreshing a MV brings it up to date to reflect the changes in the database since the view was either first created or last refreshed, whichever is more recent. More importantly a MV can can be refreshed to a specific point in time, not just “now”. This means that you can maintain multiple MVs and keep them synced to the same point in time.   There are two different methods by which a MV can be refreshed.

The first is the complete refresh method, which rebuilds the entire contents of the view from scratch. This is the less desirable method:

• During a complete refresh, the view contents are completely recalculated, which could be very expensive
• On some databases, the contents of the view are not available during complete refresh (not true of Flexviews)
• During refresh the view may occupy twice as much space (similar to ALTER TABLE)
• Supports all SQL syntax (like outer join) but can’t be refreshed to a specific point in time.
• Performs no better than CTAS, but gives a convenient method of creating and refreshing the materialized results

The second is refresh method is incremental refresh. This method updates the view. It usually examines only the rows which have changed since the view was last refreshed.

Incremental refresh has obvious benefits which include:

• Refreshing large views is orders of magnitude faster than complete refresh.
• When updating the view, only a subset of the database rows must be examined
• The rows examined are related to the rows which have changed since the last refresh
• The view can be refreshed forward to to a specific transactional point in time
• Multiple views can be rolled forward to the exact same consistent point in time, with respect to committed transactions
• The processing can be done on a slave dedicated to such work.

And some drawbacks:

• Not all SQL syntax supported (no outer join), no non-deterministic functions, etc.
• There is overhead for change-data capture.
• Some extra storage is used for the changes and deltas.
• Creating an MV is not as simple as I’d like.

Incremental refresh capabilities imply that Flexviews has some way to capture the changes that happen in the database and then apply those changes to the views.

These capabilities break down into three main categories which will be blogged about in subsequent posts:

• Change Data Capture – How Flexviews figures out what changed in the database
• Delta computation – How it uses those changes to compute the differences between the old results and the new results
• Delta application – How Flexviews uses the deltas to actually make the changes
• Finally, the last blog post will describe how to use Flexviews for online schema change, or as an ELT tool and it will cover how to create the materialized views which Flexviews manages.

View full post on MySQL Performance Blog