View full post on PHP Classes: Latest entries

Amazon DynamoDB is a fully managed NoSQL database service that provides fast performance at any scale. Today’s web-based applications often encounter database scaling challenges when faced with growth in users, traffic, and data. With Amazon DynamoDB, developers scaling cloud-based applications can start small with just the capacity they need and then increase the request capacity of a given table as their app grows in popularity. Their tables can also grow without limits as their users store increasing amounts of data. Behind the scenes, Amazon DynamoDB automatically spreads the data and traffic for a table over a sufficient number of servers to meet the request capacity specified by the customer. Amazon DynamoDB offers low, predictable latencies at any scale. Customers can typically achieve average service-side in the single-digit milliseconds. Amazon DynamoDB stores data on Solid State Drives (SSDs) and replicates it synchronously across multiple AWS Availability Zones in an AWS Region to provide built-in high availability and data durability.

History of NoSQL at Amazon – Dynamo

The Amazon.com ecommerce platform consists of hundreds of decoupled services developed and managed in a decentralized fashion. Each service encapsulates its own data and presents a hardened API for others to use. Most importantly, direct database access to the data from outside its respective service is not allowed. This architectural pattern was a response to the scaling challenges that had challenged Amazon.com through its first 5 years, when direct database access was one of the major bottlenecks in scaling and operating the business. While a service-oriented architecture addressed the problems of a centralized database architecture, each service was still using traditional data management systems. The growth of Amazon’s business meant that many of these services needed more scalable database solutions.

In response, we began to develop a collection of storage and database technologies to address the demanding scalability and reliability requirements of the Amazon.com ecommerce platform. We had been pushing the scalability of commercially available technologies to their limits and finally reached a point where these third party technologies could no longer be used without significant risk. This was not our technology vendors’ fault; Amazon’s scaling needs were beyond the specs for their technologies and we were using them in ways that most of their customers were not. A number of outages at the height of the 2004 holiday shopping season can be traced back to scaling commercial technologies beyond their boundaries.

Dynamo was born out of our need for a highly reliable, ultra-scalable key/value database. This non-relational, or NoSQL, database was targeted at use cases that were core to the Amazon ecommerce operation, such as the shopping cart and session service. Any downtime or performance degradation in these services has an immediate financial impact and their fault-tolerance and performance requirements for their data systems are very strict. These services also require the ability to scale infrastructure incrementally to accommodate growth in request rates or dataset sizes. Another important requirement for Dynamo was predictability. This is not just predictability of median performance and latency, but also at the end of the distribution (the 99.9th percentile), so we could provide acceptable performance for virtually every customer.

To achieve all of these goals, we needed to do groundbreaking work. After the successful launch of the first Dynamo system, we documented our experiences in a paper so others could benefit from them. Since then, several Dynamo clones have been built and the Dynamo paper has been the basis for several other types of distributed databases. This demonstrates that Amazon is not the only company than needs better tools to meet their database needs.

Lessons learned from Amazon’s Dynamo

Dynamo has been in use by a number of core services in the ecommerce platform, and their engineers have been very satisfied by its performance and incremental scalability. However, we never saw much adoption beyond these core services. This was remarkable because although Dynamo was originally built to serve the needs of the shopping cart, its design and implementation were much broader and based on input from many other service architects. As we spoke to many senior engineers and service owners, we saw a clear pattern start to emerge in their explanations of why they didn’t adopt Dynamo more broadly: while Dynamo gave them a system that met their reliability, performance, and scalability needs, it did nothing to reduce the operational complexity of running large database systems. Since they were responsible for running their own Dynamo installations, they had to become experts on the various components running in multiple data centers. Also, they needed to make complex tradeoff decisions between consistency, performance, and reliability. This operational complexity was a barrier that kept them from adopting Dynamo.

During this period, several other systems appeared in the Amazon ecosystem that did meet their requirements for simplified operational complexity, notably Amazon S3 and Amazon SimpleDB. These were built as managed web services that eliminated the operational complexity of managing systems while still providing extremely high durability. Amazon engineers preferred to use these services instead of managing their own databases like Dynamo, even though Dynamo’s functionality was better aligned with their applications’ needs.

With Dynamo we had taken great care to build a system that met the requirements of our engineers. After evaluations, it was often obvious that Dynamo was ideal for many database use cases. But … we learned that engineers found the prospect of running a large software system daunting and instead looked for less ideal design alternatives that freed them from the burden of managing databases and allowed them to focus on their applications.

It became obvious that developers strongly preferred simplicity to fine-grained control as they voted “with their feet” and adopted cloud-based AWS solutions, like Amazon S3 and Amazon SimpleDB, over Dynamo. Dynamo might have been the best technology in the world at the time but it was still software you had to run yourself. And nobody wanted to learn how to do that if they didn’t have to. Ultimately, developers wanted a service.

History of NoSQL at Amazon – SimpleDB

One of the cloud services Amazon developers preferred for their database needs was Amazon SimpleDB. In the 5 years that SimpleDB has been operational, we have learned a lot from its customers.

First and foremost, we have learned that a database service that takes away the operational headache of managing distributed systems is extremely powerful. Customers like SimpleDB’s table interface and its flexible data model. Not having to update their schemas when their systems evolve makes life much easier. However, they most appreciate the fact that SimpleDB just works. It provides multi-data center replication, high availability, and offers rock-solid durability. And yet customers never need to worry about setting up, configuring, or patching their database.

Second, most database workloads do not require the complex query and transaction capabilities of a full-blown relational database. A database service that only presents a table interface with a restricted query set is a very important building block for many developers.

While SimpleDB has been successful and powers the applications of many customers, it has some limitations that customers have consistently asked us to address.

Domain scaling limitations. SimpleDB requires customers to manage their datasets in containers called Domains, which have a finite capacity in terms of storage (10 GB) and request throughput. Although many customers worked around SimpleDB’s scaling limitations by partitioning their workloads over many Domains, this side of SimpleDB is certainly not simple. It also fails to meet the requirement of incremental scalability, something that is critical to many customers looking to adopt a NoSQL solution.

Predictability of Performance. SimpleDB, in keeping with its goal to be simple, indexes all attributes for each item stored in a domain. While this simplifies the customer experience on schema design and provides query flexibility, it has a negative impact on the predictability of performance. For example, every database write needs to update not just the basic record, but also all attribute indices (regardless of whether the customer is using all the indices for querying). Similarly, since the Domain maintains a large number of indices, its working set does not always fit in memory. This impacts the predictability of a Domain’s read latency, particularly as dataset sizes grow.

Consistency. SimpleDB’s original implementation had taken the “eventually consistent” approach to the extreme and presented customers with consistency windows that were up to a second in duration. This meant the system was not intuitive to use and developers used to a more traditional database solution had trouble adapting to it. The SimpleDB team eventually addressed this issue by enabling customers to specify whether a given read operation should be strongly or eventually consistent.

Pricing complexity. SimpleDB introduced a very fine-grained pricing dimension called “Machine Hours.” Although most customers have eventually learned how to predict their costs, it was not really transparent or simple.

Introducing DynamoDB

As we thought about how to address the limitations of SimpleDB and provide 1) the most scalable NoSQL solution available and 2) predictable high performance, we realized our goals could not be met with the SimpleDB APIs. Some SimpleDB operations require that all data for a Domain is on a single server, which prevents us from providing the seamless scalability our customers are demanding. In addition, SimpleDB APIs assume all item attributes are automatically indexed, which limits performance.

We concluded that an ideal solution would combine the best parts of the original Dynamo design (incremental scalability, predictable high performance) with the best parts of SimpleDB (ease of administration of a cloud service, consistency, and a table-based data model that is richer than a pure key-value store). These architectural discussions culminated in Amazon DynamoDB, a new NoSQL service that we are excited to release today.

Amazon DynamoDB is based on the principles of Dynamo, a progenitor of NoSQL, and brings the power of the cloud to the NoSQL database world. It offers customers high-availability, reliability, and incremental scalability, with no limits on dataset size or request throughput for a given table. And it is fast – it runs on the latest in solid-state drive (SSD) technology and incorporates numerous other optimizations to deliver low latency at any scale.

Amazon DynamoDB is the result of everything we’ve learned from building large-scale, non-relational databases for Amazon.com and building highly scalable and reliable cloud computing services at AWS. Amazon DynamoDB is a NoSQL database service that offers the following benefits:

• Managed. DynamoDB frees developers from the headaches of provisioning hardware and software, setting up and configuring a distributed database cluster, and managing ongoing cluster operations. It handles all the complexities of scaling and partitions and re-partitions your data over more machine resources to meet your I/O performance requirements. It also automatically replicates your data across multiple Availability Zones (and automatically re-replicates in the case of disk or node failures) to meet stringent availability and durability requirements. From our experience of running Amazon.com, we know that manageability is a critical requirement. We have seen many job postings from companies using NoSQL products that are looking for NoSQL database engineers to help scale their installations. We know from our Amazon experiences that once these clusters start growing, managing them becomes the same nightmare that running large RDBMS installations was. Because Amazon DynamoDB is a managed service, you won’t need to hire experts to manage your NoSQL installation—your developers can do it themselves.

• Scalable. Amazon DynamoDB is designed to scale the resources dedicated to a table to hundreds or even thousands of servers spread over multiple Availability Zones to meet your storage and throughput requirements. There are no pre-defined limits to the amount of data each table can store. Developers can store and retrieve any amount of data and DynamoDB will spread the data across more servers as the amount of data stored in your table grows.

• Fast. Amazon DynamoDB provides high throughput at very low latency. It is also built on Solid State Drives to help optimize for high performance even at high scale. Moreover, by not indexing all attributes, the cost of read and write operations is low as write operations involve updating only the primary key index thereby reducing the latency of both read and write operations. An application running in EC2 will typically see average service-side latencies in the single-digit millisecond range for a 1KB object. Most importantly, DynamoDB latencies are predictable. Even as datasets grow, latencies remain stable due to the distributed nature of DynamoDB’s data placement and request routing algorithms.

• Durable and Highly Available. Amazon DynamoDB replicates its data over at least 3 different data centers so that the system can continue to operate and serve data even under complex failure scenarios.

• Flexible. Amazon DynamoDB is an extremely flexible system that does not force its users into a particular data model or a particular consistency model. DynamoDB tables do not have a fixed schema but instead allow each data item to have any number of attributes, including multi-valued attributes. Developers can optionally use stronger consistency models when accessing the database, trading off some performance and availability for a simpler model. They can also take advantage of the atomic increment/decrement functionality of DynamoDB for counters.

• Low cost. Amazon DynamoDB’s pricing is simple and predictable: Storage is $1 per GB per month. Requests are priced based on how much capacity is reserved: $0.01 per hour for every 10 units of Write Capacity and $0.01 per hour for every 50 units of Read Capacity. A unit of Read (or Write) Capacity equals one read (or write) per second of capacity for items up to 1KB in size. If you use eventually consistent reads, you can achieve twice as many reads per second for a given amount of Read Capacity. Larger items will require additional throughput capacity.

In the current release, customers will have the choice of using two types of keys for primary index querying: Simple Hash Keys and Composite Hash Key / Range Keys:

Simple Hash Key gives DynamoDB the Distributed Hash Table abstraction. The key is hashed over the different partitions to optimize workload distribution. For more background on this please read the original Dynamo paper.

Composite Hash Key with Range Key allows the developer to create a primary key that is the composite of two attributes, a “hash attribute” and a “range attribute.” When querying against a composite key, the hash attribute needs to be uniquely matched but a range operation can be specified for the range attribute: e.g. all orders from Werner in the past 24 hours, all log entries from server 16 with clients IP addresses on subnet 192.168.1.0

Performance Predictability in DynamoDB

In addition to taking the best ideas of Dynamo and SimpleDB, we have added new functionality to provide even greater performance predictability.

Cloud-based systems have invented solutions to ensure fairness and present their customers with uniform performance, so that no burst load from any customer should adversely impact others. This is a great approach and makes for many happy customers, but often does not give a single customer the ability to ask for higher throughput if they need it.

As satisfied as engineers can be with the simplicity of cloud-based solutions, they would love to specify the request throughput they need and let the system reconfigure itself to meet their requirements. Without this ability, engineers often have to carefully manage caching systems to ensure they can achieve low-latency and predictable performance as their workloads scale. This introduces complexity that takes away some of the simplicity of using cloud-based solutions.

The number of applications that need this type of performance predictability is increasing: online gaming, social graphs applications, online advertising, and real-time analytics to name a few. AWS customers are building increasingly sophisticated applications that could benefit from a database that can give them fast, predictable performance that exactly matches their needs.

Amazon DynamoDB’s answer to this problem is “Provisioned Throughput.” Customers can now specify the request throughput capacity they require for a given table. Behind the scenes, DynamoDB will allocate sufficient resources to the table to predictably achieve this throughput with low-latency performance. Throughput reservations are elastic, so customers can increase or decrease the throughput capacity of a table on-demand using the AWS Management Console or the DynamoDB APIs. CloudWatch metrics enable customers to make informed decisions about the right amount of throughput to dedicate to a particular table. Customers using the service tell us that it enables them to achieve the appropriate amount of control over scaling and performance while maintaining simplicity. Rather than adding server infrastructure and re-partitioning their data, they simply change a value in the management console and DynamoDB takes care of the rest.

Summary

Amazon DynamoDB is designed to maintain predictably high performance and to be highly cost efficient for workloads of any scale, from the smallest to the largest internet-scale applications. You can get started with Amazon DynamoDB using a free tier that enables 40 million of requests per month free of charge. Additional request capacity is priced at cost-efficiently hourly rates as low as $.01 per hour for 10 units of Write Capacity or 50 strongly consistent units of Read Capacity (if you use eventually consistent reads you can get twice the throughput at the same cost, or the same read throughput at half the cost) Also, replicated solid state disk (SSD) storage is $1 per GB per month. Our low request pricing is designed to meet the needs of typical database workloads that perform large numbers of reads and writes against every GB of data stored.

To learn more about Amazon DynamoDB its functionality, APIs, use cases, and service pricing, please visit the detail page at aws.amazon.com/DynamoDB and also the Developer Guide. I am excited to see the years of experience with systems such as Amazon Dynamo result in an innovative database service that can be broadly used by all our customers.

View full post on All Things Distributed

View full post on PHP Classes: Latest entries

For most people, managing a database server during its lifetime comes down to this rather boring process – install, configure, tune, start/stop MYSQL, upgrade MySQL, 10x start/stop MySQL, upgrade MySQL, downgrade MySQL, upgrade MySQL, install a security fix for something, 20x start/stop MySQL, upgrade MySQL, … . Restarting and changing MySQL version might just be the most critical operations you ever do once server is moved into production. Success and efficiency of these operations may directly impact service availability, which means that what you could wish for the most are quick and problem free restarts and upgrades (or downgrades).

From my experience, and I have done thousands upgrades and downgrades in my life, the least number of problems come from RPM packages available in RedHat, CentOS and SuSE. In fact, I cannot recall encountering any serious problem with that package management system. Moreover, I have not seen broken systems, where installing or updating a RPM package would be impossible without resolving tons of problems first. It can obviously mean that RPM has flaws and does not verify consistency very carefully, but it never turned out to be any problem.

On the other end there are Debian and Ubuntu. Both use tool called dpkg for package management. There isn’t a month that I log in to a system based on either distribution where there are no issues with packages consistency. Unfinished installations, unresolved conflicts are so common that it’s just beyond simple negligence. The packaging system is just not robust enough. Another problem is that one broken package may block you from installing or uninstalling anything else. Imagine that someone left system in such shape, you prepared for downtime, stopped MySQL and… error – text editor has not been properly installed, so you cannot upgrade MySQL either until the problem is fixed. In a stressful situation when downtime clock ticks – annoying at best. Resolving problems can easily lead to unexpected consequences. Here’s a different scenario:

# apt-get install binutils
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.


While everyone should stop and check what this means instead of following the advice blindly, many still would just do as they were instructed. In this particular case the problems were with MySQL packages upgraded from 5.1 to 5.5. The former were not fully uninstalled, the latter were in half-configured state. Obviously binutils has nothing to do with MySQL, but it would not install anyway. The larger problem – chances are that fixing this with ‘dpkg –configure -a’ would cause MySQL to shut down. The MySQL package scripts for Debian force MySQL stop/start upon completion and unlike RPM, dpkg does not have any opt-out option.

More problems can come from the standard Debian init (startup) script for MySQL. By default it not only starts database, but also updates system tables (if needed), scans all tables for consistency problems, etc. I mean, that works great for a computer at home or a tiny and unimportant system, but any of these operations may have severe consequences on a large production system. This is why all of this extra functionality was stripped from Percona Server. This behavior actually appears to be a general problem with Debian – it wants to be smarter than you. This might work for desktops, but anywhere else it is plain stupid and makes you waste time on outsmarting a wise-ass system. Of course I do know some people who like such approach, but he is Belgian and they were unable to form a government for 541 days… (congrats that you finally made it this week, btw! )

The init scripts for RedHat, CentOS and SuSE are simple and do only what’s required of them – stop or start MySQL. No problems there.

It’s now clear that I never recommend Debian or Ubuntu, because I do not like some of the “mechanics” and I feel people are generally safer if they do not use any of these two. With the choice left between RedHat/CentOS and SuSE, I lean towards the former. Why? RedHat and CentOS are the only platforms getting packages from all MySQL and MySQL-fork vendors – Oracle, MariaDB and Percona. By my observations RedHat and CentOS are also much more frequently used with MySQL, so there will be more resources available online.

But if you are a skilled systems administrator or your company hires one, then you could use pretty much anything, including my favorites such as Slackware and Gentoo.

In the beginning I mentioned that individual circumstances may influence the decision. One example of such specific case, which limits your options, is for example a requirement for commercial support for the operating system. You will have to choose between RedHat or SuSE. Another such case can be related to hardware. Any newly released component may simply lack support from the Linux kernel, but this can also be the case for any high-end/enterprise class equipment. Hardware vendors may release Linux drivers on their own, but often only for very specific Linux distributions.

View full post on MySQL Performance Blog

Just in case anybody missed it: http://blogs.oracle.com/MySQL/entry/new_commercial_extensions_for_mysql

MySQL has long been an open source product, not an open source project…. and this really is the final nail in that.

To me, this was expected, but it’s still sad to see it.

I am very, very glad we have diverse copyright ownership in Drizzle so that this could not happen easily at all.

View full post on Planet Drizzle

AWS Database Services is responsible for setting the database strategy and delivering distributed structured storage services to our AWS customers. This team is constantly rethinking the assumptions behind how traditional databases were built and constantly working on building the right database architectures suited for the Cloud environment. The database services organization is looking for senior leaders who will be able to hire and lead a large software development team that is responsible for designing and running services that are at the cutting edge of distributed database technology that helps our customers to build scalable database-driven applications in the cloud and have a significant bottom-line impact on our business.

The ideal candidate will be someone who has built and ran large scale distributed systems and/or databases. She (or he) will be able to reason about the standard tradeoffs in building large scale distributed databases and is capable of guiding the team to make these tradeoffs.

For more information: Head of Software Development 

View full post on All Things Distributed

It has been shown that combinatorial algebraics can be applied to all aggregate functions (Flexviews uses only composable operations even for deletes over “non-distributable aggregate functions”), and Flexviews includes this capability today. Because Flexviews can move database objects forward in time synchronously with each other to specific points in time in database state, invalid database state is not possible, particularly if there are thorough database checks.

Flexviews is:
lock free for read
fully log based
supports all DML
supports all aggregate functions
supports join
supports NULL values
supports simple projection
only rolls forward in time right now
MVCC
row level locking with innodb

Flexviews was based on the following research:
pages.cs.wisc.edu/~beyer/
citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7788

Also includes my own personal and extensive research into combinatorial algebra over non-distributable aggregate functions. I was not aware of research into combinatorial algebra and developed these techniques in isolation from other researchers as I am an amateur.

View full post on MySQL Performance Blog

View full post on PHP Classes: Latest entries

Of course, since I am a QA Engineer, I could not resist the urge to try to break it >:)  As my friend and former MySQL colleage, Kostja says, “QA Engineers are a unique breed…they like to push all the buttons” : )  I would say that the QA mindset goes a bit further than that, but it is something I will delve into in another post.  I will only say that there is a reason that Microsoft recognizes QA software engineering as a distinct and specialized discipline.

So, let’s get back to Xeround.  It was the first database as a service that caught my eye and I just had to test it!  They are currently offering a free beta.  It is remarkably easy and fast to get set up with a test database and the web-based dashboard they provide is pretty interesting and offers some good information (though some of it is confusing…more on that in a bit)

It was my intent to run a small handful of tests with the mighty, mighty randgen!

My tests were as follows:

1. outer_join grammar – creates seriously nasty JOIN queries that can use up to 20 tables
2. transactional grammar – we have a grammar that creates a variety of transactions.  Some good, some bad, with lots of ROLLBACKs and SAVEPOINTs sprinkled in for spice.
3. subqueries – the nastiest grammar I have created and as I have mentioned elsewhere, it is also part of why we are just now seeing optimizer features like index condition pushdown (ICP) being reintroduced to MySQL >: )

My thoughts were that these could be quickly executed and point out any serious problems in basic functionality.  MySQL and Drizzle both use these grammars as part of their testing.  Drizzle must survive these tests on every push to trunk, so these seem like reasonable stressors for a new engine >: )

It should be noted that I had to modify the test grammars to accomodate some Xeround limitations, the modified randgen branch I used is here.  It can be branched via bzr branch lp:~patrick-crews/randgen/randgen_drizzle_exp

Each grammar would be run with the randgen’s –debug option.  This is because the user is presented with a nice report at the end of the run which indicates:  query_count:row_count (ie how many queries returned how many rows):

# 2011-04-27T20:40:18 Rows returned:
$VAR1 = {
‘    0′ => 59,
‘    1′ => 2,
‘    4′ => 1,
‘    9′ => 1,
‘   -1′ => 35,
‘>100′ => 1
};

I would use this as a comparison point against MySQL 5.1.  Granted, I could use the –Validator=ResultsetComparatorSimplify option, but then I would have an actual bug report that I would feel compelled to file and this would feel less like fun and more like work ; )  However, I have been in contact with engineers from Xeround and have shared my findings with them.

For the transactional grammar, I would run the grammar on each system and then do a diff of mysqldump files from each database.  As Xeround is a MySQL engine, this could cause some differences, but the data in the tables should be consistent.

Before I get into the testing results, I’ll provide some overall impressions:
As I said, the web interface is pretty nice and provides you with a lot of useful information.  It allows you to easily create a new named database instance and provides you with data such as status, scale, uptime, cpu utilization, memory utilization, number of connections, ops/sec, and message count.  Scale refers to the autoscale capabilities that Xeround advertises.  For the beta, you are allowed to scale from 3 to 4 servers.  3 servers is considered 100%, adding the extra server (when certain user-specified CPU or Memory limits are hit) equates to 133% .  Interestingly enough, I observed that there were always 6 active connections when the database was idle (probably some of the Xeround ‘secret sauce‘ working…).

The control panel also allows the user to set the CPU, memory, and connections limits that will trigger scale up (and possibly scale down).  In my experiments, I never seemed to tax memory or connections, but CPU limits were hit and auto-scale did trigger, though I will admit that I didn’t observe any noticeable change in the test execution.

There are also tabs for backup (not available in the free beta, though mysqldump does work against a Xeround instance), general monitoring which provides real-time information about cpu, memory and connections, and an events (messages tab).  The one thing I noted about the events tab was that I received a number of warning messages about the health of my database during times I wasn’t using it.  However, it is a beta service for general evaluation and certain oddities are to be expected.

Here is what I found with my tests:
1)  Xeround is a MySQL engine.  They do advertise this, but the main reason I noticed that all of my created test tables were now ‘Engine=Xeround’ was that I was unable to create a varchar_1024 indexed column.  Xeround is limited to 255 characters max:

# 2011-04-27T19:50:27 key (`col_char_1024_key` ))  failed: 1074 Column length too big for column 'col_char_1024' (max = 255); use BLOB or TEXT instead


This limitation required modification of the randgen grammars and gendata files to limit char columns to 255.  As noted above, you can find the modified version of the randgen here.

2)  Tables with an ENGINE=$engine_name argument are processed without an issue (ie you should be able to use a dumpfile without problems) and are converted to Xeround tables.  One thing to note is that dumpfiles *from* Xeround have ENGINE=Xeround for the CREATE TABLE statements


create table t1 (a int not null auto_increment, primary key(a)) engine=innodb;
Query OK, 0 rows affected, 2 warnings (0.702761 sec)
drizzle> show create table t1;
+-------+---------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                          |
+-------+---------------------------------------------------------------------------------------------------------------------------------------+
| t1    | CREATE TABLE `t1` (
`a` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`a`)
) ENGINE=Xeround DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
+-------+---------------------------------------------------------------------------------------------------------------------------------------+


3)  outer_join grammar:
I used the following command line:

./gentest.pl --gendata=conf/drizzle/outer_join_drizzle.zz --grammar=conf/drizzle/outer_join_drizzle.yy --queries=100 --threads=1 --dsn=dbi:mysql:host=00.00.00.00:port=9999:user=USER:password=PASSWORD:database=test --sqltrace --debug


The test is designed to generate queries with large numbers of tables (up to ~20).  The test ran without much incident.  The Xeround server monitor indicated that the CPU was hovering near 80% for most of the time, but again…beta test setup, so I’ll give them some leeway.

The big trouble is what follows.  Remember those randgen summary reports I mentioned earlier?  Below is a comparison of Xeround vs. MySQL for the same command line.  The values are row_count’ => number_of_queries_returning_said_row_count.  What this means is that for the same set of queries, Xeround and MySQL do not always return the same result sets.  I did not note any differences in query failures, so this simply indicates that results processing is differing somewhere : (  To elaborate, Xeround had 56 queries that returned 0 rows, for the same workload, MySQL only had 39.  A row count of -1 indicates that there was an error with the query, such as referencing a table or column that doesn’t exist.  Somehow, Xeround hit fewer errors than MySQL, though that is also worrisome – why do they register errors differently?
Xeround:

# 2011-04-27T20:11:05 Rows returned:
$VAR1 = {
'    0' => 56,
'    1' => 16,
'    2' => 6,
'    3' => 2,
'    5' => 1,
'    6' => 1,
'    7' => 1,
'    8' => 1,
'   -1' => 13,
'   10' => 2,
'>10' => 1
};


MySQL 5.1

$VAR1 = {
'    0' => 39,
'    1' => 15,
'    2' => 2,
'    3' => 2,
'    4' => 1,
'    7' => 2,
'    8' => 1,
'   -1' => 32,
'   10' => 1,
'>10' => 5
};


4)  transactional grammar:
I used the following command line:

./gentest.pl --gendata=conf/drizzle/translog_drizzle.zz --grammar=conf/drizzle/translog_concurrent1.yy --queries=100 --threads=1 --dsn=dbi:mysql:host=00.00.00.00:port=9999:user=USER:password=PASSWORD:database=test --sqltrace --debug


This grammar generates a variety of transactions and standalone queries.  The queries generated consist of both good and invalid SQL with lots of ROLLBACK’s and SAVEPOINT’s here and there.  Unfortunately, I noticed a large number of differences.  We’ll start with the easiest one:

< DROP TABLE IF EXISTS `A`;
< CREATE TABLE `A` (
---
> DROP TABLE IF EXISTS `a`;
> CREATE TABLE `a` (
50c50
< ) ENGINE='InnoDB' AUTO_INCREMENT=105 COLLATE='utf8_general_ci';
---
> ) ENGINE='Xeround' COLLATE='utf8_bin';


It isn’t huge, but Xeround apparently auto-converts tables names to lower-case.  The randgen attempts to create table `A`, but it is stored as table `a`.  This could be an issue for some people, but Xeround does say that the beta is for people to evaluate the system’s suitability for their purposes.

The big issue is that Xeround appears to not have registered a lot of the transactions issued by the randgen.  The Xeround dumpfile only contained the original 10 rows from table `a`, while the MySQL 5.1 version I ran locally had 94 rows by the end of the randgen run : (

Further research of the randgen logs indicate the following issue:

# 2011-04-27T20:06:56 Query:  INSERT INTO `d` ( `col_char_10` , `col_char_10_key` , `col_char_10_not_null` , `col_char_10_not_null_key` , `col_char_255` , `col_char_255_key` , `col_char_255_not_null` , `col_char_255_not_null_key` , `col_int` , `col_int_key` , `col_int_not_null` , `col_int_not_null_key` , `col_bigint` , `col_bigint_key` , `col_bigint_not_null` , `col_bigint_not_null_key` , `col_enum` , `col_enum_key` , `col_enum_not_null` , `col_enum_not_null_key` , `col_text` , `col_text_key` , `col_text_not_null` , `col_text_not_null_key` ) SELECT `col_char_10` , `col_char_10_key` , `col_char_10_not_null` , `col_char_10_not_null_key` , `col_char_255` , `col_char_255_key` , `col_char_255_not_null` , `col_char_255_not_null_key` , `col_int` , `col_int_key` , `col_int_not_null` , `col_int_not_null_key` , `col_bigint` , `col_bigint_key` , `col_bigint_not_null` , `col_bigint_not_null_key` , `col_enum` , `col_enum_key` , `col_enum_not_null` , `col_enum_not_null_key` , `col_text` , `col_text_key` , `col_text_not_null` , `col_text_not_null_key` FROM `bb`  ORDER BY `col_bigint`,`col_bigint_key`,`col_bigint_not_null`,`col_bigint_not_null_key`,`col_char_10`,`col_char_10_key`,`col_char_10_not_null`,`col_char_10_not_null_key`,`col_char_255`,`col_char_255_key`,`col_char_255_not_null`,`col_char_255_not_null_key`,`col_enum`,`col_enum_key`,`col_enum_not_null`,`col_enum_not_null_key`,`col_int`,`col_int_key`,`col_int_not_null`,`col_int_not_null_key`,`col_text`,`col_text_key`,`col_text_not_null`,`col_text_not_null_key`,`pk` LIMIT 50 /*Generated by THREAD_ID 1*/  failed: 1038 Out of sort memory; increase server sort buffer size


So, it would appear that transactions are failing for some reason or another.  However, I repeat the disclaimer about this being a beta and not a production deployment.  It could have something to do with the resources allocated for each beta user.

5)  Subquery grammar
This was the initial test I ran, but I have saved it for last.  First of all, the command line:

./gentest.pl --gendata=conf/drizzle/drizzle.zz --grammar=conf/drizzle/optimizer_subquery_drizzle.yy --queries=100 --threads=1 --dsn=dbi:mysql:host=00.00.00.00:port=9999:user=USER:password=PASSWORD:database=test --sqltrace --debug


This test generates some very nasty subquery-laded queries (see below).  The first thing I noticed on the single-threaded run was that Xeround seemed to not like this query very much at all:

SELECT    table2 . `col_int` AS field1 FROM ( CC AS table1 STRAIGHT_JOIN ( ( CC AS table2 STRAIGHT_JOIN CC AS table3 ON (table3 . `col_bigint_key` = table2 . `col_int_not_null_key`  ) ) ) ON (table3 . `col_text_not_null_key` = table2 . `col_char_10_key`  ) ) WHERE (  table1 . `col_int` NOT IN ( SELECT   SUBQUERY1_t1 . `col_int_not_null_key` AS SUBQUERY1_field1 FROM ( BB AS SUBQUERY1_t1 INNER JOIN ( CC AS SUBQUERY1_t2 INNER JOIN BB AS SUBQUERY1_t3 ON (SUBQUERY1_t3 . `col_char_10_key` = SUBQUERY1_t2 . `col_char_10_key`  ) ) ON (SUBQUERY1_t3 . `col_char_10_not_null_key` = SUBQUERY1_t2 . `col_char_10`  ) ) WHERE SUBQUERY1_t2 . `col_bigint` != table1 . `pk` OR SUBQUERY1_t2 . `pk` >= table2 . `pk` ) ) OR ( table1 . `col_int_key`  BETWEEN 48 AND ( 48 + 183 ) OR table1 . `pk`  BETWEEN 48 AND ( 48 + 104 ) )  GROUP BY field1  ;


Now it is quite nasty, but standard MySQL executes it with a minimum of fuss (though it does take a moment to handle this monster as well).

The other thing is that Xeround took an exceedingly long time to execute this workload.  While the other grammars executed in moderate amounts of time (my testing was from a hotel room in Santa Clara while the instance is in Chicago), the subquery test was noticeably slow.  I was able to walk down to the lobby, buy something, and return to my room while it was dealing with the nasty query above : (  For some context, running the same command line on my laptop took 8 seconds, Xeround took 14 minutes, but again…beta test setup and hardware, so YMMV.

Finally, we have the dreaded row count report:
Xeround:

# 2011-04-27T20:45:19 Rows returned:
$VAR1 = {
'    0' => 59,
'    1' => 2,
'    4' => 1,
'   -1' => 35,
'>10' => 1,
'>100' => 1
};


MySQL 5.1:

# 2011-04-27T20:40:18 Rows returned:
$VAR1 = {
'    0' => 59,
'    1' => 2,
'    4' => 1,
'    9' => 1,
'   -1' => 35,
'>100' => 1
};


As we can see, there is 1 query out of the 100 issued where result sets differed (returning 9 rows in MySQL vs. >10 rows in Xeround).

I also tried using –threads=10 to really stress the Xeround system (I didn’t bother with MySQL, it handles 10 threads of nasty subqueries like a champ…incidentally, so does Drizzle) ; ) Xeround was able to handle the workload and did so in 27 minutes. Since single-threaded took 14 minutes, perhaps Xeround doesn’t really begin to shine until we start hitting large numbers of concurrent connections?

So what can I say from the results of these informal tests?  Personally, I would hesitate to say that Xeround is a drop-in replacement.  The limitations on column sizes, changes in table naming, and differing result sets are a bit worrisome.  However, I will say that the Xeround engineers I met at the UC were very engaged and interested in my findings and have made significant strides in subquery processing since my initial tests.  I believe that with time these issues will be fixed and that not every customer will run into them (I know I’m beating this into the ground, but I was using a beta test system).  Behavior may be different on a production machine and not every MySQL user will generate such workloads and every customer should perform their own careful testing and evaluation before making any changes to their systems.

My personal interest ends here.  The UC introduced me to a number of interesting new storage engines and I was mainly curious about ways of evaluating them.  This was a quick and dirty bit of testing just to see if I could produce any interesting pyrotechnics ; )  Go go randgen!

I really want this picture to be shown when anyone searches for ‘randgen’ ; )

In all seriousness, I highly recommend adoption of the random query generator.  It offers a modular and customizable system for creating evaluation tools (like result set comparison, execution time comparison, replication validation, etc, etc) and has been used in production-level testing for MySQL, MariaDB and Drizzle for some time.  It also plays with Postgresql and Java DB (kind of scary that 40% of that list is owned by Oracle…), so please give it a spin and see what kinds of pretty explosions you can make…who knows, testing might actually become fun for non-QA folks >; )

Additionally, these tests only took me about half an hour to setup and execute.  Granted, I have been using the tool for some time, but 30 minutes to identify a number of potential problem areas seems pretty awesome to me, but then again, I am a QA Engineer and we live for such things.

View full post on Planet Drizzle

View full post on PHP Classes: Latest entries