Weez.com

testpost

Abidoon — Fri, 03 Sep 2010 18:37:13 +0000

New test post physics physics

Cool Handle Hosting Doubles Size of Data Center

Abidoon — Fri, 03 Sep 2010 18:20:48 +0000

Cool Handle Hosting Doubles Size of Data Center
LOS ANGELES, Sept. 1 /PRNewswire/ — Web hosting provider Cool Handle (http://www.coolhandle.com) has doubled their data center size and infrastructure. The upgrade allows Cool Handle to offer dedicated servers and virtual private servers at an amazing 50% off for a limited time only. Not only is this an incredibly low price, but the offer also includes 100Mbit network …

IPnet Calculator

Abidoon — Fri, 03 Sep 2010 18:03:19 +0000

Package:

IPnet Calculator

Summary:

Compute the common mask from multiple IP addresses

Groups:

Networking, PHP 5

Author:

Alexander Over

Description:

This class can be used to compute the common mask from multiple IP addresses.

It takes two or more IP addresses and determines the bits with the common values in order to determine what should be the applicable network address mask.

The class supports network addresses either in the IP version 4 and version 6 formats.

View full post on PHP Classes: Latest entries

Drizzle build 1742 tarball has been released

Abidoon — Fri, 03 Sep 2010 18:03:18 +0000

Drizzle source tarball based on build 1742 has been released.

In this release:

*Continued work on logging statistics – added a table to the scoreboard that gives memory usage and size
print stack trace and stack dump have been removed. As we use gcc for a majority of our builds, we are now using the built-in backtrace ability. Added crash and shutdown functions for use in testing. See blueprint for more details
FOREIGN KEY info is now put into the table proto on CREATE TABLE, no longer just passed to the engine
SHOW CREATE TABLE now uses the table proto
Packages available for Debian and Ubuntu
Removed static instances from plugins
Continued code cleanup
Various bug fixes

The Drizzle download file can be found here

View full post on DrizzleBlog

Patrick Crews: Testing the data dictionary in a concurrent environment

Abidoon — Fri, 03 Sep 2010 18:03:15 +0000

So, Brian wrote a bug the other day, asking me to do more testing of Drizzle’s data dictionary. Specifically, we wanted to look for how things behaved in a concurrent environment as this is often a killer for table functions / what people are most likely to forget.

What we came up with was the following plan:

1) Generate a test that only looked at data dictionary tables, with several users generating the same queries.

2) If step 1 looks good, we will slowly introduce background workloads (SELECT / UPDATE / etc) while we continue with the workload from step 1.

This resulted in a couple of new randgen grammars:

data_dict_concurrent_drizzle.yy – this grammar generates nothing but queries against the data dictionary tables. At present, these are mostly of the variety:

SELECT * FROM data_dictionary_table
SHOW PROCESSLIST | VARIABLE | TABLE STATUS | etc

This is designed to stress the data dictionary, either alone or with another randgen process generating a background workload.

proclist_subquery_drizzle.yy – this grammar is the same as optimizer_subquery_drizzle (generating *nasty*, subquery-heavy SELECTs), but also allows for SHOW PROCESSLIST commands. This is mainly designed to stress the server / PROCESSLIST. This grammar is nice as it is a single test that can just be run with several threads.

I am happy to report that in a data dictionary-only environment, the server was able to handle things very well. I was running up to 100 connections, 100k queries per connection and things looked good.

The other tests are another matter. While these are somewhat simple tests, they have proven highly effective so far:

Bug #627733: Crash in InnodbTrxTool::Generator::populate_innodb_locks (this=0x7f26140046f0) at plugin/innobase/handler/data_dictionary.cc:269
Bug #627742: Assertion failed – in drizzled::plugin::TableFunction::Generator::push (this=0x23876c0, arg=, length=) at drizzled/plugin/table_function.cc:185
Bug #628398: Crash / segfault in copy_fields (join=0x284f868, end_of_records=false) at drizzled/sql_select.cc:6228
Bug #628891: Crash / assertion failed – in drizzled::Diagnostics_area::set_eof_status (this=0x7f9f3c2c4258, session=0x7f9f3c2c3b10) at drizzled/diagnostics_area.cc:120

For each of these bugs, data dictionary queries were being executed while another query was also being processed. It should be noted that our newest team member, Andrew Hutchings, had Bug#627742 fixed in less than 24 hours : )

I still have a few more scenarios to run through, but it appears that we have shaken out most of the bugs in this area. Our next steps will be to install such tests in our build and test system to prevent regressions / catch new bugs and to fix the remaining crashes noted above.

View full post on Planet Drizzle

Zookeeper experience

Abidoon — Fri, 03 Sep 2010 18:03:12 +0000

While working on Kafka, a distributed pub/sub system (more on that later) at LinkedIn, I need to use Zookeeper (ZK) to implement the load-balancing logic. I’d like to share my experience of using Zookeeper. First of all, for those of you who don’t know, Zookeeper is an Apache project that implements a consensus service based on a variant of Paxos (it’s similar to Google’s Chubby). ZK has a very simple, file system like API. One can create a path, set the value of a path, read the value of a path, delete a path, and list the children of a path. ZK does a couple of more interesting things: (a) one can register a watcher on a path and get notified when the children of a path or the value of a path is changed, (b) a path can be created as ephemeral, which means that if the client that created the path is gone, the path is automatically removed by the ZK server. However, don’t let the simple API fool you. One needs to understand a lot more than those APIs in order to use them properly. For me, this translates to weeks asking the ZK mailing list (which is pretty responsive) and our local ZK experts.

To get started, it’s important to understand the state transitions and the associated watcher events inside a ZK client. A ZK client can be in one of the 3 states, disconnected, connected, and closed. When a client is created, it’s in the disconnected state. Once a connection is established, the client is moved to the connected state. If the client loses its connection to a server, it switches back to the disconnected state. If it can’t connect to any server within some time limit, it’s eventually transitioned to the closed state. For each state transition, a state changing event (disconnected, syncconnected and expired) is sent to the client’s watcher. As you will see, those events are critical to the client. Finally, if one performs an operation on ZK when the client is in the disconnected state, a ConnectionLossException (CLE) is thrown back to the caller. More detailed information can be found at the ZK site. A lot of the subtleties when using ZK are to deal with those state changing events.

The first tricky issue is related to CLE. The problem is that when a CLE happens, the requested operation may or may not have taken place on ZK. If the connection was lost before the request reached the server, the operation didn’t take place. On the other hand, it can happen that the request did reach the server and got executed there. However, before the server can send a response back, the connection was lost. If the request is a read or an update, one can just keep retrying until the operation succeeds. It becomes a problem if the request is a create. If you simply retry, you may get a NodeExistsException and it’s not clear whether it’s you or someone else have created the path. What one can do is to set the value of the path to a client specific value during creation. If a NodeExistsException is thrown, read the value back to check who actually created it. One can’t use this approach for sequential paths (a ZK feature that creates a path with a generated sequential id) though. If you retry, a different path will be created. You also can’t check who created the path, since if you get a CLE, you don’t know the name of the path that gets created. For this reason, I think that sequential paths have very limited benefit since it’s very hard to use them correctly.

The second tricky issue is to distinguish between a disconnect and an expired event. The former happens when the ZK client can’t connect to the server. This is because either (1) the ZK server is down, or (2) the ZK server is up, but the ZK client is partitioned from the server or it is in a long GC pause and can’t send the heartbeat in time. In case (1), when the ZK server comes back, the client watcher will get a syncconnected event and everything is back to normal. Surprisingly, in this case, all the ephemeral paths and the watchers are still kept at the server and you don’t have to recreate them. In case (2), when the client finally reconnects to the server, it will get back an expired event. This implies that the server thinks the client is dead and has taken the liberty to delete all the ephemeral paths and watchers created by that client. It’s the responsibility of the client to start a new ZK session and to recreate the ephemeral paths and the watchers.

To deal with the above issues, one has to write additional code that keeps track of the ZK client state, starts a new session when the old one expires, and handles the CLE appropriately. For my application, I find the ZKClient package quite useful. ZKClient is a wrapper of the original ZK client. It maintains the current state of the ZK client, hides the CLE from the caller by retrying the request when the state is transitioned to connected again, and reconnects when necessary. ZKClient has an Apache license and has been used in Katta for quite some time. Even with the help of ZKClient, I still have to handle things like who actually created a path when a NodeExistsException occurs and re-registering after a session expires.

Finally, how do you test your ZK application, especially the various failure scenarios? One can use utilities like “ifconfig down/up” to simulate network partitioning. Todd Lipcon’s Gremlins seems very useful too.

View full post on SNA Projects Blog

Eight great virtual appliances for VMware, free for the downloading

Abidoon — Fri, 03 Sep 2010 18:03:07 +0000

Eight great virtual appliances for VMware, free for the downloading
The combination of free open source and virtual machines is hard to beat; here are some of the handiest virtual appliances you’ll find

lighttpd 1.4.28 released

Abidoon — Fri, 03 Sep 2010 18:03:05 +0000

Just in case you didn’t see the announcement on the main page: click

Please head over there for more information and comments.

If you want to get the latest source for any branch, you can get it from our svn repository.
Documentation to do so can be obtained from this page:
http://redmine.lighttpd.net/projects/lighttpd/wiki/DevelSubversion
Bug reports or feature requests can be filed in our ticket system:
http://redmine.lighttpd.net/projects/lighttpd/issues/new
Please make sure to check if there isn’t a ticket already here:
http://redmine.lighttpd.net/projects/lighttpd/issues

Thank you for flying light.

View full post on lighty’s life

New committer: Glen Barber (full doc/www)

Abidoon — Fri, 03 Sep 2010 18:03:03 +0000

View full post on FreeBSD News Flash

How long Innodb Shutdown may take

Abidoon — Fri, 03 Sep 2010 18:02:45 +0000

How long it may take MySQL with Innodb tables to shut down ? It can be quite a while.
In default configuration innodb_fast_shutdown=ON the main job Innodb has to do to complete shutdown is flushing dirty buffers. The number of dirty buffers in the buffer pool varies depending on innodb_max_dirty_pages_pct as well as workload and innodb_log_buffer_size and can be anywhere from 10 to 90% in the real life workloads. Innodb_buffer_pool_pages_dirty status will show you the actual data. Now the flush speed also depends on number of factors. First it is your storage configuration – you may be looking at less than 200 writes/sec for single entry level hard drive to tens of thousands of writes/sec for high end SSD card. Flushing can be done using multiple threads (in XtraDB and Innodb Plugin at least) so it scales well with multiple hard drives. The second important variable is your workload, especially how dirty pages would line up on the hard drive. If there are a lot of sequential pages which are dirty Innodb will be able to use larger size IOs – up to 1MB flushing dirty pages which can be a lot faster than flushing data page by page.

So if we have system with single hard drive doing 200 IO/ssc, 48G buffer pool which is 90% dirty and completely random page writes we’ll look at 13500 seconds or about 5min per 1GB of Buffer pool size.
This is worse case scenario though it is quite common in practice to see shutdown time of about 1min per GB of buffer pool per hard drive.

Baron has written a nice post how to decrease innodb shutdown time which you may want to read on this topic.

Entry posted by peter |
7 comments

Add to: | | | |

View full post on MySQL Performance Blog