Shard-Query introduces set based processing, which on the surface appears to be similar to other technologies on the market today. However, the scaling features of Shard-Query are just a side effect of the fact that it operates on sets in parallel. Any set can be operated on to any arbitrary degree of parallelism up to, and including, the cardinality of the set.
This is because:

1. It is often possible to arbitrarily transform one type of expression into a different, but compatible type for computational purposes as long as the conversion is bidirectional
2. An range operation over a set of integers or dates can be transformed into one or more discrete sub-ranges
3. Any operation on an entire set is the same as running that operation on each item individually.
4. After expanding a set operation into N discrete dimensions, it can always be collapsed back into a one dimensional set.
5. Arrays are sets

Treating a general purpose computer as an SIMD computer is possible because in a set, you can perform operations on all of the items independently. The SIMD processor simply needs to wait for all parallel operations on its input to complete. Parallelism is embarrassing and the maximum degree of parallelism is easily enforced with queue.

Today I am going to show you how to take almost any function, and treat any size cluster of Turing computers as a specialized purpose SIMD computer with respect to your function. The SQL interface to Shard-Query imposes a wait for all the workers to complete, but you can register a callback function to handle the output of each input asynchronously, if you like.

Right now I believe this only works on finite sets. I’ve decided to show how to count the number of unique words, an how many times those words appear in a document. Set based processing of course works on sets. A document is a set of words.

Before you read further, I want to tell you why I’ve decided to use the words of the Constitution of the United States of America as an example. It is my favourite document in the world. It speaks of honesty, and integrity, truth and openness. I believe in all of these things. I believe, that with cheap computation, our world can become an amazing place. Please use this technology constructively and for peaceful purposes. Love one another and let’s solve all the complex problems in the world together in peace and harmony.

The following is a somewhat naive example, since grammar will not be taken into account. I start by splitting our document into a list of words:
cat /tmp/constitution.txt | sed -e’s/ /\n/g’ > words

I am going to perform the following operations on every word in the constitution:
1) compute the md5 of every item
2) compute the md5 on the reverse of every item
3) count the total number of words
4) count the frequency of words
5) order by the frequency of words, then by the md5 of the word, then by the md5 of the reverse of the word.
6) determine the number of unique words. This is not projected, but you can infer it from the number of items in the output set.

The US Constitution is not very large. I inflated the document size significantly to over 3 million “words” by duplicating the entire set multiple times.
mysql> load data infile ‘/tmp/words’ into table words (chars);
Query OK, 6033 rows affected (0.01 sec)
Records: 6033 Deleted: 0 Skipped: 0 Warnings: 0

I blow up the size of the words table and I create words2. This is the data upon which we will operate:

create table words2 partition by hash(bucket) partitions 12 as select id % 6 bucket, chars words from words;
Query OK, 3088896 rows affected (2.41 sec)
Records: 3088896  Duplicates: 0  Warnings: 0

Here is the serial version as run by the native database interface (MySQL):

mysql> select word, md5(word), md5(reverse(word)), count(*)
from words2 group by 1,2,3 order by 3,1 desc;
...
| Legislature      | 4380f755e4150b1c11f0ae9ca1910bcb | fecd2758f3c64c8176ce60c4ff7c1cf3 |     3072 |
| consent          | 9d721d9a89406a2a6861efaae44a785f | fede6baff4c3716c37a3c60bf4051b3f |      512 |
| admit,           | d803450bb41af1f7372af6ddc8e42d14 | fee1e6f166edfccd849fe4438eb1924f |      512 |
| Affirmation.     | 9568b7e19ee3da70d3e486134add2743 | fee5d3a27ec5be41941b5689f70c5587 |      512 |
| may,             | 289cf5ceddb80bab96c92de0a918e122 | fee80b247ce32faca9de1a031119533c |     1024 |
| legislatures     | 0640c734a3d25eed18126c7db6a39523 | ff238c73fea4086c10cda4a46aeb9d9a |     1024 |
| Time             | a76d4ef5f3f6a672bbfab2865563e530 | ff38a346616fc8a4df42c7f6c95bf1cc |     2048 |
| Congress:        | 873c419d2c2139bc8bbc3cbaffcc3473 | ff592a4dac2aa93c8a0589898885fe48 |      512 |
| Charles          | 399423ff652ebb6a6701be7ec3202fc6 | ffac637b74c0f062904ab466d9bf9e01 |     1024 |
| impairing        | 1c718d732bc6f6805835f8be6ef6e43e | ffc86c559e06009a743d891ce1e4fc4f |      512 |
+------------------+----------------------------------+----------------------------------+----------+
1427 rows in set (4.94 sec)

1427 rows in set (5.00 sec)
1427 rows in set (5.03 sec)
1427 rows in set (5.00 sec)

Since the data fits in memory speed is near constant and the single threaded operation burns one CPU.

To help completely demonstrate how Shard-Query makes parallel set operations work, I’ll operate in only one dimension for the first example, just like the MySQL client. This will be a linear operation because Shard-Query has no idea how to add parallelism in this case. It is data set agnostic, operating only on sets, not relations. If it were smarter it would ask the data dictionary about partitioning.

Array
(
    [word] => Congress:
    [md5(word)] => 873c419d2c2139bc8bbc3cbaffcc3473
    [md5(reverse(word))] => ff592a4dac2aa93c8a0589898885fe48
    [count(*)] => 512
)
Array
(
    [word] => Charles
    [md5(word)] => 399423ff652ebb6a6701be7ec3202fc6
    [md5(reverse(word))] => ffac637b74c0f062904ab466d9bf9e01
    [count(*)] => 1024
)
Array
(
    [word] => impairing
    [md5(word)] => 1c718d732bc6f6805835f8be6ef6e43e
    [md5(reverse(word))] => ffc86c559e06009a743d891ce1e4fc4f
    [count(*)] => 512
)
1427 rows returned (5.0057470798492s, 4.9994130134583s, 0.0063340663909912s)

The set of three numbers are wall clock time (as calculated by microtime()), SQL execution time, and parse time, respectively.

Actually performance is a little worse. This is not unexpected. Since Shard-Query must add at small amount of overhead, a single threaded operation may be slower than the same operation on the native database.

That doesn’t matter because Shard-Query is a smart database proxy that can add parallelism. In this mode it will add additional six degrees of parallelism the query:

Array
(
    [word] => Congress:
    [md5(word)] => 873c419d2c2139bc8bbc3cbaffcc3473
    [md5(reverse(word))] => ff592a4dac2aa93c8a0589898885fe48
    [count(*)] => 342
)
Array
(
    [word] => Charles
    [md5(word)] => 399423ff652ebb6a6701be7ec3202fc6
    [md5(reverse(word))] => ffac637b74c0f062904ab466d9bf9e01
    [count(*)] => 598
)
Array
(
    [word] => impairing
    [md5(word)] => 1c718d732bc6f6805835f8be6ef6e43e
    [md5(reverse(word))] => ffc86c559e06009a743d891ce1e4fc4f
    [count(*)] => 512
)
1427 rows returned (0.87930011749268s, 0.87229418754578s, 0.0070059299468994s)

Why six degrees of parallelism? Because that is how many physical cores are connected to my bus, and because I chose to create six hash “buckets” in the table. This allows MySQL to set up a sequential scan over the items in this bucket, particularly since we are examining all the items. We operate on all the buckets and then use intelligent expression substitution to put the results back together, when necessary. When sorting or grouping are used, a final pass over the final result may be necessary, and this may add a small amount of serialization at the end.

How does this work?

Here is the most important part of the explain plan in the mode without parallelism. Notice that there is only one query. If your database system can not provide native parallelism, then performance will be poor.

-- SQL TO SEND TO SHARDS:
Array
(
    [0] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` GROUP BY 1,2,3 ORDER BY NULL
)

The other important optimization combines results from multiple queries together. This query is single threaded, and thus this serves no purpose. It will be much more important in a moment.

-- AGGREGATION SQL:
SELECT `word`,`md5(word)`,SUM(`count(*)`) AS `count(*)`
FROM `aggregation_tmp_39323566`  GROUP BY 1,2 ORDER BY 1 ASC
ON DUPLICATE KEY UPDATE
`word`=VALUES(`word`),
`md5(word)`=VALUES(`md5(word)`),
`count(*)`=`count(*)` +  VALUES(`count(*)`)

Now, consider the query with BETWEEN 1 and 6 added to the where clause. This creates boundary conditions for our query. Any set of integers can be broken up into as many items are the set contains, and thus it is possible to convert the BETWEEN expression into a partition elimination expression.

Here is the output from the parallel version:

-- SQL TO SEND TO SHARDS:
Array
(
    [0] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 1   GROUP BY 1,2,3 ORDER BY NULL
    [1] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 2   GROUP BY 1,2,3 ORDER BY NULL
    [2] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 3   GROUP BY 1,2,3 ORDER BY NULL
    [3] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 4   GROUP BY 1,2,3 ORDER BY NULL
    [4] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 5   GROUP BY 1,2,3 ORDER BY NULL
    [5] => SELECT word AS `word`,md5(word) AS `md5(word)`,md5(reverse(word)) AS `md5(reverse(word))`,COUNT(*) AS `count(*)`
FROM words2 AS `words2` WHERE bucket  = 6   GROUP BY 1,2,3 ORDER BY NULL
)

This powers the UPSERT. The results from the six branches are combined with this.

-- AGGREGATION SQL:
SELECT `word`,`md5(word)`,SUM(`count(*)`) AS `count(*)`
FROM `aggregation_tmp_27656998`  GROUP BY 1,2 ORDER BY 1 ASC
ON DUPLICATE KEY UPDATE
`word`=VALUES(`word`),
`md5(word)`=VALUES(`md5(word)`),
`count(*)`=`count(*)` +  VALUES(`count(*)`)

All six branches compute fully in parallel.

Set processing also allows us to capture and effectively aggregate and manage every single change in any database. The entire collection of human intelligence can be stored an efficiently reused over and over again, conserving resources now lost through wasted re-computation. I will be sharing more information about this in the future.

Finally, I would like to note that this set has a cardinality of 3088896. This is the maximum theoretical degree of parallelism that this data set can achieve with my method. Likely current network technologies can not support such a degree.

mysql> select min(id),max(id) from words;
+---------+---------+
| min(id) | max(id) |
+---------+---------+
|       1 | 3088896 |
+---------+---------+
1 row in set (0.00 sec)

View full post on MySQL Performance Blog

The error messages generally occur due to problems with system data structures, system files, and other resources, which make your system and data inaccessible. This behavior leads to significant data loss from your Linux hard drive and requires Data Recovery Linux to be sorted out.

In a practical scenario, you may come across the below error message while working on your Linux machine:

“Make Says, Error 139″

After the above error, Linux system does not work properly and shuts down. The problem occurs every time when you boot your system or perform a specific set of operations. This error renders your system inaccessible and causes serious data loss. In order to get your business-critical data recovered, identify the cause of this problem and perform Linux Data Recovery by resolving it.

Root of the problem

This error occurs when your compiler dumps the core. You have a damaged, old, or buggy version of the GCC (GNU Compiler Collection). The compiler corruption may occur due to numerous reasons, such as unexpected system shutdown, operating system malfunction, virus infection, file system corruption, and more.

You may also face this situations due to problems with the disk or memory corruption. The disk corruption is resulted when metadata structure of the hard drive become faulty. In such cases, operating system can not locate and access data from the hard drive.

Resolution

Go through any of the below steps to sort out this issue:

•    Install latest release of Linux compiler or EGCS (Experimental/Enhanced GNU Compiler System).

•    Check wait states, clock rate, and the refresh timing for SIMMS and cache. Use fsck command-line utility to scan the disk and fix errors.

•    Use powerful and advanced third-party Linux Recovery software to recover lost, missing, deleted, and inaccessible data from Linux hard drive.

Linux Data Recovery tools employ high-end scanning techniques to methodically scan whole hard drive and extract all data from it. With simple user interface and read-only behavior, they perform easy and safe recovery in all cases of data loss.

Stellar Phoenix Linux Data Recovery software recovers lost data from Linux hard drive, USB drive, and network mapped drive. The software works well with all major distributions of Linux operating system, including Red Hat, SUSE, Debian, Ubuntu, and Mandriva. It supports recovery from Ext4, Ext3, Ext2, FAT32, FAT16, and FAT12 file system volumes.

Actually there is a simple and logical answer to this question. When the Windows was launched some 20 plus years ago, it caught like wild fire because it was all of a sudden so easy to use and so convenient to install. The best things that it maintained a constant identity – as the product of Windows, while on the other hand Linux went along under many names, such as Knoppix, Mandrake, SuSe, Windows, and so on. With each name (different companies) Linux as an operating system had slight variations. Hence, it was bothersome for people to keep track, and sometime use Linux due to this aspect.

Then comes the inherent difference between these operation systems. Technically speaking, you will find the GUI is optional in the Linux operations system while it is part and parcel of the Windows. The separation of the GUI directly affects the reliability and speed – not to mention efficiency – of a server and hence here Linux scores over the Windows.

Another major difference is in the way these operating systems handle their command prompts. While Windows uses similar command interpreters for its Windows 9x versions, the NT series have a different style but common to each other. Linux on the other hand, being a UNIX version has the capability of handling multiple command interpreters, which can be a boon, though it mostly favors Bourne Again Shell (BASH). The other interpreters that you will find in Linux are C Shell, ash, Korn shell and so on.

The best and most highlighted difference is the price tag attached to the operating systems. Windows is an expensive to say the least, operating system which comes with a copyrighted license, while Linux is free for all, easily downloadable anytime you want it. The downside here is that Linux has instruction which will tell you what to do – which may be why people are indeed reluctant to move away from Windows; though there are a few low cost Linux versions which come with automation and manuals.

Another plus in favor of Linux is the lack of any security requirement. The viruses, spy wares, malawares and what not you have it are all a product of Microsoft, for Windows. Hence, while you have to use high security firewalls with Windows, you are free from any such headaches with Linux.

Lastly, you have the difference of availability of software. The majority of the software that you find on the market are for Windows and they will not run on Linux unless Windows is somewhere configured as a subsystem which actually goes against the reason of using Linux in the first place.

Hence, till a suitable interface is found where Linux can be compatible with Windows without using Windows itself, have a few clear-cut instructions for first time users and sufficient information on its technicalities, people will prefer Windows to Linux, even if they have to pay to use it.

Read more on Courier-Post

I. Slick and go: You can install it on your hard disk and completely erase the current disk, which will wipe out your Windows install. This isn’t recommended unless you are installing Linux on an older computer that isn’t critical to your business. Or if you are really ready for a change and are into new and exciting things…

II. Dual-Boot: Another way is to install Linux alongside your current Windows install and then decide which one you want to use at bootup. This is a popular route because you can always go back to Windows if you need something. This is recommended because there will be growing pains when you start to use Linux. Linux can do anything that Windows can do, but it is not Windows. Once you get used to Linux, you’ll never go back, but sometimes getting to that point is difficult. Dual-booting can ease the transition and give the new Linux user a safety net.

III. Boot from USB flash drive: You can alternatively install Linux on a USB stick and eliminate the risk of hurting your current Windows setup. This is a nice option because it has all of the advantages to the Dual-Boot option, and you can carry your Linux distro with you whereever you go. Then any computer at work or home or your friend’s house is your Linux box.

IV. New HD: Another “safe” way to get into Linux is to either buy a new harddrive, or grab a used one out of your spare computer parts drawer and install Linux on the new HD. (You do have a spare computer parts drawer, right?) Take the old drive out, carefully store it somewhere safe, put the “new” drive in and install Linux. If at any point you want to go back to your Windows system, then just swap out drives. This is a nice way to go because it is safe, and it is easy.

V. Super Easy: The final way to install Linux is the super easy method. Simply use the Wubi installer. Wubi is a Windows program that makes installing Linux very easy. Just download Wubi and it will install Linux on your Windows computer with a few easy clicks. For more about Wubi, go here: http://en.wikipedia.org/wiki/Wubi_ (Ubuntu_installer).

The great thing about Wubi is that a Windows user can try out Linux without doing a complex install involving partitioning or formatting. Wubi is an easy way to get Linux going but there are a few limitations. Hibernation is not supported. If Windows crashes and the hard drive is unmounted uncleanly, there can be issues. The system may be a little slower compared to normal install methods. But for someone who wants to give Linux a test drive, this is a great way to go.