You could use this in a Vagrant setup if you like (I’ve done so for testing).

Step 1) Set the following in your Vagrantfile:

Vagrant::Config.run do |config|
  config.vm.box = "lucid32"
  config.vm.box_url = "http://files.vagrantup.com/lucid32.box"
  config.vm.provision :puppet
end

Step 2) Get puppet-apt helper.

I used https://github.com/evolvingweb/puppet-apt and put it in a manifests/ directory like so:

$ mkdir manifests
$ cd manifests
$ git clone git://github.com/evolvingweb/puppet-apt.git

Step 3) Write your puppet manifest:

import "puppet-apt/manifests/init.pp"
import "puppet-apt/manifests/ppa.pp"

class drizzlebuild {
        apt::ppa { "ppa:drizzle-developers/ppa": }
        package { "drizzle-dev":
                  ensure => latest,
        }
}

include drizzlebuild

Step 4) “vagrant  up” and you’re done! Feel free to build Drizzle inside this VM.

I’m sure there may be some more proper way to do it all, but that was a pretty neat first intro to me to Puppet and friends

View full post on Planet Drizzle

Ideas and decisions:

We want to ship the frontend with the backend code as one package with no extra setup, and minimal code (no “nonsense xml” servlet configurations).
We chose Scalatra, Scalatra is to scala what sinatra is for Ruby, it allows you to define the resources with very minimal code, for example:
get("/salute") {
"Hello World"
}


It also offers powerful templating features through Scalate, which allows you to render JSP, HAML, Mustache, etc.

Realtime updating on the page will require some Ajax and DOM manipulation.
Dealing with browser inconsistencies is a pain, JQuery makes it a breeze and also offers powerful ajax functions, it is also used as the glue for other libraries like backbone.js and mustache.js.

Javascript code tends to get messy and big over time, let’s use something that allows the frontend to be a good starting point and scale the code over time.
Backbone.js is a MVC pattern implementation for javascript apps, very minimal, from all the options out there Backbone seems to be the cleanest dont-try-to-do-everything option. While for this frontend is still a bit of an overkilll, it offers a good starting point for a larger project. Also enforces writing code in a specific style wich makes it easy to know where all the pieces are put together.

The backbone layer holds a model (Tweet), a collection of tweets (TweetList), and two views, one for the visual representation of a Tweet model (TweetView), and one for the left sidebar that contains the search box and list of results (AppView).

Do take a look at the commented code for the app.js and the index.ssp file to learn more about how everything works in more detail.

Some UI niceties like good looking buttons, realtime update of timestamps, and animation.
A nice touch would be to refresh the time associated with a tweet, for that we used a JQuery plugin called EasyDate. For the buttons and layout we used CSS3 properties that degrade fine to older browsers (and IE). For the images we used one sprited png file.

And that’s it!, it’s a very simple page, that in order to build fast we leveraged lots of great opensource software that is out there, check out the code and give it a try locally. It’s really simple.

View full post on SNA Projects Blog

A challenge we gave ourselves was to see how quickly we can build a scalable Tweet search system. Some requirements:

• The entire stack must be scalable, e.g. by adding  commodity hardware, it should theoretically handle all tweets with real internet traffic.
• The part of the system should be able to scale differently, e.g. there is no reason the messaging system needs to scale the same way as the storage system.
• Fault tolerant – The system should be able to tolerate system failures as well as data corruption.
• Only 1 day to build it.

We scribbled together the following architecture:

Every component here is an open-source project we have deployed into our production environment with its own scaling characteristics.

Some design considerations and implementation details:

TweetStreamer is a twitter streamer using the twitter API written in Scala. We chose to use it because it is light-weight, simple and frankly, does exactly what we wanted. It can be easily configured to handle different types of twitter feeds.

We now have a data feed, one obvious thing to do is to store it. Voldemort seemed to be an obvious choice. Key = twitter id, value = tweet status (json string).

We also send the feed to a Kafka topic to be consumed by down-stream services, e.g. Sensei (our search system). You can find the code here.

The Sensei/search integration is more work: (code here), We need to setup of the following pieces:

• How we would parse the query.
• How to consume indexing events: e.g. extract from a tweet json string pieces of data we want to index. The search node consumes from Kafka, and builds an index-able object from a json string. In this case, we are only interested in 3 pieces of data: unique tweet id, the actual tweet text and the time (to be able to sort on): see (here)
• Index data retention policy as we accumulate more and more tweets. For this case, we configured index retention to be 7 days, with the index rolling forward nightly. Here is a wiki on how this works under-the-hood.

The result set contains an array of search hits, each hit containing the tweet id, relevance score and the time. We then for each tweet id, get the original tweet status object from Voldemort and build the result json object. This is done in a Scalatra servlet: see code here.

This pretty much describes the system. Let’s understand some design decisions that were made:

• One design choice was letting the process that writes to Voldemort also be a Kafka consumer. Although this would be cleaner, we would risk a data-race where search may return hit array before they are yet added to Voldemort. By making sure it is first added to Voldemort, we can rely on it being an authoritative storage for our tweets.
• You may have already realized Kafka is acting as a proxy for twitter stream, and we could have also streamed tweets directly into the search systems, bypassing the Kafka layer. What we would be missing is the ability to play back tweet events from a specific check-point. One really nice feature about Kafka is that you can keep a consumption point to have data replayed. This makes reindexing for cases such as data corruption and schema changes, etc., possible. Furthermore, to scale search, we would have a growing number of search nodes consume from the same Kafka stream. Kafka is written in a way where adding consumers does not affect through-put of the system really helps in scaling the entire system.
• Another important design decision was on using Voldemort for storage. One solution would be instead store tweets in the search index, e.g. Lucene stored fields. The benefits with this approach would be stronger consistency between search and store, and also the stored data would follow the retention policy of that’s defined by the search system. However, other than the fact that Lucene stored field is no-where near as optimal comparing to a Voldemort cluster (an implementation issue), there are more convincing reasons:
  • We can first see the consistency benefit for having search and store be together is negligible. Actually, if we follow our assumption of tweets being append-only and we always write to Voldemort first, we really wouldn’t have consistency issues. Yet, having data storage reside on the same search system would disproportionally introduce contention for IO bandwidth and OS cache, as data volume increases, search performance can be negatively impacted.
  • The point about retention is rather valid. As search index guarantees older tweets to be expired, Voldemort store would continue to grow. Our decision ultimately came down to two points: 1) Voldemort’s growth factor is very different, e.g. adding new records into the system is much cheaper, so it is feasible to have a much longer data retention policy. 2) Having have cluster of tweet storage allows us to integrate with other systems if desired for analytics, display etc.

We tossed up what we built on Github (Chirper Project). As you can see, the amount of code is rather minimal, and yes, we were able to build it in one day.

Here is a screenshot:

chirper screenshot

As you can see, the UI is rather fancy. Read on for part 2 of 2 of this post where Alejandro will detail out the UI portion of this application.

View full post on SNA Projects Blog

This is our Drizzle7 beta release.  Please test away!

In this release:

• kernel options uses boost program_options, replaced my_getopt
• csv, archive, and blackhole tables are not enabled by default, need –plugin-add <name> to use them
• significant work on the transaction log – testing + bug fixes.  This provides the infrastructure for replication.
• drizzledump can now be used to migrate MySQL->Drizzle directly (no intermediate dump files)
• session/memory uses boost specific
• thread system based on boost
• improvements in configuration file processing
• merge of latest from PBXT 1.0.11-7 + new PBXT tests and bug fixes
• introduction of session performance table (not built by default)
• initial work on catalog support
• introduction of Sphinx-based documentation within the source tree
• stress testing of the data_dictionary tables (high concurrency scenarios) via the randgen
• timestamps are now 64 bit and ISO compliant
• 4 byte enum – able to handle very large enum operations
• dates are now 4 bytes – allows us to support a full range of values
• IO cleanup
• 40 bug fixes

The Drizzle download file can be found here

View full post on Planet Drizzle

This is our Drizzle7 beta release.  Please test away!

In this release:

• kernel options uses boost program_options, replaced my_getopt
• csv, archive, and blackhole tables are not enabled by default, need –plugin-add <name> to use them
• significant work on the transaction log – testing + bug fixes.  This provides the infrastructure for replication.
• drizzledump can now be used to migrate MySQL->Drizzle directly (no intermediate dump files)
• session/memory uses boost specific
• thread system based on boost
• improvements in configuration file processing
• merge of latest from PBXT 1.0.11-7 + new PBXT tests and bug fixes
• introduction of session performance table (not built by default)
• initial work on catalog support
• introduction of Sphinx-based documentation within the source tree
• stress testing of the data_dictionary tables (high concurrency scenarios) via the randgen
• timestamps are now 64 bit and ISO compliant
• 4 byte enum – able to handle very large enum operations
• dates are now 4 bytes – allows us to support a full range of values
• IO cleanup
• 40 bug fixes

View full post on DrizzleBlog

In this release:

• Using boost for locks in table_share
• Initial work on moving to Sphinx-based documentation
• random_number, random_string and counter tables added to data_dictionary
• libdrizzle has been moved into the drizzle tree
• variable defaults no longer shown automatically via drizzled –help.  To view variable values, it is suggested to use SHOW VARIABLES after server startup.  Variable default values will also be available via documentation, once it is ready.
• All engine-specific tests are now included in their respective plugin test directories (test suites)
• Continued work on command line options
• Continued code refactoring
• Various bug fixes

The Drizzle download file can be found here

View full post on DrizzleBlog

In this release:

• Using boost for locks in table_share
• Initial work on moving to Sphinx-based documentation
• random_number, random_string and counter tables added to data_dictionary
• libdrizzle has been moved into the drizzle tree
• variable defaults no longer shown automatically via drizzled –help.  To view variable values, it is suggested to use SHOW VARIABLES after server startup.  Variable default values will also be available via documentation, once it is ready.
• All engine-specific tests are now included in their respective plugin test directories (test suites)
• Continued work on command line options
• Continued code refactoring
• Various bug fixes

The Drizzle download file can be found here

View full post on Planet Drizzle

The Drizzle download file can be found here

View full post on Planet Drizzle

In this release:

• *Continued work on logging statistics – added a table to the scoreboard that gives memory usage and size
• print stack trace and stack dump have been removed. As we use gcc for a majority of our builds, we are now using the built-in backtrace ability. Added crash and shutdown functions for use in testing. See blueprint for more details
• FOREIGN KEY info is now put into the table proto on CREATE TABLE, no longer just passed to the engine
• SHOW CREATE TABLE now uses the table proto
• Packages available for Debian and Ubuntu
• Removed static instances from plugins
• Continued code cleanup
• Various bug fixes

The Drizzle download file can be found here

View full post on Planet Drizzle

In this release:

• *Continued work on logging statistics – added a table to the scoreboard that gives memory usage and size
• print stack trace and stack dump have been removed. As we use gcc for a majority of our builds, we are now using the built-in backtrace ability. Added crash and shutdown functions for use in testing. See blueprint for more details
• FOREIGN KEY info is now put into the table proto on CREATE TABLE, no longer just passed to the engine
• SHOW CREATE TABLE now uses the table proto
• Packages available for Debian and Ubuntu
• Removed static instances from plugins
• Continued code cleanup
• Various bug fixes

The Drizzle download file can be found here

View full post on DrizzleBlog