View full post on PHP Classes: Latest entries

Kafka
Neha Narkhede (LinkedIn)
Wednesday, July 27, 2011

ABSTRACT

Kafka is a distributed publish-subscribe messaging system aimed at providing a scalable, high-throughput, low latency solution for log aggregation and activity stream processing for LinkedIn. Built on Apache Zookeeper in Scala, Kafka aims at providing a unified stream for both real-time and offline consumption. We provide a mechanism for parallel data load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines. Kafka combines the benefits of traditional log aggregators and messaging systems and has been used successfully in production for 8 months. It provides API similar to that of a messaging system and allows applications to consume log events in real-time.

Written by the SNA team at LinkedIn, Kafka is open sourced under the Apache 2.0 License and is an Apache incubator project. In this presentation, we will highlight the core design principles for this system, and how this system fits into LinkedIn’s data ecosystem as well as some of the products and monitoring applications it supports in our usage.

BIOGRAPHY

Neha Narkhede is a Senior Software Engineer in the Search, Network and Analytics Team at LinkedIn, focusing on Distributed Systems. She is one of the initial contributors to Project Kafka. In the past she’s worked on search systems in large scale databases and has been an active contributor to several projects LinkedIn has open sourced, including Voldemort, Bobo and Zoie.

View full post on SNA Projects Blog

In all seriousness, it’s kind of neat that we’re using dbqp to run some of our staging tests and we gain a few neat things:

Speed

Here are the trend charts for randgen and crashme.  While it doesn’t look like randgen is showing much of an improvement, it is worth mentioning that this job now runs both the standard and the transaction log tests in a single run >: )  Previously, we had a separate drizzle-automation job for the transaction log.  Just the trx_log tests took ~30 minutes to run (plus build time).  Long story short, we’re saving about 30-40 minutes on randgen testing per staging run and only needing to build once!

Maintainability

The jobs we run are in the tree and anyone can easily repeat them.  While Drizzle-automation kicks major butt (and I have taken many ideas from it), it is a separate piece of software that requires setup and maintenance.  Basing things around an in-tree setup means that you only need the code and any required bits and pieces.  Now if we need to set up a new randgen machine, we only need the randgen and dbd::drizzle installed (and we plan on including randgen in-tree soon, so you won’t even need that!).  If we need to set up a new crash-me machine, we only need dbd::drizzle – and everyone should have dbd::drizzle installed! ; )

Ease of use

Pretty much all tests provide the same standard output:

dtr mode

From the command:

./dbqp

Our default mode is dtr (aka using drizzletest.cc to execute standard .test files). To run all available tests, use the make target – make test-dbqp

20110621-081404  trigger_dictionary.loaded                  [ pass ]       43
20110621-081408  logging_stats.cumulative                   [ pass ]     1045
20110621-081412  errmsg_stderr.stderr                       [ pass ]       36
20110621-081412  ===============================================================
20110621-081412 INFO Test execution complete in 496 seconds
20110621-081412 INFO Summary report:
20110621-081412 INFO Executed 566/566 test cases, 100.00 percent
20110621-081412 INFO STATUS: PASS, 566/566 test cases, 100.00 percent executed
20110621-081412 INFO Spent 254 / 496 seconds on: TEST(s)
20110621-081412 INFO Test execution complete
20110621-081412 INFO Stopping all running servers...


randgen mode

From the command:

./dbqp --mode=randgen --randgen-path=/path/to/your/randgen


20110621-170141  main.subquery                              [ pass ]     3780
20110621-170148  main.subquery_semijoin                     [ pass ]     3016
20110621-170156  main.subquery_semijoin_nested              [ pass ]     3750
20110621-170202  main.varchar                               [ pass ]     2658
20110621-170202  ===============================================================
20110621-170202 INFO Test execution complete in 147 seconds
20110621-170202 INFO Summary report:
20110621-170202 INFO Executed 19/19 test cases, 100.00 percent
20110621-170202 INFO STATUS: PASS, 19/19 test cases, 100.00 percent executed
20110621-170202 INFO Spent 77 / 147 seconds on: TEST(s)
20110621-170202 INFO Test execution complete
20110621-170202 INFO Stopping all running servers...


crashme mode

From the command:

./dbqp --mode=crashme


20110621-181515  main.crashme                               [ fail ]   149840
20110621-181515  func_extra_to_days=error        # Function TO_DAYS
20110621-181515  ###
20110621-181515  ###<select to_days('1996-01-01') from crash_me_d
20110621-181515  ###>2450084
20110621-181515  ###We expected '729024' but got '2450084'
20110621-181515  func_odbc_timestampadd=error        # Function TIMESTAMPADD
20110621-181515  ###
20110621-181515  ###<select timestampadd(SQL_TSI_SECOND,1,'1997-01-01 00:00:00')
20110621-181515  ###>1997-01-01 00:00:01.000000
20110621-181515  ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
20110621-181515  ###
20110621-181515  ###<select {fn timestampadd(SQL_TSI_SECOND,1,{ts '1997-01-01 00:00:00'}) }
20110621-181515  ###>1997-01-01 00:00:01.000000
20110621-181515  ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
20110621-181515
20110621-181515 ERROR Failed test.  Use --force to execute beyond the first test failure
20110621-181515  ===============================================================
20110621-181515 INFO Test execution complete in 153 seconds
20110621-181515 INFO Summary report:
20110621-181515 INFO Executed 1/1 test cases, 100.00 percent
20110621-181515 INFO STATUS: FAIL, 1/1 test cases, 100.00 percent executed
20110621-181515 INFO FAIL tests: main.crashme
20110621-181515 INFO Spent 149 / 153 seconds on: TEST(s)
20110621-181515 INFO Test execution complete
20110621-181515 INFO Stopping all running servers...


While this isn’t a huge feature, it is nice to have a standardized report for knowing if something failed, what failed and how (we always dump test tool output on test failures).  Why is this nice?  Well, the world is a busy place and only needing to know one way of reading test output simplifies things just a teensy little bit.  This small improvement becomes a huge benefit over time if you happen to spend good chunks of your day looking at test output like me : )

Other than that, I’m still working on teaching dbqp interesting new tricks that will help me in testing SkySQL‘s Reference Architecture – expect to hear more about that next month!

View full post on Planet Drizzle

View full post on PHP Classes: Latest entries

View full post on PHP Classes: Latest entries

Ideas and decisions:

We want to ship the frontend with the backend code as one package with no extra setup, and minimal code (no “nonsense xml” servlet configurations).
We chose Scalatra, Scalatra is to scala what sinatra is for Ruby, it allows you to define the resources with very minimal code, for example:
get("/salute") {
"Hello World"
}


It also offers powerful templating features through Scalate, which allows you to render JSP, HAML, Mustache, etc.

Realtime updating on the page will require some Ajax and DOM manipulation.
Dealing with browser inconsistencies is a pain, JQuery makes it a breeze and also offers powerful ajax functions, it is also used as the glue for other libraries like backbone.js and mustache.js.

Javascript code tends to get messy and big over time, let’s use something that allows the frontend to be a good starting point and scale the code over time.
Backbone.js is a MVC pattern implementation for javascript apps, very minimal, from all the options out there Backbone seems to be the cleanest dont-try-to-do-everything option. While for this frontend is still a bit of an overkilll, it offers a good starting point for a larger project. Also enforces writing code in a specific style wich makes it easy to know where all the pieces are put together.

The backbone layer holds a model (Tweet), a collection of tweets (TweetList), and two views, one for the visual representation of a Tweet model (TweetView), and one for the left sidebar that contains the search box and list of results (AppView).

Do take a look at the commented code for the app.js and the index.ssp file to learn more about how everything works in more detail.

Some UI niceties like good looking buttons, realtime update of timestamps, and animation.
A nice touch would be to refresh the time associated with a tweet, for that we used a JQuery plugin called EasyDate. For the buttons and layout we used CSS3 properties that degrade fine to older browsers (and IE). For the images we used one sprited png file.

And that’s it!, it’s a very simple page, that in order to build fast we leveraged lots of great opensource software that is out there, check out the code and give it a try locally. It’s really simple.

View full post on SNA Projects Blog

A challenge we gave ourselves was to see how quickly we can build a scalable Tweet search system. Some requirements:

• The entire stack must be scalable, e.g. by adding  commodity hardware, it should theoretically handle all tweets with real internet traffic.
• The part of the system should be able to scale differently, e.g. there is no reason the messaging system needs to scale the same way as the storage system.
• Fault tolerant – The system should be able to tolerate system failures as well as data corruption.
• Only 1 day to build it.

We scribbled together the following architecture:

Every component here is an open-source project we have deployed into our production environment with its own scaling characteristics.

Some design considerations and implementation details:

TweetStreamer is a twitter streamer using the twitter API written in Scala. We chose to use it because it is light-weight, simple and frankly, does exactly what we wanted. It can be easily configured to handle different types of twitter feeds.

We now have a data feed, one obvious thing to do is to store it. Voldemort seemed to be an obvious choice. Key = twitter id, value = tweet status (json string).

We also send the feed to a Kafka topic to be consumed by down-stream services, e.g. Sensei (our search system). You can find the code here.

The Sensei/search integration is more work: (code here), We need to setup of the following pieces:

• How we would parse the query.
• How to consume indexing events: e.g. extract from a tweet json string pieces of data we want to index. The search node consumes from Kafka, and builds an index-able object from a json string. In this case, we are only interested in 3 pieces of data: unique tweet id, the actual tweet text and the time (to be able to sort on): see (here)
• Index data retention policy as we accumulate more and more tweets. For this case, we configured index retention to be 7 days, with the index rolling forward nightly. Here is a wiki on how this works under-the-hood.

The result set contains an array of search hits, each hit containing the tweet id, relevance score and the time. We then for each tweet id, get the original tweet status object from Voldemort and build the result json object. This is done in a Scalatra servlet: see code here.

This pretty much describes the system. Let’s understand some design decisions that were made:

• One design choice was letting the process that writes to Voldemort also be a Kafka consumer. Although this would be cleaner, we would risk a data-race where search may return hit array before they are yet added to Voldemort. By making sure it is first added to Voldemort, we can rely on it being an authoritative storage for our tweets.
• You may have already realized Kafka is acting as a proxy for twitter stream, and we could have also streamed tweets directly into the search systems, bypassing the Kafka layer. What we would be missing is the ability to play back tweet events from a specific check-point. One really nice feature about Kafka is that you can keep a consumption point to have data replayed. This makes reindexing for cases such as data corruption and schema changes, etc., possible. Furthermore, to scale search, we would have a growing number of search nodes consume from the same Kafka stream. Kafka is written in a way where adding consumers does not affect through-put of the system really helps in scaling the entire system.
• Another important design decision was on using Voldemort for storage. One solution would be instead store tweets in the search index, e.g. Lucene stored fields. The benefits with this approach would be stronger consistency between search and store, and also the stored data would follow the retention policy of that’s defined by the search system. However, other than the fact that Lucene stored field is no-where near as optimal comparing to a Voldemort cluster (an implementation issue), there are more convincing reasons:
  • We can first see the consistency benefit for having search and store be together is negligible. Actually, if we follow our assumption of tweets being append-only and we always write to Voldemort first, we really wouldn’t have consistency issues. Yet, having data storage reside on the same search system would disproportionally introduce contention for IO bandwidth and OS cache, as data volume increases, search performance can be negatively impacted.
  • The point about retention is rather valid. As search index guarantees older tweets to be expired, Voldemort store would continue to grow. Our decision ultimately came down to two points: 1) Voldemort’s growth factor is very different, e.g. adding new records into the system is much cheaper, so it is feasible to have a much longer data retention policy. 2) Having have cluster of tweet storage allows us to integrate with other systems if desired for analytics, display etc.

We tossed up what we built on Github (Chirper Project). As you can see, the amount of code is rather minimal, and yes, we were able to build it in one day.

Here is a screenshot:

chirper screenshot

As you can see, the UI is rather fancy. Read on for part 2 of 2 of this post where Alejandro will detail out the UI portion of this application.

View full post on SNA Projects Blog

View full post on PHP Classes: Latest entries

Mirage: A New Multi-Scale Operating System for Clouds and Crowds
Anil Madhavapeddy (University of Cambridge)
Friday, October 22, 2010

ABSTRACT

Applications run on all kinds of environments these days: multicore desktops, virtual cloud infrastructures, smart-phones, and web browsers. These diverse environments make it worth rethinking the long-term future of our software stacks; do we really want to continue bundling gigabytes of general-purpose OS software with every single cloud image? Is there any point holding onto decades-old interfaces such as POSIX any more?

I will introduce Mirage, a new operating system built in the statically type-safe OCaml functional language. Mirage compiles high-level functional source code directly into a variety of targets such as:
(i) small microkernels that run directly on the “bare-metal” Xen hypervisor;
(ii) Javascript for web browsers; or
(iii) embedded ARM devices;
and of course normal operating systems such as Linux.

Mirage provides a consistent, simple programming API across all of these diverse backends, which makes it a powerful foundation for constructing safe, complex distributed systems across a heterogeneous set of modern compute resources such as mobile devices or cloud computing infrastructure. Also, it’s just plain fun programming in OCaml.

BIOGRAPHY

Dr. Anil Madhavapeddy is a Senior Research Fellow at Wolfson College Cambridge based in the Systems Research Group, investigating programming models for cloud computing. Dr. Madhavapeddy was on the original team at Cambridge that developed the Xen hypervisor, and subsequently served as the senior architect and product director for XenSource/Citrix before returning to academia.

Prior to obtaining his PhD in 2006 from the University of Cambridge, Dr. Madhavapeddy had a diverse background in industry at Network Appliance, NASA, and Internet Vision. In addition to professional and academic activities, Dr. Madhavapeddy is an active member of the open source development community with the OpenBSD operating system, a member of the steering committee for Commercial Uses of Functional Programming, and on the boards of startup companies such as Limbe Labs and Ashima Arts.

View full post on SNA Projects Blog

Read more on ZDNet UK