Using Mappers to Read and Partition Large amounts of Data from Kafka into Hadoop

Posted by Xuyen On on April 8, 2014 in Big Data

In my previous posts, I outlined how to import data into Hive tables using Hive scripts and dynamic partitioning. However, we’ve found that this only works for small batch sizes and it is not scalable for larger jobs. Instead, we found that it is faster and more efficient to partition the data as they are… Read more

DNA and the Masses: The Science and Technology Behind Discovering Who You Really Are

Posted by Melissa Garrett on March 12, 2014 in Analytics, Big Data, DNA, Science

Originally published on Wired Innovation Insights, 3-12-14. There is a growing interest among mainstream consumers to learn more about who they are and where they came from. The good news is that DNA tests are no longer reserved for large medical research teams or plot lines in CSI. Now, the popularity of direct-to-consumer (DTC) DNA tests… Read more

Ancestry.com to Lead Core Conversation at SXSW

Posted by Melissa Garrett on March 6, 2014 in Big Data, Technology Conferences

Headed to SXSW Interactive? Join EVP of Product, Eric Shoup and Senior Director of Product at Tableau, Francois Ajenstat, for an engaging Core Conversation about how using big data can tell personalized stories. Big Data is a game changer for storytelling. Too often, the data we pull is cold, factual and dehumanized. Technologies can now… Read more

Inferring Familiar Relationships From Historical Data Features (Part 2)

Posted by Laryn Brown on February 28, 2014 in Big Data

In my previous post, I outlined some of the problems and strategies we use at Ancestry.com to determine if two people who appear in the same household are related. As promised, I want to focus this time on how to resolve ambiguous results. In my early days of doing family history research, I made an… Read more

Video Q&A with Lead Engineer at Ancestry.com

Posted by Melissa Garrett on February 21, 2014 in Big Data, CSS/HTML/JavaScript, DNA, Technology Conferences

Jeremy Pollack, a lead engineer at Ancestry.com, answers questions on the technical backend of AncestryDNA in a video interview with InfoQ. The interview took place after his presentation with Bill Yetman on scaling AncestryDNA using Hadoop and HBase at QConSF in 2013. Check it out!

Adventures in Big Data: Join the Community, Share, and Give Back

Posted by Bill Yetman on February 19, 2014 in Big Data, Development

Ancestry.com to Host HBase Meetup on March 12th at our SF office  If you are thinking about starting a Big Data Initiative, you may want to consider its affect across the organization. At Ancestry.com, we have been a very traditional Microsoft .NET and SQL Server shop for a long time. Several Initiatives, two which involve… Read more

Handling Dynamic JSON Schemas

Posted by Xuyen On on February 5, 2014 in Big Data

In my last post, I introduced our first steps in creating a scalable, high volume messaging system and would like to provide an update on our progress. We have built out a 0.7.2 Kafka cluster to start ingesting data from our servers. The cluster consists of the following: 5 x  Kafka nodes •    Dual 6… Read more

IT Transformation as a “Business” Discipline

Posted by Deal Daly on January 22, 2014 in Big Data, IT

This series of essays will explore IT transformation (“futurization”) as a functional discipline of the business. IT transformation has clear business purposes.   The transformational activity works to: increase speed to market for the business’s products and services, provide new and fulfilling career paths to IT engineers, increase reliability, availability and performance of systems and… Read more

Visualizing Family Trees

Posted by Leonid Zhukov on January 17, 2014 in Big Data

A company’s data set is a unique asset and it is very advantageous for companies to know what one of its most valuable assets looks like to make product and business decisions.  That is where data scientists come in: we like to study data.  At Ancestry.com, we have a large and unique set of data, which… Read more

On Track to Data-Driven

Posted by Aaron Ling on December 25, 2013 in Big Data, Distributed Computing, Technical Management

Ancestry.com becomes more and more aware of the value of the data our website generates every single day. We have a lot of customers coming to the website to discover, preserve and share their family history. They come from different parts of the world and are looking for information that helps them tell the story… Read more