Maintaining Balance by Using Feedback Loops in Software

Posted by cgroneman on July 29, 2014 in C#, Performance

Maintaining Balance by Using Feedback Loops in Software

Feedback is an important part of daily life.  Used wisely, feedback loops can help us make better decisions, resulting in overall improvements.  Feedback loops are useful in all sorts of situations, from relationships to what we eat for dinner.  Software can also be made to take advantage of feedback loops.

Feedback, Feedback loops, and Apple Pie

The difference between feedback and a feedback loop is an important one.  To illustrate, let’s say I get a new apple pie recipe I want to try out.  I follow the directions exactly, and the pie comes out burnt.  I see that it’s burnt (feedback), so I throw it out rather than eat it.  But I really like apple pie, so I the next day I try again, with the same result.  Using feedback alone, I can throw the pie out again and again until the day I die.  What is needed is a feedback loop.  A feedback loop is where existing result(s) are used as input.  Where feedback prevented me from eating the burnt pie, a feedback loop can be used to make adjustments to the baking time and/or temperature.  After a few iterations of learning from the feedback, I’ll be able to enjoy a tasty apple pie, cooked to perfection.

Feedback in Software

Simple Feedback LoopThere are many forms of feedback in software, including exceptions, return values, method timings, and other measurements.  Whether it will be useful to create a feedback loop in your application is up to you – often times, there is little value in creating an automated feedback loop.  However, there are situations where feedback loops can be very helpful.  Software that is resource-bound or dependent on external systems are examples of software that can benefit from feedback loops.  Here at, much work has been done to create feedback loops based on service level agreements between dependent systems.  We’ve seen a lot of benefit, but we’ve also had a learning experience or two.  If you’ve ever heard screeching because of feedback with a microphone, you know that not all feedback loops produce improved results.  Yes, it’s possible to produce bad feedback loops in software.  This can happen with a single feedback loop, but it’s especially true if you have multiple feedback loops — multiple feedback loops with bidirectional interactions can quickly become a debugging nightmare.  The more you can compartmentalize the moving parts and limit interactions, the easier maintenance will be.

A real-world example

One of the projects I work on is a system that finds and fixes problems.  Most of the jobs this system runs are low priority.  The system uses a shared compute cluster to do most of the work, and we don’t want it to use the cluster if it is loaded up with higher priority work.  Rather than queue up jobs in the cluster for days at a time, a feedback loop is used to determine whether jobs should run on the cluster or local processors.  The system monitors the cluster by sending a “ping” type task periodically, and tracking that feedback (timings, exceptions, etc.) along with the feedback from real tasks.  With that feedback, the system determines when the cluster should be used, and will dynamically change the compute destination based on that feedback.  Meanwhile, another feedback loop that tracks how much work is queued up in memory, and automatically adjusts how quickly work is produced.  We have other feedback loops around the data sources, to take action if we suspect this application is slowing down production systems.  The entire system works pretty well.

My team has seen improved performance and stability as feedback loops have been judiciously added.  We’ve found them to be worthwhile on any system we’ve applied them to, critical or not.  While it seems there’s always more we can do, it’s nice to have systems do some self-tuning to keep things efficiently running.

Leave a comment

Past Articles

Building an Operationally Successful Component – Part 3: Robustness

Posted by Geoff Rayback on July 23, 2014 in DevOps, Uncategorized

Building an Operationally Successful Component – Part 3: Robustness My previous two posts discussed building components that are “operationally successful.”  To me, a component cannot be considered successful unless it actually operates as expected when released into the wild.  Something that, “works on my machine,” cannot be considered a success unless it also works on… Read more

Lessons Learned Building a Messaging Framework

Posted by Xuyen On on July 1, 2014 in Big Data

We have built out an initial logging framework with Kafka 0.7.2, a messaging system developed at LinkedIn. This blog post will go over some of the lessons we’ve learned by building out the framework here at Most of our application servers are Windows-based and we want to capture IIS logs from these servers. However,… Read more

Adventures in Big Data: Commodity Hardware Blues

Posted by Bill Yetman on June 20, 2014 in Big Data

One of the real advantages of a system like Hadoop is that it runs on commodity hardware. This will keep your hardware costs low. But when that hardware fails at an unusually high rate it can really throw a wrench into your plans. This was the case recently when we set up a new cluster… Read more

Website Performance 101

Posted by Jeremy Johnson on June 17, 2014 in Performance, Web

Here at, we have a team dedicated to monitoring, measuring, and helping the company improve the performance of the website. Trying to do this is a very fun and interesting challenge. With a website that has many billions of records and other content (10 petabytes), making it fast is no small task! To illustrate… Read more

Building an Operationally Successful Component – Part 2: Self Correction

Posted by Geoff Rayback on June 10, 2014 in DevOps

Building an Operationally Successful Component – Part 2: Self-correction In my last post I talked about building components that are “operationally successful,” by which I mean that the software functions correctly when it is deployed into production.  I suggested that there are three things that a software component must have, to some degree, in order… Read more

Featured Article: Want Great APIs? Start With Training

Posted by Harold Madsen on June 3, 2014 in API, has awesome software engineers, products, and APIs. However, programmers are not always trained as API designers and when it comes to API development, consistency matters. As companies build their API programs using multiple teams, APIs tend to develop their own personalities and become radically different from one another. That’s a problem. Fortunately, it doesn’t… Read more

Reinventing Search with Smart Filtering

Posted by Sam Smith on May 30, 2014 in Search

Last summer introduced a new but lesser known feature called “Smart Filtering”.  Smart Filtering is a technology that applies only when searching for someone in one of your trees.  When you search for someone in your tree, a lot more data becomes available to the search engine including life events (birth, marriage, death, etc…),… Read more to Present at Hadoop Summit

Posted by Bill Yetman on May 27, 2014 in Technology Conferences

Interest in direct-to-consumer DNA testing has grown dramatically in the past few years. When you’re measuring more than 700,000 DNA markers for each individual, how do you analyze all that data across a rapidly growing database, while providing actionable results for your customers? At the Hadoop Summit next week,

Find A Grave Engineering

Posted by Robert Schultz on May 21, 2014 in Development, Mobile, Web

Last October acquired a very exciting property called Find A Grave which focused on collecting content around the graves of family, loved ones and famous people. With the acquisition we wanted to take Find A Grave to the next level and provide the current users new and better experiences around consuming and contributing content.… Read more

Learning Alternative Name Spellings Technical Report

Posted by Jeffrey Sukharev on May 1, 2014 in Data Science Publishes Technical Report on data-driven technique for finding alternative name spellings.    In this article we discuss the problem of finding alternative name spelling, an important component of name matching (part of the record linkage field). We started this project primarily because of real issues that we encountered while working on name matching for… Read more

Migrating From TFS to Git-based Repositories (Part I)

Posted by Seng Lin Shee on April 29, 2014 in Agile, Development, Technical Management

Git, a distributed revision control and source code management system has been making waves for years, and many software houses have been slowly adopting this system as not only their source code repository, but also as a way software development projects are managed. There is much debate about using either a centralized or distributed revision… Read more to Present Jermline on DNA Day at the Global Big Data Conference

Posted by Jeremy Pollack on April 9, 2014 in Big Data, Data Science, Development, DNA, Science

Interested in genealogy?  Curious about DNA?  Fascinated by the world of big data?  If so, come check out my talk  at the Global Big Data Conference on DNA day this Friday, April 25 at 4pmPT in the Santa Clara Convention Center!  I’ll cover Jermline, our massively-scalable DNA matching application.  I’ll talk about our business, give a run-through… Read more