Maintaining Balance by Using Feedback Loops in Software
Feedback is an important part of daily life. Used wisely, feedback loops can help us make better decisions, resulting in overall improvements. Feedback loops are useful in all sorts of situations, from relationships to what we eat for dinner. Software can also be made to take advantage of feedback loops.
Feedback, Feedback loops, and Apple Pie
The difference between feedback and a feedback loop is an important one. To illustrate, let’s say I get a new apple pie recipe I want to try out. I follow the directions exactly, and the pie comes out burnt. I see that it’s burnt (feedback), so I throw it out rather than eat it. But I really like apple pie, so I the next day I try again, with the same result. Using feedback alone, I can throw the pie out again and again until the day I die. What is needed is a feedback loop. A feedback loop is where existing result(s) are used as input. Where feedback prevented me from eating the burnt pie, a feedback loop can be used to make adjustments to the baking time and/or temperature. After a few iterations of learning from the feedback, I’ll be able to enjoy a tasty apple pie, cooked to perfection.
Feedback in Software
There are many forms of feedback in software, including exceptions, return values, method timings, and other measurements. Whether it will be useful to create a feedback loop in your application is up to you – often times, there is little value in creating an automated feedback loop. However, there are situations where feedback loops can be very helpful. Software that is resource-bound or dependent on external systems are examples of software that can benefit from feedback loops. Here at Ancestry.com, much work has been done to create feedback loops based on service level agreements between dependent systems. We’ve seen a lot of benefit, but we’ve also had a learning experience or two. If you’ve ever heard screeching because of feedback with a microphone, you know that not all feedback loops produce improved results. Yes, it’s possible to produce bad feedback loops in software. This can happen with a single feedback loop, but it’s especially true if you have multiple feedback loops — multiple feedback loops with bidirectional interactions can quickly become a debugging nightmare. The more you can compartmentalize the moving parts and limit interactions, the easier maintenance will be.
A real-world example
One of the projects I work on is a system that finds and fixes problems. Most of the jobs this system runs are low priority. The system uses a shared compute cluster to do most of the work, and we don’t want it to use the cluster if it is loaded up with higher priority work. Rather than queue up jobs in the cluster for days at a time, a feedback loop is used to determine whether jobs should run on the cluster or local processors. The system monitors the cluster by sending a “ping” type task periodically, and tracking that feedback (timings, exceptions, etc.) along with the feedback from real tasks. With that feedback, the system determines when the cluster should be used, and will dynamically change the compute destination based on that feedback. Meanwhile, another feedback loop that tracks how much work is queued up in memory, and automatically adjusts how quickly work is produced. We have other feedback loops around the data sources, to take action if we suspect this application is slowing down production systems. The entire system works pretty well.
My team has seen improved performance and stability as feedback loops have been judiciously added. We’ve found them to be worthwhile on any system we’ve applied them to, critical or not. While it seems there’s always more we can do, it’s nice to have systems do some self-tuning to keep things efficiently running.