GitHub’s Scientist puts Ruby rewrites to the test
In a blog post, GitHub engineer Jesse Toth talked about the need to replace or rewrite part of a system and how GitHub has had many systems that scaled far beyond their original design. Eventually a large component of the application has to be redone, and Scientist was built to help test production data and behavior to ensure correctness.
"It works by creating a lightweight abstraction called an experiment around the code that is to be replaced," Toth said. "The original code -- the control -- is delegated to by the experiment abstraction, and its result is returned by the experiment. The rewritten code is added as a candidate to be tried by the experiment at execution time."
When the experiment is called at runtime, both code paths are run, with the order randomized to avoid ordering issues. Scientist compares the results of both the control and the candidate and records any differences. "The duration of execution for both code blocks is also recorded. Then the result of the control code is returned from the experiment."
Scientist is not opinionated about what users should do with the data it produces, Toth said. "Implement the publish method in your experiment class to record metrics and store mismatches. Scientist passes an experiment's result to this method." A Scientist::Result contains information about an experiment, including the results of the control and candidate blocks if there was a difference and whether an experiment matched, mismatched, or was ignored.
GitHub is encouraging use of Scientist. "Even if Ruby isn't your language of choice, we'd still encourage you to apply Scientist's experiment pattern to your system," said Toth. Scientist has been used at GitHub for such purposes as a multi-year-long rewrite to clean up permission code, switching to a new code search cluster, optimizing queries, and refactoring risky parts of the code base.
Toth cautioned that Scientist is not meant to be used for any code that has side effects. "A candidate code path that writes to the same database as the control, invalidates a cache, or otherwise modifies data that affects the original, production behavior is dangerous and incorrect. For this reason, we only use Scientist on read operations." Users also will experience a performance hit when using Scientist in production.