Scaling Up Evaluation of Code Search Tools Through Developer Usage Metrics
Author : Damevski, Kostadin; Shepherd, David; Pollock, Lori
Booktitle : IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER)
Date : Mar 2015
Publisher : IEEE
Keyword(s) : measurement, code search, feature location, field studies
Document Type : In Conference Proceedings
Code search is a fundamental part of program understanding and software maintenance and thus researchers have developed many techniques to improve its performance, such as corpora preprocessing and query reformulation. Unfortunately, to date, evaluations of code search techniques have largely been in lab settings, while scaling and transitioning to e ective practical use demands more empirical feedback from the eld. This paper addresses that need by studying metrics based on automatically-gathered anonymous eld data from code searches to infer user satisfaction. We describe techniques for addressing important concerns, such as how privacy is retained and how the overhead on the inter- active system is minimized. We perform controlled user and eld studies which identify a metric that correlates with user satisfaction, enabling the future evaluation of search tools through anonymous usage data. As we further explore the data, we also present a predictive multi-metric model that achieves accuracy of over 70% in determining query satisfaction.