An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering

Author : Damevski, Kostadin; Shepherd, David; Pollock, Lori
Booktitle : First Workshop on the Next 5 Years of Textual Analysis in Software Maintenance
Date : Jun 2012
Keyword(s) : Evaluation, annotated corpora, feature location, paired interleaving
Document Type : In Conference Proceedings

Abstract :

One of the key challenges facing the community of researchers in text-based analysis for software engineering is evaluation of new techniques and tools that leverage those techniques. In this paper, we explore the paired interleaving approach, used to evaluate Internet search engines, as an alternative to creating annotated corpora for gold sets. In particular, we examine this approach in the context of evaluating feature location techniques, which collectively are a key software engineering client application for text-based analysis of software artifacts. This paper describes the paired interleaving approach and presents the challenges in customizing this technique for comparative evaluation of feature location techniques with different underlying text analysis and preprocessing. Better evaluations of text-based analyses and their ultimate client tool applications would help remove the barriers to industry adoption of text analysis in software tools.

Paper Link