One of the key challenges facing the community of researchers in text-based analysis for software engineering is evaluation of new techniques and tools that leverage those techniques. In this paper, we explore the paired interleaving approach, used to evaluate Internet search engines, as an alternative to creating annotated corpora for gold sets. In particular, we examine this approach in the context of evaluating feature location techniques, which collectively are a key software engineering client application for text-based analysis of software artifacts. This paper describes the paired interleaving approach and presents the challenges in customizing this technique for comparative evaluation of feature location techniques with different underlying text analysis and preprocessing. Better evaluations of text-based analyses and their ultimate client tool applications would help remove the barriers to industry adoption of text analysis in software tools.