Robust Search Evaluation is both a “must have” for any modern day Search team and an “after thought” that never gets the team’s full attention. This is especially true with the various open source search engines. Most teams build their own data collection and eval tools. Some use standalone open source tools. We present a better solution!