usnistgov/trec_eval

trec_eval is the standard tool used by the TREC community for
evaluating an ad hoc retrieval run, given the results file and a
standard set of judged results.  

------------------------------------------------------------------------------
Installation: Should be as easy as typing "make" in the source directory.
If you wish the trec_eval binary to be placed in a standard location, alter
the first line of Makefile appropriately.

------------------------------------------------------------------------------
Testing: sample input and output files are included in the directory test.
"make quicktest" will perform some sample simple evaluations and compare
the results.

------------------------------------------------------------------------------
Usage:  Most options can be ignored.  The only one most folks will need
is the "-q" flag, to indicate whether to output official results for individual 
queries as well as the averages over all queries.  Official TREC usage
might be something like 
	trec_eval -q -c -M1000 official_qrels submitted_results 
to ensure correct evaluation if submitted_results doesn't have results
for all queries, or returns more than 1000 documents per query.
If you wish to output only one particular measure:
        trec_eval -m measure[.params] rel_info_file results_file


------------------------------------------------------------------------------
Change Log  (only recent)
------------------------------------------------------------------------------
12/31/08
       comments and documentation of Zscore file format corrected.
       trec_eval.c get_zscores.c
2/25/08 Version 9.0alpha.  
       Complete rewrite of entire trec_eval (needed for a long time!).  
       Complete separation of individual measure calculations -
       computers are now fast enough so can afford recalculation of lots
       of intermediate values.  
       Should be much easier to add measures to, and much easier to add
       new input file formats with associated measures.
       Parameters for measures (eg, cutoffs for P) can be specified on the
       command line.  
       Choice of measures can be specifed on the command line.
       An initial set of preference evaluation measures (with their own
       input rel_info format) have been added.
       Help now gives targeted measure and format descriptions.  Try
          trec_eval -h -m all_prefs -R prefs
       to get info on preference measures and formats, for instance.
       All internal calculations are in double rather than float. Yields
       minor variations in output at rare times; mostly when going from a
       double percentage to a corresponding doc cutoff (eg, in iprec_at_recall).
       All globally known procedure names or variables now begin with 'te_'
       to allow incorporation of procedures in other programs.
       
       Measures added:
       	        ndcg, ndcg_cut, set_F, success, map_avgjg, P_avgjg,
		various preference evaluation measures.
       Measures renamed:
                Rprec-mult_*            was *R-prec
                set_P			was exact_prec
	        set_recall 		was exact_recall
	        set_relative_P		was exact_relative_prec
	        set_recall		was exact_recall
	        set_map			was exact_unranked_avg_prec
		gm_map                  was gm_ap
		11pt_avg                was 11-pt_avg
		P_*                     was P*
                recall_*                was recall*
		relative_P_*            was relative_P*
		iprec_at_recall_*       was ircl_prn.*
       Measures dropped for now:
       		3-pt_avg       	
		avg_doc_prec   	
		avg_relative_prec	
		exact_relative_unranked_avg_prec	
		map_at_R       	
                int_map        	
                exact_int_R_rcl_prec
                int_map_at_R   	
                unranked_avg_prec*
                relative_unranked_avg_prec*
                rcl_at_142_nonrel	
                fallout_recall_*
                int_*R-prec
                micro_prec     	
                micro_recall   	
                micro_bpref    	
		bpref variants
		time base measures.
	Input formats added:
	        prefs - allows expression of preferences
		qrels_prefs - same as standard qrels, except treated as prefs
		qrels_jg - same as standard qrels, except allows judgment sets
                           from multiple users (judgment groups).

Version 8.1, Added infAP, minor bug fixes
7/24/06 Improved infAP comments (implementation verified by Yilmaz).
        trec_eval_help.c: allow longer measure explanations.
6/27/06 get_opt.c Fixed error message
6/22/06 Added measure infAP (Aslam et al) to allow judging only sample 
        of pools.  -1 for rel in qrels file interpreted as pool doc not judged.
6/22/06 trvec_teval.c: fixed bugs in calculation of bpref if multiple
	relevance levels were used and a non-default relevance level
	was given. (Eg. A doc with rel level of 2 was counted as unjudged
	rather than judged nonrel if a relevance level of 3 was needed
	to consider relevant.)
4/5/06  Changed comments in README, trec_eval.c, trec_eval_help.c files 
        which incorrectly claimed queries with no relevant docs are 
        ignored (this was true with very old versions of trec_eval).  Now
        reads that queries with no relevance information are ignored.
        Giorgio Di Nunzio and Nicola Ferro,
------------------------------------------------------------------------------
Version 8.0, full bpref bug fix, see file bpref_bug.  I decided to up the 
        version number since bpref results are incompatible with previous 
        results (though the changes are small).
------------------------------------------------------------------------------
------------------------------------------------------------------------------

Files:
Main procedure:
trec_eval.c
---
Procedures to read input files of various formats:
formats.c          Mapping names of input file formats to input procedures
get_qrels.c        Read the standard judged documents (qrels format)
get_qrels_jg.c     Read qrels format with multiple judgment groups per query
get_prefs.c        Read preferences judgments instead of doc judgments - see
		   formats.c for full description.
get_qrels_prefs.c  Read qrels_jg format file, interpret as prefs file.
get_trec_results.c Read the standard result file (trec_results format).
---
Procedures to merge rel_info and results from input form into form that measures
can easily use, if they wish:
form_res_rels.c    'qrels' and 'trec_results' into RES_RELS format.
form_res_rels_jg.c 'qrels_jg' and 'trec_results' into RES_RELS_JG format.
form_pref_counts.c ('prefs' or 'qrels_prefs') and 'trec_results' format
---
The actual measures:
measures.c   Associates measure name with parameters and
             init, calculation, accumulation, printing procedures
meas_*.c     Common procedures used by many measures for init, acc, printing.
m_<measure_name>.c  measure specific procedures
---
Miscellaneous:
Makefile     Compile and test trec_eval
README       This file
CHANGELOG    Recent changes
test         Directory of collection of sample input and output for trec_eval
trec_eval.h  Basic evaluation structures.
functions.h  Prototype decorations of measure procedures.
sysfunc.h
common.h
bpref_bug:   Description of bug in bpref that existed in trec_eval versions 6
             through 7.3.

------------------------------------------------------------------------------
------------------------------------------------------------------------------
Adding a new measure.

Assuming it uses standard input files:
1. In m_<new_measure>.c, write any needed measure specific procedures needed to
      initialize measure
      calculate measure
      accumulate measure (adding one topic's value to summary value over topics)
      calculate the ending average for a measure.
      print single
      print final query measure value (and cleanup if needed)
   Most measures require only a new calculate measure procedure - the
   other procedures are generic and already implemented depending on the type of
   measure (has cutoffs and value for each cutoff, parameters, is a float, etc).
   See functions.h to see fit for these generic procedures.
2. In same file, construct TREC_MEAS te_meas* entry pointing to above 
   procedures and any default cutoffs or parameters.
3. Add pointer to that TREC_MEAS entry in "measures.c"
4. Add measure source file to Makefile

------------------------------------------------------------------------------
Adding a new file format
1. Implement reading of new format in get_<new_format>.c, with returned top
   level output of type ALL_REL_INFO or ALL_RESULTS.  The individual topic
   returned values will be in a format dependent form which will be passed
   to the appropriate measures.
2. Add format to appropriate format list in formats.c
3. Add measures to take advantage of format (see above)
4. To use, invoke trec_eval with -R or -T values, and -m measures that
   are appropriate.