## UAI 2014 Inference CompetitionOrganizer: Vibhav Gogate (Email: vgogate at hlt dot utdallas dot edu)
## Evaluation Criteria## MAP TaskFor Evaluating the results produce by different MAP solvers we have used following two metrics – ## Relative Gap From the LeaderFor each problem for a solver the score is computed as follows – where is the value of the result returned by and is the same for the best solver for this problem instance (i.e.– solver with highest MAP value)
## RankFor each problem each solvers are ranked in-terms of its solution quality, i.e.– solvers with higher MAP value gets smaller rank and the winner(s) for that problem gets the rank 1. Finally we add all the ranks of a solver in all the problems. The winner of the competition is selected using this metric. The solver which has lowest cumulative score is decided as a winner. ## MMAP TaskFor Evaluating the results produce by different MMAP solvers we have used following two metrics – ## Relative Gap From the LeaderFor each problem for a solver the score is computed as follows – where is the log MMAP value of the result returned by and is the same for the best solver for this problem instance (i.e.– solver with highest MMAP value)
## RankFor each problem each solvers are ranked in-terms of its solution quality, i.e.– solvers with higher MMAP value gets smaller rank and the winner(s) for that problem gets the rank 1. Finally we add all the ranks of a solver in all the problems. The winner of the competition is selected using this metric. The solver which has lowest cumulative score is decided as a winner. ## PR TaskFor evaluating the results of different PR solvers we compute the errors of the partition function. The error of a solver on instance is computed as follows - where is the true partition function and the approximate partition function computed by the solver. The final error for a solver is given by . The solver with least final error is decided as a winner. ## MAR TaskFor Evaluating the results produce by different MAR solvers we have used following two metrics – ## Hellinger ErrorFor the problem the Hellinger error corresponding to a solver is computed as follows – where is the total number of variables, is the Hellinger distance between the true probability distribution corresponding to the variable () and the approximate one returned by the solver (). The final error for a solver is given by ## Max-Absolute ErrorFor the problem the max absolute error corresponding to a solver is computed as follows – where the true probability distribution corresponding to the variable is () and the approximate one returned by the solver is (). The final error for a solver is given by |