Definite Pronoun Resolution Dataset
This page is a distribution site of data created for intra-sentential definite pronoun resolution experiments. We target pronouns whose resolution requires semantic and world knowledge, rather than those that can be resolved using traditional grammatical and syntactic constraints on coreference such as Binding constraints and agreement on gender and number. A restricted version of this task is sometimes referred to as the Winograd Schema Challenge (Levesque, 2011; Levesque et al., 2012), owing to Terry Winograd's pioneering attempt to use the resolution of such pronouns to illustrate the difficulty of natural language understanding (Winograd, 1972).
Composed by 30 students from one of my undergraduate classes, the dataset comprises 943 sentence pairs and is divided into a training set and a test set. These sentence pairs cover topics ranging from real events (e.g., Iran's plan to attack the Saudi ambassador to the U.S.) to events/characters in movies (e.g., Batman) and purely imaginary situations, largely reflecting the pop culture as perceived by the American kids born in the early 90s. Each annotated example spans four lines: the first line contains the sentence, the second line contains the target pronoun, the third line contains the two candidate antecedents, and the fourth line contains the correct antecedent. If the target pronoun appears more than once in the sentence, its first occurrence is the one to be resolved.
Please acknowledge your use of this dataset by citing the following paper, which describes a pronoun resolver that significantly outperforms the Stanford resolver and the mention-ranking model (as implemented in the Cherrypicker resolver) on our dataset:
Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge.
Altaf Rahman and Vincent Ng.
Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 777-789, 2012.
To give you an idea of how (incredibly) interesting this resolution task is, here are some sample sentence pairs taken from our dataset. In each sentence, the target pronoun is italized and its antecedent is boldfaced:
|The city councilmen refused the demonstrators a permit because they advocated violence.||The city councilmen refused the demonstrators a permit because they feared violence.|
|James asked Robert for a favor, but he refused.||James asked Robert for a favor, but he was refused.|
|Keith fired Blaine but he did not regret.||Keith fired Blaine although he is diligent.|
|Emma did not pass the ball to Janie, although she was open.||Emma did not pass the ball to Janie, although she should have.|
|Medvedev will cede the presidency to Putin because he is more popular.||Medvedev will cede the presidency to Putin because he is less popular.|
Hector Levesque, The Winograd Schema Challenge, Tenth International Symposium on Logical Formalizations of Commonsense Reasoning (Commonsense-2011).
Hector Levesque, Ernest Davis, and Leora Morgenstern, The Winograd Schema Challenge, Thirteenth International Conference on Knowledge Representation and Reasoning (KR-2012).
Understanding Natural Language,
Academic Press, 1972.