Annotating Inter-Sentence Temporal Relations in Clinical Notes

Jennifer D'Souza and Vincent Ng.
Proceedings of the 9th International Conference on Language Resources and Evaluation, pp. 2758-2765, 2014.

Click here for the PostScript or PDF version. The talk slides are available here.


Owing in part to the surge of interest in temporal relation extraction, a number of datasets manually annotated with temporal relations between event-event pairs and event-time pairs have been produced recently. However, it is not uncommon to find missing annotations in these manually annotated datasets. Many researchers attributed this problem to "annotator fatigue". While some of these missing relations can be recovered automatically, many of them cannot. Our goals in this paper are to (1) manually annotate certain types of missing links that cannot be automatically recovered in the i2b2 Clinical Temporal Relations Challenge Corpus, one of the recently released evaluation corpora for temporal relation extraction; and (2) empirically determine the usefulness of these additional annotations. We will make our annotations publicly available, in hopes of enabling a more accurate evaluation of temporal relation extraction systems.


The human annotation used in this paper is available from this page.

BibTeX entry

  author = {Jennifer D'Souza and Vincent Ng},
  title = {Annotating Inter-Sentential Temporal Relations in Clinical Notes},
  booktitle = {Proceedings of the 9th Language Resources and Evaluation Conference},
  pages = {2758--2765}, 
  year = 2014}