The following pages describe the NTCIR-2 Automatic Text Summarization Task. Please keep in mind that the pages may not be final in its content since we need to finalize some of the details later on. The additions and updates will be announced in this page, so please check it from time to time.

(1) Objectives

We have two objectives in Automatic Text Summarization task. The first is to collect summarized texts.
The second is evaluation of automatic text summarization systems.

I) Collection of summarized texts
Texts are summarized by hand. The human summarizers (annotators) are persons who have enough experience in producing summaries.

There are going to be two types of summaries.

1. Extract-type summary
The summarizers extract important sentences of the text as well as part(s) of a sentence (smaller than a sentence) thought to be important.

2. Abstract-type summary
At the same time, we will have the articles summarized freely, without worrying about keeping original sentences intact.

We use newspaper articles (Mainichi Shinbun, The Mainichi: you need to pay for a license to use) as original texts for these tasks. They are not limited to business domain, and articles of other domains such as editorials, columns will be included.

We would like to have several hundred articles summarized in these ways, and make them ready for the intrinsic evaluation (see below).

II) Evaluation of Automatic Text Summarization Systems We plan to have two kinds of evaluation.

1. Intrinsic evaluation
We use the summaries that we collect in the first task for the intrinsic evaluation. Evaluation of extract-type summaries (in which important sentences are extracted) will be conventional: we measure how much matching there is between the human and machine summaries. As for the "free" summaries, we conduct a subjective evaluation, measuring how close the human and machine summaries are by analyzing contents of the summaries, readability, and acceptability as a combination of the first two measurements.

2. Extrinsic evaluation
We will use IR (Information Retrieval) as the task to be done for measuring automatic text summarization systems.

The evaluation method we use is actually employed at SUMMAC evaluation.
First, the query and the summaries of the retrieved documents are shown to the human judge, and he/she will measure how well each summary answers to the query. We would like to measure how long it takes to complete the task, as well as recall, precision, and F-measures.

(2) Participation
If you are interested in taking part in the Task, please look at the following Web page ( http://www.rd.nacsis.ac.jp/~ntcadm/workshop/work-en.html) for how to apply.
You may participate in one or more evaluations.

(3) Tentative Schedule

July 20, 2000 : deadline for call for participation
September, 2000 : Dryrun
November or December, 2000 : Evaluation
February 1, 2001 : paper (to report the results) due
February 21-23, 2001 : NTCIR2 workshop in Tokyo
March 1, 2001 : final version of the paper due

Co-chairs of the Text Summarization Task
Manabu Okumura : oku@pi.titech.ac.jp
Takahiro Fukusima : fukusima@res.otemon.ac.jp