- Organizers
- What's new
- Task description
- References
- How the training/test data will be built
- The evaluation methodology
- The availability of the resources(copyright, costs)
Organizers
- Manabu Okumura(Tokyo Institute of Technology)
- Kiyoaki Shirai(Japan Advanced Institute of Science and Technology)
What's new
2010.1.7
The schedule of the task was decided as follows:
- Release of traning data: February 1st
- Release of test data: March 24th
- Submission deadline of the result: March 31st
- Release of the answer data for the test data: April 2nd
We appreciate for your cooperation, though the schedule is tight.
2009.8.30
Trial data for our task is now ready for you. We hope those who are now planning to participate in the task will kindly send a mail to us in the following address, and can get the data:
oku atmark pi.titech.ac.jp
The steps to get the data are the following:
- (You) send a mail to us in order to let us know you are planning to participate in the task,
- (We) send an agreement form to you,
- (You) send back the form to us, after signing it,
- (We) tell the method to get the data.
The trial data will include the data of Iwanami Kokugo Jiten and a sample corpus, accompanying with a scorer.
Task description
This task can be considered an extension of SENSEVAL-2 JAPANESE LEXICAL SAMPLE Monolingual dictionary-based task. Word senses are defined according to the Iwanami Kokugo Jiten, a Japanese dictionary published by Iwanami Shoten. Please refer to that task for reference. We think that our task has the following two new characteristics:
1) All previous Japanese sense-tagged corpora were from newspaper articles, while sense-tagged corpora have been constructed in English on balanced corpora, such as Brown corpus and BNC corpus. The first balanced corpus of contemporary written Japanese (BCCWJ corpus) is now being constructed as part of a national project in Japan [Maekawa, 2008], and we are now constructing a sense-tagged corpus on it. Therefore, the task will use the first balanced Japanese sense-tagged corpus.
2) In previous WSD tasks, systems have been required to select a sense from a given set of senses in a dictionary for a word in one context (an instance). However, the set of senses in the dictionary is not always complete. New word senses sometimes appear after the dictionary has been compiled. Therefore, some instances might have a sense that cannot be found in a set in the dictionary. The task will take into account not only the instances having a sense in the given set but also the instances having a sense that cannot be found in the set. In the latter case, systems should output that the instances have a sense that is not in the set.
In summary, systems in the task will receive the following input and should output as follows:
[Input]
Test documents with marked target words from the BCCWJ corpus, where
the genre of documents is also provided, because of their
diversity. Examples include books, newspaper articles, white papers,
blogs, magazines, and documents from a Q&A site on the WWW.
[Output]
The sense ID of each target word in the Iwanami Kokugo Jiten if the
sense is in the dictionary. If systems find that the sense is not in
the dictionary, say `new sense.'
References
Kikuo Maekawa. "Balanced Corpus of Contemporary Written Japanese." Proceedings of the 6th Workshop on Asian Language Resources (ALR), Hyderabad, pp.101-102, 2008:01.
How the training/test data will be built
The BCCWJ corpus is now under construction. Since part of it is already available for our sense tagging, we are now constructing a sense-tagged corpus on it. Sense tagging has been manually done. Because a fragment of the corpus was tagged by multiple taggers, the Kappa statistics can be calculated. For the instances that have a `new sense,' two other taggers will reexamine carefully whether those instances really have a new sense.
The evaluation methodology
Organizers will return the evaluation in two ways:
a. evaluating the outputted sense IDs, assuming the `new sense' as another sense ID. The outputted sense IDs will be compared to the given gold standard word senses, and the usual precision measure for supervised word sense disambiguation systems will be computed using the standard SENSEVAL scorer. The Iwanami Kokugo Jiten has three levels for sense IDs, and we use the middle-level sense in the task. Therefore, we can call the scoring in the task `middle-grained scoring.'
b. evaluating the ability of finding the instances of new senses, assuming the task as classifying each instance into a `known sense' or `new sense' class. The outputted sense IDs (same as in a.) will be compared to the given gold standard word senses, and the usual accuracy for binary classification will be computed, assuming all sense IDs in the dictionary are in the `known sense' class.
The availability of the resources (copyright, costs)
The Iwanami Kokugo Jiten will be available soon from GSK (http://www.gsk.or.jp/). A corpus annotated with sense IDs will also be distributed as training data. Each article will be assigned its genre code. Participants in this task are required to submit a copyright agreement form to the National Institute of Japanese Language.
