NTCIR STC2 Japanese Subtask

Shared Task on Short Text Conversation

Welcome to NTCIR STC2 Japanese Subtask

Overview

NTCIR STC2 is the second round of NTCIR Short Text Conversation. This page overviews the Japanese subtask. If you are interested in the Chinese subtask, please refer to this page.

In the Japanese subtask, given an input comment, the task is to retrieve or generate a suitable response to that comment. If we can create reasonable responses, it will lead to the improvement in the language generation capability of dialogue systems. Unlike STC1, both retrieval and generative methods are allowed for response generation.

Note that, in the Japanese subtask, comments posted at Yahoo! News are provided as a pool of comments. This is different from STC1, in which Twitter was used. In Yahoo! News, users post comments about an article. Users can also post comments to other users’ comments as responses. We use these comment-response pairs as our dataset in STC2.

Dataset of Japanese task

The following data will be provided:

  1. Yahoo! News comments data; approximately around one million comment-response pairs.
  2. Development data. Input samples and output samples annotated with reference labels.
  3. Test data. The test data will be provided at the time of the formal run.

The input data for 2 and 3 are those sampled from Yahoo! News comments. The data will be provided to participants upon signing an agreement with NII.

Evaluation

As in STC1, retrieved/generated responses will be evaluated by human judges. The evaluation is done by the organizers. In the evaluation process, each retrieved/generated response is labeled with 0 (inappropriate), 1 (appropriate in some context), and 2 (appropriate) by multiple judges. The rate of appropriate answers and information retrieval (IR) related measures (graded relevance IR measures) will be used as evaluation metrics.

Registration (still open)

Register through the official site of NTCIR-13.

http://ntcir.nii.ac.jp/NTCIR13Regist/

We will provide details to the address you provide to this site. We will accept registration until the formal run begins. See the schedule below.

Schedule of Japanese Subtask

Organizers

Contact Us

Regarding the Japanese task, please contact the organizers of the Japanese task

With regards to the information about STC, please contact the following address.