Welcome to NTCIR STC2 Japanese Subtask

Overview

NTCIR STC2 is the second round of NTCIR Short Text Conversation. This page overviews the Japanese subtask. If you are interested in the Chinese subtask, please refer to this page.

In the Japanese subtask, given an input comment, the task is to retrieve or generate a suitable response to that comment. If we can create reasonable responses, it will lead to the improvement in the language generation capability of dialogue systems. Unlike STC1, both retrieval and generative methods are allowed for response generation.

Note that, in the Japanese subtask, comments posted at Yahoo! News are provided as a pool of comments. This is different from STC1, in which Twitter was used. In Yahoo! News, users post comments about an article. Users can also post comments to other users’ comments as responses. We use these comment-response pairs as our dataset in STC2.

Dataset of Japanese task

The following data will be provided:

1. Yahoo! News comments data; approximately around one million comment-response pairs.
2. Development data. Input samples and output samples annotated with reference labels.
3. Test data. The test data will be provided at the time of the formal run.

The input data for 2 and 3 are those sampled from Yahoo! News comments. The data will be provided to participants upon signing an agreement with NII.

Evaluation

As in STC1, retrieved/generated responses will be evaluated by human judges. The evaluation is done by the organizers. In the evaluation process, each retrieved/generated response is labeled with 0 (inappropriate), 1 (appropriate in some context), and 2 (appropriate) by multiple judges. The rate of appropriate answers and information retrieval (IR) related measures (graded relevance IR measures) will be used as evaluation metrics.

Registration

Registration is closed.

Schedule of Japanese Subtask

• Release of the Yahoo! News comments data and starts accepting registration: Jun 11, 2017
• Release of the development data: Jul 1, 2017 Jul 21, 2017
• Task registration due: July 23, 2017 (Important: submission & evaluation)
• Formal run start (test data release): July 24, 2017
• STC run submission deadline: July 31, 2017 August 13
• Relevance assessments: August 1-31, 2017 August 14-31, 2017
• Results and Draft Task overview released to participants: Sep 1 2017
• Participants draft papers due: Oct 1 2017
• All camera ready papers due: Nov 1 2017
• NTCIR-13 Conference: Dec 2017

Organizers

• Ryuichiro Higashinaka, Nippon Telegraph and Telephone Corporation, Japan
• Yusuke Miyao, National Institute of Informatics, Japan
• Yuki Arase, Osaka University, Japan
• Masako Nomoto, Yahoo Japan Corporation, Japan

Contact Us

Regarding the Japanese task, please contact the organizers of the Japanese task

• stc-japanese<at>nii.ac.jp

With regards to the information about STC, please contact the following address.

• stc-org<at>list.waseda.jp