Call For Papers/How to Apply

Challenge Goal

Recent advances in machine learning have led to the application of artificial intelligence (AI) technology to a variety of social problems. Accordingly, being able to explain the reason for an AI decision is becoming important to ensure the secure and safe use of AI techniques.

On this background, the Knowledge Graph Reasoning Challenge has been organized from 2018 in Japan. It aims to promote techniques for explainable AI using knowledge graphs. In the challenge, the task is to estimate the culprits with a reasonable explanation using a dataset of knowledge graphs representing eight Sherlock Holmes mystery stories. The challenge has been held four times in Japan, and 24 approaches have been proposed and fruitfully discussed, including approaches to solving the constraint satisfaction problem, logical inference, and machine learning techniques including knowledge graph embedding.

In the workshop, we would like to organize the First International Knowledge Graph Reasoning Challenge (IKGRC 2023) to discuss a wider variety of knowledge graph reasoning technologies for explainable AI.

Challenge Task

The subject set for the Challenge is the cases in Sherlock Holmes short stories in which Holmes solved mysterious crimes.
The task is to reason and/or estimate the truth of the case while providing a reasonable explanation based on the relevant open knowledge graphs.
The right answer of the task is to arrive at the same conclusion as Holmes, that is, to explain the truth of the case that Holmes solved.

We are soliciting entries for the dataset in the following three divisions.

• (1) Main track: Development of a system that leads to one or more culprits in one or more of the target stories.
• (2) Tool track: Development of tools to solve partial tasks required in the process of finding the culprit, e.g., suspect estimation, alibi verification, motive explanation, etc.
• (3) Idea track: Ideas for how to realize 1 and 2 (without implementation)

Target Stories and Tasks

The Speckled Band: Who killed Julia? (criminal & explanation)
The Devil's Foot: Who killed the victims? (criminal & explanation)
The Crooked Man: Why did Barclay die? (explanation)
The Dancing Men: Break the codes (code breaking)
The Abbey Grange: Who killed Lord Blackenstall? (criminal & explanation)
The Resident Patient: Who killed Blessington? (criminal & explanation)
Silver Blaze: Who took out the White Silver Blaze? (criminal & explanation)

Task Execution Conditions

• 1. Reasoning should be based on the given knowledge graphs, with various range of use:
  • a）Cover-all: use the whole knowledge graph
  • b）Cover-a-part (90%): use the triples from the fist ID up to 10 % before the last ID
  • c）Cover-a-part (75%): use the triples from the first ID up to 25% before the last ID
    (The reference ID numbers are to be published).
• 2. You can extend the knowledge graphs by yourself (e.g. complement of common sense knowledge).
• 3. Pull requests are welcome in case you find any error in the given knowledge graphs (e.g. typos, inconsistency, etc.).

Submissions

• 1. Please submit(*) a short paper (2 pages, IEEE format) with an outline of each of the submitted categories by [November 30 December 17, 2022]. Systems and tools may be under development. After peer review, accepted papers will be published in the Proceedings of the ICSC.
  (*) The EaasyChair URL for the IKGRC 2023 is https://easychair.org/conferences/?conf=ikgrc2023.
• 2. If your paper is accepted, you will be required to send us* materials explaining your system, tools, and ideas (formats for the materials: .doc .pdf) at least January 22th, 2023. *Please send the material to kgrc@knowledge-graph.jp via email.
• 3. On the day of the challenge, you will be asked to present your system, tool, or idea with a demonstration (online participation is possible). Prizes will be awarded to the best entries.
  *Time for the presentation is 20 min. including 5 min. QA.

Evaluation Criteria (Details to be announced)

• 1. Can the approach reproduce Holmes’s reasoning?
• 2. Is the approach logically persuasive?
• 3. How unique is the approach?
  • -way of reasoning
  • -extended knowledge
• 4. How applicable or feasible is the approach?
  • -applicable to all the tasks vs. specific to one task
  • -good use of small amount of knowledge vs. needs for huge amount of knowledge
• 5. How good is the idea presented?
  • -Examination of submissions and presentation

Contact