How to Apply

How to Apply

TBD
(Please see below for information about last year's challenge.)

Challenge Goal

This Challenge aims to promote the development of AI system that can explain the reason for the conclusion.

Challenge Task

The subject set for the Challenge is the cases in Sherlock Holmes short stories in which Holmes solved mysterious crimes.
The task is to reason and/or estimate the truth of the case while providing a reasonable explanation based on the relevant open knowledge graphs.
The right answer of the task is to arrive at the same conclusion as Holmes, that is, to explain the truth of the case that Holmes solved.

Update history
In addition to the knowledge graph of “the Speckled Band” provided for Challenge 2018, we provided the knowledge graphs of four additional stories; “The Devil's Foot,” “A Case of Identity,” “The Crooked Man” and “The Dancing Men.” The entrants can use all the five graphs or just one of them as their choice.
In Challenge 2019, a new Tool Category Prize will be awarded. In the Tool Category, submissions can be the development of a tool with which a task is partially solved.
For example: estimation of the criminal, review of the alibis, explanation of the motives, etc.
　　*Submissions of tool that supports knowledge graph construction by “triplification of natural language“ are also acceptable.
In Challenge 2020, we provided the knowledge graphs of three additional stories; "The Abbey Grange," "The Resident Patient," and "Silver Blaze."
In Challenge 2021, we held KGR4XAI: The 1st International Workshop on Knowledge Graph Reasoning for Explainable Artificial Intelligence co-located with 10th International Joint Conference on Knowledge Graphs (IJCKG 2021). In addition to this workshop, we held the 1st Knowledge Graph Reasoning Challenge for Students.
In Challenge 2022, we will hold the 1st International Knowledge Graph Reasoning Challenge.

Story-specific Tasks

The Speckled Band: Who killed Julia? (criminal & explanation)
The Devil's Foot: Who killed the victims? (criminal & explanation)
A Case of Identity: Why did the bridegroom disappear? (explanation)
The Crooked Man: Why did Barclay die? (explanation)
The Dancing Men: Break the codes (code breaking)
The Abbey Grange: Who killed Lord Blackenstall? (criminal & explanation)
The Resident Patient: Who killed Blessington? (criminal & explanation)
Silver Blaze: Who took out the White Silver Blaze? (criminal & explanation)

Task Execution Conditions

  • 1. Reasoning should be based on the given knowledge graphs, with various range of use:
    • a）Cover-all: use the whole knowledge graph
    • b）Cover-a-part (90%): use the triples from the fist ID up to 10 % before the last ID
    • c）Cover-a-part (75%): use the triples from the first ID up to 25% before the last ID
      (The reference ID numbers are to be published).
  • 2. You can extend the knowledge graphs by yourself (e.g. complement of common sense knowledge).
  • 3. Pull requests are welcome in case you find any error in the given knowledge graphs (e.g. typos, inconsistency, etc.).

Categories

  • (1) Main Category: development of a system that can accomplish the task of one or more subject stories
  • (2) Tool Category: development of a tool with which a task can be partially solved
    For example: estimation of the criminal, review of the alibis, explanation of the motives, etc.
    * Submissions of tool that supports knowledge graph construction by "triplification of natural language" are also acceptable.
  • (3) Idea Category: development of idea to realize (1) or (2) (no actual implementation is necessary)

Submissions

Provide the following information in line with the entry form (doc, pdf).
The Challenge consists of three categories: Main Category, Tool Category and Idea Category. To apply for Idea Category, submit (1), (2) and (4) below only.

  • (1) Personal profile
  • (2) Explanation of the reasoning/estimating process
    • -Showcase the main points using program logs, for example. For Idea Category entry, any format is acceptable.
    • -Provide to what extend (or up to which ID) you used the knowledge: show which knowledge you added for reasoning/estimating, if any.
      (Disclosure of the added knowledge is optional.)
  • (3) Program to execute 　
    • - Stand-alone application or service.
      (Submission of the source code is optional.)
  • (4) Whether or not to give the consent to share the submitted materials

-With the entrants’ consent, their entry forms and programs may be published on the website of the Knowledge Graph Reasoning Challenge*. Please give written consent in the entry form.

  • *Part of submitted materials may also be used in events related to Knowledge Graph Reasoning Challenge.
  • *Submit the completed entry form in PDF through e-mail. (Any software can be used to make a form)
    Email address：kgrc@knowledge-graph.jp

Evaluation Criteria (Details to be announced)

  • 1. Can the approach reproduce Holmes’s reasoning?
  • 2. Is the approach logically persuasive?
  • 3. How unique is the approach?
    • -way of reasoning
    • -extended knowledge
  • 4. How applicable or feasible is the approach?
    • -applicable to all the tasks vs. specific to one task
    • -good use of small amount of knowledge vs. needs for huge amount of knowledge
  • 5. How good is the idea presented?
    • -Examination of submissions and presentation
  • *The final review will include a peer review among the entrants.