Data Set

Data Set Publication

Table of Contents

  1. How to Use the Knowledge Graph and Videos
  2. Structure of the Knowledge Graph
  3. Sample SPARQL Queries

How to Use the Knowledge Graph and Videos


The knowledge graph is published in Resource Description Framework (RDF) format on GitHub. Corresponding videos are also available.

The Knowledge Graph is available in TTL format and in N-TRIPLE format for easy handling as a machine learning data set.

Videos: or in zip format:

Scene graph data for video is also available on Github in the ActionGenome format.

Viewing the Knowledge Graph in a Web Browser

You can view the knowledge graph in a web browser. It is available at, please refer to the GraphDB manual for more details.

SPARQL Endpoint

We provide an endpoint for searching the knowledge graph using SPARQL queries. Please refrain from making a large number of inquiries that could overload the server. Using as an API Parameters: query={URL-encoded SPARQL query}, format={data format (json, xml, csv, ...)}


Example Using Python (rdflib) in Google Colaboratory

Here is an example of using the above SPARQL endpoint with a program (Python) in Google Colab. The same applies to your own Jupyter notebook.

  1. Execution Example Using Python (rdflib)

Structure of the Knowledge Graph

The knowledge graph provided for the reasoning challenge is expressed with "events" and corresponding "actions," "main objects," "target objects," "agents," "relationships between scenes (nextEvent)," "times," etc. Please see the slides below for more details.

  • Schema (correspondence with video)
    Slide 1
  • Example of Knowledge Graph
    Slide 2
  • Slide 3

Sample SPARQL Queries

Retrieve events during the activity of "browsing the internet"

PREFIX vh2kg: 
  ex:browse_internet_scene1 vh2kg:hasEvent ?event .
  ?event vh2kg:action ?action .

Retrieve a list of activities

PREFIX vh2kg: 
  ?activity vh2kg:virtualHome ex:scene1 .