Institut für Kognitionswissenschaft

Institute of Cognitive Science


Osnabrück University navigation and search


Main content

Top content

Providing Video Annotations in Multimedia Containers for Visualization and Research

WACV17 - Demonstration Video and Datasets

An ever increasing amount of video data sets which comprise additional meta data, such as labelled objects, tagged events, or gaze data become avaible every day. Unfortunately, these meta data are usually stored in separate files in custom-made data formats, which reduces accessibility even for experts and makes the data effectively inaccessible for non-experts. Within our paper as well as the demonstration video and example data sets on this website as supplementary material, we want to promote the use of existing multimedia container formats to establish a standardized method of incorporating content and meta data. This will facilitate visualization in standard multimedia players, streaming via the internet, and easy use without conversion. The demonstration video below compare the current methods and our new approaches.

This video show the difficulaty to visualze meta data of video data in comparision with our new approch of embedding meta data in multimedia containers.

Source Code

VLC 3.0.0 patch

Modified version of the subsusf.c. Necessary for the playback of the USF files.

USF to ASS translation

XSL file for USF to ASS translation usf2ass.xsl.

Eye tracking data to USF converter

Currently, the eye tracking data to USF converter is only available as command line implementation. A version with a user interface is planned to release in the upcoming months. If you want to get the source code of the command line converter nevertheless, please write an email to Julius Schöning.

VLC Visual Analytics Plugins

To install the extensions under Linux, please copy the SimSub.lua and/or MergeSub.lua into ~/.local/share/vlc/lua/extensions/ for the current user or /usr/lib/vlc/lua/extensions/ for all users.

SimSub: Visualization of different eye tracking datasets in multiple windows
MergeSub: Visualization of different eye tracking datasets in a single window

Converted Data Sets with Instantaneous Visualizations

The following data sets are provided for research purposes. By using these data sets in the proposed multimedia container format, please cite [S], [S1], [S2], or [S3] and also the original dataset [A], [B], [C], [K], or [R].

Kurzhals et al. [K]

ID Images Stimulus Setting Task Induced Patterns
K1 01a.png Car Pursuit
(Rectangle)
ASS
USF
Panning camera follows a red car while it was going through a roundabout. Follow the red car. Potential smooth pursuit with long time spans of attentional synchrony on the red car.
01b.png Car Pursuit
(Polygon)
ASS
USF
K2 02.png Turning Car
(Rectangle)
ASS
USF
Camera follows turning car. The movement of the car describes the shape of an eight. Recognize the shape that is described by the movement of the car. Attentional synchrony on the car with potential smooth pursuit eye movement.
02.png Turning Car
(Polygon)
ASS
USF
K3 03.png Dialog
ASS
USF
Two persons talk to each other in front of the camera. Follow the dialog attentively. Switching focus between the faces of both persons. Label on shirt (right person) attracts additional attention.
K4 04.png Thimblerig
ASS
USF
A thimblerig with three cups and a marble. Find the cup with the marble. Attentional synchrony mainly on the cup with the marble.
K5 05.png Memory
ASS
USF
A 4x4 memory game. Pairwise flipping of cards is performed until all pairs are found. After one card is flipped, focus on the corresponding card of the pair. Increasing attention on matching cards after several turns and switching focus during the search.
K6 06.png UNO
ASS
USF
Two persons play UNO card game until the right player wins. For each player's turn, focus on the playable cards on the hand. Switching focus and attention mainly distributed between both hands and the stack of played cards.
K7 07.png Kite
ASS
USF
Person on a meadow steers a kite. The kite repeatedly leaves the field of view. Follow the flight path of the kite if possible. Smooth pursuit if the kite is visible. Otherwise, the participants either tried to estimate the position of the kite, or focused on the person.
K8 08.png Case-Exchange
ASS
USF
Various persons crossing the field of view while a text ribbon in the lower part is showing further information. Task is provided by the text ribbon: Look for metal case. Attentional synchrony on the text ribbon until the metal case appears and the task is readable.
K9 09.png Ball Game
ASS
USF
Three players with orange shirts and one player with a white shirt pass a ball around. Task group A: Count ball contacts of the white player. Task group B: Count passes between orange players. Attentional synchrony often on the ball, independent from the task.
K10 10.png Bag Search
ASS
USF
Various persons carrying different bags are crossing the field of view. Look for a specfic bag. Two groups: two different search targets, presented before the video started. Switching focus on new bags in the scene. Depending on the group, the search targets attract more attention.
K11 11.png Person Search
ASS
USF
People with different clothing cross the field of view. Task group A: Find the person with a hooded sweater. Task group B: Find the person with a red shirt and a headgear. Switching focus on new persons. After identification, search targets become less important than new persons.

Benchmark Data [B]

ID Images Stimulus Annotated Objects
B1 01a.png Airplane
(Rectangle)
ASS
USF
6 Objects:
  • Yellow Plane
  • Blue Plane
  • Silver Plane
  • River
  • City left
  • City right
01b.png Airplane
(Polygon)
ASS
USF
B2 02a.png Alec Baldwin
(Rectangle)
ASS
USF
1 Object:
  • Alec Baldwin
02b.png Alec Baldwin
(Polygon)
ASS
USF
B3 03a.png Arctic Kayak
(Rectangle)
ASS
USF
9 Objects:
  • Blue Kayak
  • Red Kayak
  • Red Kayak 2
  • Man 1
  • Man 2
  • Man 3
  • Man 4
  • Man 5
  • Man 7
03b.png Arctic Kayak
(Polygon)
ASS
USF
B4 04a.png Dominoes
(Rectangle)
ASS
USF
8 Objects:
  • Man
  • Slab 1
  • Slab 2
  • Slab 3
  • Slab 4
  • Slab 5
  • Slab 6
  • Slab 7
04b.png Dominoes
(Polygon)
ASS
USF
B5 03a.png Avalanche
(Rectangle)
ASS
USF
3 Objects:
  • Avalanche
  • Mountain Near
  • Mountain Far
03b.png Avalanche
(Polygon)
ASS
USF
B6 03a.png Big Wheel
(Rectangle)
ASS
USF
3 Objects:
  • Tree
  • Accident Driver
  • Road
03b.png Big Wheel
(Polygon)
ASS
USF
B7 04a.png Campanile
(Rectangle)
ASS
USF
2 Objects:
  • High rise building
  • Tower
04b.png Campanile
(Polygon)
ASS
USF
B8 ... ... in progress ...
B... ... ... in progress ...

References

[S2] J. Schöning, P. Faion, G. Heidemann & U. Krumnack.
Providing Video Annotations in Multimedia Containers for Visualization and Research.
In IEEE Winter Conference on Applications of Computer Vision (WACV) 2017. IEEE.
| PDF | DOI | URL | BibTeX
[K] K. Kurzhals, C.F. Bopp, J. Bässler, F. Ebinger & D. Weiskopf.
Benchmark data for evaluating visualization and analysis techniques for eye tracking for video stimuli.
In ACM Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), pages: 54-60, 2014. ACM Press.
| DOI | BibTeX
[B] P. Sundberg, T. Brox, M. Maire, P. Arbelaez & J. Malik.
Occlusion boundary detection and figure/ground assignment from optical flow.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 2233-2240, 2011. IEEE.
| DOI | BibTeX