Institut für Kognitionswissenschaft

Institute of Cognitive Science


Osnabrück University navigation and search


Main content

Top content

Instantaneous visualization of gaze data, here show as yellow dots: (a) visualization of gaze data on a certain frame with VLC player — (b) like normal subtitles one can easily change between the gaze data of subjects — (c) exemplary gaze data on another video sequence.

Eye Tracking Data in Multimedia Containers for Instantaneous Visualizations

ETVIS16 - Demonstration Video and Datasets

Nowadays, the amount of gaze data records of subjects associated with video sequences increase daily. These eye tracking data are unfortunately stored in separate files in custom-made data formats, which reduces accessibility even for experts and makes the data effectively inaccessible for non-experts. Consequently, we still lack interfaces for many common use cases, such as visualization, streaming, data analysis, high level understanding, and semantic web integration of eye tracking data. To overcame these deficits, we want to promote the use of existing multimedia container formats to establish a standardized method of incorporating content videos with eye tracking metadata. This will facilitate instantaneous visualization in standard multimedia players, streaming via the Internet, and easy use without conversion. Using our prototype software, we embed gaze data from eye tracking studies and the corresponding video into a single multimedia container, which can be visualized by any media player. Based on this prototype implementation, we discuss the benefit of our approach as a possible standard for storing eye tracking metadata including the corresponding video.

Demonstration video showing instantaneous visualization of gaze data on a data set [K].

Source Code

VLC 3.0.0 patch

Modified version of the subsusf.c. Necessary for the playback of the USF files.

USF to ASS translation

XSL file for USF to ASS translation usf2ass.xsl.

Eye tracking data to USF converter

Currently, the eye tracking data to USF converter is only available as command line implementation. A version with a user interface is planned to release in the upcoming months. If you want to get the source code of the command line converter nevertheless, please write an email to Julius Schöning.

VLC Visual Analytics Plugins

To install the extensions under Linux, please copy the SimSub.lua and/or MergeSub.lua into ~/.local/share/vlc/lua/extensions/ for the current user or /usr/lib/vlc/lua/extensions/ for all users.

SimSub: Visualization of different eye tracking datasets in multiple windows
MergeSub: Visualization of different eye tracking datasets in a single window

Converted Eye Tracking Data Sets with Instantaneous Visualizations

The following data sets are provided for research purposes. By using these data sets in the proposed multimedia container format, please cite [S], [S1], [S2], or [S3] and also the original dataset [A], [B], [C], [K], or [R].

Coutrot & Guyader [C]

ID Images Stimulus Note
C1 01.png Video 1:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C2 02.png Video 2:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C3 03.png Video 3:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C4 04.png Video 4:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C5 05.png Video 5:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C6 06.png Video 6:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C7 07.png Video 7:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C8 04.png Video 8:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C9 09.png Video 9:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C10 10.png Video 10:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C11 11.png Video 11:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C12 12.png Video 12:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C13 13.png Video 13:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C14 14.png Video 14:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C15 15.png Video 15:
ASS
USF
Observers: 18 per all four auditory condition
Class: one moving object
C16 16.png Video 16:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C17 17.png Video 17:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C18 16.png Video 18:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C19 19.png Video 19:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C20 20.png Video 20:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C21 21.png Video 21:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C22 22.png Video 22:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C23 23.png Video 23:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C24 24.png Video 24:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C25 25.png Video 25:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C26 26.png Video 26:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C27 27.png Video 27:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C28 28.png Video 28:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C29 29.png Video 29:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C30 30.png Video 30:
ASS
USF
Observers: 18 per all four auditory condition
Class: several moving objects
C31 31.png Video 31:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C32 32.png Video 32:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C33 33.png Video 33:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C34 34.png Video 34:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C35 35.png Video 35:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C36 36.png Video 36:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C37 37.png Video 37:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C38 38.png Video 38:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C39 39.png Video 39:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C40 40.png Video 40:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C41 41.png Video 41:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C42 42.png Video 42:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C43 43.png Video 43:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C44 44.png Video 44:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C45 45.png Video 45:
ASS
USF
Observers: 18 per all four auditory condition
Class: landscapes
C46 46.png Video 46:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C47 47.png Video 47:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C48 48.png Video 48:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C49 49.png Video 49:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C50 50.png Video 50:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C51 51.png Video 51:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C52 52.png Video 52:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C53 53.png Video 53:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C54 54.png Video 54:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C55 55.png Video 55:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C56 56.png Video 56:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C57 57.png Video 57:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C58 58.png Video 58:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C59 59.png Video 59:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces
C60 60.png Video 60:
ASS
USF
Observers: 18 per all four auditory condition
Class: faces

Kurzhals et al. [K]

ID Images Stimulus Setting Task Induced Patterns
K1 01a.png Car Pursuit
(Rectangle)
ASS
USF
Panning camera follows a red car while it was going through a roundabout. Follow the red car. Potential smooth pursuit with long time spans of attentional synchrony on the red car.
01b.png Car Pursuit
(Polygon)
ASS
USF
K2 02.png Turning Car
(Rectangle)
ASS
USF
Camera follows turning car. The movement of the car describes the shape of an eight. Recognize the shape that is described by the movement of the car. Attentional synchrony on the car with potential smooth pursuit eye movement.
02.png Turning Car
(Polygon)
ASS
USF
K3 03.png Dialog
ASS
USF
Two persons talk to each other in front of the camera. Follow the dialog attentively. Switching focus between the faces of both persons. Label on shirt (right person) attracts additional attention.
K4 04.png Thimblerig
ASS
USF
A thimblerig with three cups and a marble. Find the cup with the marble. Attentional synchrony mainly on the cup with the marble.
K5 05.png Memory
ASS
USF
A 4x4 memory game. Pairwise flipping of cards is performed until all pairs are found. After one card is flipped, focus on the corresponding card of the pair. Increasing attention on matching cards after several turns and switching focus during the search.
K6 06.png UNO
ASS
USF
Two persons play UNO card game until the right player wins. For each player's turn, focus on the playable cards on the hand. Switching focus and attention mainly distributed between both hands and the stack of played cards.
K7 07.png Kite
ASS
USF
Person on a meadow steers a kite. The kite repeatedly leaves the field of view. Follow the flight path of the kite if possible. Smooth pursuit if the kite is visible. Otherwise, the participants either tried to estimate the position of the kite, or focused on the person.
K8 08.png Case-Exchange
ASS
USF
Various persons crossing the field of view while a text ribbon in the lower part is showing further information. Task is provided by the text ribbon: Look for metal case. Attentional synchrony on the text ribbon until the metal case appears and the task is readable.
K9 09.png Ball Game
ASS
USF
Three players with orange shirts and one player with a white shirt pass a ball around. Task group A: Count ball contacts of the white player. Task group B: Count passes between orange players. Attentional synchrony often on the ball, independent from the task.
K10 10.png Bag Search
ASS
USF
Various persons carrying different bags are crossing the field of view. Look for a specfic bag. Two groups: two different search targets, presented before the video started. Switching focus on new bags in the scene. Depending on the group, the search targets attract more attention.
K11 11.png Person Search
ASS
USF
People with different clothing cross the field of view. Task group A: Find the person with a hooded sweater. Task group B: Find the person with a red shirt and a headgear. Switching focus on new persons. After identification, search targets become less important than new persons.

Riche et al. [R]

ID Images Stimulus Note
R1 01.png Video 1:
ASS
USF
Observers: 10
Class: Surveillance
R2 02.png Video 2:
ASS
USF
Observers: 10
Class: Abnormal
R3 03.png Video 3:
ASS
USF
Observers: 10
Class: Surveillance
R4 04.png Video 4:
ASS
USF
Observers: 10
Class: Abnormal
R5 05.png Video 5:
ASS
USF
Observers: 10
Class: Surveillance
R6 06.png Video 6:
ASS
USF
Observers: 10
Class: Moving
R7 07.png Video 7:
ASS
USF
Observers: 10
Class: Noise
R8 08.png Video 8:
ASS
USF
Observers: 10
Class: Crowd
R9 09.png Video 9:
ASS
USF
Observers: 10
Class: Surveillance
R10 10.png Video 10:
ASS
USF
Observers: 10
Class: Crowd
R11 11.png Video 11:
ASS
USF
Observers: 10
Class: Noise
R12 12.png Video 12:
ASS
USF
Observers: 10
Class: Crowd
R13 13.png Video 13:
ASS
USF
Observers: 10
Class: Noise
R14 14.png Video 14:
ASS
USF
Observers: 10
Class: Crowd
R15 15.png Video 15:
ASS
USF
Observers: 10
Class: Noise
R16 16.png Video 16:
ASS
USF
Observers: 10
Class: Abnormal
R17 17.png Video 17:
ASS
USF
Observers: 10
Class: Noise
R18 18.png Video 18:
ASS
USF
Observers: 10
Class: Abnormal
R19 19.png Video 19:
ASS
USF
Observers: 10
Class: Moving
R20 20.png Video 20:
ASS
USF
Observers: 10
Class: Abnormal
R21 21.png Video 21:
ASS
USF
Observers: 10
Class: Crowd
R22 22.png Video 22:
ASS
USF
Observers: 10
Class: Moving
R23 23.png Video 23:
ASS
USF
Observers: 10
Class: Noise
R24 24.png Video 24:
ASS
USF
Observers: 10
Class: Moving

References

[S] J. Schöning, C. Gundler, G. Heidemann, P. König & U. Krumnack.
Visual Analytics of Gaze Data with Standard Multimedia Players.
Journal of Eye Movement Research, 10(5) : 1-14, 2017.
| DOI | URL | BibTeX
[S1] J. Schöning, P. Faion, G. Heidemann & U. Krumnack.
Eye Tracking Data in Multimedia Containers for Instantaneous Visualizations.
In IEEE VIS Workshop on Eye Tracking and Visualization (ETVIS), pages: 74-78, 2016. IEEE.
| PDF | DOI | URL | BibTeX
[C] A. Coutrot & N. Guyader.
How saliency, faces, and sound influence gaze in dynamic social scenes.
Journal of Vision, 14(8) : 5-5, 2014. Association for Research in Vision and Ophthalmology (ARVO).
| DOI | BibTeX
[K] K. Kurzhals, C.F. Bopp, J. Bässler, F. Ebinger & D. Weiskopf.
Benchmark data for evaluating visualization and analysis techniques for eye tracking for video stimuli.
In ACM Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), pages: 54-60, 2014. ACM Press.
| DOI | BibTeX
[R] N. Riche, M. Mancas, D. Culibrk, V. Crnojevic, B. Gosselin & T. Dutoit.
Dynamic Saliency Models and Human Attention: A Comparative Study on Videos.
Lecture Notes in Computer Science, pages: 586-598, ISBN: 978-3-642-37431-9, 2013. Springer Science + Business Media.
| DOI | BibTeX