Institut für Kognitionswissenschaft

Institute of Cognitive Science


Osnabrück University navigation and search


Main content

Top content

Structure from Neural Networks (SfN²)

Abstract

For 3D reconstruction from a set of images or video frames, the Structure from Motion (SfM) approach is still the dominant method. The SfM reconstruction pipeline has become quite robust and accurate. Recently, artificial neural networks (ANNs) have been successfully applied for 3D reconstruction of a priori known object categories. We want to go beyond the restriction of pre-defined object categories by emulating SfM using of ANNs for reconstruction of arbitrary 3D objects. We have developed a synthetic scalable cube dataset for training, testing and validating ANNs. We will show that it is possible to reconstruct 3D shapes by ANNs, and we will benchmark three different ANN architectures inspired by the three most important strategies of SfM pipelines --- the global, hierarchical, and incremental key point matching. Along with this benchmark, we introduce our dataset and its generator and discuss how it can be used for capturing the complexity of real world images.

Dataset

  Cube Setup Link
voxelcubes voxelcubes voxelcubes voxelcubes ... voxelcubes 3x3x3
100 000 cubes
Download - jpg views + txt
Download - obj files
voxelcubes voxelcubes voxelcubes voxelcubes ... voxelcubes 4x4x4
300 000 cubes
Download - jpg views + txt
Download - obj files
voxelcubes voxelcubes voxelcubes voxelcubes ... voxelcubes 8x8x8
430 000 cubes
Download - jpg views + txt
Download - obj files
This work is licensed under a Creative Commons Attribution 4.0 International License.
For giving appropriate credit, cite one of the publication from the references below.
Creative Commons License

Generator Tools

Python Voxelizer

This pyhton script create voxelized objects incl. a voxel set list out of ply, off or stl 3D object files.

Download Python Voxelizer

Cube Generator

This generator, written in Matlab, randomly generates n 3D objects. Each such object is created by taking a unit cube in R^3 and subdividing it into a r x r x r subcubes. The parameter r can be defined by the user. By ensuring the uniqueness of the cube distribution in the voxel grid, this generator is able to generate 2^(r^3) different 3D objects and export them as 3D *.obj object files.

Download Cube Generator

Views Generator

This generator, written in Matlab, (can be optionally used for voxelization of 3D objects and) renders w input images with a pixel resolution x by x. Where the w different viewpoints are uniformly distributed around the object by using the Fibonacci lattice

Download Views Generator

Visualized Results

Visualization of the accuracy of the global, hierarchical, and incremental over all 1000 epochs on the training samples. Filter size of each convolutional layer = 32.

Visualization of the accuracy of the global, hierarchical, and incremental over all 1000 epochs on the validation samples. Filter size of each convolutional layer = 32.

Visualization of the accuracy of the global, hierarchical, and incremental over all 1000 epochs on the training samples. Filter size of each convolutional layer = 16.

Supplementary Plots and Network Architectures

Download supplementary Plots of the network performance and network architectures. PDF