SolAR architecture

Introduction

Augmented Reality (AR) applications developers and end-users face a dilemma.

On one hand, major IT actors have released toolkits to develop AR applications. Nevertheless, they do not always meet the specific needs required by dedicated use cases or contextual environments (localization accuracy, lighting conditions, indoor/outdoor environments, tracking area range, dynamic scenes, etc.).

No solution fits all, and generally, these toolkits do not provide the level of tuning required to optimally adapt the vision based localization solution to the use case. Moreover, these closed solutions do not always ensure the confidentiality of data, and can store information concerning your environment (3D maps, key frames) or the augmentations (3D meshes, procedure scenarios) that could contain crucial intellectual property and private information.

On the other hand, open source vision libraries and SLAM implementations can generally be modified and configured to optimally meet AR applications requirements. However, many SLAM implementations generally developed by academic actors do not provide the license or the level of maturity required for the development of commercial solutions. Likewise, open source vision libraries offer a huge number of low-level functions but require a huge expertise and important development resources to obtain a usable camera pose estimation pipeline ready for commercial use.

To that end, SolAR offers an alternative to current commercial AR SDKs or existing open-source solutions, providing the benefits of both worlds – openness, ease of use, efficiency, adaptiveness. It aims at creating an ecosystem bringing researchers, developers, and end-users together to help the adoption of augmented reality.

The purpose of this documentation is to describe the architecture of the SolAR framework: the modules that compose this framework with their interfaces and data structures, the external libraries used, the links between all these elements and the way these dependences are managed. Then, the notion of AR pipeline will be presented through concrete examples. Finally, we show how to plug SolAR pipelines in Unity to create AR applications.

Presentation of SolAR Framework

SolAR is an open-source framework under Apache v2 licence dedicated to Augmented Reality.

SolAR is modular and evolutive. It allows, in various contexts of use, the construction of customized computer vision pipelines targeting Augmented Reality applications.

This framework offers a C++ SDK to easily and quickly develop and use custom solutions for camera pose estimation or 3D mapping. It provides developers with a full chain from low-level vision components development to camera pose estimation pipelines and AR service development.

SolAR contains:

  • interfaces promoting interoperability

  • computer vision components constructed from third-party bricks

  • plugins for third-party applications

According to user needs, SolAR framework offers several profiles:

  • SolAR pipeline user: use of prebuilt standalone or Unity integrated AR pipelines to design AR applications

  • SolAR pipeline assembler: assembly of components to build new customized AR pipelines

  • SolAR component developer: creation of customized bricks responding to specific needs

  • SolAR framework contributor: contribution to SolAR expansion

The SolAR Framework addresses the full chain of AR applications development related to computer vision:

SolAR Framework global process for AR application development
Figure 1. SolAR Framework global process for AR application development
  1. Component creation: SolAR defines a unique API for basic components required for computer vision pipelines (features detection, descriptors extraction, matching, Perspective N Points, homography, image filters, bundle adjustment, etc.). The SolAR community can implement new components compliant with the SolAR API, or wrap components available in third party libraries (e.g. OpenCV, Ceres, G2O, or Point Cloud Library).

  2. Component publication: A set of components are packaged into modules to ease their publication. SolAR modules, whether royalty free or under a commercial license, are stored on an artefact repository to be available to the SolAR community of pipeline assemblers.

  3. Vision pipeline assembling: Modules of components published by the SolAR community can be downloaded from the modules repositories. Then, the SolAR framework provides developers with a pipeline mechanism allowing them to assemble SolAR components to define their own vision pipelines such as a camera pose estimation solution.

  4. Vision pipeline publication: When a vision pipeline has been assembled and configured by default, it can be published in a repository of artefacts dedicated to SolAR vision pipelines to make it accessible to the SolAR pipeline users.

  5. AR service development: Thanks to a SolAR plugin for Unity, XR service developers can download SolAR pipelines stored on the dedicated artefact repository and integrate them in few clicks to their applications. They can simply develop AR applications as with any AR SDK and deploy them. Since SolAR is based on a unified interface, AR application developers will be able to easily make their applications evolve with the new solutions developed by the SolAR community.

The SolAR Framework is made up of several parts:

  • SolAR Core offers all features to create AR components (as a component developer) and to manage vision pipelines (as a pipeline assembler)

  • SolAR Components are concrete implementations of vision processing algorithms: each component can define its own parameters that will be used to fine tune the vision pipeline
    Also, a SolAR component of higher level can embed a low-level component based on a injection mechanism

  • SolAR Modules are shared libraries embedding a set of SolAR components to ease their management and publication

  • SolAR Pipelines are chains of connected components designed to process and analyze image data, step by step, and can be used to create AR applications (as a pipeline user)

  • Unity Plugin is dedicated to pipelines built to be used in Unity

The SolAR Framework uses several third-party dependencies, coming from b<>com or other parties, such as XPCF, Boost, spdlog, Eigen, which will be detailed in this document.

SolAR Framework global view
Figure 2. SolAR Framework global view

The SolAR initiative was launched by the b<>com Institute of Research and Technology, and is open to any contributors or users who share the SolAR goals.

SolAR encapsulation concepts

The SolAR Framework offers 3 levels of encapsulation to model:

The relationships between these 3 concepts can be represented like this:

Components Modules and Pipelines
Figure 3. Components, Modules and Pipelines

A pipeline corresponds to a sequence of functions provided by several components, which are embedded in different modules. Each module contains many components, and an assembler has to choose among all the proposed modules the components he needs to build his own pipeline.

To create and manage components, modules or pipelines, "best-practices" are presented on SolAR web site: http://www.solarframework.org/community/best_practices/

SolAR Components

SolAR components are elements embedding processing or storage capabilities. Components are designed to be connected together in order to create a full pipeline. For this reason, a component ideally defines one (as an exception several) processing function generally defining input(s) (the data to process) and ouptut(s) (the processed data). For interoperability purposes, a component must implement a SolAR component interface defined by the SolAR Framework API (see Framework API).

Each module implementation and interface is identified by a Universally Unique IDentifier (UUID) to nearly ensure the uniqueness of a component implementation when the system instantiates it.

As each component implementation can require dedicated configuration parameters, the SolAR framework provides an easy-to-use mechanism, based on XPCF tool, to initialize them at load-time with an external XML configuration file.

Finally, when a SolAR component is implemented, it has to be embedded in a SolAR module for its publication to the SolAR pipeline assemblers.

All this steps are discribed on the SolAR web site: http://www.solarframework.org/create/component/#create_component

SolAR Modules

A SolAR module consists of a shared library embedding a set of SolAR components as well as the corresponding factories to instantiate them. The management of modules is based on the XPCF tool, which provides the following features:

  • introspection to figure out which components are available in a module

  • separate implementation from interface to create the concrete implementation of a SolAR component and to bind it to an abstract SolAR and XPCF component interfaces

  • component creation

Each module implementation is identified with a Universally Unique IDentifier (UUID) to nearly ensure the uniqueness of the module when the system loads it.
As mentionned previoulsy, the third party tool XPCF can introspect the shared library of a module to obtain information about the embedded components and the interfaces they implement. As a result, the introspection of a shared library requires to load it, which could become tricky when the shared library has been built on a different platform (useful for authoring tools supporting cross-platform compilation such as Unity).

For this reason, XPCF proposes to associate to each module a registry file in xml format that describes the module with:

  • the module UUID

  • the components embedded in the module with their UUID

  • the abstract component interfaces implemented by the components

Thus, any system will be able to introspect a module without the need to load the corresponding shared library.

The SolAR module management is presented on the web site: http://www.solarframework.org/create/module/

Some modules are already provided with the SolAR Framework, and will be presented below.

SolARModuleCeres

This module is based on a third-party library called Ceres Solver.
Ceres Solver is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems.

Ceres Solver is licensed under the BSD license.

Ceres Solver web site: http://ceres-solver.org

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleCeres

SolARModuleFBOW

This module is based on a third-party library called FBOW.
FBOW (Fast Bag of Words) is an extremmely optimized version of the DBow2/DBow3, which are open source C++ libraries for indexing and converting images into a bag-of-word representation. The library is highly optimized to speed up the Bag of Words creation using AVX,SSE and MMX instructions. It requires OpenCV library.

FBOW is licensed under the MIT license.

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleFBOW

SolARModuleG2O

This module aggregates components dedicated to bundle adjustement optimization based on g2o library.
g2o is an open-source C++ framework for optimizing graph-based nonlinear error functions. g2o has been designed to be easily extensible to a wide range of problems and a new problem typically can be specified in a few lines of code. The current implementation provides solutions to several variants of SLAM and BA.

g2o is licenced under LGPL v3.

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleG2O

SolARModuleOpenCV and SolARModuleNonFreeOpenCV

These two SolAR modules are based on OpenCV (Open Source Computer Vision Library), which is a library of programming functions mainly aimed at real-time computer vision.

OpenCV has a modular structure, which means that the package includes several shared or static libraries. The following modules are available:

  • Core functionality (core) - a compact module defining basic data structures, including the dense multi-dimensional array Mat and basic functions used by all other modules.

  • Image Processing (imgproc) - an image processing module that includes linear and non-linear image filtering, geometrical image transformations (resize, affine and perspective warping, generic table-based remapping), color space conversion, histograms, and so on.

  • Video Analysis (video) - a video analysis module that includes motion estimation, background subtraction, and object tracking algorithms.

  • Camera Calibration and 3D Reconstruction (calib3d) - basic multiple-view geometry algorithms, single and stereo camera calibration, object pose estimation, stereo correspondence algorithms, and elements of 3D reconstruction.

  • 2D Features Framework (features2d) - salient feature detectors, descriptors, and descriptor matchers.

  • Object Detection (objdetect) - detection of objects and instances of the predefined classes (for example, faces, eyes, mugs, people, cars, and so on).

  • High-level GUI (highgui) - an easy-to-use interface to simple UI capabilities.

  • Video I/O (videoio) - an easy-to-use interface to video capturing and video codecs.

  • …​ some other helper modules, such as FLANN and Google test wrappers, Python bindings, and others.

The library is cross-platform and free for use under the open-source BSD license.

However, note that there are algorithms in OpenCV that have been patented and hence require a separate licence agreement if used for commercial purposes: that is the reason why a SolARModuleNonFreeOpenCV module has been designed, to group these "non free" part of the library.

OpenCV web site: https://opencv.org/

The SolARModuleOpenCV module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleOpenCV

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The SolARModuleNonFreeOpenCV module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleNonFreeOpenCV

SolARModuleOpenGL

This SolAR module is based on OpenGL (Open Graphics Library) which is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. The API is typically used to interact with a graphics processing unit (GPU), to achieve hardware-accelerated rendering.

The OpenGL specification describes an abstract API for drawing 2D and 3D graphics. Although it is possible for the API to be implemented entirely in software, it is designed to be implemented mostly or entirely in hardware.

OpenGL is is a Free Software License B.

OpenGL web site: https://www.opengl.org/

SolARModuleOpenGL also uses the freeglut library, which is a free-software/open-source alternative to the OpenGL Utility Toolkit (GLUT) library.

GLUT (and hence freeglut) takes care of all the system-specific chores required for creating windows, initializing OpenGL contexts, and handling input events, to allow for trully portable OpenGL programs.

freeglut is released under the MIT license.

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The SolARModuleOpenGL module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleOpenGL

SolARModuleOpenGV

This module is based on OpenGV (Open Geometric Vision), a C++ library designed to solve geometric computer vision problems. It contains efficient implementations of absolute-pose, relative-pose, triangulation, and point-cloud alignment methods for the calibrated case. All problems can be solved for central or non-central cameras, and embedded into a random sample consensus or nonlinear optimization scheme. The library is relying on the adapter pattern, and thus may easily be included into other projects. It furthermore contains a Matlab interface and a full benchmark kit for testing and comparing algorithms against each other.

OpenGV is released under the FreeBSD license.

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The SolARModuleOpenGV module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleOpenGV

SolARModulePCL

This module is based on PCL (Point Cloud Library), a C++ library designed to point cloud processing. The PCL framework contains numerous state-of-the art algorithms including filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation. These algorithms can be used, for example, to filter outliers from noisy data, stitch 3D point clouds together, segment relevant parts of a scene, extract keypoints and compute descriptors to recognize objects in the world based on their geometric appearance, and create surfaces from point clouds and visualize them. It requires Flann (Fast Library for Approximate Nearest Neighbors).

PCL is released under the terms of the BSD license and is open source software. It is free for commercial and research use.

PCL web site: https://pointclouds.org

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The SolARModulePCL module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModulePCL

SolARModuleRealSense

The SolAR Module RealSense is designed to use Intel® RealSense™ depth sensors throught its RealSense SDK. Intel® RealSense™ technologies offer a variety of vision‑based solutions designed to understand the world in 3D.

RealSense SDK is open-source and licensed under the Apache v2 licence.

RealSense SDK web site: https://dev.intelrealsense.com/docs

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The SolARModuleRealSense module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleRealSense

SolARModuleTools

This module implements some generic features used in vision processing, such as 2D or 3D transformations, matches between keypoints, homography checking, keyframe searching, 3D points filtering, etc.

SolARModuleTools is open-source, designed by b<>com, under Apache v2 licence.

Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/

The SolARModuleTools module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleTools

SolAR Pipelines

Definition

The SolAR framework has been designed to simplify the creation of vision pipelines adressing Augmented Reality applications.

Pipeline creation flow using SolAR Framework
Figure 4. Pipeline creation flow using SolAR Framework

A SolAR vision pipeline is a chain of connected vision processing components generally executed in parallel. For instance, a SolAR vision pipeline may take as input images captured by one or more cameras as well as data from an inertial sensor. These data will be processed by a set of SolAR components to estimate the pose of an AR device.

As a vision pipeline assembler, a SolAR user can easily:

  • download existing SolAR vision components created and published by the SolAR component creators

  • assemble the vision components to create a vision pipeline

  • update the vision pipeline architecture

  • swap components by other ones

  • configure components

  • test the vision pipeline

  • publish it for AR pipeline users who will develop AR services

To do that, the SolAR framework handles everything that will allow a modular assembling of a vision pipeline:

  • SolAR Data Structures define the information that flow in a pipeline and are exchanged between components

  • SolAR Component Interfaces define for now more than 50 abstract interfaces for the different categories of vision processing components required to implement vision pipelines

  • SolAR Components are concrete implementations of vision processing components compliant with SolAR components interfaces

  • SolAR Modules are shared libraries embedding a set of SolAR components to ease their management and publication

  • Component management based on XPCF

Here is a documentation on standalone C++ pipeline creation: http://www.solarframework.org/assemble/standalone_c_plus_plus/

A simple pipeline example: Natural Image Marker

The SolAR Natural Image Marker sample shows a SolAR pipeline for augmented reality based on a natural image. This pipeline loads a reference image marker, then tries to detect it on real time camera images and to estimate the pose of the camera in relation to the coordinate system of the image marker. If the marker is detected, the pipeline draws a 3D cube on its position, renders the 3D cube from a virtual camera which pose corresponds to the one estimated by the pipeline, and displays the rendered 3D cube over the current camera image.

Here is a screenshot example of the Natural Image Marker pipeline: