Augmented Reality (AR) applications developers and end-users face a dilemma.
On one hand, major IT actors have released toolkits to develop AR applications. Nevertheless, they do not always meet the specific needs required by dedicated use cases or contextual environments (localization accuracy, lighting conditions, indoor/outdoor environments, tracking area range, dynamic scenes, etc.).
No solution fits all, and generally, these toolkits do not provide the level of tuning required to optimally adapt the vision based localization solution to the use case. Moreover, these closed solutions do not always ensure the confidentiality of data, and can store information concerning your environment (3D maps, key frames) or the augmentations (3D meshes, procedure scenarios) that could contain crucial intellectual property and private information.
On the other hand, open source vision libraries and SLAM implementations can generally be modified and configured to optimally meet AR applications requirements. However, many SLAM implementations generally developed by academic actors do not provide the license or the level of maturity required for the development of commercial solutions. Likewise, open source vision libraries offer a huge number of low-level functions but require a huge expertise and important development resources to obtain a usable camera pose estimation pipeline ready for commercial use.
To that end, SolAR offers an alternative to current commercial AR SDKs or existing open-source solutions, providing the benefits of both worlds – openness, ease of use, efficiency, adaptiveness. It aims at creating an ecosystem bringing researchers, developers, and end-users together to help the adoption of augmented reality.
SolAR is an open-source framework under Apache v2 licence dedicated to Augmented Reality.
SolAR is modular and evolutive. It allows, in various contexts of use, the construction of customized computer vision pipelines targeting Augmented Reality applications.
This framework offers a C++ SDK to easily and quickly develop and use custom solutions for camera pose estimation or 3D mapping. It provides developers with a full chain from low-level vision components development to camera pose estimation pipelines and AR service development.
SolAR contains:
interfaces promoting interoperability
computer vision components constructed from third-party bricks
plugins for third-party applications
According to user needs, SolAR framework offers several profiles:
SolAR pipeline user: use of prebuilt standalone or Unity integrated AR pipelines to design AR applications
SolAR pipeline assembler: assembly of components to build new customized AR pipelines
SolAR component developer: creation of customized bricks responding to specific needs
SolAR framework contributor: contribution to SolAR expansion
The SolAR Framework addresses the full chain of AR applications development related to computer vision:
Component creation: SolAR defines a unique API for basic components required for computer vision pipelines (features detection, descriptors extraction, matching, Perspective N Points, homography, image filters, bundle adjustment, etc.). The SolAR community can implement new components compliant with the SolAR API, or wrap components available in third party libraries (e.g. OpenCV, Ceres, G2O, or Point Cloud Library).
Component publication: A set of components are packaged into modules to ease their publication. SolAR modules, whether royalty free or under a commercial license, are stored on an artefact repository to be available to the SolAR community of pipeline assemblers.
Vision pipeline assembling: Modules of components published by the SolAR community can be downloaded from the modules repositories. Then, the SolAR framework provides developers with a pipeline mechanism allowing them to assemble SolAR components to define their own vision pipelines such as a camera pose estimation solution.
Vision pipeline publication: When a vision pipeline has been assembled and configured by default, it can be published in a repository of artefacts dedicated to SolAR vision pipelines to make it accessible to the SolAR pipeline users.
AR service development: Thanks to a SolAR plugin for Unity, XR service developers can download SolAR pipelines stored on the dedicated artefact repository and integrate them in few clicks to their applications. They can simply develop AR applications as with any AR SDK and deploy them. Since SolAR is based on a unified interface, AR application developers will be able to easily make their applications evolve with the new solutions developed by the SolAR community.
The SolAR Framework is made up of several parts:
SolAR Core offers all features to create AR components (as a component developer) and to manage vision pipelines (as a pipeline assembler)
SolAR Components are concrete implementations of vision processing algorithms: each component can define its own parameters that will be used to fine tune the vision pipeline
Also, a SolAR component of higher level can embed a low-level component based on a injection mechanism
SolAR Modules are shared libraries embedding a set of SolAR components to ease their management and publication
SolAR Pipelines are chains of connected components designed to process and analyze image data, step by step, and can be used to create AR applications (as a pipeline user)
Unity Plugin is dedicated to pipelines built to be used in Unity
The SolAR Framework uses several third-party dependencies, coming from b<>com or other parties, such as XPCF, Boost, spdlog, Eigen, which will be detailed in this document.
The SolAR initiative was launched by the b<>com Institute of Research and Technology, and is open to any contributors or users who share the SolAR goals.
SolAR web site: http://www.solarframework.org
The SolAR Framework offers 3 levels of encapsulation to model:
unitary vision processing: components
components organization: modules
vision processing chain: pipelines
The relationships between these 3 concepts can be represented like this:
A pipeline corresponds to a sequence of functions provided by several components, which are embedded in different modules. Each module contains many components, and an assembler has to choose among all the proposed modules the components he needs to build his own pipeline.
To create and manage components, modules or pipelines, "best-practices" are presented on SolAR web site: http://www.solarframework.org/community/best_practices/
SolAR components are elements embedding processing or storage capabilities. Components are designed to be connected together in order to create a full pipeline. For this reason, a component ideally defines one (as an exception several) processing function generally defining input(s) (the data to process) and ouptut(s) (the processed data). For interoperability purposes, a component must implement a SolAR component interface defined by the SolAR Framework API (see Framework API).
Each module implementation and interface is identified by a Universally Unique IDentifier (UUID) to nearly ensure the uniqueness of a component implementation when the system instantiates it.
As each component implementation can require dedicated configuration parameters, the SolAR framework provides an easy-to-use mechanism, based on XPCF tool, to initialize them at load-time with an external XML configuration file.
Finally, when a SolAR component is implemented, it has to be embedded in a SolAR module for its publication to the SolAR pipeline assemblers.
All this steps are discribed on the SolAR web site: http://www.solarframework.org/create/component/#create_component
A SolAR module consists of a shared library embedding a set of SolAR components as well as the corresponding factories to instantiate them. The management of modules is based on the XPCF tool, which provides the following features:
introspection to figure out which components are available in a module
separate implementation from interface to create the concrete implementation of a SolAR component and to bind it to an abstract SolAR and XPCF component interfaces
component creation
Each module implementation is identified with a Universally Unique IDentifier (UUID) to nearly ensure the uniqueness of the module when the system loads it.
As mentionned previoulsy, the third party tool XPCF can introspect the shared library of a module to obtain information about the embedded components and the interfaces they implement. As a result, the introspection of a shared library requires to load it, which could become tricky when the shared library has been built on a different platform (useful for authoring tools supporting cross-platform compilation such as Unity).
For this reason, XPCF proposes to associate to each module a registry file in xml format that describes the module with:
the module UUID
the components embedded in the module with their UUID
the abstract component interfaces implemented by the components
Thus, any system will be able to introspect a module without the need to load the corresponding shared library.
The SolAR module management is presented on the web site: http://www.solarframework.org/create/module/
Some modules are already provided with the SolAR Framework, and will be presented below.
This module is based on a third-party library called Ceres Solver.
Ceres Solver is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems.
Ceres Solver is licensed under the BSD license.
Ceres Solver web site: http://ceres-solver.org
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleCeres
This module is based on a third-party library called FBOW.
FBOW (Fast Bag of Words) is an extremmely optimized version of the DBow2/DBow3, which are open source C++ libraries for indexing and converting images into a bag-of-word representation. The library is highly optimized to speed up the Bag of Words creation using AVX,SSE and MMX instructions. It requires OpenCV library.
FBOW is licensed under the MIT license.
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleFBOW
This module aggregates components dedicated to bundle adjustement optimization based on g2o library.
g2o is an open-source C++ framework for optimizing graph-based nonlinear error functions. g2o has been designed to be easily extensible to a wide range of problems and a new problem typically can be specified in a few lines of code. The current implementation provides solutions to several variants of SLAM and BA.
g2o is licenced under LGPL v3.
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleG2O
These two SolAR modules are based on OpenCV (Open Source Computer Vision Library), which is a library of programming functions mainly aimed at real-time computer vision.
OpenCV has a modular structure, which means that the package includes several shared or static libraries. The following modules are available:
Core functionality (core) - a compact module defining basic data structures, including the dense multi-dimensional array Mat and basic functions used by all other modules.
Image Processing (imgproc) - an image processing module that includes linear and non-linear image filtering, geometrical image transformations (resize, affine and perspective warping, generic table-based remapping), color space conversion, histograms, and so on.
Video Analysis (video) - a video analysis module that includes motion estimation, background subtraction, and object tracking algorithms.
Camera Calibration and 3D Reconstruction (calib3d) - basic multiple-view geometry algorithms, single and stereo camera calibration, object pose estimation, stereo correspondence algorithms, and elements of 3D reconstruction.
2D Features Framework (features2d) - salient feature detectors, descriptors, and descriptor matchers.
Object Detection (objdetect) - detection of objects and instances of the predefined classes (for example, faces, eyes, mugs, people, cars, and so on).
High-level GUI (highgui) - an easy-to-use interface to simple UI capabilities.
Video I/O (videoio) - an easy-to-use interface to video capturing and video codecs.
… some other helper modules, such as FLANN and Google test wrappers, Python bindings, and others.
The library is cross-platform and free for use under the open-source BSD license.
However, note that there are algorithms in OpenCV that have been patented and hence require a separate licence agreement if used for commercial purposes: that is the reason why a SolARModuleNonFreeOpenCV module has been designed, to group these "non free" part of the library.
OpenCV web site: https://opencv.org/
The SolARModuleOpenCV module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleOpenCV
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The SolARModuleNonFreeOpenCV module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleNonFreeOpenCV
This SolAR module is based on OpenGL (Open Graphics Library) which is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. The API is typically used to interact with a graphics processing unit (GPU), to achieve hardware-accelerated rendering.
The OpenGL specification describes an abstract API for drawing 2D and 3D graphics. Although it is possible for the API to be implemented entirely in software, it is designed to be implemented mostly or entirely in hardware.
OpenGL is is a Free Software License B.
OpenGL web site: https://www.opengl.org/
SolARModuleOpenGL also uses the freeglut library, which is a free-software/open-source alternative to the OpenGL Utility Toolkit (GLUT) library.
GLUT (and hence freeglut) takes care of all the system-specific chores required for creating windows, initializing OpenGL contexts, and handling input events, to allow for trully portable OpenGL programs.
freeglut is released under the MIT license.
freeglut web site: http://freeglut.sourceforge.net/
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The SolARModuleOpenGL module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleOpenGL
This module is based on OpenGV (Open Geometric Vision), a C++ library designed to solve geometric computer vision problems. It contains efficient implementations of absolute-pose, relative-pose, triangulation, and point-cloud alignment methods for the calibrated case. All problems can be solved for central or non-central cameras, and embedded into a random sample consensus or nonlinear optimization scheme. The library is relying on the adapter pattern, and thus may easily be included into other projects. It furthermore contains a Matlab interface and a full benchmark kit for testing and comparing algorithms against each other.
OpenGV is released under the FreeBSD license.
OpenGV web site: http://laurentkneip.github.io/opengv/
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The SolARModuleOpenGV module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleOpenGV
This module is based on PCL (Point Cloud Library), a C++ library designed to point cloud processing. The PCL framework contains numerous state-of-the art algorithms including filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation. These algorithms can be used, for example, to filter outliers from noisy data, stitch 3D point clouds together, segment relevant parts of a scene, extract keypoints and compute descriptors to recognize objects in the world based on their geometric appearance, and create surfaces from point clouds and visualize them. It requires Flann (Fast Library for Approximate Nearest Neighbors).
PCL is released under the terms of the BSD license and is open source software. It is free for commercial and research use.
PCL web site: https://pointclouds.org
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The SolARModulePCL module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModulePCL
The SolAR Module RealSense is designed to use Intel® RealSense™ depth sensors throught its RealSense SDK. Intel® RealSense™ technologies offer a variety of vision‑based solutions designed to understand the world in 3D.
RealSense SDK is open-source and licensed under the Apache v2 licence.
RealSense SDK web site: https://dev.intelrealsense.com/docs
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The SolARModuleRealSense module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleRealSense
This module implements some generic features used in vision processing, such as 2D or 3D transformations, matches between keypoints, homography checking, keyframe searching, 3D points filtering, etc.
SolARModuleTools is open-source, designed by b<>com, under Apache v2 licence.
Its API is documented on the following link: http://www.solarframework.org/assemble/moduleapi/
The SolARModuleTools module is available on SolAR Git Hub: https://github.com/SolarFramework/SolARModuleTools
The SolAR framework has been designed to simplify the creation of vision pipelines adressing Augmented Reality applications.
A SolAR vision pipeline is a chain of connected vision processing components generally executed in parallel. For instance, a SolAR vision pipeline may take as input images captured by one or more cameras as well as data from an inertial sensor. These data will be processed by a set of SolAR components to estimate the pose of an AR device.
As a vision pipeline assembler, a SolAR user can easily:
download existing SolAR vision components created and published by the SolAR component creators
assemble the vision components to create a vision pipeline
update the vision pipeline architecture
swap components by other ones
configure components
test the vision pipeline
publish it for AR pipeline users who will develop AR services
To do that, the SolAR framework handles everything that will allow a modular assembling of a vision pipeline:
SolAR Data Structures define the information that flow in a pipeline and are exchanged between components
SolAR Component Interfaces define for now more than 50 abstract interfaces for the different categories of vision processing components required to implement vision pipelines
SolAR Components are concrete implementations of vision processing components compliant with SolAR components interfaces
SolAR Modules are shared libraries embedding a set of SolAR components to ease their management and publication
Component management based on XPCF
Here is a documentation on standalone C++ pipeline creation: http://www.solarframework.org/assemble/standalone_c_plus_plus/
The SolAR Natural Image Marker sample shows a SolAR pipeline for augmented reality based on a natural image. This pipeline loads a reference image marker, then tries to detect it on real time camera images and to estimate the pose of the camera in relation to the coordinate system of the image marker. If the marker is detected, the pipeline draws a 3D cube on its position, renders the 3D cube from a virtual camera which pose corresponds to the one estimated by the pipeline, and displays the rendered 3D cube over the current camera image.
Here is a screenshot example of the Natural Image Marker pipeline:
To perform this vision pipeline, the Natural Image Marker project uses several SolAR modules and some of their components:
SolARModuleOpenCV:
SolARCameraOpencv
SolARMarker2DNaturalImageOpencv
SolARKeypointDetectorOpencv
SolARKeypointDetectorRegionOpencv
SolARDescriptorsExtractorAKAZE2Opencv
SolARDescriptorMatcherKNNOpencv
SolARGeometricMatchesFilterOpencv
SolARPoseEstimationPlanarPointsOpencv
SolAROpticalFlowPyrLKOpencv
SolARProjectOpencv
SolARUnprojectPlanarPointsOpencv
SolAR2DOverlayOpencv
SolARSideBySideOverlayOpencv
SolAR3DOverlayOpencv
SolARImageViewerOpencv
SolARModuleTools
SolARBasicMatchesFilter
SolARKeypointsReIndexer
SolARImage2WorldMapper4Marker2D
To manage SolAR modules and components needed to process this pipeline (definitions, interfaces, configurations, bindings, etc.), XPCF uses a dedicated XML file ('conf_NaturalImageMarker.xml' for this project).
Here is an extract of this file:
Install
Use
Assemble
Create
Community
Documentation
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<xpcf-registry autoAlias="true" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<module uuid="15e1990b-86b2-445c-8194-0cbe80ede970" name="SolARModuleOpenCV" path="$REMAKEN_PKG_ROOT/packages/SolARBuild/win-cl-14.1/SolARModuleOpenCV/0.7.0/lib/x86_64/shared" description="moduleOpenCV">
<component uuid="5B7396F4-A804-4F3C-A0EB-FB1D56042BB4" name="SolARCameraOpencv" description="SolARCameraOpencv">
<interface uuid="125f2007-1bf9-421d-9367-fbdc1210d006" name="IComponentIntrospect" description="IComponentIntrospect"/> +
<interface uuid="5DDC7DF0-8377-437F-9C81-3643F7676A5B" name="ICamera" description="ICamera"/>
</component>
<component uuid="efcdb590-c570-11e7-abc4-cec278b6b50a" name="SolARMarker2DNaturalImageOpencv" description="SolARMarker2DNaturalImageOpencv">
<interface uuid="125f2007-1bf9-421d-9367-fbdc1210d006" name="IComponentIntrospect" description="IComponentIntrospect"/> +
<interface uuid="3c9cee8a-e9ca-4c16-851a-669a94c2a68d" name="IMarker" description="IMarker"/> +
<interface uuid="e9cdcf6e-c54c-11e7-abc4-cec278b6b50a" name="IMarker2Dquared" description="IMarker2Dquared"/> +
<interface uuid="8fed06f8-c54d-11e7-abc4-cec278b6b50a" name="IMarker2DNaturalImage" description="IMarker2DNaturalImage"/>
</component>
...
</module>
<factory>
<bindings>
<bind interface="ICamera" to="SolARCameraOpencv" />
</bindings>
</factory>
<properties>
<configure component="SolARCameraOpencv">
<property name="calibrationFile" type="string" value="camera_calibration.yml"/>
<property name="deviceID" type="UnsignedInteger" value="0"/>
</configure>
<configure component="SolARMarker2DNaturalImageOpencv">
<property name="filePath" type="string" value="grafMarker.yml"/>
</configure>
<configure component="SolARKeypointDetectorOpencv">
<property name="type" type="string" value="AKAZE2"/>
<property name="imageRatio" type="Float" value="0.5"/>
<property name="nbDescriptors" type="Integer" value="1000"/>
</configure>
...This XML file is used to:
declare the modules used by the pipeline and, for each module, the components required with their interfaces
bind components to specific interfaces
define properties of each component
The following diagram shows the chaining between the components for the pose estimation based on planar points:
And here is the detailed sequence diagrams that shows, step by step, data exchange between the components, and the main features processed by each component:
These pipeline sample is available on GitHub: https://github.com/SolarFramework/Sample-NaturalImageMarker/tree/master/SolARSample_NaturalImageMarker_Mono
For AR applications, otimizing latency time is essential to deliver a better end-user experience. So, each pipeline involved in an AR application should optimize its processings to provide the resulting data in minimum time. To this end, the SolAR Framework makes possible to implement pipelines by paralleling the processing of components, when possible.
The multithreaded Natural Image Marker pipeline exemple provides the same feature as the pipeline detailed in the previous paragraph: it loads a reference image marker, then tries to detect it on real time camera images and to estimate the pose of the camera in relation to the coordinate system of the image marker.
To perform this vision pipeline, this project uses the same SolAR modules and components than those listed previously. So, the XML file used by XPCF to manage modules and components is exactly the same ("conf_NaturalImageMarker_Multi.xml").
The main difference between the multithreaded pipeline and the "simple" one (see paragraph A simple pipeline example: Natural Image Marker) lies in the use of components:
In the "simple" pipeline example, the components are called sequentially, the data produced by one component being used as input data for the next one (as you can see on this diagram).
In the multithreaded pipeline, the vision process is divided into several tasks that run in parallel and exchange the data they produce through shared buffers.
The tasks identified to process the Natural Image Marker pipeline are:
camera image capture: gets an image from the camera
marker detection: tries to detect the reference image (the natural image marker) in a camera image and to determine the camera pose
display: draws a 3D cube on marker position, renders the 3D cube from a virtual camera which pose corresponds to the one estimated by the pipeline, and displays the rendered 3D cube over the current camera image
These three tasks are started simultaneously and run in parallel for the duration of the pipeline process, by executing their processing in a loop. They use asynchronous shared buffers to get the data they need for their processing (from other tasks) and to make the data they produce available for other tasks.
The following diagram shows the chaining between the components inside each task, and the data exchanged asynchronously between tasks through buffers:
To implement the notions of task and shared buffer, SolAR pipelines use parallel helpers provided by XPCF:
DelegateTask: task that executes a function in a loop, in its own thread
example of DelegateTask declaration:
xpcf::DelegateTask taskCamImageCapture(camImageCapture);example of loop function declaration:
std::function<void(void)> camImageCapture = [&stop,camera,&m_dropBufferCamImageForDetection, &m_dropBufferCamImageForTracking](){ ...example of starting a DelegateTask instance:
taskCamImageCapture.start();example of stopping a DelegateTask instance:
taskCamImageCapture.stop();DropBuffer: asynchronous shared buffer which can contain a single object or a pair, a tuple of objects
examples of DropBuffer declaration:
xpcf::DropBuffer<SRef<Image>> m_dropBufferCamImageForDetection;
xpcf::DropBuffer<std::tuple<SRef<Image>, Transform3Df, bool>> m_dropBufferPoseForDisplay;examples of data storage in a DropBuffer:
m_dropBufferCamImageForDetection.push(camImage);
m_dropBufferPoseForDisplay.push(std::make_tuple(camImage, Transform3Df::Identity(), false));examples of data recovery from a DropBuffer:
m_dropBufferCamImageForDetection.tryPop(camImage);
m_dropBufferPoseForDisplay.tryPop(resultFromDetection);These pipeline sample is available on GitHub: https://github.com/SolarFramework/Sample-NaturalImageMarker/tree/master/SolARSample_NaturalImageMarker_Multi
The SolAR Data Structure define the information that flows in a pipeline and are exchanged between components. These information are used to connect components.
Data Structure classes are designed to model all kinds of data needed in a vision pipeline, such as:
camera parameters
generic descriptors, set of descriptors, match between 2 descriptors
2D geometric structures: point, edge, rectangle, size
3D geometric strutures: point, map of 3D points, cloud of 3D points
frame, key frame
2D image
etc.
All of the Data Structure classes are described on the SolAR web site: http://www.solarframework.org/create/api/
The Data Structure code is available on the SolAR GitHub repository: https://github.com/SolarFramework/SolARFramework/tree/master/interfaces/datastructure
The SolAR Framework API has been designed to avoid interoperability issues between components. For this purpose, any component has to implement this API, which can be considered as the SolAR Component Interface.
This API is split into several parts:
'display': to draw 2D information, 3D contents, to display point clouds, images, etc.
'features': offers different functions related to visual features (e.g. points or lines) such as detection of keypoints from an image or a region of an image, extraction of descriptors from a set of keypoints, matches between sets of descriptors, extraction of contours from an image, a squared binary pattern or an image of squared binary pattern, and so on…
'fusion': to fuse different kinds of data such as inertial and visual data
'geom': a complete set of geometrical functions such as 2D or 3D transformations of set of points, projection of 3D points on 2D image plane, undistorsion applying to set of points, etc.
'image': functions dedicated to images (conversion, filtering, loading, warping and cropping…)
'input': to access devices (camera calibration and images capture, data of IMU) or files (to load different kinds of markers)
'loop': to detect a loop closure from a given keyframe, to optimize a system of 3D points and keyframes from a loop closing detection
'pipeline': to define a global pose estimation pipeline
'pointCloud': provides several filters to prune a point cloud
'reloc': to retrieve a set of pre-recorded keyframes close to a given frame, or to get a camera pose giving a frame
'sink': to store a synchronized pose and image from the pipeline to make it available to a third party application, and to update a texture buffer with a new image
'slam': functions dedicated to SLAM processing (SLAM initialization using image stream of a camera, mapping, tracking)
'solver': functions for modeling and solving large, complicated optimization problems, whether related to a 3D map (e.g. 3D map filtering, 3D mapping and triangulation), to camera pose (e.g. pose estimation from 2D-2D, 2D-3D or 3D-3D correspondences or from a marker), or to both (e.g. bundle adjustment)
'source': to set a new image coming from a third party
'storage': to store the covisibility graph between keyframes, a set of keyframes or a point cloud (to share persistent data between processing components)
'tracking': to estimate the optical flow between two images
This API is detailed on the SolAR web site: http://www.solarframework.org/create/api/
The API code is available on the SolAR GitHub repository: https://github.com/SolarFramework/SolARFramework/tree/master/interfaces/api
This part of the SolAR Framework is currently used to run SolAR pipelines in the Unity environment. But more generally, its goal is to manage pipelines in different environments, Unity being one of them. It does not offer an access to the whole SolAR API, but just to the API allowing to load and initialize an existing pipeline implementing the IPipeline interface, to start and stop it, to optionally feed it with images, and to get access to the pose of the camera estimated by the pipeline.
As Unity only supports the C# language, the Pipeline Manager must first use the SolAR Wrapper (see the next paragraph) to wrap the C++ code of a pipeline in C#. Then, this manager will take the right interfaces to make this pipeline compatible with the Unity environment.
The Pipeline Manager code is available on the SolAR GitHub repository: https://github.com/SolarFramework/SolARPipelineManager
This wrapper is made of several SWIG scripts which allow to use the whole SolAR API in languages other than C++, such as C# for Unity environment. But we can imagine other languages like Java, Python…
The Simplified Wrapper and Interface Generator (SWIG) is an open-source software tool used to connect computer programs or libraries written in C or C++ with scripting languages such as Lua, Perl, PHP, Python, R, Ruby, Tcl, and other languages like C#, Java, JavaScript, Go, D, OCaml, Octave, Scilab and Scheme. Output can also be in the form of XML.
The export of SolAR pipelines to Unity is described here: http://www.solarframework.org/assemble/pipeline/#export_your_pipeline_for_unity
The SolAR Wrapper project is available on the SolAR Git Hub repository: https://github.com/SolarFramework/SwigWrapper
To process data transformations, geometric operations, component, module and pipeline management, and so on, the SolAR Framework needs some external tools or libraries described below.
The Cross-Platform Component Framework (XPCF) is a lightweight framework designed to provide dependency injection, with clarity, simplicity and light code.
This framework offers several major features:
an "in-code" dependency injection factory to create components and bind them to a specific interface
a component abstract factory to load components at runtime from modules (named ModuleManager)
increase modularity with better separation of concerns through interface discovery (using component introspection)
a centralized description of modules, components and interfaces (with a dedicated configuration file)
module introspection to figure out the components contained and the interfaces they implement
secure memory management using smart pointers for components (prevents from memory leaks, dangling null pointer)
component and service configuration at creation through abstract variant parameters
support for both components and services
support for Domain Specific Language through the "IFeatures" interface (for instance, a component can declare "opencv" and "cuda" features as it is an opencv/cuda accelerated implementation of a keypointdetector)
XPCF defines some important concepts that will be used in the SolAR Framework, such as:
Module: a shared library hosting a set of factories to create components/services instances
Component: a class that can have several instances over the application lifetime, each instance is unloaded (destroyed) when no more reference exists upon one of its interfaces
Service: a component singleton that only has one instance at any given time, and is never unloaded
The component, module and pipeline management of the SolAR Framework is based on XPCF:
it provides an "in-code" dependency injection factory to create concrete component instances and bind them to abstract SolAR component interfaces
it handles the loading of modules and components at runtime
it provides interfaces to load pipelines, used modules and components at run-time, etc.
All these mechanisms are described in the chapters dedicated to SolAR Components, Modules and Pipelines.
XPCF is an open-source framework designed by b<>com, under Apache v2 licence.
XPCF presentation and code are available on this Git Hub repository: https://github.com/b-com-software-basis/xpcf
Boost is a set of libraries for the C++ programming language that provides support for tasks and structures such as linear algebra, pseudorandom number generation, multithreading, image processing, regular expressions, serialization and unit testing.
Most of the Boost libraries are licensed under the Boost Software License, designed to allow Boost to be used with both free and proprietary software projects. Many of Boost’s founders are on the C++ standards committee, and several Boost libraries have been accepted for incorporation into the C\+ Technical Report 1, the C+11 standard (e.g. smart pointers, thread, regex, random, ratio, tuple) and the C++17 standard (e.g. filesystem, any, optional, variant, string_view).
Boost web site: https://www.boost.org/
spdlog is a C++ logging library. Its main features are:
Very fast – performance is the primary goal
Headers only
No dependencies
Cross platform - Linux / Windows on 32/64 bits
Multi/Single threaded loggers
Rotating log files
and more…
spdlog is licensed under the MIT license.
spdlog GitHub repository: https://github.com/gabime/spdlog
Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers, and related algorithms.
Eigen is Free Software. Starting from the 3.1.1 version, it is licensed under the MPL2, which is a simple weak copyleft license.
Eigen web site: http://eigen.tuxfamily.org
The SolAR framework is using the meta dependencies management tool Remaken supporting Conan, vcpkg and its native C++ packaging structure based on pkg-config description files.
Remaken relies on other tools and package managers, as shown on this diagram:
Remaken is a meta dependencies management tool which supports different package formats.
Cross platforms packaging systems:
Conan
vcpkg
Remaken (pkg-config based)
Dedicated system tools:
apt-get (debian and derivatives)
pacman (archlinux)
yum (redhat, fedora, centos)
zypper (openSUSE)
pkg (freebsd)
pkgutil (solaris)
Homebrew (mac OS X)
chocolatey (windows)
Remaken uses a packagedependencies.txt file defining all dependencies of a project. Thus, by maintaining a simple dependency file describing the third parties used by a project, it will be easier to download the dependencies, link and build this project, and install and deploy the solution with all its dependencies in a very simple way.
For each project, a packagedependencies.txt file can be created in the root project folder. Each line of this file must follow the pattern:
framework#channel | version | [condition]#library name | identifier@repository_type | repository_url | link_mode | optionsFor example:
opencv | 3.4.3 | opencv | thirdParties@github | https://github.com/SolarFramework/binaries/releases/download
xpcf | 2.1.0 | xpcf | bcomBuild@github | https://github.com/SolarFramework/binaries/releases/download | static
spdlog | 0.14.0 | spdlog | bincrafters@conan | conan-center | na
freeglut#testing | 3.0.0 | freeglut | user@conan | https://github.com/SolarFramework/binaries/releases/downloadgithub, artifactory, nexus and path dependencies are installed using remaken packaging format through an url or filesystem repository.
System dependencies are installed using operating system dependent package manager (apt for linux debian and derivatives, Homebrew for Mac OS X, chocolatey for windows…).
Conan dependencies are installed using packaging format with Conan package manager.
vcpkg dependencies are installed using vcpkg packaging format with vcpkg package manager.
Remaken is an open-source framework designed by b<>com, under Apache v2 licence.
The Remaken tool and its documentation are available on Git Hub repository: https://github.com/b-com-software-basis/remaken
pkg-config is a tool that defines and supports a unified interface for querying installed libraries for the purpose of compiling software that depends on them. It allows programmers and installation scripts to work without explicit knowledge of detailed library path information.
The primary use of pkg-config is to provide the necessary details for compiling and linking a program to a library. This metadata is stored in pkg-config files. These files have the suffix '.pc' and reside in specific locations known to the pkg-config tool.
The '.pc' file format contains predefined metadata keywords and freeform variables. For example, here is the SolAR Framework configuration file:
libname=SolARFramework
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/interfaces
Name: SolARFramework
Description:
Version: 0.7.0
Requires:
Libs: -L${libdir} -l${libname}
Libs.private: ${libdir}/${pfx}${libname}.${lext}
Cflags: -I${includedir}Here is a short description of the keyword fields:
Name: A human-readable name for the library or package. This does not affect usage of the pkg-config tool, which uses the name of the .pc file.
Description: A brief description of the package.
URL: An URL where people can get more information about and download the package.
Version: A string specifically defining the version of the package.
Requires: A list of packages required by this package. The versions of these packages may be specified using the comparison operators '=', '<', '>', '⇐' or '>='.
Requires.private: A list of private packages required by this package but not exposed to applications. The version specific rules from the 'Requires' field also apply here.
Conflicts: An optional field describing packages that this one conflicts with. The version specific rules from the 'Requires' field also apply here. This field also takes multiple instances of the same package. E.g., 'Conflicts: bar < 1.2.3, bar >= 1.3.0'.
Cflags: The compiler flags specific to this package and any required libraries that don’t support pkg-config. If the required libraries support pkg-config, they should be added to 'Requires' or 'Requires.private'.
Libs: The link flags specific to this package and any required libraries that don’t support pkg-config. The same rule as 'Cflag’s applies here.
Libs.private: The link flags for private libraries required by this package but not exposed to applications. The same rule as 'Cflags' applies here.
pkg-config is licenced under GNU GPL.
pkg-config web site: https://www.freedesktop.org/wiki/Software/pkg-config/
Conan is a portable package manager, intended for C and C++ developers, but it is able to manage builds from source, dependencies, and precompiled binaries for any language.
Conan is a decentralized package manager with a client-server architecture. This means that clients can fetch packages from, as well as upload packages to, different servers (“remotes”), similar to the “git” push-pull model to/from git remotes.
One of the most powerful features of Conan is that it can create and manage pre-compiled binaries for any possible platform and configuration. Using pre-compiled binaries and avoiding repeatedly building from source, save a lot of time to developers and Continuous Integration servers, while also improving the reproducibility and traceability of artifacts.
Installation of packages from servers is also very efficient. Only the necessary binaries for the current platform and configuration are downloaded, not all of them. If the compatible binary is not available, the package will be built from sources in the client. This package, built for a specific configuration, can then be uploaded on the server to make it available to other developers.
Conan works with any build system too. There are built-in integrations with most popular ones, like CMake, Visual Studio (MSBuild), Autotools and Makefiles, SCons, etc. But it is not a requirement to use any of them. It is not even necessary that all packages use the same build system, every package can use their own build system, and depend on other packages using different build systems. It is also possible to integrate with any build system, including proprietary ones.
SolAR has its own Conan remote server to store recipes corresponding to binaries related to SolAR third parties and modules. Thus, binaries used by SolAR on the recommended platforms can be directly downloaded without the need to build them on the client. The “conan-solar” remote server is accessible at the following URL: https://repository.solarframework.org/conan
Conan web site: https://conan.io/
Conan is Free and Open Source, with a permissive MIT license.
vcpkg is a cross-platform open source package manager by 'Microsoft', for 'Windows', 'Linux' and 'MacOS'. It provides access to C and C++ libraries to its supported platforms.
Vcpkg is licensed under MIT license.
Vcpkg is available here: https://github.com/microsoft/vcpkg
A package manager is a software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer’s operating system in a consistent manner.
A package manager deals with packages, distributions of software and data in archive files. Packages contain metadata, such as the software’s name, description of its purpose, version number, vendor, checksum (preferably a cryptographic hash function), and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database. Package managers typically maintain a database of software dependencies and version information to prevent software mismatches and missing prerequisites. They work closely with software repositories, binary repository managers, and app stores.
Package managers are designed to eliminate the need for manual installs and updates.
Package managers currently used by the SolAR Frame work are: - Chocolatey (for 'Windows'): https://chocolatey.org/ - Homebrew (for 'MacOS'): https://brew.sh/ - APT (for 'Linux' distributions such as 'Debian' or 'Ubuntu'): https://en.wikipedia.org/wiki/APT_(software) - yum(for 'Linux' distributions such as 'Fedora', 'CentOS', 'Red Hat'): http://yum.baseurl.org/
To manage libraries dependencies during the build process, SolAR projects use builddefs-qmake: a set of cross-platform rules based on qmake used to provide homogeneous C++ builds, ensuring build rules for shared libraries and executables are the same along the builds. It supports either dynamic or static libraries builds.
These rules are defined in a set of qmake project include files (".pri" files), and are used for different purposes. These rules can be installed by remaken by running "remaken init" command (with option "–tag" if a specific version of the rules is expected).
First, the rules can set the include and link flags based on the packagedependencies.txt file for static (DEPENDENCIESCONFIG = static) and dynamic (DEPENDENCIESCONFIG = shared) dependencies. It also supports recursive search of dependencies (DEPENDENCIESCONFIG = recursive). This feature avoids the need for the developer to maintain the flags in the project file.
Secondly, it can handle the installation in a dedicated folder (INSTALLSUBDIR) of the resulting binaries (DEPENDENCIESCONFIG = install), and can also install all its dependencies in a recursive way (DEPENDENCIESCONFIG = install_recurse). The dependencies ignored during installation can be defined in a dedicated file (packageignoreinstall.txt).
Finally, if the dependencies are installed thanks to Conan, the rules will automatically download them to your machine, and even build them if the binaries are not available on the Conan remote server for the requested configuration.
Also, a specific flag called QTVS allows the loading of the qmake projects in Visual Studio with the plugin QT Visual Studio Tools.
To sum up, here is the global sequence diagram of SolAR dependency management:
To assemble C++ pipelines for Unity, SolAR Framework provides a dedicated plugin. The SolAR plugin for Unity is a simple interface to load and run in Unity environment any SolAR pipelines and use them to design AR services with Unity. Integration of SolAR pipelines in Unity Engine gives opportunities to developers of AR services to get access to fully configurable SolAR based solutions to design AR applications (for Windows, Linux and Android).
While SolAR is dedicated to computer vision pipelines, for now, the pipeline interface is only dedicated to pose estimation pipelines. New pipeline interfaces will be soon added for other vision tasks such as 3D reconstruction, object recognition, etc.
SolAR uses SolAR Pipeline Manager and SolAR Wrapper to manage the Unity plugin.
In that way, pipelines should first be created as C++ SolAR pipelines, and then be exported for Unity.
A C++ SolAR pipeline is a shared library embedding an implementation of the IPipeline interface defined by the SolAR Framework. Thus, this pipeline can be loaded at run-time and ran by any third party application such as Unity.
Any SolAR pipeline must implement IPipeline class, which means that six methods must be implemented:
init: Initializes the pipeline by providing a reference to the component manager loaded by the PipelineManager. You will have to instantiate all the components used by your pipeline and initialize them.
getCameraParameters: Provides the camera parameters.
start: Starts the pipeline with a texture buffer that will be updated when a new frame will be processed and ready for display.
stop: Stops the pipeline.
update: A methode that provides the new pose of the camera.
loadSourceImage: If there is no camera in your pipeline, you can feed it with an external image.
Once the C++ SolAR pipeline is built, you can import the DLL in Unity, select it in the SolARPipelineLoader object and run Unity Editor. Pipeline parameters are easily editable from Unity Editor.
The steps to use pipelines for Unity are described on SolAR web site: http://www.solarframework.org/use/unity/
The steps to assemble pipelines for Unity are described on SolAR web site: http://www.solarframework.org/assemble/pipeline/
Technology Research Institute focused on innovation in areas like Artificial Intelligence, immersive video and audio, content protection, 5G networks, Internet of Things, cognitive technologies
Application Programming Interface
Augmented Reality
Bundle Adjustment
Dynamic Link Library
Graphics Processing Unit
Graphical User Interface
Information Technology
Platform as a Service
Software Development Kit
Simultaneous Localization And Mapping
Solution for Augmented Reality
Simplified Wrapper and Interface Generator
Universally Unique IDentifier
Extensible Markup Language
Cross-Platform Component Framework