SolAR is a solution to easily and quickly implement custom camera pose estimation pipelines based on a set of vision and mathematics components in order to make them available to AR application developers.
The SolAR core handles everything that will allow the implementation of a pipeline:
SolAR Data Structures define the information that flow in the pipelines and are exchanged between components.
SolAR Components Interfaces define a standardized interface for the different categories of basic processing elements required to implement camera pose estimation (i.e.: key point extractor, descriptor computation, etc.). This unified interface is required to ensure interoperability between components allowing to easily swap one with another to improve the final camera pose estimation pipeline.
The SolAR Pipeline Manager handles the implementation of a pipeline by loading components, connecting them, and running the pipeline.
The SolAR Components are implementations of basic processing elements compliant with the SolAR components interfaces. Several components can implement the same component interface. In general, these components are created by wrapping computer vision libraries that could be either open-source (OpenCV, PCL, ROS, etc.) or proprietary. Each component can define its own parameters that will be used to fine tune the camera pose estimation solution.
Coming soon: configure components on the fly, pipeline, multithread, metadata.
The SolAR pipeline consists of a chain of basic processing elements linked together so that the outputs of each element are the inputs of the next one. In order to provide modularity, SolAR specifies standard data structures defining the information exchanged between components required to implement a camera pose estimation solution.
The following categories describe the current SolAR data structures, with more to come.
A SolAR image is an array of pixels where each pixel is defined by a set of color components coded with a given number of bits. Each pixel can be encoded with a dedicated colorimetric space (RGB, YUV, LAB, etc.). Contrary to OpenCV, an image is not just a matrix or an array. A SolAR image is clearly defined as an image object to ensure that a component waiting for an image as an input will not receive a random matrix such as a pose. Thus, an image is defined by:
The image size (width, height)
The image layout such as LAYOUT_GRB, LAYOUT_BGR, LAYOUT_GREY, LAYOUT_RGBA, LAYOUT_RGBX, LAYOUT_UNDEFINED
The pixel order either INTERLEAVED or PER_CHANNEL
The size of each color component, also called data type which can be either TYPE_8U, TYPE_16U, TYPE_32U, TYPE_64U
A pointer to the raw data
In order to limit memory allocations and copies, which can drastically impact the efficiency of the camera pose estimation implementation, an image can point to the raw data instantiated by third party libraries such as OpenCV or OpenVX.
The SolAR implementation of matrices is based on the open source Eigen library for linear algebra. The SolAR framework redefines the following matrices:
SolARVector4f equates to a 4x1 matrix of floats
SolARVector4d equates to a 4x1 matrix of doubles
SolARVector3f equates to a 3x1 matrix of floats
SolARVector3d equates to a 3x1 matrix of doubles
SolARTranslationVectorf equates to a 3x1 matrix of floats used to define a translation
SolARPoint_3Df equates to a 3x1 matrix of floats used to define a 3D point
SolARPoint_2Df equates to a 2x1 matrix of floats used to define a 2D point
SolARPoseMatrixf equates to a 4x4 matrix of floats used to define a pose
SolARMatrix33d equates to a 3x3 matrix of doubles
SolARRotationMatrixf equates to a 3x3 matrix of floats used to define a rotation
SolARQuaternionf equates to a Quaternion in floats used to define a rotation
The SolaARPose data structure is based on a standard row major homogeneous 4x4 matrix. Euler angles and quaternions can be directly extracted from this structure built using Eigen algorithms.
A SolAR Marker is a structure describing a convex area in a SolAR Image. A SolAR Marker allows managing different types of markers such ad fiducial markers or checker boards. It encodes shape and pose information as follows:
Edges: a set of SolARPoint_2Df defining the convex area. A fiducial marker can be described with four edges. The order of the points is important and impacts the structure of the marker
Translation: a SolARVector3f defining the translation of the marker relative to the camera
Rotation: a SolARRotationMatrixf defining the rotation of the marker relative to the camera
The SolAR Marker embeds information about the marker in a minimalist way. The reason is directly linked to the desired genericity across all the existing markers in the state-of-the-art methods.
The current implementation of the SolAR Features data structure takes advantage of the OpenCV Features2D definition. However, we are currently working on an abstract API to handle multiple keypoints, features descriptors and extractors implementations using technologies such as CUDA and OpenVX. We will be especially focusing on proper memory management and avoiding useless data copying.
The current SolAR Features data structure is not the final implementation we are aiming for and will be undergoing changes as we actively work on it.
SolAR components are vision or mathematics processing elements compliant with the SolAR component interface defined by the SolAR framework.
The Acquisition Interface manages the capturing strategy from a given device and provides the input signal to the next processing component.
The ISolARCamera interface is dedicated to acquiring images from external device. It allows grabbing the current image and sets/extracts some parameters related to the projection and acquisition models as follows:
get/set camera acquisition parameters: get default acquisition setting parameters (image size, frame rate..). The user can also set custom parameters which force the camera grabbing parameters to that value.
get/set camera projection parameters: get camera projection parameters including camera matrix and distortion parameters. It assumes that camera is already correctly calibrated. The user can set these parameters by performing a preliminary calibration.
The Conversion Interface manages conversion for different data structure.
The ISolaARImageConvertor performs the conversion between several image formats. It allows converting both color and type value using one unified method which automatically detects the color and/or type layers of the input and output.
The Filtering Interface manages filtering operations for different data structure.
The ISolARImageFilter is dedicated to perform various filtering operations assuming input/output consistency. The interface embeds the following operations:
Binarizing
Equalizing
Blurring
Erosion
Dilation
Gradient
Laplacian
The ISolARFeaturesFilter is dedicated to filter different extracted/computed features.
The ISolARMarkerFilter is dedicated to enhance marker filtering.
The Estimation Interface manages several pose estimation methods for cameras, objects and markers.
The ISolARKeypointDetector, ISolARDescriptorExtractor and ISolARDescriptorMatcher allow to detect geometrically relevant points in the image using known algorithms such as SURF or SIFT. (SURF, SIFT..). They also compute descriptors around these points and correlate sets of matches.
The ISolARHomographyFinder performs an homography transformation estimation and extracts a camera pose.
The ISolARMarkerFinder allows to perform marker finding operations on different types of markers. It detects, segments and recognizes the marker and estimates its pose relative to the camera.
The SolAR third party connector is a special SolAR Component used to exchange internal SolAR pose data with any third party outside of the SolAR framework. It is used to safely provide access to a SolAR Image and its associated SolAR Pose in a thread safe race free fashion. It internally implements a mutex locked circular buffer, allowing safe producer/consumer access. It allows synchronous and asynchronous read access for the consumer.
The internal buffer size of the container is currently set at one which means only the latest image and pose are available to the consumer at any given time. In the near future, SolAR will allow users to provide initialization parameters when creating components so the users will be able to specify the size of the buffer they desire.
Any component wishing to expose its pose data to an external third party must use a Third Party Connector. It can then use the set method to set the current SolAR Image and its associated SolAR Pose. The set method is thread safe and blocking. It will not return until the lock on the internal buffer is released by the reader.
Any third party external to SolAR that needs access to internal pose data of a component must connect to its Third Party Connector. It can then use either the get method or the tryGet method to get the latest SolARImage and its associated SolARPose. The get method is thread safe and blocking. This means it will not return until there is available data to read and the lock on the internal buffer is released by the writer. The tryGet method is also thread safe but non blocking. It will return immediately after being called. It will return true if it was able to acquire the lock and read the data, or false if it was unable to obtain the lock or if there was nothing to read.
SolAR relies on the Cross Platform Component Framework (XPCF), a versatile framework that provides dynamic component instantiation and loading from modules. Basically, modules are libraries that collect components sharing a common element: purpose, framework, etc. XPCF also provides uniform component parametrization. Using XPCF, and thanks to the unified SolAR interfaces, it will be possible to build applications by picking and choosing different components in different modules at runtime without prior knowledge of the components when the application itself is compiled.
For more details, please refer to the XPCF website (coming soon).
What is SolAR?
SolAR is an open source initiative to develop a framework for pose estimation that can be used for Augmented Reality (AR) applications.
Who initiated the SolAR project?
The b<>comInstitute of Research and Technology initiated the SolAR project after collecting the feedback of AR actors, AR end-users (industry, real-estate, medical, etc.), AR developers (essentially SMEs) and computer vision experts (academic researchers and research engineers from companies).
Who can contribute to the SolAR project?
Anyone can contribute to the SolAR project. Contributors only need to be proficient in C++ and computer vision skills.
Can I use the SolAR framework for my AR applications?
Of course! Just be wary of the licenses of the components used by the pose estimation pipelines you choose as each component can be under its own license, independently of the encompassing SolAR Apache v2 license.
For any requests, please contact us.
Discover
Use it
Contribute
Applications