What is a C++ pipeline for Unity ?

A C++ pipeline is a shared library embedding an implementation of the IPipeline interface defined by the SolAR framework. Thus, this pipeline can be loaded at run-time and run by any third party application such as a C++ application or by Unity. To provide configuration capabilities to a pipeline, it is considered by SolAR as a component embedded in a module. For this reason, readers are encouraged to have a look at the create a module section.

While SolAR is dedicated to computer vision pipelines, for now, the pipeline interface is only dedicated to pose estimation pipeline. To be more specific, the pipelines built to be used in Unity are pipelines that take on hand camera frames and that deliver on the other hand camera pose estimation.

New pipeline interfaces will be soon added for other vision task such as 3D reconstruction, object recognition, etc.

Initialize your pipeline step-by-step

Create your pipeline project

  • QT Creator

Open QTCreator and create a new project (in File menu).

App wizard QT

Then, set the name of your application (e.g. MySolARPipeline) that will use a SolAR pipeline, and its location.

Next, define your build system with qmake. For the next step, choose your development kits. We recommend to use MSVC 2017 64bit on Windows or Clang on Linux.

Your project is now created and the result is a .pro file like following:

MyPipeline.pro
## remove Qt dependencies
QT       -= core gui
CONFIG -= qt

## global defintions : target lib name, version
TARGET = MyPipeline
INSTALLSUBDIR = SolARBuild
FRAMEWORK = $$TARGET
VERSION=X.X.X (1)

DEFINES += MYVERSION=$${VERSION}
DEFINES += TEMPLATE_LIBRARY
CONFIG += c++1z

include(findremakenrules.pri) (2)



CONFIG(debug,debug|release) {
    DEFINES += _DEBUG=1
    DEFINES += DEBUG=1
}

CONFIG(release,debug|release) {
    DEFINES += _NDEBUG=1
    DEFINES += NDEBUG=1
}

DEPENDENCIESCONFIG = sharedlib recursive install (3)

## Configuration for Visual Studio to install binaries and dependencies. Work also for QT Creator by replacing QMAKE_INSTALL
PROJECTCONFIG = QTVS (4)

#NOTE : CONFIG as staticlib or sharedlib, DEPENDENCIESCONFIG as staticlib or sharedlib, QMAKE_TARGET.arch and PROJECTDEPLOYDIR MUST BE DEFINED BEFORE templatelibconfig.pri inclusion
include ($$shell_quote($$shell_path($${QMAKE_REMAKEN_RULES_ROOT}/templatelibconfig.pri)))  # Shell_quote & shell_path required for visual on windows



## DEFINES FOR MSVC/INTEL C++ compilers
msvc {
DEFINES += "_BCOM_SHARED=__declspec(dllexport)"
}

INCLUDEPATH += interfaces/

HEADERS += interfaces/MyPipelineAPI.h \

SOURCES +=    src/MyPipeline_main.cpp \
              src/component.cpp

unix:!android {
    QMAKE_CXXFLAGS += -Wignored-qualifiers
}

macx {
    DEFINES += _MACOS_TARGET_
    QMAKE_MAC_SDK= macosx
    QMAKE_CFLAGS += -mmacosx-version-min=10.7 -std=c11 #-x objective-c++
    QMAKE_CXXFLAGS += -mmacosx-version-min=10.7 -std=c11 -std=c++11 -O3 -fPIC#-x objective-c++
    QMAKE_LFLAGS += -mmacosx-version-min=10.7 -v -lstdc++
    LIBS += -lstdc++ -lc -lpthread
}

win32 {
    LIBS += User32.Lib
    DEFINES += WIN64 UNICODE _UNICODE
    QMAKE_COMPILER_DEFINES += _WIN64
    QMAKE_CXXFLAGS += -wd4250 -wd4251 -wd4244 -wd4275
}

android {
    ANDROID_ABIS="arm64-v8a"
}

header_files.path = $${PROJECTDEPLOYDIR}/interfaces
header_files.files = $$files($${PWD}/interfaces/*.h*)

configuration_files.path = $${PROJECTDEPLOYDIR}/configuration
#configuration_files.files = $$files($${PWD}/tests/TestMyPipeline/PipelineMyPipeline.xml)

INSTALLS += header_files
INSTALLS += xpcf_xml_files
INSTALLS += configuration_files

OTHER_FILES += \
    packagedependencies.txt

#NOTE : Must be placed at the end of the .pro
include ($$shell_quote($$shell_path($${QMAKE_REMAKEN_RULES_ROOT}/remaken_install_target.pri)))) # Shell_quote & shell_path required for visual on windows (5)
1 Set the version number of your pipeline,
2 This .pri file has been installed by the wizard. It will allow to find the remaken folder depending on the OS you are using.
3 The dependencies of your pipeline will be installed recursively. More details are available on the builddefs-qmake project on GitHub.
4 The installation of your pipeline will also work with Visual Studio. Warning, in QTCreator, this will replace the usual QMAKE_INSTALL.
5 Place at the end the .pri file to install your pipeline.

Visual Studio

You can also simply create your application with Visual Studio by using the .pro file (see above for more details how to configure the .pro file).

Microsoft Visual Studio provides a Qt Visual Studio Tools. This enables developers to import QT project files (.pro) into Visual Studio.

Install QT Visual Studio Tools:

  • In Visual Studio, select Tools > Extensions and Updates > Online to install and update QT Visual Studio Tools.

Import the .pro. file into Visual Studio:

  • Select Qt VS Tools > Open Qt Project File (.pro) and choose your .pro file.

Right now, your project is configured.

Select your dependencies

As mentionned previously, SolAR framework provides developers with a building pipeline allowing among other things to easily manage dependencies (download, version management, packaging during deployment step, etc.).

To define the dependencies used by your pipeline, just add to your packagedependencies.txt a reference to the SolARFramework as shown below:

/.packagedependencies.txt

SolARFramework|0.8.0|SolARFramework|SolARBuild@github|https://github.com/SolarFramework/SolarFramework/releases/download

Here is the syntax for each dependency (more information available on the Remaken project on GitHub):

framework#channel | version | [condition]#library name | identifier or local folder@repository_type | repository_url | link_mode | options

As the component manager provided by XPCF can load at runtime the modules used by your pipeline as defined into your configuration file presented next, you do not need to add them into the packagedependencies.txt file. Also, XPCF and the build pipeline handle dependencies recursivity, meaning that you do not need to add the dependencies of the SolAR framework as well as of the modules.

Your dependencies, also called artifacts, should be available in your ${USER_HOME}/.remaken/packages/<yourCompiler>/ folder. To automatically download your dependencies, just run remaken where your packagedependencies.txt is located.

Refer to https://github.com/b-com-software-basis/remaken for more information.

Numerous samples of packagedependencies.txt files can be found with the SolAR Samples you have surely installed on your machine.

Create the configuration file

This file will be used at run-time. It will allow XPCF to load Module components and configure them at run-time. So you can experiment different implementations and configurations of a pipeline without the need to recompile your application.

Creating a configuration file is very easy. Once you have identified the modules and the components required to assemble your pipeline, just record them in the xml configuration file.

Following, an example of a configuration file for a pipeline using the camera and image viewer components embedded in the SolARModuleOpenCV module :

MyStandalonePipelineConfiguration.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<xpcf-registry autoAlias="true">

<module uuid="15e1990b-86b2-445c-8194-0cbe80ede970" name="SolARModuleOpenCV" path="$REMAKEN_PKG_ROOT/packages/SolARBuild/win-cl-14.1/SolARModuleOpenCV/0.8.0/lib/x86_64/shared" description="OpenCV"> (1)
  <component uuid="5B7396F4-A804-4F3C-A0EB-FB1D56042BB4" name="SolARCameraOpencv" description="SolARCameraOpencv"> (2)
    <interface uuid="125f2007-1bf9-421d-9367-fbdc1210d006" name="IComponentIntrospect" description="IComponentIntrospect"/>
    <interface uuid="5DDC7DF0-8377-437F-9C81-3643F7676A5B" name="ICamera" description="ICamera"/>
  </component>
  <component uuid="fa4a780a-9720-11e8-9eb6-529269fb1459" name="SolARVideoAsCameraOpencv" description="SolARVideoAsCameraOpencv">
      <interface uuid="125f2007-1bf9-421d-9367-fbdc1210d006" name="IComponentIntrospect" description="IComponentIntrospect"/>
      <interface uuid="5DDC7DF0-8377-437F-9C81-3643F7676A5B" name="ICamera" description="ICamera"/>
  </component>
  <component uuid="19ea4e13-7085-4e3f-92ca-93f200ffb01b" name="SolARImageViewerOpencv" description="SolARImageViewerOpencv">
    <interface uuid="125f2007-1bf9-421d-9367-fbdc1210d006" name="IComponentIntrospect" description="IComponentIntrospect"/>
    <interface uuid="b05f3dbb-f93d-465c-aee1-fb58e1480c42" name="IImageViewer" description="IImageViewer"/>
  </component>
</module>
<factory>
    <bindings>
    <bind interface="ICamera" to="SolARCameraOpencv" /> (3)
    <bind name="VideoAsCamera" interface="ICamera" to="SolARVideoAsCameraOpencv" /> (4)
    </bindings>
</factory>

<properties> (5)
  <configuration component="SolARCameraOpencv">
    <property name="calibrationFile" type="string" value="camera_calibration.yml"/>
    <property name="deviceID" type="UnsignedInteger" value="0"/>
  </configuration>
  <configuration component="SolARCameraOpencv" name="Configuration1"> (6)
    <property name="calibrationFile" type="string" value="camera_calibration.yml"/>
    <property name="deviceID" type="UnsignedInteger" value="1"/>
  </configuration>
  <configure component="SolARImagesAsCameraOpencv">
    <property name="calibrationFile" type="string" value="camera_calibration.yml"/>
    <property name="imagesDirectoryPath" type="string" value="..\Images\frame_%04d.png"/>
  </configure>
  <configuration component="SolARImageViewerOpencv">
    <property name="title" type="string" value="Original Image"/>
    <property name="exitKey" type="Integer" value="27"/>
    <property name="width" type="Integer" value="0"/>
    <property name="height" type="Integer" value="0"/>
  </configuration>
</properties>

</xpcf-registry>
1 Add the modules used by your pipeline. To get information concerning the modules, have a look to their registry file in the ${HOME_DIR}/.xpcf/SolAR/.
2 For each module, add the components used by your pipeline. To get information concerning the components, have a look to the registry file of their module available in the ${HOME_DIR}/.xpcf/SolAR/.
3 Thanks to the factory field, you can bind by default a specific component implementation (here a camera openCV) on a given SolAR interface (here the ICamera).
4 You can give a name to a binding. In this case, in your code you will be able to ask for this specific binding when you will xant to instantiate a ICamera component.
5 Optionally, add a configuration to your components. The name of your configuration parameters are generally the name of the variable in the class definition of the component without the prefix "m_". The type of the configuration parameters are simple types such as "String", "Integer", "Float", "Double", or array of these simple types.
6 A second configuration is defined for the SolARCameraOpenCV with a specific name if you want to instanciate a second component with a specific configuration

Examples of such configuration files are available in the samples directory that is provided with the SolAR installation.

Implement your pipeline

Using QT Creator Wizard

Refer to Create/Create_Component part. You can create a Pipeline like a component using Qt creator wizard. This component has to inherit of IPiepline interface. Then, follow below instructions.

IPipeline API

A virtual class named IPipeline has been defined in SolAR framework and is located in api/pipeline/IPipeline.h .
Any SolAR pipeline should implement this class, that means that six methods are to be implemented in the corresponding source code :

  • init: Initialize the pipeline by providing a reference to the component manager loaded by the PipelineManager. You will have to instantiate all the components used by your pipeline and initialize by for instance starting the camera, initialize component with intrinsic parameter of the camera, etc.

  • getCameraParameters: Provide third party applications with the calibration parameters of the camera. Useful hen you want to set the parameter of the virtual camera with the ones from the real one.

  • start: Start the pipeline with a texture buffer that will be updated when a new frame will be processed and ready for display.

  • stop: Stop the pipeline.

  • update: A methode that provide the new pose of the camera.

  • loadSourceImage: If there is no camera in your pipeline, you can feed it with an external image (for instance, an image capture by the Unity web camera).

A full description of these methods is available in the api/pipeline/IPipeline.h file.

Pipeline Template

To implement your pipeline, you have to fill in the header file MyPipeline.h and the source file MyPipeline.cpp. Next, we provide you with templates for these to files.

MyPipeline.h
/**
 * Information concerning the copyright and licence of your pipeline
 */

#ifndef MYPIPELINE_H
#define MYPIPELINE_H

(1)
#if _WIN32
#ifdef MyPipeline_API_DLLEXPORT
#define MYPIPELINE_EXPORT_API __declspec(dllexport)
#else //MYPIPELINE_API_DLLEXPORT
#define MYPIPELINE_EXPORT_API __declspec(dllimport)
#endif //MYPIPELINE_API_DLLEXPORT
#else //_WIN32
#define MYPIPELINE_EXPORT_API
#endif //_WIN32

// Mandatory include file (2)
#include "xpcf/core/traits.h"
#include "xpcf/component/ConfigurableBase.h"
#include "api/pipeline/IPipeline.h"
#include "xpcf/threading/BaseTask.h"

// Add here the header file for the data structures required by your pipeline
// e.g. #include "datastructure/Image.h"
(3)

// Add here the header file for the component interfaces required by your pipeline
// e.g. #include "api/input/devices/ICamera.h"
(4)

namespace SolAR { (5)
using namespace datastructure;
using namespace api;
namespace PIPELINES {

/**
 * @class MyPipeline
 * @brief A short description of your pipeline
 */
(6)
class MYPIPELINE_EXPORT_API MyPipeline : public org::bcom::xpcf::ConfigurableBase,
    public api::pipeline::IPipeline
{
public:
    MyPipeline();
    ~MyPipeline();

    //// @brief Initialization of the pipeline
    /// Initialize the pipeline by providing a reference to the component manager loaded by the PipelineManager.
    /// @param[in] componentManager a shared reference to the component manager which has loaded the components and configuration in the pipleine manager
    FrameworkReturnCode init(SRef<xpcf::IComponentManager> xpcfComponentManager) override;

    /// @brief Provide the camera parameters
    /// @return the camera parameters (its resolution and its focal)
    pipeline::CameraParameters getCameraParameters() override;

    /// @brief Starts the pipeline and provides a texture buffer which will be updated when required.
    /// @param[in] textureHandle a pointer to the texture buffer which will be updated at each call of the update method.
    FrameworkReturnCode start(void* imageDataBuffer) override;

    /// @brief Stop the pipeline.
    FrameworkReturnCode stop() override;

    /// @brief update the pipeline
    /// Get the new pose and update the texture buffer with the image that has to be displayed
    SinkReturnCode update(Transform3Df& pose) override;

    /// @brief load the source image
    SourceReturnCode loadSourceImage(void* sourceTextureHandle, int width, int height) override;

    void unloadComponent () override final;

private:
  // Add here the declaration of the shared references on components required by your pipeline (e.g. SRef<input::device::ICamera> m_camera;)
  (7)

  // Threads (8)
  bool pipelineLoop();
  xpcf::DelegateTask* m_pipelineLoopTask = nullptr;

  // optionally, add here some parameters of you rmodule that can be configured
  // int m_myPipelineParameter; (9)
};

}
}


XPCF_DEFINE_COMPONENT_TRAITS(SolAR::PIPELINES::MyPipeline, (10)
                             "<yourUUID>",
                             "My pipeline",
                             "A sample pipeline used for documentation");

#endif // MYPIPELINE_H
1 Add the required declarations to export the pipeline functions embedded in your shared library. Just replace in a case sensitive manner "MyModule" by the name of your module.
2 These include files are mandatory. They correspond to the xpcf traits file as here we declare the module traits directly in this file, to the XPCF configurableBase header file if you want to configure your pipeline through an external xml file, and to the SolAR IPipeline header file, as your pipeline implement this SolAR interface.
3 The header files of the SolAR data structures required to connect your components.
4 The header files of the SolAR component interfaces required by your pipeline.
5 SolAR and XPCF namespaces directives to shorten the calls to SolAR api and data structures.
6 Declare your pipeline class, and add the functions defined as abstract in the IPipeline interface.
7 Declaration of the components used by the pipeline (thanks to their abstract interfaces). We are using a shared reference of the components to ease their use in multi-threaded pipelines.
8 Declare a xpcf task and the function running the pipeline loop in a dedicated thread.
9 You can declare parameters for the pipeline. Some of these parameters may be configurable.
10 Finally, add the trait of your pipeline. To generate an UUID, you can use a online UUID generator
MyPipeline.cpp
/**
 * Information concerning the copyright and licence of your pipeline
 */

// Mandatory header file (1)
#include "xpcf/module/ModuleFactory.h"
#include "core/Log.h"

// Header to your pipeline declaration (2)
#include "MyPipeline.h"

namespace xpcf=org::bcom::xpcf;

// Declaration of the module embedding MyPipeline (3)
XPCF_DECLARE_MODULE("<yourUUID>", "MyPipelineModule", "The module embedding MyPipeline")

// Add this function required by the XPCF component factory (4)
extern "C" XPCF_MODULEHOOKS_API xpcf::XPCFErrorCode XPCF_getComponent(const boost::uuids::uuid& componentUUID,SRef<xpcf::IComponentIntrospect>& interfaceRef)
{
    xpcf::XPCFErrorCode errCode = xpcf::XPCFErrorCode::_FAIL;
    errCode = xpcf::tryCreateComponent<SolAR::PIPELINES::MyPipeline>(componentUUID,interfaceRef);

    return errCode;
}

// Add the declaration of the component (here a pipeline) to XPCF (5)
XPCF_BEGIN_COMPONENTS_DECLARATION
XPCF_ADD_COMPONENT(SolAR::PIPELINES::MyPipeline)
XPCF_END_COMPONENTS_DECLARATION

// Macro used by the XPCF factory to define a component/pipeline (6)
XPCF_DEFINE_FACTORY_CREATE_INSTANCE(SolAR::PIPELINES::MyPipeline)

namespace SolAR { (7)
using namespace datastructure;
using namespace api::pipeline;
namespace PIPELINES {

MyPipeline::MyPipeline():ConfigurableBase(xpcf::toUUID<MyPipeline>())
{
   addInterface<api::pipeline::IPipeline>(this);
   // Add here the declaration of the component implementation you want to inject in your interfaces
   // e.g. declareInjectable<input::device::ICamera>(m_camera);
   // If a sepcific named binding is needed
   // e.g. declareInjectable<input::device::ICamera>(m_camera, "VideoAsCamera");
   (8)
   // Add here the mapping of your pipeline properties with your variables of your pipeline claa
   // e.g. declareProperty("myPipelineProperty", m_MyPipelineVariable);
   (9)
}

MyPipeline::~MyPipeline()
{

}

void MyPipeline::onInjected()
{
    // ADD HERE: Things to do when your pipline has been injected
}

xpcf::XPCFErrorCode MyPipeline::OnConfigured()
{
  // ADD HERE: Things to do when the variables of your pipeline class have just been set according to the properties defined in your configuration file.
}

bool MyPipeline::pipelineLoop() (10)
{
  // ADD HERE: the code connecting the component of your pipeline

  return true;
}

FrameworkReturnCode MyPipeline::init(SRef<xpcf::IComponentManager> xpcfComponentManager)
{
  try {
    // ADD HERE: Thinks to do to initialize your pipeline
    //e.g. load a marker or set camera parameters for components requiring them.
    (11)

  }
  catch (xpcf::Exception e)
  {
     LOG_WARNING("One or more components cannot be created: {}", e.what());
     return FrameworkReturnCode::_ERROR_;
  }

  return FrameworkReturnCode::_SUCCESS;
}

CameraParameters MyPipeline::getCameraParameters()
{
  CameraParameters camParam;
  // ADD HERE the code to return camera parameters

  return camParam;
}

SourceReturnCode MyPipeline::loadSourceImage(void* sourceTextureHandle, int width, int height)
{
   // ADD HERE the code to take external image as input of your pipeline

   return SourceReturnCode::_NOT_IMPLEMENTED;
}


FrameworkReturnCode MyPipeline::start(void* imageDataBuffer)
{
  // ADD HERE the code to start your pipeline
  // e.g. start the camera

  // create and start a thread for the loop of the pipeline (12)
  auto pipelineLoopThread = [this](){;pipelineLoop();};
  m_pipelineLoopTask = new xpcf::DelegateTask(pipelineLoopThread);
  m_pipelineLoopTask->start();

  return FrameworkReturnCode::_SUCCESS;
}

FrameworkReturnCode MyPipeline::stop()
{
  // ADD HERE the code to stop your pipeline
  if (m_pipelineLoopTask != nullptr)
    m_pipelineLoopTask->stop(); (13)
  return FrameworkReturnCode::_SUCCESS;
}

SinkReturnCode MyPipeline::update(Transform3Df& pose)
{
  // ADD HERE the code to update the pose of the camera

  return SinkReturnCode::_NOT_IMPLEMENTED;
}

}
}
1 Required header file corresponding to the XPCF module factory.
2 Obviously, add the header file corresponding to your pipeline declaration.
3 Declare your module to XPCF. generate a UUID for your module by using an online UUID generator.
4 Implement the method XPCF_getComponent required by the XPCF factory to instantiate the componen t(here a pipeline).
5 Add your component/pipeline to XPCF.
6 Add the definition of your pipeline to the XPCF component factory.
7 Embed your pipeline in the SolAR/PIPELINES namespace.
8 If you have declare the component interfaces required for your pipeline in its header file, you can map component implementatations on them. The choice of the component implementations to map on your interfaces will be solved at runtime according to what is described in your xml configuration file. By default, the resolution chooses the first component alias declared in the configuration file that fits your interface. If you have several components alias declared in your configuration file, you can force the choice by defining a specific binding. In your pipeline, if you need several instances of a component based on different implementations, you can name your binding and pass it to the DeclareInjectable function. Do not forget to add _autoalias=true" at the beginning of your xml configuration file if you want to automatically create aliases for all declared components based on their names.
9 To set a variable of the pipeline class as configurable and map on it a property defined in the xml configuration file, you have to declare the property. Here, if an xml element with the name "myPipelineProperty" is specified in the configuration file, when exiting the constructor, its value will be set to the variable m_myPipelineVariable.
10 Implement the loop of the pipeline by connecting components. This loop will run in a dedicated thread.
11 If required, add the code to initialize your pipeline (e.g. load a reference image, initialize some component with the intrinsic parameters of the camera, etc.).
12 Start the pipeline loop thread when the start method is called.
13 Stop the pipeline loop thread when the stop method is called.

Pipeline sample

We will present next the implementation of the simplest pipeline that consists in capturing an image from a camera and make it available for a third party application. For this implementation, we will need only one module: SolARModuleOpenCV.

MyPipeline.h
/**
 * Information concerning the copyright and licence of your pipeline
 */

#ifndef MYPIPELINE_H
#define MYPIPELINE_H

#if _WIN32
#ifdef MyPipeline_API_DLLEXPORT
#define MYPIPELINE_EXPORT_API __declspec(dllexport)
#else //MYPIPELINE_API_DLLEXPORT
#define MYPIPELINE_EXPORT_API __declspec(dllimport)
#endif //MYPIPELINE_API_DLLEXPORT
#else //_WIN32
#define MYPIPELINE_EXPORT_API
#endif //_WIN32

// Mandatory include file
#include "xpcf/core/traits.h"
#include "xpcf/component/ConfigurableBase.h"
#include "api/pipeline/IPipeline.h"

// Add here the header file for the data structures required by your pipeline
// e.g. #include "datastructure/Image.h"
#include "datastructure/Image.h" (1)

// Add here the header file for the component interfaces required by your pipeline
// e.g. #include "api/input/devices/ICamera.h" (2)
#include "api/input/devices/ICamera.h"
#include "api/sink/ISinkPoseImage.h"
#include "api/source/ISourceImage.h"



namespace SolAR {
using namespace datastructure;
using namespace api;
namespace PIPELINES {

/**
 * @class MyPipeline
 * @brief A short description of your pipeline
 */
class MYPIPELINE_EXPORT_API MyPipeline : public org::bcom::xpcf::ConfigurableBase,
    public api::pipeline::IPipeline
{
public:
    MyPipeline();
    ~MyPipeline();

    //// @brief Initialization of the pipeline
    /// Initialize the pipeline by providing a reference to the component manager loaded by the PipelineManager.
    /// @param[in] componentManager a shared reference to the component manager which has loaded the components and configuration in the pipleine manager
    FrameworkReturnCode init(SRef<xpcf::IComponentManager> xpcfComponentManager) override;

    /// @brief Provide the camera parameters
    /// @return the camera parameters (its resolution and its focal)
    pipeline::CameraParameters getCameraParameters() override;

    /// @brief Starts the pipeline and provides a texture buffer which will be updated when required.
    /// @param[in] textureHandle a pointer to the texture buffer which will be updated at each call of the update method.
    FrameworkReturnCode start(void* imageDataBuffer) override;

    /// @brief Stop the pipeline.
    FrameworkReturnCode stop() override;

    /// @brief update the pipeline
    /// Get the new pose and update the texture buffer with the image that has to be displayed
    SinkReturnCode update(Transform3Df& pose) override;

    /// @brief load the source image
    SourceReturnCode loadSourceImage(void* sourceTextureHandle, int width, int height) override;

    void unloadComponent () override final;

private:
  // Add here the declaration of the shared references on components required by your pipeline (e.g. SRef<input::device::ICamera> m_camera;)
  SRef<input::devices::ICamera> m_camera; (3)
  SRef<sink::ISinkPoseImage> m_sink;
  SRef<source::ISourceImage> m_source;

  // Threads
  bool pipelineLoop();
  xpcf::DelegateTask* m_pipelineLoopTask = nullptr;

  // optionally, add here some parameters of your module that can be configured
  // int m_myPipelineParameter;

  // Other attributes
  bool m_externalInputImageMode = false; (4)
};

}
}


XPCF_DEFINE_COMPONENT_TRAITS(SolAR::PIPELINES::MyPipeline,
                             "855c83b7-f4ec-48ab-8e89-56018ea9e169",
                             "My pipeline",
                             "A sample pipeline used for documentation");

#endif // MYPIPELINE_H
1 include the header file of a SolAR image to connect the camera component to the sink component.
2 include the component interface headers for a camera component, a sink component and a source component. A sink component handles an output buffer feed by the pipeline a read by external third parties. Reciprocally, a source component handles an input buffer feeds by external third parties and used by the pipeline.
3 Declaration of the three components used by this pipeline, a camera, a sink component handling the pose and the output image, and a source component handling an input image.
4 Add an attribute to know if the pipeline use as input an image coming from a third party.
MyPipeline.cpp
/**
 * Information concerning the copyright and licence of your pipeline
 */

// Mandatory header file
#include "xpcf/module/ModuleFactory.h"
#include "core/Log.h"

// Header to your pipeline declaration
#include "MyPipeline.h"

namespace xpcf=org::bcom::xpcf;

// Declaration of the module embedding MyPipeline
XPCF_DECLARE_MODULE("855c83b7-f4ec-48ab-8e89-56018ea9e169", "MyPipelineModule", "The module embedding MyPipeline")

// Add this function required by the XPCF component factory
extern "C" XPCF_MODULEHOOKS_API xpcf::XPCFErrorCode XPCF_getComponent(const boost::uuids::uuid& componentUUID,SRef<xpcf::IComponentIntrospect>& interfaceRef)
{
    xpcf::XPCFErrorCode errCode = xpcf::XPCFErrorCode::_FAIL;
    errCode = xpcf::tryCreateComponent<SolAR::PIPELINES::MyPipeline>(componentUUID,interfaceRef);

    return errCode;
}

// Add the declaration of the component (here a pipeline) to XPCF
XPCF_BEGIN_COMPONENTS_DECLARATION
XPCF_ADD_COMPONENT(SolAR::PIPELINES::MyPipeline)
XPCF_END_COMPONENTS_DECLARATION

// Macro used by the XPCF factory to define a component/pipeline
XPCF_DEFINE_FACTORY_CREATE_INSTANCE(SolAR::PIPELINES::MyPipeline)

namespace SolAR {
using namespace datastructure;
using namespace api::pipeline;
namespace PIPELINES {

MyPipeline::MyPipeline():ConfigurableBase(xpcf::toUUID<MyPipeline>())
{
   addInterface<api::pipeline::IPipeline>(this);
   // Add here the declaration of the component implementation you want to inject in your interfaces
   // e.g. declareInjectable<input::device::ICamera>(m_camera);
   // If a sepcific named binding is needed
   // e.g. declareInjectable<input::device::ICamera>(m_camera, "VideoAsCamera");
   (1)
   DeclareInjectable<input::devices::ICamera>(m_camera);
   DeclareInjectable<sink::ISinkPoseImage>(m_sink);
   DeclareInjectable<source::ISourceImage>(m_source);

   // Add here the mapping of your pipeline properties with your variables of your pipeline claa
   // e.g. declareProperty("myPipelineProperty", m_MyPipelineVariable);

}

MyPipeline::~MyPipeline()
{

}

void MyPipeline::onInjected()
{
    // ADD HERE: Things to do when your pipline has been injected
}

xpcf::XPCFErrorCode MyPipeline::OnConfigured()
{
  // ADD HERE: Things to do when the variables of your pipeline class have just been set according to the properties defined in your configuration file.
}

bool MyPipeline::pipelineLoop() (2)
{
  // ADD HERE: the code connecting the component of your pipeline
  SRef<Image> image;

  if (m_externalInputImageMode)
  {
    if (m_source->getNextImage(image) == SourceReturnCode::_NEW_IMAGE)
      m_sink->set(image);
  }
  else
  {
    if (m_camera->getNextImage(image) == SolAR::FrameworkReturnCode::_ERROR_LOAD_IMAGE)
      return false;
    m_sink->set(image);
  }
  return true;
}

FrameworkReturnCode MyPipeline::init(SRef<xpcf::IComponentManager> xpcfComponentManager)
{
  try {
    // ADD HERE: Thinks to do to initialize your pipeline
    //e.g. load a marker or set camera parameters for components requiring them.

  }
  catch (xpcf::Exception e)
  {
     LOG_WARNING("One or more components cannot be created: {}", e.what());
     return FrameworkReturnCode::_ERROR_;
  }

  return FrameworkReturnCode::_SUCCESS;
}

CameraParameters MyPipeline::getCameraParameters()
{
  CameraParameters camParam;
  // ADD HERE the code to return camera parameters
  if (m_camera) (3)
    {
        camParam  m_camera->getCameraParameters();
    }
  return camParam;
}

SourceReturnCode MyPipeline::loadSourceImage(void* sourceTextureHandle, int width, int height)
{
   // ADD HERE the code to take external image as input of your pipeline
   m_externalInputImageMode = true; (4)
   return m_source->setInputTexture((unsigned char *)sourceTextureHandle, width, height);
}


FrameworkReturnCode MyPipeline::start(void* imageDataBuffer)
{
  // ADD HERE the code to start your pipeline
  m_sink->setImageBuffer((unsigned char*)imageDataBuffer); (5)

  if (m_camera->start() != FrameworkReturnCode::_SUCCESS) (6)
    return FrameworkReturnCode::_ERROR_;

  // create and start a thread for the loop of the pipeline
  auto pipelineLoopThread = [this](){;pipelineLoop();};
  m_pipelineLoopTask = new xpcf::DelegateTask(pipelineLoopThread);
  m_pipelineLoopTask->start();

  return FrameworkReturnCode::_SUCCESS;
}

FrameworkReturnCode MyPipeline::stop()
{
  // ADD HERE the code to stop your pipeline
  if (m_pipelineLoopTask != nullptr)
    m_pipelineLoopTask->stop();
  return FrameworkReturnCode::_SUCCESS;
}

SinkReturnCode MyPipeline::update(Transform3Df& pose)
{
  // ADD HERE the code to update the pose of the camera

  return SinkReturnCode::_NOT_IMPLEMENTED;
}

}
}
1 We declare the injection of the 3 components required for this pipeline.
2 Implementation of the pipeline loop. If an input image has been updated, we will access to it through the source component and we pass it to the sink component. If not, we read the last image captured by the camera and we pass it to the sink component.
3 A simple code to get camera parameters and return it.
4 Set the m_externalInputImageMode to true to inform the pipeline that it run with external input images, and put the image in the buffer of the source component.
5 Set the image buffer for the sink component.
6 Start the camera.

Export your pipeline for Unity

General

First of all, you must follow the installation instructions of Unity and of the SolAR plugin for unity available in the section Use Unity.

Then, copy the shared binaries of your pipeline, of its modules and of the dependencies of its modules in the Assets/Plugins folder of your Unity project.

Copy the configuration file of your pipeline in the Assets/SolAR/Pipelines folder of your Unity project. When copied, edit the configuration file and replace all module paths so they are relative to your Unity project (./Assets/Plugins).

Now, in the Unity editor, you can select again your pipeline folder in the inspector of your SolARPipelineLoader, and select in the pipeline list your pipeline.

If you want to export your pipeline, create a new Unity package with:

  • The shared binaries stored in the Unity plugin folder relative to:

    • Your pipeline

    • The modules used by your pipeline

    • The third parties of the modules used by your pipeline

  • The pipleline configuration file stored under Assets/SolAR/Pipelines/.

Unity Android Deployment

For Unity platform we proceed to a change in the path of the pipeline configuration file to match applications public path. Your pipeline configuration will be available from the device and could be edited directly to manage its components, parameters, etc.

Deployment details

/!\ Only arm64-v8a architecture are supported by SolAR.

While you launch build for Android platform :

  • On Unity

    • Unity pre-build process will be called to generate ./Assets/StreamingAssets/SolAR/Android/android.xml. This file will list every file in under your ./Assets/StreamingAssets/.

    • Path of pipelines xml in ./Assets/StreamingAssets/SolAR/Pipelines/ are set to match Android filesystem to the public application directory. This will let your xml available on your device and you could edit them.

    • This file will be read to clone every asset included in the Android private JAR to Android application public path

    • APK is built and it includes Unity StreamingAssets and libraries

  • On Android

    • While the application is launched for the first time, it will look for android.xml in the private application JAR. Then if this file doesn’t exist in the device application public path (/storage/emulated/0/Android/com.bcom/SolARDemo/files/StreamingAssets/SolAR/Android) it will be read and all of his content lists will be cloned into this path. Otherwise, the already present one will be read and cloned. If you don’t want to overwrite an asset at each application launching you can set the overwrite attribute for a dedicated file to false.

    • The pipeline selected by the application will be load (from public application path) and his path to ./Assets/Plugins will be matched with Android private JAR path (/data/app/com.bcom/SolARUnityPlugin-[only-known-on-running]==/lib/arm64/).

    • Application is initialized correctly. You can change the pipeline selected with SolARMenu in the right-hand corner.

Android pipeline
Figure 1. Android pipeline

Test your pipeline in C++

To test your pipeline in C++, have a look to the section use in C++.