SwiftFusion
A differentiable sensor fusion library written in Swift.
Think factor graphs (Ã la GTSAM) coupled with deep learning (via Swift for TensorFlow).
This project is in an early phase, but feel free to explore! Subject to massive changes. :-)
Getting started
Installation
macOS
Download a Swift for TensorFlow macOS toolchain and follow the macOS installation instructions. Installing the latest development snapshot is recommended. Check here for the "required" version of Xcode for using nightly toolchains. Sometimes toolchains may also work with older Xcode minor versions than the one listed.
To use Swift for TensorFlow, add the toolchain to PATH
:
export PATH="/Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/bin/:$PATH"
Linux
Download a Swift for TensorFlow Linux toolchain and follow the Linux installation instructions. Installing the latest development snapshot is recommended.
NVIDIA Jetson
Download the Swift for TensorFlow NVIDIA Jetson toolchain and follow the Linux installation instructions.
Testing
swift test --enable-test-discovery
Benchmarking
swift run -c release -Xswiftc -cross-module-optimization SwiftFusionBenchmarks
Development
Xcode (macOS)
Xcode provides a great Swift development experience on macOS.
To open SwiftFusion in Xcode, run open Package.swift
.
Visual Studio Code
To enable Swift autocomplete in Visual Studio Code, install the Maintained Swift Development Environment plugin, and set the following plugin settings:
"sde.languageServerMode": "sourcekit-lsp",
"sourcekit-lsp.serverPath": "<your toolchain path>/usr/bin/sourcekit-lsp",
"sourcekit-lsp.toolchainPath": "<your toolchain path>",
"swift.path.swift_driver_bin": "<your toolchain path>/usr/bin/swift",
The CodeLLDB plugin enables debugging within Visual Studio Code. You need to set the following setting:
"lldb.library": "/swift-tensorflow-toolchain/usr/lib/liblldb.so"
A sample launch.json
file:
{
"version": "0.2.0",
"configurations": [
{
"type": "lldb",
"request": "launch",
"name": "Debug",
"program": "${workspaceFolder}/.build/x86_64-unknown-linux-gnu/debug/SwiftFusionPackageTests.xctest",
"args": [
"--enable-test-discovery"
],
"cwd": "${workspaceFolder}"
}
]
}
Code overview
The main code is in Sources/SwiftFusion
, organized under subdirectories.
Core
The core abstractions of SwiftFusion:
Vector.swift
: protocol that formalizes a Euclidean vector space with standard orthonormal basis, and defines a great number of default methods for it. Also defines theScalarsInitializableVector
andFixedSizeVector
protocols, and a collectionStandardBasis<V: Vector>
for the orthogonal bases.Manifold.swift
: protocol that inherits fromDifferentiable
that addsretract
andlocalCoordinate
, which convert between aManifold
structure and itsLocalCoordinate
.LieGroup.swift
: protocol that formalizes making a differentiable manifold into a Lie group with composition operator*
. Also definesLieGroupCoordinate
protocol.
Specific Vector
-like data structures:
VectorN.swift
: generated byVectorN.swift.gyb
, implements small fixed-size vectors conforming toAdditiveArithmetic
,Vector
, andFixedSizeVector
.FixedSizeMatrix.swift
: matrix whose dimensions are known at compile time. Conforms toEquatable, KeyPathIterable, CustomStringConvertible, AdditiveArithmetic, Differentiable, FixedSizeVector
.
Additional utilities:
DataTypes.swift
: for now, just implementsLieGroup
protocol forVector5
Dictionary+Differentiable.swift
: conformDictionary
toDifferentiable
.MathUtil.swift
: for now justpinv
(pseudo-inverse), usingTensor.svd
.TensorVector.swift
: a view of aTensor
that conforms to theVector
protocol.TrappingDouble.swift
: a wrapper forDouble
that traps instead of allowingNaN
.Tuple+Vector.swift
: (undocumented)TypeKeyedArrayBuffers.swift
: related to storage ofValues
andFactors
.
Inference
FactorGraph.swift
: the main factor graph structure storing factors.Factor.swift
: defines the core factor protocol hierarchyFactor
→VectorFactor
→LinearizableFactor
→GaussianFactor
, as well as theVariableTuple
andDifferentiableVariableTuple
protocols andTuple
conformances.GaussianFactorGraph.swift
: A factor graph whose factors are allGaussianFactor
s.
A factor graph stores factors in a storage array of type [ObjectIdentifier: AnyFactorArrayBuffer]
. ObjectIdentifier
is a built-in Swift identifier type and AnyFactorArrayBuffer
is defined in FactorsStorage.swift.
Optimizers
GradientDescent.swift
: a basic gradient descent optimizer.CGLS.swift
: a conjugate gradient solver for least-squares problems.LM.swift
: a Levenberg-Marquardt nonlinear optimizer.
MCMC
RandomWalkMetropolis.swift
: implements a Metropolis sampler modeled after its equivalent in TensorFlow Probability.
Geometry
This directory defines rotation and pose types.
We use the GTSAM naming scheme: Pose2
, Rot2
, Pose3
, and Rot3
.
Pose2.swift
: a 2D pose.Rot2.swift
: a 2D rotation.Pose3.swift
: a 3D pose.Rot3.swift
: a 3D rotation.
2D and 3D points are simply represented as Vector2
and Vector3
.
Image
ArrayImage.swift
:Differentiable
multi-channel image stored as[Double]
. Has a differentiabletensor
method andupdate
function.OrientedBoundingBox.swift
: a rectangular region of an image, not necessarily axis-aligned.Patch.swift
: a differentiablepatch
method forArrayImage
, that returns a resampled image for the givenOrientedBoundingBox
.
Datasets
This directory primarily contains support for reading G2O files.
LICENSE
Copyright 2020 The SwiftFusion Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.