• Stars
    star
    196
  • Rank 198,524 (Top 4 %)
  • Language
    C++
  • License
    MIT License
  • Created about 1 year ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

C++ library for Fearless Timeseries Logging

Data Tamer

cmake Ubuntu ros2 codecov

DataTamer is a library to log/trace numerical variables over time and takes periodic "snapshots" of their values, to later visualize them as timeseries.

It works great with PlotJuggler, the timeseries visualization tool (note: you will need PlotJuggler 3.8.2 or later).

DataTamer is "fearless data logger" because you can record hundreds or thousands of variables: even 1 million points per second should have a fairly small CPU overhead.

Since all the values are aggregated in a single "snapshot", it is usually meant to record data in a periodic loop (a very frequent use case, in robotics applications).

Kudos to pal_statistics, for inspiring this project.

How it works

architecture

DataTamer can be used to monitor multiple variables in your applications.

Channels are used to take "snapshots" of a subset of variables at a given time. If you want to record at different frequencies, you can use different channels.

DataTamer will forward the collected data to 1 or multiple sinks; a sink may save the information immediately in a file (currently, we support MCAP) or publish it using an inter-process communication, for instance, a ROS2 publisher.

You can easily create your own, specialized sinks.

Use PlotJuggler to visualize your logs offline or in real-time.

Features

  • Serialization schema is created at run-time: no need to do code generation.
  • Suitable for real-time applications: very low latency (on the side of the callee).
  • Multi-sink architecture: recorded data can be forwarded to multiple "backends".
  • Very low serialization overhead, in the order of 1 bit per traced value.
  • The user can enable/disable traced variables at run-time.

Limitations

  • Traced variables can not be added (registered) once the recording starts (first takeSnapshot).
  • Focused on periodic recording. Not the best option for sporadic, asynchronous events.
  • If you use DataTamer::registerValue you must be careful about the lifetime of the object. If you prefer a safer RAII interface, use DataTamer::createLoggedValue instead.

Examples

Basic example

#include "data_tamer/data_tamer.hpp"
#include "data_tamer/sinks/mcap_sink.hpp"

int main()
{
  // Multiple channels can use this sink. Data will be saved in mylog.mcap
  auto mcap_sink = std::make_shared<DataTamer::MCAPSink>("mylog.mcap");

  // Create a channel and attach a sink. A channel can have multiple sinks
  auto channel = DataTamer::LogChannel::create("my_channel");
  channel->addDataSink(mcap_sink);

  // You can register any arithmetic value. You are responsible for their lifetime!
  double value_real = 3.14;
  int value_int = 42;
  auto id1 = channel->registerValue("value_real", &value_real);
  auto id2 = channel->registerValue("value_int", &value_int);

  // If you prefer to use RAII, use this method instead
  // logged_real will unregister itself when it goes out of scope.
  auto logged_real = channel->createLoggedValue<float>("my_real");

  // Store the current value of all the registered values
  channel->takeSnapshot();

  // You can disable (i.e., stop recording) a value like this
  channel->setEnabled(id1, false);
  // or, in the case of a LoggedValue
  logged_real->setEnabled(false);

  // The next snapshot will contain only [value_int], i.e. [id2],
  // since the other two were disabled
  channel->takeSnapshot();
}

How to register custom types

Containers such as std::vector and std::array are supported out of the box. You can also register a custom type, as shown in the example below.

#include "data_tamer/data_tamer.hpp"
#include "data_tamer/sinks/mcap_sink.hpp"
#include "data_tamer/custom_types.hpp"

// a custom type
struct Point3D
{
  double x;
  double y;
  double z;
};

namespace DataTamer
{
template <> struct TypeDefinition<Point3D>
{
  // Provide the name of the type
  std::string typeName() const { return "Point3D"; }
  // List all the member variables that you want to be saved (including their name)
  template <class Function> void typeDef(Function& addField)
  {
    addField("x", &Point3D::x);
    addField("y", &Point3D::y);
    addField("z", &Point3D::z);
  }
}
} // end namespace DataTamer

int main()
{
  auto channel = DataTamer::LogChannel::create("my_channel");
  channel->addDataSink(std::make_shared<DataTamer::MCAPSink>("mylog.mcap"));

  // Array/vectors are supported natively
  std::vector<double> values = {1, 2, 3, 4};
  channel->registerValue("values", &values);

  // Requires the implementation of DataTamer::TypeDefinition<Point3D>
  Point3D position = {0.1, -0.2, 0.3};
  channel->registerValue("position", &position);

  // save the data as usual ...
  channel->takeSnapshot();
}

Compilation

Compiling with ROS2

Just use colcon :)

Compiling with Conan (not ROS2 support)

Note that the ROS2 publisher will NOT be built when using this method.

Assuming conan 2.x installed. From the source directory.

Release:

conan install . -s compiler.cppstd=gnu17 --build=missing -s build_type=Release
cmake -S . -B build/Release -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_TOOLCHAIN_FILE="build/Release/generators/conan_toolchain.cmake"
cmake --build build/Release --parallel

Debug:

conan install . -s compiler.cppstd=gnu17 --build=missing -s build_type=Debug
cmake -S . -B build/Debug -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_TOOLCHAIN_FILE="build/Debug/generators/conan_toolchain.cmake"
cmake --build build/Debug --parallel

How to deserialize data recorded with DataTamer

I will write more extensively about the serialization format used by DataTamer, but for the time being I created a single header file without external dependencies that you can just copy into your project: data_tamer_parser.hpp

You can see how it is used in this example: mcap_reader

More Repositories

1

rviz_visual_tools

C++ API wrapper for displaying shapes and meshes in Rviz
C++
747
star
2

ros_control_boilerplate

Provides a simple simulation interface and template for setting up a hardware interface for ros_control
C++
244
star
3

generate_parameter_library

Declarative ROS 2 Parameters
Python
228
star
4

roscpp_code_format

Auto formatting script for ROS C++ Style Guidelines
226
star
5

deep_grasp_demo

Deep learning for grasp detection within MoveIt.
C++
104
star
6

abb_ros2

C++
86
star
7

pick_ik

Inverse Kinematics solver for MoveIt
C++
68
star
8

topic_based_ros2_control

ros2_control hardware interface that uses topics to command the robot and publish its state
C++
58
star
9

tf_visual_tools

Manually manipulate TFs in ROS using this Rviz plugin.
C++
51
star
10

ros2_robotiq_gripper

C++
48
star
11

RSL

ROS Support Library
C++
46
star
12

rosparam_shortcuts

Quickly load variables from rosparam with good command line error checking.
C++
38
star
13

descartes_capability

Drop-in capability for MoveIt's move_group that uses Descartes
C++
37
star
14

ros_testing_templates

C++
28
star
15

tf_keyboard_cal

Allows manual control of a TF through the keyboard or interactive markers
C++
27
star
16

quick-ros-install

Instant install script for ROS
Shell
25
star
17

ros2_package_template

A ROS2 package cookiecutter template
C++
22
star
18

moveit_studio_ur_ws

Workspace containing example MoveIt Studio configuration packages.
C++
21
star
19

UR10e_welding_demo

C++
21
star
20

crac_build_system

The unified Catin / ROSBuild / Ament / Colcon build system
Shell
21
star
21

snowbot_operating_system

The weather outside is frightful
C++
18
star
22

moveit_sim_controller

A simulation interface for a hardware interface for ros_control, and loads default joint values from SRDF
C++
14
star
23

ik_benchmarking

Utilities for IK solver benchmarking with MoveIt 2
C++
13
star
24

moveit_shelf_picking

Benchmarking suite for dual arm manipulation
C++
12
star
25

ros2_epick_gripper

ROS2 driver for the Robotiq EPick gripper.
C++
11
star
26

ROStoROS2

A living documentation of automate-able and manual steps for porting ROS packages to ROS2
10
star
27

moveit_studio_sdk

MoveIt Studio SDK
Python
9
star
28

trackjoint

C++
9
star
29

ros_reflexxes

Reflexxes Type II provides acceleration-limited trajectory smoothing
C++
9
star
30

ruckig_traj_smoothing

A jerk-limited trajectory smoothing plugin
8
star
31

graph_msgs

ROS messages for publishing graphs of different data types
CMake
8
star
32

launch_param_builder

Python library for loading parameters in launch files
Python
8
star
33

moveit_boilerplate

Basic functionality for easily building applications on MoveIt! in C++
C++
6
star
34

stretch_moveit_plugins

Package for the inverse kinematics solver for stretch from Hello Robot Inc.
C++
5
star
35

tool_changing

MoveGroup capability to dynamically change end-effectors
C++
4
star
36

test_examples

Examples of different test techniques (integration, unit, gtest, dependency injection...)
Python
4
star
37

moveit_studio_example_behaviors

C++
4
star
38

CMakeGuidelines

Collection of useful cmake tips
4
star
39

moveit_differential_ik_plugin

A plugin to provide differential kinematics. Used in admittance control.
C++
4
star
40

pronto

C++
4
star
41

light_painting_demo

C++
3
star
42

cpp_polyfills

Vendored Pollyfills for C++
C++
3
star
43

ocs2_robotic_assets

CMake
2
star
44

rqt2_setup

2
star
45

Moveit2_Tutorials

test deployment of moveit 2 tutorials
HTML
2
star
46

cartesian_msgs-release

1
star
47

moveit_sim_controller-release

1
star
48

ros_control_boilerplate-release

1
star
49

cirs_girona_cala_viuda

ROS2 version of girona cala viuda dataset
Python
1
star
50

picknik_controllers

PickNik's ros2_controllers
C++
1
star
51

rviz_visual_tools-release

1
star
52

rosparam_shortcuts-release

1
star
53

tf_keyboard_cal-release

1
star
54

rqt2_demo_nodes

Python
1
star
55

picknik_accessories

Various accessories used for studio configuration packages
CMake
1
star
56

moveit_studio_kortex_ws

C++
1
star
57

moveit_camera_survey

Code for survey of RGB-D cameras for manipulation
Dockerfile
1
star
58

picknik_ur5_moveit_config

CMake
1
star
59

beta_moveit_website

Demo site for major changes to moveit.ros.org. Found at http://moveit_beta.picknik.ai/
JavaScript
1
star
60

robot_on_rails

MoveIt Studio package for a UR robot on a linear rail
C++
1
star
61

c3pzero

Python
1
star
62

picknik_ament_copyright

Python
1
star
63

boost_sml

State Machine Library ROS Package
C++
1
star
64

cartesian_msgs

Experimental messages for sending Cartesian commands to a robot.
CMake
1
star