ActiveNick/HoloBot ActiveNick/HoloBot

  • Stars
    star
    124
  • Rank 286,896 (Top 6 %)
  • Language
    C#
  • License
    MIT License
  • Created about 8 years ago
  • Updated over 3 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
ActiveNick/HoloBot
github

ActiveNick

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

HoloBot is a reusable 3D interface that allows HoloLens & VR users to interact with any bot using Mixed Reality & Speech.

HoloBot

Take your bots beyond Skype, Slack, Microsoft Teams and Facebook and bring them into the real world with Mixed Reality. Why simply type-chat with a bot when you could actually look at them, talk to them and listen to their answers?

HoloBot is reusable Windows Mixed Reality Unity project for Microsoft HoloLens v1, v2 and immersive VR headsets. The app acts as a holographic client for a chatbot. This 3D app lets you interact with a floating robot head using speech recognition, getting answers spoken back by the "bot" using Text-to-Speech. The commands sent to the bot are powered by the Microsoft Bot Framework and LUIS from Microsoft Cognitive Services.

  • Unity version: 2018.4.12f1
  • Mixed Reality Toolkit (MRTK) version: v2.1.0

To get started with HoloLens & Windows Mixed Reality development, visit the Windows Mixed Reality Dev Center. The HoloLens Developer Kit is available for sale in several countries at http://hololens.com.

Project Update

2021: I am not actively working on this given a lack of free cycles, but I do plan on revisiting this soon enough to upgrade to the latest version of Unity, test with the latest Speech SDK, and test on the latest build of HoloLens 2. New features are in planning too. If there are pressing issues, you can post them here and I will do my best to respond, but I cannot offer any guarantees at this time.

This current version is technically still a work in progress as I was migrating HoloBot to Unity 2018.4, the latest version of MRTKv2 (at the time), HoloLens 1 & 2, and use the new unified Speech SDK from Microsoft Cognitive Services. Note that builds in this branch might sometimes cause errors or have missing features given how MRTK and the speech SDK are evolving fast.

Note for users of older versions of Unity: Now that HoloBot has been upgraded to Unity 2018.4, I have archived the previous versions in a separate dev branches. Note that these legacy versions of HoloBot are only provided for archival purposes and these branches will no longer be worked on or supported:

  • Unity 5.6.3p2
  • Unity 2017.4

Requirements

The following plugins/packages are dependencies that must be imported manually in this project:

Features

  • Hovering bot head (aka HoloBot) with looping ambient sound.
  • Tap & hold the HoloBot to drag it to a different location, drop the hold to place.
  • Gaze at the HoloBot to trigger the speech recognizer, you will hear a "ping" sound.
  • Speak "commands" to HoloBot using natural language. HoloBot has only been tested with English for now.
  • The HoloBot speaks back to you using Speech Synthesis (aka Text-to-Speech, or TTS).
  • All sounds and speech use spatial sound that originate from the HoloBot's location in the room.
  • The "brain" of HoloBot can be any public bot built with the Microsoft Bot Framework. Build your Bot using C# or Node. See bot integration instructions below.

Video Demonstration

ScreenShot

The bot demonstrated in this video is The Maker Show Bot, also found here on GitHub. Feel free to fork the code and plug HoloBot to your own chatbot.

Instructions / Implementation Notes

  • The HoloBot model and sounds come from the Holographic Academy tutorial: [Holograms 240: Sharing Holograms].(https://developer.microsoft.com/en-us/windows/holographic/holograms_240).
  • HoloBot has finally been upgraded to a recent version of the HoloToolkit for Unity. See above for the specific version.
  • Uses the InputManager from MRTK for Unity (prefab) for Gaze & Gesture management.
  • Now using the Hand Draggable script instead of my own custom Tap to Place.
  • MicrophoneManager.cs now implements IFocusable for Gaze/Input Controller events (enter/leave), which triggers the speech recording.
  • Uses Text to Speech Manager from MRTK for Unity (Utilities scripts).
  • It is no longer needed to use your own Newtonsoft.Json.dll in the /Plugins folder of the HoloBot Unity project since the MRTK already includes it.

Connecting your Bot to HoloBot

  • Create and register your bot as per the intructions at https://dev.botframework.com. Bots can be built with C# & ASP.NET WebAPI or Javascript & Node.js. Since HoloBot uses free natural language dictation, it is highly recommended that your bot support NLP via the Language Understanding Intelligent Service (LUIS) from Microsoft Cognitive Services.
  • From the Bot Connector portal, enable the Direct Line channel on your bot, and enable version 3.0 of the Direct Line API.
  • Generate and copy your Direct Line secret (aka API key)
  • Open BotService.cs in the /Scripts folder of the HoloBot Unity project and paste your Direct Line secret in the _APIKEY private string

All

Acknowledgments

I want to offer special thanks to the following people who have helped in various ways over the years: Jared Bienz, Tobiah Zarlez, Etienne Margraff, Vanessa Arnauld, Sara Nagy, Phil Cherner, Kat Haris, and the original Microsoft Mixed Reality Academy team & mentors from 2016 - especially Pat - for their awesome training, resources, patience and help.

Follow Me

More Repositories

1

Unity-Text-to-Speech

Sample app used to demonstrate the use of Microsoft Cognitive Services Text-to-Speech APIs (aka Speech Synthesis) from within Unity.
C#
99
star
2

Unity-MS-SpeechSDK

Sample Unity project used to demonstrate Speech Recognition using the new Microsoft Speech Service (Preview) via WebSockets.
C#
62
star
3

Unity-SpeechWithLUIS

Sample Unity project used to demonstrate the integration of Speech Recognition and Language Understanding using the new Microsoft Speech Service (Preview) and LUIS from Microsoft Cognitive Services.
C#
28
star
4

AzureChatrforWindows

Windows Clients for a Cloud-based Cross-Platform Chat App for Windows Phone and Windows Tablets, Laptops and Desktops. Written in C# as a native Universal app using Visual Studio.
C#
19
star
5

TheMakerShowBot

This bot helps you get started as a maker by recommending online resources about project ideas, where to buy hardware boards, sensors and other parts, and more. This bot also recommends specific episodes of The Maker Show to watch based on your interests.
C#
17
star
6

MyWeather

Cross-Platform Mobile Weather App for iOS, Android, Windows 10 (UWP) and WP8 built in C# with Xamarin. Uses Xamarin.Forms to share 100% of XAML UI code.
C#
15
star
7

AzureChatrforAndroid

Android Client for a Cloud-based Cross-Platform Chat App for Smartphones and Tablets. Written in Java as a native app using Eclipse.
Java
13
star
8

Whiteboxing-UnityMLAgents

Experimental testbed where I test various Machine Learning & AI concepts using Unity ML Agents.
C#
10
star
9

Planetarium

A sample HoloLens app that uses spatial mapping to automatically place holograms within a room. This sample uses the HoloToolkit for Unity.
C#
8
star
10

WeatherTalk

Windows 10 / UWP sample weather app for phones, tablets, desktops, IoT devices, Xbox & HoloLens
C#
7
star
11

AzureStorageDemoUnity3D

Sample Unity project used to demonstrate the use of Azure Storage services in games and Mixed Reality projects.
C#
6
star
12

AzureChatrforiOS

iOS Client for a Cloud-based Cross-Platform Chat App for iPhone and iPad. Written in Objective-C as a native app using Xcode.
Objective-C
6
star
13

BingTranslateDemo

Mobile Cross-Platform Translator Demo powered by Bing. Contains 3 solutions: a PCL solution for the Bing Tranlator API wrapper, a Xamarin.Forms solution for iOS, Android and Windows Phone 8 Silverlight, and a Universal Windows (WinRT) app for Windows Store 8.1 and Windows Phone 8.1.
C#
5
star
14

BotFrameworkTestClient

Sample UWP project in C# for Windows 10 to test Direct Line calls to communicate with any bot created with the Microsoft Bot Framework.
C#
4
star
15

HoloModelShowcase

Mixed Reality demo project for HoloLens used to showcase a new product in 3D.
C#
4
star
16

Xamarin-Layouts

Sample Xamarin solution to experiment with Xamarin.Forms layouts.
C#
3
star
17

NodeChatBot

Basic ChatBot samples for the Microsoft Bot Framework, built with Node.js
JavaScript
3
star
18

MotionMatch

Motion Matching application using Microsoft Band, UWP, Azure IoT and Machine Learning
C#
3
star
19

SpeechAmbiance10

Universal Windows Platform (UWP) project that demonstrates development using Speech Synthesis (aka Text-to-Speech) using SSML on Windows 10 in C#.
C#
3
star
20

MSSpeechServiceWebSocketConsole

Sample applications (.NET console & UWP app) used to test Speech Recognition in the Microsoft Speech SDK using WebSockets.
C#
3
star
21

CognitiveServices-Demo-Face

Demo of Microsoft Cognitive Services using the Face API, built in C# for Windows 10 / UWP.
C#
3
star
22

DesertShooterVR

Sample first-person-style game for virtual & mixed reality, optimized for Microsoft HoloLens
C#
3
star
23

HoloExplorer-Mars

Mars exploration simulation for Windows Mixed Reality and HoloLens. Work in progress, stay tuned.
C#
2
star
24

HelloWeather

Mobile Weather Demo for Windows Phone 8.1
C#
2
star
25

KosmicWarz-WP

From my archives, Kosmic Warz is a 3D shooter game inspired by Space Invaders, built in C# & XNA 4.0 for the Windows Phone 7.5 Silverlight platform.
C#
2
star
26

dotNet-Text-to-Speech

Sample .NET application used to demonstrate the use of Microsoft Cognitive Services Text-to-Speech APIs.
C#
2
star
27

HoloDemo

Technical demo used for HoloLens & Mixed Reality developer training
C#
2
star
28

Holograms101-Origami

The completed version of the Holograms 101 tutorial from the Holographic Academy, which walks developers through a complete project, built in Unity, that demonstrates core Windows Holographic Platform and HoloLens features including gaze, gestures, voice input, spatial sound and spatial mapping.
C#
2
star
29

UnityStudy-360VideoSkybox

Windows Mixed Reality demo project for immersive VR headsets used to showcase 360 degrees video as the Skybox background
C#
1
star
30

ImmersiveMRTest

Testing playground for Immersive Mixed reality experiences using MRTK
C#
1
star
31

SpaceBallsVR

Sample VR game for Windows Mixed Reality immersive headsets where you shoot asteroids at planets.
C#
1
star
32

FiveOclock

Ever wanted to have a drink but it's too early in the day and you're afraid it's not socially acceptable to drink so early? What is it we tell ourselves to justify that early beer or glass of wine? It's five o'clock somewhere!!!
C#
1
star
33

HoloToolkitTankDemo

Technical demo that showcases how to easily build holographic experiences for Windows Mixed Reality & HoloLens using the Open Source HoloToolkit for Unity
C#
1
star
34

LearningUnity

Simple 3D pickup game used to teach the basics of the Unity Game Engine step-by-step.
C#
1
star
35

dotNetCore-Demos

A collection of samples used to demonstrate development with .NET Core and ASP.NET Core in my conference & meetup tech talks.
C#
1
star
36

Windows10-IoT-Core-Demos

Various demos for learning maker & IoT projects on Windows 10 IoT Core using UWP & C#, Arduino, etc.
C#
1
star
37

UnityLearningMecanim

Unity 5 playground to test humanoid animations with Mecanim.
C#
1
star
38

SpeechRecognitionAndTTSQuickstart

Windows Phone 8.1 Speech SDK Sample from MSDN - Reposted here as a convenience for viewers of the Cortana MVA. See the original sample at http://code.msdn.microsoft.com/Speech-Recognition-And-TTS-483fe645.
CSS
1
star
39

WeatherExperience

Simple demo app used to test the integration of UWP libraries as Unity plugins
C#
1
star
40

SpeechSynthesisWin81

Windows 8.1 Speech Synthesis Sample from MSDN - Reposted here as a convenience for viewers of the Cortana MVA. See the original sample at http://code.msdn.microsoft.com/Speech-synthesis-sample-6e07b218.
CSS
1
star
41

HelloHolograms

A variation of the Holograms 100 tutorial from the Holographic Academy. This one uses the HoloToolkit for Unity.
C#
1
star
42

CameraSimpleViewer

A simple Windows 10 camera viewer app written in C# for the Universal Windows Platform (UWP). I created this to easily show external cameras on screen when recording or presenting maker/IoT stuff.
C#
1
star
43

MyWeatherNUI

Cross-Platform Mobile Weather App for iOS, Android and Windows 10 built in C# with Xamarin. Uses a custom UI for each platform.
C#
1
star