• This repository has been archived on 12/Jan/2021
  • Stars
    star
    110
  • Rank 316,770 (Top 7 %)
  • Language
    JavaScript
  • License
    Other
  • Created over 8 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Discover dark data in videos with IBM Watson and IBM Cloud Functions

Cloud Functions Dark Vision - Discover dark data in videos with IBM Watson and IBM Cloud Functions

Build Status

Dark Vision is a technology demonstration leveraging Cloud Functions and Watson services. If you are looking for an official and supported IBM offering head over to the Watson Video Enrichment product. This product uses Watson APIs and additional technology to enrich video assets.

Think about all the videos individuals and companies (Media and Entertainment) accumulate every year. How can you keep track of what's inside of them so you can quickly search and find what you're looking for? Show me all the videos that have Arc De Triomphe in it. or Show me the all the videos that talk about peaches

What if we used artificial intelligence to process these videos to tell us which video has what we're looking for without us having to watch all of them.

Dark Vision is an application that processes videos to discover what's inside of them. By analyzing individual frames and audio from videos with IBM Watson Visual Recognition and Natural Language Understanding, Dark Vision builds a summary with a set of tags, famous people or landmarks detected in the video. Use this summary to enhance video search and categorization.

Watch this Youtube video to learn more about the app

Dark Vision

Overview and Architecture

Built using IBM Cloud, the application uses:

Extracting frames and audio from a video

The user uploads a video or image using the Dark Vision web application, which stores it in a Cloudant Database (1). Once the video is uploaded, Cloud Functions detects the new video (2) by listening to Cloudant changes (trigger). Cloud Functions then triggers the video and audio extractor action (3). During its execution, the extractor produces frames (images) (4), captures the audio track (5) and stores them in Cloudant (6, 7). The frames are then processed using Watson Visual Recognition, the audio with Watson Speech to Text and Natural Language Understanding. The results are stored in the same Cloudant DB. They can be viewed using Dark Vision web application or the iOS application.

Object Storage can complement Cloudant. When doing so, video and image medadata are stored in Cloudant and the media files are stored in Object Storage.

Architecture

extract_video digraph G { node [fontname = "helvetica"] rankdir=LR /* stores a video */ user -> storage [label="1"] /* cloudant change sent to openwhisk */ storage -> openwhisk [label="2"] /* openwhisk triggers the extractor */ openwhisk -> extractor [label="3"] /* extractor produces image frames and audio */ extractor -> frames [label="4"] extractor -> audio [label="5"] /* frames and audio are stored */ frames -> storage [label="6"] audio -> storage [label="7"] /* styling ****/ frames [label="Image Frames"] audio [label="Audio Track"] storage [shape=circle style=filled color="%234E96DB" fontcolor=white label="Data Store"] openwhisk [shape=circle style=filled color="%2324B643" fontcolor=white label="Cloud Functions"] } extract_video )

Processing frames and standalone images

Whenever a frame is created and uploaded (1), Cloudant emits a change event (2) and Cloud Functions triggers the analysis (3). The analysis (4) is persisted with the image (5).

Architecture

image_analysis digraph G { node [fontname = "helvetica"] /* stores a image */ frame -> storage [label="1"] /* cloudant change sent to openwhisk */ storage -> openwhisk [label="2"] /* openwhisk triggers the analysis */ openwhisk -> analysis [label="3"] /* extractor produces image frames */ {rank=same; frame -> storage -> openwhisk -> analysis -> watson [style=invis] } /* analysis calls Watson */ analysis -> watson [label="4"] /* results are stored */ analysis -> storage [label="5"] /* styling ****/ frame [label="Image Frame"] analysis [label="analysis"] storage [shape=circle style=filled color="%234E96DB" fontcolor=white label="Data Store"] openwhisk [shape=circle style=filled color="%2324B643" fontcolor=white label="Cloud Functions"] watson [shape=circle style=filled color="%234E96DB" fontcolor=white label="Watson\nVisual\nRecognition"] } image_analysis )

Processing audio

Whenever the audio track is extracted (1), Cloudant emits a change event (2) and Cloud Functions triggers the audio analysis (3).

Extract the audio transcript

Extracting the transcript from an audio track using the Speech to Text service may take more than 5 minutes depending on the video. Because Cloud Functions actions have a 5 minutes limit, waiting in the action for the audio processing to complete is not possible for longer videos. Fortunately the Speech to Text service has a very nice asynchronous API. Instead of waiting for Speech to Text to process the audio, Dark Vision sends the audio file to Speech to Text (4) and Speech to Text will notify Dark Vision with the transcript when it is done processing the audio (5). The result is attached to the audio document (6).

Architecture

extract_audio digraph G { node [fontname = "helvetica"] audio -> storage [label="1"] storage -> openwhisk [label="2"] openwhisk -> speechtotext [label="3"] speechtotext -> watson [label="4 - Start Recognition"] watson -> speechtotext [label="5 - Receive transcript"] speechtotext -> storage [label="6 - Store transcript"] audio [label="Audio Track"] {rank=same; audio -> storage -> openwhisk -> speechtotext -> watson [style=invis] } storage [shape=circle style=filled color="%234E96DB" fontcolor=white label="Data Store"] openwhisk [shape=circle style=filled color="%2324B643" fontcolor=white label="Cloud Functions"] speechtotext [label="speechtotext"] watson [shape=circle style=filled color="%234E96DB" fontcolor=white label="Watson\nSpeech to Text"] } extract_audio )

Analyze the transcript

Once the transcript is stored (1), the text analysis (3) is triggered (2) to detect concepts, entities and emotion (4). The result is attached to the audio (5).

Architecture

audio_analysis digraph G { node [fontname = "helvetica"] transcript -> storage [label="1"] storage -> openwhisk [label="2"] openwhisk -> textanalysis [label="3"] textanalysis -> nlu [label="4"] textanalysis -> storage [label="5"] /* extractor produces image frames */ {rank=same; transcript -> storage -> openwhisk -> textanalysis -> nlu [style=invis] } /* styling ****/ transcript [label="Transcript"] textanalysis [label="textanalysis"] storage [shape=circle style=filled color="%234E96DB" fontcolor=white label="Data Store"] openwhisk [shape=circle style=filled color="%2324B643" fontcolor=white label="Cloud Functions"] nlu [shape=circle style=filled color="%234E96DB" fontcolor=white label="Natural\nLanguage\nUnderstanding"] } ) audio_analysis

Prerequisites

  • IBM Cloud account. Sign up for Bluemix, or use an existing account.
  • Docker Hub account. Sign up for Docker Hub, or use an existing account.
  • Node.js >= 6.9.1
  • XCode 8.0, iOS 10, Swift 3 (For iOS application)

Deploying Dark Vision automatically in IBM Cloud

Dark Vision comes with a default toolchain you can use to deploy the solution with few clicks. If you want to deploy it manually, you can skip this section.

  1. Ensure your organization has enough quota for one web application using 256MB of memory and 4 services.

  2. Click Deploy to IBM Cloud to start the IBM Cloud DevOps wizard:

Deploy to IBM Cloud

โš ๏ธ Dark Vision can currently only be deployed in the US South region.

  1. Select the GitHub box.

    1. Decide whether you want to fork/clone the Dark Vision repository.
    2. If you decide to Clone, set a name for your GitHub repository.
  2. Select the Delivery Pipeline box.

    1. Enter an IBM Cloud API key.
    2. Select the resource group, region, organization and space where you want to create services and deploy the web application.
    3. Set the name of the Dark Vision web application. Pick a unique name to avoid conflicts.
    4. Optionally set the admin username and password for the application. When set, the application will prompt for this username and password when uploading videos/images, when resetting a video or an image. If the username and password are not defined, any visitor can upload videos/images for processing.
    5. If you already have a Watson Visual Recognition service instance you want to reuse, retrieve its API key from the credentials and set the value in the form. If you leave the field empty, the pipeline will create a new service instance automatically.
  3. Click Create.

  4. Select the Delivery Pipeline named darkvision.

    1. In the Environment Properties, you can change the service plans for the services to be created. COS_PLAN can be changed to Standard if you are already using the Lite plan in your account.
    2. When ready, press the Run Stage button in the DEPLOY stage to run the pipeline.
  5. Wait for the Deploy job to complete.

  6. Access the Dark Vision app when it's ready and start uploading videos and images!

iOS application to view the results (Optional)

The iOS application is a client to the API exposed by the web application to view the results of the analysis of videos. It is an optional piece.

To configure the iOS application, you need the URL of the web application deployed above. The web app exposes an API to list all videos and retrieve the results.

  1. Open ios/darkvision.xcworkspace with XCode

  2. Open the file darkvision/darkvision/model/API.swift

  3. Set the value of the constant apiUrl to the application host previously deployed.

  4. Save the file

Running the iOS application in the simulator

  1. Start the application from XCode with iPad Air 2 as the target

  1. Browse uploaded videos

  1. Select a video

Results are made of tags returned by Watson. The tags with the highest confidence score are shown. Tap a tag to change the main image to the frame where this tag was detected.

Code Structure

Cloud Functions - Deployment script

File Description
deploy.js Helper script to install, uninstall, update the Cloud Functions trigger, actions, rules used by Dark Vision.

Cloud Functions - Change listener

File Description
changelistener.js Processes Cloudant change events and calls the right actions. It controls the processing flow for videos and frames.

Cloud Functions - Frame extraction

The frame extractor runs as a Docker action created with the Cloud Functions Docker SDK:

  • It uses ffmpeg to extract frames and audio from the video.
  • It is written as a nodejs app to benefit from several nodejs helper packages (Cloudant, ffmpeg, imagemagick)
File Description
Dockerfile Docker file to build the extractor image. It pulls ffmpeg into the image together with node. It also runs npm install for both the server and client.
extract.js The core of the frame extractor. It downloads the video stored in Cloudant, uses ffmpeg to extract frames and video metadata, produces a thumbnail for the video. By default it produces around 15 images for a video. This can be changed by modifying the implementation of getFps. First 15 min of audio is also exported.
app.js Adapted from the Cloud Functions Docker SDK to call the extract.js node script.

Cloud Functions - Frame analysis

analysis.js holds the JavaScript code to perform the image analysis:

  1. It retrieves the image data from the Cloudant document. The data has been attached by the frame extractor as an attachment named "image.jpg".
  2. It saves the image file locally.
  3. If needed, it resizes the image so that it matches the requirements of the Watson service
  4. It calls Watson
  5. It attachs the results of the analysis to the image and persist it.

The action runs asynchronously.

The code is very similar to the one used in the Vision app.

Cloud Functions - Audio analysis

File Description
speechtotext.js Uses Speech to Text to transcript the audio. It acts as the callback server for the asynchronous API of Speech to Text service. The speechtotext action is exposed as a public HTTP endpoint by the deploy.js script.
textanalysis.js Calls Natural Language Understanding on the transcript.

Web app

The web application allows to upload videos (and images). It shows the video and image catalog and for each video the extracted frames.

File Description
app.js The web app backend handles the upload of videos/images, and exposes an API to retrieve all videos, their frames, to compute the summary
Services Services used by controllers
Home page Controller and view for the home page
Video page Controller and view for the video detail page

Shared code between Cloud Functions actions and web app

These files are used by the web application and the Cloud Functions actions. They are automatically injected in the Cloud Functions actions by the deploy.js script and during the build of the Docker image. These scripts have dependencies on Cloudant, async, pkgcloud which are provided by default in Cloud Functions Node.js actions.

File Description
cloudantstorage.js Implements API on top of Cloudant to create/read/update/delete video/image metadata and to upload files
cloudobjectstorage.js Implements the file upload operations on top of Cloud Object Storage. Used by cloudantstorage.js when Cloud Object Storage is configured.
cloudant-designs.json Design documents used by the API to expose videos and images. They are automatically loaded into the database when the web app starts for the first time.

iOS

The iOS app is an optional part of the Dark Vision sample app. It uses the API exposed by the web application to display the videos in the catalog and their associated tags.

File Description
API.swift Calls the web app API. Update the constant apiUrl to map to the location of your web app.

Contribute

Please create a pull request with your desired changes.

Troubleshooting

Dark Vision correctly processes video frames but does not process the audio track

This has been reported several times when using the toolchain. It is tracked as issue 51. Make sure to look at the toolchain DEPLOY log to confirm the problem. Locate the line Registering Speech to Text callback... to identify the error.

Cloud Functions

Polling activations is good start to debug the Cloud Functions action execution. Run

ibmcloud fn activation poll

and upload a video for analysis.

Web application

Use

ibmcloud cf logs <appname>

to look at the live logs for the web application

License

See License.txt for license information.

More Repositories

1

terraform-provider-ibm

https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs
Go
341
star
2

redli

Redli - A humane alternative to the Redis-cli and TLS
Go
273
star
3

phonebot

Slackbot using IBM Watson and Twilio to make phone calls via slack commands
JavaScript
237
star
4

todo-apps

Sample ToDo application (various languages) running on IBM Cloud
Java
207
star
5

chatbot-watson-android

An Android ChatBot powered by Watson Services - Assistant, Speech-to-Text and Text-to-Speech on IBM Cloud.
Java
196
star
6

ibm-cloud-cli-release

156
star
7

docs

IBM Cloud product documentation.
149
star
8

ibm-cloud-developer-tools

Installation scripts for new IBM Cloud Developer CLI Plugin and it's dependencies.
Shell
131
star
9

get-started-python

A Python application and tutorial that use Flask framework to provide a REST API to receive requests from the UI. The API then persists the data to a Cloudant database.
JavaScript
120
star
10

kube-samples

HCL
117
star
11

logistics-wizard

The Logistics Wizard is an end-to-end, smart supply chain management solution that showcases how to execute hybrid cloud, microservices, and predictive data analytics in the real world.
112
star
12

vscode-log-output-colorizer

Language extension for VSCode that adds syntax colorization for both the output/debug/extensions panel and *.log files.
JavaScript
101
star
13

get-started-node

Sample and tutorial to help you get started with Express, REST API and a database.
JavaScript
78
star
14

ansible-collection-ibm

https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs
Python
70
star
15

bluechatter

Deploy & Scale a chat app using Cloud Foundry, Docker Container and Kubernetes
JavaScript
65
star
16

election-insights

Sentiment analysis and visualization on all things election
JavaScript
63
star
17

openwhisk-workshops

Series of IBM Bluemix OpenWhisk workshops
58
star
18

node-mqtt-for-anki-overdrive

Node.js Controller and MQTT API for Anki Overdrive
JavaScript
55
star
19

cloudco-insurance

A modern insurance company. The application showcases cognitive and cloud computing ideas in the context of insurance.
45
star
20

real-time-tone-analysis

Allows users to analyze the tone of their speech in real-time using Watson Speech-to-Text and Tone Analysis technologies
CSS
45
star
21

openwhisk-visionapp

A sample iOS app for image tagging and face detection built with IBM Bluemix OpenWhisk
Swift
43
star
22

python-iot-raspberry-pi

A demo/sample python app to demonstrate using inputs and outputs on a Raspberry Pi
Python
43
star
23

nodejs-MEAN-stack

Solution tutorial: Modern Web Application using MEAN stack on IBM Cloud
JavaScript
41
star
24

watson-conversation-variables

Samples using context variables and (system) entities in IBM Watson Assistant (formerly Conversation) service on IBM Cloud
40
star
25

bluemix-go

Go library for accessing the Bluemix API
Go
37
star
26

health-blockchain

A blockchain for fitness data demo
CSS
37
star
27

insurance-bot

A message based insurance experience
CSS
36
star
28

skylink

Skylink is a sample application that connects drones to the cloud with near realtime image analysis leveraging IBM Cloudant, OpenWhisk, IBM Watson, and IBM Bluemix.
JavaScript
36
star
29

node-helloworld

Sample/tutorial hello world in node
HTML
35
star
30

go-etcd-rules

A simple rules engine for use with etcd
Go
35
star
31

ibm-cloud-cli-sdk

Go
34
star
32

iot-sensor-tag

Sample code for connecting to a TI Sensor Tag and using that data in apps deployed to Bluemix.
JavaScript
33
star
33

github-traffic-stats

Manage and automatically collect Github traffic statistics for repositories
Python
32
star
34

ziggy

JavaScript
32
star
35

idemix-issuer-verifier

Sample app built to demonstrate how to integrate IBM Identity Mixer privacy-preserving authentication API using Bluemix:
JavaScript
32
star
36

Kitura-Starter

Kitura-based server starter application for the IBM Cloud.
Swift
30
star
37

MERN-app

[DEPRECATED] Simple starter for the MERN stack (MongoDB, Express, React, Node.js) on Kubernetes
JavaScript
28
star
38

get-started-java

Sample and tutorial to help you get started with a Java EE app, REST API and a database.
JavaScript
28
star
39

openwhisk-slackapp

A serverless Slack app built with Slack Events API and IBM Cloud Functions
JavaScript
27
star
40

slack-chatbot-database-watson

Code for the solution tutorial "Build a database-driven Slackbot" (chatbot) with a custom extension in IBM Watson Assistant
Python
27
star
41

hackathon-prep-material

Getting ready for a Bluemix hackathon? Here is some great material to get you started.
26
star
42

doctor-watson

Bluemix application using Watson Cloud Services with Twilio to create your own doctor on the phone.
JavaScript
26
star
43

collaboration

Patterns for collaborative LoB Applications on Bluemix via the CLEAN stack
JavaScript
24
star
44

cf-manifest-generator

Generates manifest file for Cloud Foundry deployments
Java
23
star
45

HybridBanking-Android

Application that demos interaction between a customer and company support staff using Bluemix Mobile backend and Watson services
Java
23
star
46

BigInsights-on-Apache-Hadoop

Example projects for 'BigInsights for Apache Hadoop' on IBM Bluemix
23
star
47

car-data-management

A sample Watson IoT Context Mapping and Driver Behavior application that sends simulated car probe data and issues analysis request.
Java
22
star
48

swift-helloworld

Swift
22
star
49

investment-insights-for-asset-managers

Investment Insights for Asset Managers is a modern finance portfolio manager that provides real-time insight into how news all around the world can impact the investment to any given portfolio.
JavaScript
22
star
50

aspnet-core-helloworld

Sample ASP.NET Core application for Bluemix.
C#
21
star
51

capital-weather

Using ArcGIS and the Weather Channel and Google Maps Geocoding APIs, discover and interact with weather data from any city across the world.
CSS
21
star
52

jpetstore-kubernetes

Modernize and Extend: JPetStore on IBM Cloud Kubernetes Service
Java
21
star
53

intercom-nodejs

Intercom gives users the ability to remotely communicate with a chat representative through Watson speech-to-text and Twilio integrations.
JavaScript
21
star
54

nodejs-cloudant

Sample Node.js application which uses Bluemix Cloudant NoSQL service
JavaScript
20
star
55

multiple-deployment-options

Shows how one service can be deployed to multiple deployment options
JavaScript
20
star
56

golang-openssl-wrapper

Go
20
star
57

go-cloudant

go-cloudant is a Cloudant DB client written in Go
Go
20
star
58

box-watson

Sample app built to demonstrate how to integrate the Box API with Watson using Bluemix
JavaScript
20
star
59

talent-manager

Sample/Tutorial using Cloudant, Watson in a JavaEE App on IBM Bluemix
Java
19
star
60

bluemix-cloud-connectors

Simplifies the way Java developers connect to services in Bluemix.
Java
18
star
61

parrot-sample

Sample code which uses MQTT to control a Parrot AR Drone
JavaScript
18
star
62

nodejs-cloudantdb-crud-example

Example of CRUD operations on a Cloudant database using a Node.js runtime on IBM Bluemix
JavaScript
17
star
63

terraform-opa-ibm

An OPA library to develop IT Control policies, for the IBM Cloud
Open Policy Agent
17
star
64

go-hello-world

A Hello World app using Golang
Go
17
star
65

vpc-tutorials

Companion scripts to VPC tutorials
HCL
17
star
66

DevOps-Services-Docs

We can help you get it done! Bluemix DevOps Services Content
JavaScript
17
star
67

serverless-guestbook

A serverless guestbook web application and API built with Cloud Functions
JavaScript
16
star
68

terraform-ibm-openshift

Provision IBM Cloud infrastructure with Terraform, and install Red Hatยฎ OpenShift Container Platform 3.
HCL
15
star
69

multiple-environments-as-code

Plan, create and update deployment environments with the IBM Cloud CLI and Terraform
HCL
15
star
70

logistics-wizard-webui

Web UI for Logistics Wizard Showcase demo. The Logistics Wizard is an end-to-end, smart supply chain management solution that showcases how to execute hybrid cloud, microservices, and predictive data analytics in the real world.
JavaScript
14
star
71

secure-file-storage

Apply end to end security to securely store files
Shell
14
star
72

compose-mongodb-helloworld-nodejs

Shows you how to connect to an IBM Compose for MongoDB for IBM Cloud service using Node.js
JavaScript
14
star
73

logistics-wizard-erp

Defines the API used by the Logistics Wizard to access data from an ERP system. Also provides a default implementation to be used as a simulator.
JavaScript
14
star
74

facebook-photo-analyzer

A sample/demo app that uses IBM Bluemix to analyze the objects in photos on Facebook via IBM Watson and analyzes the sentiment of the comments on photos via the Alchemy API
JavaScript
14
star
75

get-started-php

A PHP application and tutorial that use slim framework to provide a REST API to receives requests from the UI. The API then uses the sag client library to persist the visitor names to a Cloudant database.
PHP
14
star
76

logistics-wizard-toolchain

Toolchain configuration files for the logistics-wizard demo
13
star
77

get-started-go

HTML
13
star
78

insights-weather

A sample application featuring the Weather Company Data for IBM Bluemix service.
JavaScript
13
star
79

ibm-developer-extension-vscode

Extension for VS Code editor to enable IBM Cloud CLI capabilities from within the editing environment.
TypeScript
13
star
80

assistant-shop.r

Demo showcasing how existing business processes and rules can be augmented with Web RTC and cognitive services to drive customer engagement and help make smarter business decisions.
JavaScript
12
star
81

sql-query-clients

Client samples for IBM Cloud SQL Query service
Jupyter Notebook
12
star
82

insurance-bot-dashboard

Admin Dashboard to monitor and analyze chats
JavaScript
12
star
83

php-mysql

Simple PHP application to demonstrate connecting and interacting with a MySQL service on Bluemix
PHP
12
star
84

cf-deployment-tracker-service

๐Ÿšซ This project is no longer maintained
JavaScript
12
star
85

openwhisk-typescript

A starter kit for using Typescript with IBM Cloud Functions
JavaScript
11
star
86

weather-company-data-demo

A demo app that uses the Weather Company Data for IBM Bluemix service.
JavaScript
11
star
87

IBM-Analytics-Engine

Tutorials, tips and code for the IBM Analytics Engine
HTML
10
star
88

java-cloudant

Sample Java application which uses Bluemix Cloudant NoSQL service.
Java
10
star
89

match_tracker

Match Tracker demo showing how to use Apache Kafka, IBM Bluemix and Twitter Insights to process real-time social media streams.
JavaScript
10
star
90

openwhisk-publisher

Static website generation with Jekyll, Object Storage and OpenWhisk in Bluemix
Shell
10
star
91

iot-device-phone-simulator

A web application which acts as an IoT device when loaded in a smart phone browser. The data from the sensors are then used for Anomaly detection.
JavaScript
10
star
92

django-boilerplate

An example application that runs Django on IBM Bluemix.
Python
9
star
93

sphero-bluemix-speech

Example/Tutorial app that uses Watson Speech to text service to control a Sphero
CSS
9
star
94

ibmcloud-image-builder

ibmcloud-image-builder
Shell
9
star
95

webapp-with-cos-and-cdn

A web application with its static files hosted in Cloud Object Storage and accessible through a Content Delivery Network
HTML
9
star
96

application-log-analysis

Code for a tutorial on how to use the IBM Log Analysis and monitoring with IBM Cloud Monitoring
Python
9
star
97

map-driver-insights

A sample Watson IoT Context Mapping and Driver Behavior application that displays analysis results on map UI.
Java
9
star
98

gp-java-tools

CLI Tools for Globalization Pipeline
Java
9
star
99

cloud-sql-database

Code for a tutorial on how to provision an SQL (relational) database on the IBM Cloud, load data and use it with a Python (Flask / Jinja) web app.
HTML
9
star
100

JavaCloudTrader

http://cloudtrader.mybluemix.net/
Java
9
star