• Stars
    star
    530
  • Rank 83,660 (Top 2 %)
  • Language
    Go
  • License
    MIT License
  • Created about 5 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Go implementation of the MediaDevices API.


Pion MediaDevices

Go implementation of the MediaDevices API

Slack Widget GitHub Workflow Status Go Reference Coverage Status License: MIT


mediadevices provides access to media input devices like cameras, microphones, and screen capture. It can also be used to encode your video/audio stream to various codec selections. mediadevices abstracts away the complexities of interacting with things like hardware and codecs allowing you to focus on building appilcations, interacting only with an amazingly simple, easy, and elegant API!

Install

go get -u github.com/pion/mediadevices

Usage

The following snippet shows how to capture a camera stream and store a frame as a jpeg image:

package main

import (
	"image/jpeg"
	"os"

	"github.com/pion/mediadevices"
	"github.com/pion/mediadevices/pkg/prop"

	// This is required to register camera adapter
	_ "github.com/pion/mediadevices/pkg/driver/camera" 
	// Note: If you don't have a camera or your adapters are not supported,
	//       you can always swap your adapters with our dummy adapters below.
	// _ "github.com/pion/mediadevices/pkg/driver/videotest"
)

func main() {
	stream, _ := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
		Video: func(constraint *mediadevices.MediaTrackConstraints) {
			// Query for ideal resolutions
			constraint.Width = prop.Int(600)
			constraint.Height = prop.Int(400)
		},
	})

	// Since track can represent audio as well, we need to cast it to 
	// *mediadevices.VideoTrack to get video specific functionalities
	track := stream.GetVideoTracks()[0]
	videoTrack := track.(*mediadevices.VideoTrack)
	defer videoTrack.Close()

	// Create a new video reader to get the decoded frames. Release is used 
	// to return the buffer to hold frame back to the source so that the buffer 
	// can be reused for the next frames.
	videoReader := videoTrack.NewReader(false)
	frame, release, _ := videoReader.Read()
	defer release()

	// Since frame is the standard image.Image, it's compatible with Go standard 
	// library. For example, capturing the first frame and store it as a jpeg image.
	output, _ := os.Create("frame.jpg")
	jpeg.Encode(output, frame, nil)
}

More Examples

  • Webrtc - Use Webrtc to create a realtime peer-to-peer video call
  • Face Detection - Use a machine learning algorithm to detect faces in a camera stream
  • RTP Stream - Capture camera stream, encode it in H264/VP8/VP9, and send it to a RTP server
  • HTTP Broadcast - Broadcast camera stream through HTTP with MJPEG
  • Archive - Archive H264 encoded video stream from a camera

Available Media Inputs

Input Linux Mac Windows
Camera ✔️ ✔️ ✔️
Microphone ✔️ ✔️ ✔️
Screen ✔️ ✔️ ✔️

By default, there's no media input registered. This decision was made to allow you to play only what you need. Therefore, you need to import the associated packages for the media inputs. For example, if you want to use a camera, you need to import the camera package as a side effect:

import (
	...
	_ "github.com/pion/mediadevices/pkg/driver/camera"
)

Available Codecs

In order to encode your video/audio, mediadevices needs to know what codecs that you want to use and their parameters. To do this, you need to import the associated packages for the codecs, and add them to the codec selector that you'll pass to GetUserMedia:

package main

import (
	"github.com/pion/mediadevices"
	"github.com/pion/mediadevices/pkg/codec/x264"      // This is required to use H264 video encoder
	_ "github.com/pion/mediadevices/pkg/driver/camera" // This is required to register camera adapter
)

func main() {
	// configure codec specific parameters
	x264Params, _ := x264.NewParams()
	x264Params.Preset = x264.PresetMedium
	x264Params.BitRate = 1_000_000 // 1mbps

	codecSelector := mediadevices.NewCodecSelector(
		mediadevices.WithVideoEncoders(&x264Params),
	)
	
	mediaStream, _ := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
		Video: func(c *mediadevices.MediaTrackConstraints) {},
		Codec: codecSelector, // let GetUsermedia know available codecs
	})
}

Since mediadevices doesn't implement the video/audio codecs, it needs to call the codec libraries from the system through cgo. Therefore, you're required to install the codec libraries before you can use them in mediadevices. In the next section, it shows a list of available codecs, where the packages are defined (documentation linked), and installation instructions.

Note: we do not provide recommendations on choosing one codec or another as it is very complex and can be subjective.

Video Codecs

x264

A free software library and application for encoding video streams into the H.264/MPEG-4 AVC compression format.

mmal

A framework to enable H264 hardware encoding for Raspberry Pi or boards that use VideoCore GPUs.

openh264

A codec library which supports H.264 encoding and decoding. It is suitable for use in real time applications.

vpx

A free software video codec library from Google and the Alliance for Open Media that implements VP8/VP9 video coding formats.

vaapi

An open source API that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities (currently support VP8/VP9).

Audio Codecs

opus

A totally open, royalty-free, highly versatile audio codec.

Benchmark

Result as of Nov 4, 2020 with Go 1.14 on a Raspberry pi 3, mediadevices can produce video, encode, send across network, and decode at 720p, 30 fps with < 500 ms latency.

The test was taken by capturing a camera stream, decoding the raw frames, encoding the video stream with mmal, and sending the stream through Webrtc.

FAQ

Failed to find the best driver that fits the constraints

mediadevices provides an automated driver discovery through GetUserMedia and GetDisplayMedia. The driver discover algorithm works something like:

  1. Open all registered drivers
  2. Get all properties (property describes what a driver is capable of, e.g. resolution, frame rate, etc.) from opened drivers
  3. Find the best property that meets the criteria

So, when mediadevices returns failed to find the best driver that fits the constraints error, one of the following conditions might have occured:

  • Driver was not imported as a side effect in your program, e.g. import _ github.com/pion/mediadevices/pkg/driver/camera
  • Your constraint is too strict that there's no driver can fullfil your requirements. In this case, you can try to turn up the debug level by specifying the following environment variable: export PION_LOG_DEBUG=all to see what was too strict and tune that.
  • Your driver is not supported/implemented. In this case, you can either let us know (file an issue) and wait for the maintainers to implement it. Or, you can implement it yourself and register it through RegisterDriverAdapter
  • If trying to use import _ github.com/pion/mediadevices/pkg/driver/screen note that you will need to use GetDisplayMedia instead of GetUserMedia

Failed to find vpx/x264/mmal/opus codecs

Since mediadevices uses cgo to access video/audio codecs, it needs to find these libraries from the system. To accomplish this, pkg-config is used for library discovery.

If you see the following error message at compile time:

# pkg-config --cflags  -- vpx
Package vpx was not found in the pkg-config search path.
Perhaps you should add the directory containing `vpx.pc'
to the PKG_CONFIG_PATH environment variable
No package 'vpx' found
pkg-config: exit status 1

There are 2 common problems:

  • The required codec library is not installed (vpx in this example). In this case, please refer to the available codecs.
  • Pkg-config fails to find the .pc files for this codec (reference). In this case, you need to find where the codec library's .pc is stored, and let pkg-config knows with: export PKG_CONFIG_PATH=/path/to/directory.

Roadmap

The library can be used with our WebRTC implementation. Please refer to that roadmap to track our major milestones.

Community

Pion has an active community on the Slack.

Follow the Pion Twitter for project updates and important WebRTC news.

We are always looking to support your projects. Please reach out if you have something to build! If you need commercial support or don't want to use public methods you can contact us at [email protected]

Contributing

Check out the contributing wiki to join the group of amazing people making this project possible: AUTHORS.txt

License

MIT License - see LICENSE for full text

More Repositories

1

webrtc

Pure Go implementation of the WebRTC API
Go
13,425
star
2

turn

Pion TURN, an API for building TURN clients and servers
Go
1,826
star
3

example-webrtc-applications

Examples of WebRTC applications that are large, or use 3rd party libraries
Go
1,031
star
4

awesome-pion

A curated list of awesome things related to Pion
Shell
723
star
5

stun

A Go implementation of STUN
Go
625
star
6

dtls

DTLS 1.2 Server/Client implementation for Go
Go
584
star
7

ice

A Go implementation of ICE
Go
430
star
8

opus

Pure Go implementation of Opus
Go
389
star
9

rtp

A Go implementation of RTP
Go
349
star
10

rtwatch

Watch videos with friends using WebRTC, let the server do the pausing and seeking.
Go
301
star
11

sctp

A Go implementation of SCTP
Go
219
star
12

offline-browser-communication

Demonstration of a browser connecting to Pion WebRTC without a signaling server.
Go
195
star
13

mdns

Pure Go implementation of Multicast DNS
Go
190
star
14

quic

A Go implementation of the QUIC API for Peer-to-peer and Client-to-Server Connections
Go
178
star
15

rtsp-bench

RTSP -> WebRTC Server that generates a CPU Usage report
Go
165
star
16

rtcp

A Go implementation of RTCP
Go
151
star
17

sdp

A Go implementation of the SDP
Go
146
star
18

srtp

A Go implementation of SRTP
Go
116
star
19

interceptor

Pluggable RTP/RTCP processors for building real time communication
Go
109
star
20

obs-wormhole

Supercharge OBS with WebRTC. Remote WebRTC Sources and Serverless Broadcasting
Go
95
star
21

datachannel

A Go implementation of WebRTC Data Channels
Go
81
star
22

transport

Transport testing for Pion
Go
78
star
23

explainer

Explainer parses WebRTC Offers/Answers then provides summaries and suggestions
Go
45
star
24

11LiveChat

Another one-one live chatroom using pion webrtc with sfu mode
Go
42
star
25

udp

Please use github.com/pion/transport/v2/udp instead
Go
36
star
26

webrtc-voicemail

Voicemail.... for the web! Create voicemails via WebRTC and Transcribe them.
Go
31
star
27

demo-conference

A simple web conferencing application built using Pion
HTML
29
star
28

webrtc-zero-downtime-restart

A playground to make WebRTC easier to deploy, safer and more robust
Go
21
star
29

webrtc-v3-design

EXPERIMENTAL - new API proposals and discussions
Go
17
star
30

signaler

The Pion Signaler
Go
15
star
31

dcnet

net interfaces over DataChannels
Go
14
star
32

portmap

An implementation of UPnP-IGD, NAT-PMP and PCP for configuring port-forwarding
Go
13
star
33

.goassets

Asset files automatically deployed to Go package repositories
Shell
9
star
34

codec

Go wrappers of multiple codecs to be go gettable
Go
7
star
35

logging

The logging library used by Pion
Go
7
star
36

website

Pion public website
HTML
7
star
37

turnc

RFC 5766 TURN client implementation in go
Go
6
star
38

producer

File to webrtc.Track playback
Go
5
star
39

talks

Content for talks given by Pion WebRTC developers
CSS
5
star
40

pion-browser-client

Browser Client for Pion
JavaScript
5
star
41

msapi

media stream api for webrtc !
Go
4
star
42

randutil

Helper library for cryptographic and mathmatical randoms
Go
4
star
43

zapion

uber-go/zap ⚡ backended pion logger
Go
3
star
44

pkg

Shared Go libraries used by Pion products
Go
3
star
45

ci-sandbox

CI testing sandbox
Go
2
star
46

simulnet

A net implementation that enables simulations
2
star
47

ion-admin

This is ion's manager
1
star
48

.github

Default community health files
1
star
49

rtpio

Go
1
star