• Stars
    star
    675
  • Rank 66,879 (Top 2 %)
  • Language
    Jupyter Notebook
  • Created over 4 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Chrome extension that adds a 3d photo effect to instagram pages.

A chrome extension that adds depth parallax (an effect similar to Facebook 3D photos) on images from instagram profile pages.

It uses 3d-photo-inpainting running in Colab (free GPU) and Cloud pubsub/storage for communication.

More info: https://twitter.com/cyrildiagne/status/1251920177782042624

Demo

Original Paper: 3D Photography using Context-aware Layered Depth Inpainting Meng-Li Shih, Shih-Yang Su, Johannes Kopf, Jia-Bin Huang

โš ๏ธ Word of caution!

๐Ÿงช This project is highly experimental and requires a strong knowledge of the Google Cloud Platform to setup & run.

Architecture

architecture

This extension works by communicating with a Colab notebook running as GPU worker to compute the 3D inpainting videos.

For this extension to work, there are 2 other applications that must be running:

  1. The Colab notebook. It must be openned in another tab with the last cell running.
  2. The proxy service. It must be running (either remotely or on your computer) to receive HTTP requests from the extension and turn them into PubSub messages that the Colab listens to.

Setup the GCP Project

Create a GCP project

First create a project on the Google Cloud Platform. Then install the Cloud SDK and initialize it in your local machine with gcloud init.

Alternatively, you can also use Cloud Shell.

In a terminal window, set the local default project:

export PROJECT_ID=<your gcp project id>
gcloud config set project $PROJECT_ID

Create a service account for colab to be able to access cloud pubsub & storage.

It will create a insta3d-colab-key.json file that you must upload on the Colab to authenticate the notebook with your project.

export COLAB_SA=insta3d-colab
export COLAB_KEY_FILE=./insta3d-colab-key.json
./scripts/create_colab_key.sh

Create the Pubsub topic and subscription

export TOPIC_NAME=insta3d
./scripts/create_pubsub.sh

Enable CORS on your bucket

For the extension to be able to load the images generated by the colab, the CORS must be enabled on a Cloud Storage Bucket.

One way to do so is to create a file cors.json

[
  {
    "origin": ["*"],
    "responseHeader": ["Content-Type"],
    "method": ["GET"],
    "maxAgeSeconds": 3600
  }
]

And apply it to your bucket by running:

export BUCKET_NAME=<your-bucket-name>
gsutil cors set cors.json gs://$BUCKET_NAME

/!\ This will enable the CORS on the entire bucket.

Run the proxy service locally

(Recommended) Create a virtual environment

virtualenv venv
source venv/bin/activate

Install the dependencies:

pip install -r proxy/requirements.txt

Start the proxy:

export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/insta3d-colab-key.json
python proxy/main.py

Run the Colab

Open the notebook in Colab and follow the instructions. Make sure you've activated a GPU Runtime.

Load the extension & enjoy

Phew, when all that is done, it's time to load the extension and profit!

  • Navigate to chrome://extensions
  • Click the Load unpacked button
  • Select the "extension" folder from this repo.

It should add a new icon in your extensions bar that you can use to trigger the extension. Keep an eye on the console to check for eventual errors.

Changes with the original 3d-photo-inpainting

  • Update argument.yml to use custom path + smaller size + custom fps/duration
  • Update straight-line and circle paths in utils.py
  • Set ffmpeg to create 1 keyframe per frame in mesh.py

Thanks and acknowledgements

3d-photo-inpainting The original paper, code and notebook.

@derek-xia for the help improving the documentation.

More Repositories

1

ar-cutpaste

Cut and paste your surroundings using AR
TypeScript
14,561
star
2

ar-cptext

AR Copy Paste - Text Proto
Java
718
star
3

ar-facedoodle

Doodle AR masks right from the browser with tensorflow.js
JavaScript
545
star
4

basnet-http

HTTP service wrapper for BASNet: Boundary-Aware Salient Object Detection
Python
453
star
5

screenpoint

Project an image centroid to another image using OpenCV
Python
444
star
6

Shine-MP3-Encoder-on-AS3-Alchemy

Flash/Alchemy port of the lightweight Shine MP3 Encoder
C
126
star
7

ofxCvCameraProjectorCalibration

OpenFrameworks addon that helps calibrating a camera and a projector using OpenCV
C++
117
star
8

u2net-http

HTTP Wrapper for U^2-Net (Qin et al, Pattern Recognition 2020)
Python
87
star
9

ofxEmbree

a photo-realistic ray tracing addon for openFrameworks (using Intel Embree engine)
C++
70
star
10

kuda

Serverless APIs on remote GPUs
Go
55
star
11

ofxSpine

Implementation of Spine's runtime to load and play 2D skeleton animations with openFrameworks
C
20
star
12

waxing-crescent

Code for the โ˜พ Waxing Crescent generative artwork by Cyril Diagne
TypeScript
17
star
13

ofxCvFeaturesTracker

Natural Features Tracking addon for OpenFrameworks using OpenCV only
C++
14
star
14

ofApps

working repository for openFrameworks tests and demos
C++
12
star
15

kinect2-socket

A little x64 windows application that streams kinect tracking datas via websocket
C++
10
star
16

ecal_handline_workshop

1 week workshop @ ECAL with 2nd year Media and Interaction Design
JavaScript
9
star
17

kinect2-socket-client

A javascript library that handles kinec2-socket streams and playback files
JavaScript
7
star
18

mobilestylegan-web-demo

Web Demo of MobileStyleGan FFHQ 1024 using the ONNX Web Runtime
JavaScript
5
star
19

MapamokStepper

An interactive projection mapping experiment with Mapamok and a Stepper motor
C++
4
star
20

kinect_three

playground for kinect body tracking experiments using threejs
CoffeeScript
4
star
21

ofxSerialLink

a simple openFrameworks addon / protocol to send commands and read values from serial interface
C++
3
star
22

starfield

Swing through the stars
C++
3
star
23

OF_tennisBallTracker

a little experiment with OF + OpenCV HoughCircles
C++
3
star
24

ECV_kinect_basics_workshops_2013

Processing sketches and support material for my Kinect Basics Workshops at ECV Paris
Processing
3
star
25

android-audioinput-videoplayer

Cordova based html player that plays videos on audio signal input (microphone or line-in)
JavaScript
2
star
26

semainebloc-S1-2020-Extra

JavaScript
2
star
27

Starfield_v2

2nd version of Starfield installation
C++
1
star
28

playtime

Playtime - automatic photo studio
1
star
29

ofxOctaniumGui

An openFrameworks addon for Octanium's app GUI using GWEN
C++
1
star
30

EO

CMake
1
star
31

ofxRazerHydra

openFrameworks addon for the Razer Hydra
C++
1
star
32

nvidiasmi-http

A simple http service that prints the output of nvidia-smi
Dockerfile
1
star