• Stars
    star
    14,561
  • Rank 2,054 (Top 0.05 %)
  • Language
    TypeScript
  • License
    MIT License
  • Created over 4 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Cut and paste your surroundings using AR

AR Cut & Paste

An AR+ML prototype that allows cutting elements from your surroundings and pasting them in an image editing software.

Although only Photoshop is being handled currently, it may handle different outputs in the future.

Demo & more infos: Thread

⚠️ This is a research prototype and not a consumer / photoshop user tool.

Update 2020.05.11: If you're looking for an easy to use app based on this research, head over to https://clipdrop.co

Modules

This prototype runs as 3 independent modules:

  • The mobile app

    • Check out the /app folder for instructions on how to deploy the app to your mobile.
  • The local server

    • The interface between the mobile app and Photoshop.
    • It finds the position pointed on screen by the camera using screenpoint
    • Check out the /server folder for instructions on configuring the local server
  • The object detection / background removal service

    • For now, the salience detection and background removal are delegated to an external service
    • It would be a lot simpler to use something like DeepLap directly within the mobile app. But that hasn't been implemented in this repo yet.

Usage

1 - Configure Photoshop

  • Go to "Preferences > Plug-ins", enable "Remote Connection" and set a friendly password that you'll need later.
  • Make sure that your PS document settings match those in server/src/ps.py, otherwise only an empty layer will be pasted.
  • Also make sure that your document has some sort of background. If the background is just blank, SIFT will probably not have enough feature to do a correct match.

2 - Setup the external salience object detection service

Option 1: Set up your own model service (requires a CUDA GPU)

  • As mentioned above, for the time being, you must deploy the BASNet model (Qin & al, CVPR 2019) as an external HTTP service using this BASNet-HTTP wrapper (requires a CUDA GPU)

  • You will need the deployed service URL to configure the local server

  • Make sure to configure a different port if you're running BASNet on the same computer as the local service

Option 2: Use a community provided endpoint

A public endpoint has been provided by members of the community. This is useful if you don't have your own CUDA GPU or do not want to go through the process of running the servce on your own.

Use this endpoint by launching the local server with --basnet_service_ip http://u2net-predictor.tenant-compass.global.coreweave.com

3 - Configure and run the local server

  • Follow the instructions in /server to setup & run the local server.

4 - Configure and run the mobile app

  • Follow the instructions in /app to setup & deploy the mobile app.

Thanks and Acknowledgements

More Repositories

1

ar-cptext

AR Copy Paste - Text Proto
Java
718
star
2

instagram-3d-photo

A Chrome extension that adds a 3d photo effect to instagram pages.
Jupyter Notebook
675
star
3

ar-facedoodle

Doodle AR masks right from the browser with tensorflow.js
JavaScript
545
star
4

basnet-http

HTTP service wrapper for BASNet: Boundary-Aware Salient Object Detection
Python
453
star
5

screenpoint

Project an image centroid to another image using OpenCV
Python
444
star
6

Shine-MP3-Encoder-on-AS3-Alchemy

Flash/Alchemy port of the lightweight Shine MP3 Encoder
C
126
star
7

ofxCvCameraProjectorCalibration

OpenFrameworks addon that helps calibrating a camera and a projector using OpenCV
C++
117
star
8

u2net-http

HTTP Wrapper for U^2-Net (Qin et al, Pattern Recognition 2020)
Python
87
star
9

ofxEmbree

a photo-realistic ray tracing addon for openFrameworks (using Intel Embree engine)
C++
70
star
10

kuda

Serverless APIs on remote GPUs
Go
55
star
11

ofxSpine

Implementation of Spine's runtime to load and play 2D skeleton animations with openFrameworks
C
20
star
12

waxing-crescent

Code for the ☾ Waxing Crescent generative artwork by Cyril Diagne
TypeScript
17
star
13

ofxCvFeaturesTracker

Natural Features Tracking addon for OpenFrameworks using OpenCV only
C++
14
star
14

ofApps

working repository for openFrameworks tests and demos
C++
12
star
15

kinect2-socket

A little x64 windows application that streams kinect tracking datas via websocket
C++
10
star
16

ecal_handline_workshop

1 week workshop @ ECAL with 2nd year Media and Interaction Design
JavaScript
9
star
17

kinect2-socket-client

A javascript library that handles kinec2-socket streams and playback files
JavaScript
7
star
18

mobilestylegan-web-demo

Web Demo of MobileStyleGan FFHQ 1024 using the ONNX Web Runtime
JavaScript
5
star
19

MapamokStepper

An interactive projection mapping experiment with Mapamok and a Stepper motor
C++
4
star
20

kinect_three

playground for kinect body tracking experiments using threejs
CoffeeScript
4
star
21

ofxSerialLink

a simple openFrameworks addon / protocol to send commands and read values from serial interface
C++
3
star
22

starfield

Swing through the stars
C++
3
star
23

OF_tennisBallTracker

a little experiment with OF + OpenCV HoughCircles
C++
3
star
24

ECV_kinect_basics_workshops_2013

Processing sketches and support material for my Kinect Basics Workshops at ECV Paris
Processing
3
star
25

android-audioinput-videoplayer

Cordova based html player that plays videos on audio signal input (microphone or line-in)
JavaScript
2
star
26

semainebloc-S1-2020-Extra

JavaScript
2
star
27

Starfield_v2

2nd version of Starfield installation
C++
1
star
28

playtime

Playtime - automatic photo studio
1
star
29

ofxOctaniumGui

An openFrameworks addon for Octanium's app GUI using GWEN
C++
1
star
30

EO

CMake
1
star
31

ofxRazerHydra

openFrameworks addon for the Razer Hydra
C++
1
star
32

nvidiasmi-http

A simple http service that prints the output of nvidia-smi
Dockerfile
1
star