• Stars
    star
    2,430
  • Rank 18,202 (Top 0.4 %)
  • Language
    Swift
  • License
    MIT License
  • Created 10 months ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Run OpenAI's CLIP model on iOS to search photos.

Queryable

download-on-the-app-store

Queryable

The open-source code of Queryable, an iOS app, leverages the OpenAI's CLIP model to conduct offline searches in the 'Photos' album. Unlike the category-based search model built into the iOS Photos app, Queryable allows you to use natural language statements, such as a brown dog sitting on a bench, to search your album. Since it's offline, your album privacy won't be compromised by any company, including Apple or Google.

Blog | Website | App Store

PicQuery(Android)

download-on-the-app-store

The Android version(Code) developed by @greyovo, which supports both English and Chinese. See details in #12.

How does it work?

  • Encode all album photos using the CLIP Image Encoder, compute image vectors, and save them.
  • For each new text query, compute the corresponding text vector using the Text Encoder.
  • Compare the similarity between this text vector and each image vector.
  • Rank and return the top K most similar results.

The process is as follows:

For more details, please refer to my blog: Run CLIP on iPhone to Search Photos.

Run on Xcode

Download the ImageEncoder_float32.mlmodelc and TextEncoder_float32.mlmodelc from Google Drive. Clone this repo, put the downloaded models below CoreMLModels/ path and run Xcode, it should work.

Core ML Export

If you only want to run Queryable, you can skip this step and directly use the exported model from Google Drive. If you wish to implement Queryable that supports your own native language, or do some model quantization/acceleration work, here are some guidelines.

The trick is to separate the TextEncoder and ImageEncoder at the architecture level, and then load the model weights individually. Queryable uses the OpenAI ViT-B/32 model, and I wrote a Jupyter notebook to demonstrate how to separate, load, and export the Core ML model. The export results of the ImageEncoder's Core ML have a certain level of precision error, and more appropriate normalization parameters may be needed.

  • Update (2023/09/22): Thanks to jxiong22 for providing the scripts to convert the HuggingFace version of clip-vit-base-patch32. This has significantly reduced the precision error in the image encoder. For more details, see #18.

Contributions

Disclaimer: I am not a professional iOS engineer, please forgive my poor Swift code. You may focus only on the loading, computation, storage, and sorting of the model.

You can apply Queryable to your own product, but I don't recommend simply modifying the appearance and listing it on the App Store. If you are interested in optimizing certain aspects(such as #4, #5, #6, #10, #11, #12), feel free to submit a PR (Pull Request).

  • Thanks to Chris Buguet, the issue (#5) where devices below iPhone 11 couldn't run has been fixed.
  • greyovo has completed the Android app(#12) development: Google Play. The author stated that the code will be released in the future.

Thank you for your contribution : )

If you have any questions/suggestions, here are some contact methods: Discord | Twitter | Reddit: r/Queryable.

License

MIT License

Copyright (c) 2023 Ke Fang

More Repositories

1

disco-diffusion-wrapper

Implementation of disco-diffusion wrapper that could run on your own GPU with batch text input.
Jupyter Notebook
571
star
2

randomCNN-voice-transfer

Audio style transfer with shallow random parameters CNN.
Python
375
star
3

PodFind

Find what podcasters think of new things: GPT-4, SVB, etc.
JavaScript
149
star
4

Proxy

A simple tool for fetching usable proxies from several websites.
Python
125
star
5

BaiduCrawler

Sample of using proxies to crawl baidu search results.
Python
118
star
6

api-usage

Track your OpenAI API token usage & cost.
HTML
58
star
7

WaveGAN-pytorch

PyTorch implementation of " Synthesizing Audio with Generative Adversarial Networks"
Python
57
star
8

teach-show-consult

Teach ChatGPT the Alda music programming language, show it some superb code, and consult with it to compose a melody.
Python
47
star
9

QLearningMouse

Cat-and-Mouse game with Reinforcement Learning (Q-Learning).
Python
24
star
10

make-CelebA-HQ

Supposed you've downloaded CelebA & CelebA-HQ dataset, and want to get HQ images from them.
Python
15
star
11

Manzarek

A tiny bot reposts blind date information from website fanfou.
Python
11
star
12

Disentangled-Sequential-Autoencoder

PyTorch Implementation of Disentangled Sequential Autoencoder
Jupyter Notebook
8
star
13

Focus

Chrome Extension: One-click to batch open websites, double-click to close them.
JavaScript
8
star
14

N-Grams-novel

An English & Chinese novel generator based on N-Grams.
Python
4
star
15

DrQAChinese

Python
3
star
16

mazz.github.io

HTML
1
star
17

mazzzystar.github.io

HTML
1
star
18

MusicGAN

Generate long-term "structure" dependency raw piano audio, result: https://soundcloud.com/mazzzystar/sets/only-1-discriminator-to-control-both-local-long-term
Python
1
star