This project integrates GPT-V with OCR to address the shortcommings of GPT-V in being able to point to something precisely on the screen. As a result, we have an AI that can fully see, understand and interact with your computer screen.
The way it works is by annotating each identified element with a label that GPT-V can use:
| Screenshot | Annotated |
|---|---|
 |
 |
demo.mp4
Clone this repo:
git clone https://github.com/rogeriochaves/driver.git
Install the dependencies:
pip install -r requirements.txt
Now make a copy of .env.example to .env, you need to have both OpenAI and Azure Vision OR Google Cloud Vision for the OCR (if you want to contribute, DM me on twitter and I can send you a key for Azure Vision)
# OpenAI key to use GPT-V to be able to "see" the screen
OPENAI_API_KEY=""
# Choose either Azure, Google Cloud Vision or Baidu (better for chinese) to use for OCR to help GPT-V finding elements on the screen
AZURE_VISION_API_KEY=""
AZURE_VISION_ENDPOINT=""
# GCLOUD_VISION_API_KEY=""
# BAIDU_OCR_API_KEY=""
# BAIDU_OCR_SECRET_KEY=""Finally, ask it to do anything you want!
python main.py "hey there, please go to my gmail and send an email to Laura with a poem declaring my love"
Thanks to @MulongXie et al for building the UIED algorithm, which is used together with OCR for identifying GUI elements
There is A LOT that can be done, the project is very very new so contributions are very welcome! If you have suggestions for improvements or new features, open an issue, and feel free to fork the repository, make your changes, and submit a pull request.
This project is open-source and available under the MIT License. See the LICENSE file for more details.