GPT Automator
Your voice-controlled Mac assistant. GPT Automator lets you perform tasks on your Mac using your voice. For example, opening applications, looking up restaurants, and synthesizing information.
Made by Luke Harries and Chidi Williams at the London EA Hackathon, February 2023.
Requirements
-
# on Ubuntu or Debian sudo apt update && sudo apt install ffmpeg # on Arch Linux sudo pacman -S ffmpeg # on MacOS using Homebrew (https://brew.sh/) brew install ffmpeg # on Windows using Chocolatey (https://chocolatey.org/) choco install ffmpeg # on Windows using Scoop (https://scoop.sh/) scoop install ffmpeg
Instructions
- Install the dependencies from the
requirements.txtorpyproject.tomlfiles. - Create a
.envfile from the.env.examplefile and fill in the OpenAI API key. - Run
python gui.pyto run the GUI and click 'Record' to say your prompt. Alternatively, runpython main.py [prompt]to run the CLI.
How it works
GPT Automator converts your audio input to text using OpenAI's Whisper. Then, it uses a LangChain Agent to choose a set of actions, including generating AppleScript (for desktop automation) and JavaScript (for browser automation) commands from your prompt using OpenAI's GPT-3 ("text-davinci-003"), and then executing the resulting script.
Example prompts
- Find the result of a calculation. Prompt: "What is 2 + 2?" -> It will write AppleScript to open up a calculator and type in 5 * 5.
- Find restaurants nearby. Prompt: "Find restaurants near me" -> It will open up Google search, read the text on the page, and say the best restaurants.
- Play a game of chess. Prompt: "Play a game of chess" -> It will open up Chess.com and start clicking around.
Learn more
Checkout our blog posts for more information:
Disclaimer
This project executes code generated from natural language and may be susceptible to prompt injection and similar attacks. This work was made as a proof-of-concept and is not intended for production use.