Wrench: control your Android phone from your Desktop
Here’s a video of Wrench: playing monument valley (with phone screen mirrored), using Lua to script your phone, and ordering coffee (kinda).
Here’s another youtube video which introduced Wrench’s most useful functions: chatting with IM and Social Media Apps, finding contacts in IM and Social Media Apps, writing emails, and so on (inputing text is best done with the PC, then sent with the phone).
Install
There are prebuilt versions for Windows, Mac and Linux (ubuntu 14.04). They are released at https://github.com/SmartisanTech/Wrench-releases/releases .
Compilation
To compilen Wrench, you need make sure SnoreNotify is compiled and installed first. It’s in the SnoreNotify
sub-directory. Please refer to the SnoreNotify/build-*.sh
for your platform.
Compiling Wrench
- Under Linux, compile Wrench with the
build-linux.sh
script. You should be able to run this script directly.Because Wrench used QHotkey, you need to install
libqt5x11extras5-dev
first if you are under Debian/Ubuntu or something similar under other Linux distributions. - For Windows version, I compile with Wine under Linux, with the Qt for msys installed as compiler. Please refer to the
./build-wine.sh
script. - For Mac version, please refer to the
./build-mac.sh
script.
You need to adapt the ./build-wine.sh
and ./build-mac.sh
scripts, there are some hardcoded parameters (such as the PATH environment and the hostname of the Mac PC that I use for compilation).
How does it work
Android has provided quite some debugging abilities through adb, for example, you can
- Inject keyboard and touch events with
input
- Start Activities and Services with
am start|startservice
- Take a screen shot with
screencap
- Record your screen with
screenrecord
- Dump window infomation with
dumpsys window
(where you can also get infomation about the Input Method)
Wrench just makes use of all the above, with a great deal of optimization.
Optimazations
better and faster adb shell input
Android’s builtin input
command allows you to inject touch, keyboard, text events, but there’re some limitations.
- You can use
adb shell input text <string>
to input a string, but the string is limited: it can’t contain spaces or any other special characters, such as quote and double quote, or Chinese characters.I solved this problem by using the Android clipboard and my own Input Method WrenchIme adapted from Google Pinyin. When Wrench need to input any string, it:
- Put the string into the phone’s clipboard.
- Trigger a special key (SCROLL_LOCK) with
adb shell input
- WrenchIme interpret the SCROLL_LOCK key to send the clipboard text to the current App, as if the text is an IME candidate and has been selected.
adb shell input
is very slow, because every timeinput
is run, aframework/input.jar
is loaded and a new java virtual machine is created by runningapp_process
.To solve this problem, Wrench comes with a Setclip.apk, which whileas being an Android Apk, it also provides the functionality of
framework/input.jar
(i.e., it can also be loaded and run byapp_process
), but instead of loading every time, it loads once and then runs as a daemon.
adb shell am
faster For the same reason as adb shell input
, am
is also very slow because an am.jar
is loaded with a new java virtual machine each time. Wrench improved it using the same trick as input
, in fact, Setclip.apk will load am.jar
in turn when it is daemonized.
(Wrench’s am.jar is from AOSP, with userdebug build to make sure the .dex java class files are not optimized away and thus dalvik loadable.)
faster phone screen mirroring
At first, Wrench used screencap
to make a screenshot of the phone screen, and displayed on the PC. This is very slow.
Now, with some phones (according to the Android SDK version and CPU arch), Wrench can use the VNC protocol to mirror your phone screen, which is a lot smoother as you can see from the video. In doing this, Wrench used source code from 3 projects: it combined the AOSP screenrecord project and the droidVncServer project to run a VNC server on the phone; and used the Qt VNC Viewer to talk to the phone’s VNC server.
How to use Wrench
Please refer to this youtube video.
There is also a blog explaining all the details, but it’s in Chinese.
Here’s the simple steps:
- Start the Wrench program
- Enable the phone’s USB debugging.
- Connect the phone to Desktop with USB.
(Wrench is packaged with the AOSP adb
program, so you need to make sure you are not running a different version of adb, as it may cause connection trouble.)
How to compile droidVncServer
- Note
- If your phone is deeply customized, it might not work if you compile the droidVncServer in AOSP source code. I found this out on Smartisan (the company that I’m working at)’s U1 phone, where I compiled droidVncServer with aosp-lollipop-mr1-dev, which has the same SDK version as U1, but sadly the program crashes when run. I also compiled the AOSP screenrecord program with aosp-lollipop-mr1-dev, then run it on U1, and found that it can do .mp4 file recording, but also crashes when run for streaming (with
--output-format frames
).In this case, the only way to make it work is to compile under the same source code where your phone’s system image is compiled, which is unlikely unless you are a developper of this phone hardware company. Sorry.
If your phone’s Android SDK version/CPU arch is not supported, you can compile it yourself.
First you need get AOSP source code. SDK/NDK is not supported to compile droidVncServer because some system APIs are used which is not available in SDK/NDK.
Next copy the droidVncServer
directory into AOSP’s external/droidVncServer
, set up AOSP’s build appropriately, and run mma
in external/droidVncServer
.
Note that the source code for droidVncServer
may need some porting if your phone’s SDK version is not 22 or 23. The changes may be small or big or even impossible depending on how different your phone’s SDK version is.
When build is finished, the binary is in external/droidVncServer
, you need copy it back into your Wrench executable directory, with a rename (for e.g., androidvncserver-arm64.sdk23
should be renamed to androidvncserver-aarch64.sdk23
, where aarch64
is the output from your phone’s adb shell uname -m
), and generate a .md5 file for it (Please refer to release/androidvncserver-aarch64.sdk23.md5
).
Security Risks
Wrench need to use adb to connect to your phone. So it has all the security risks that comes with leaving adb open on your phone.
Wrench input text by putting it into the phone’s clipboard, so if there are clipboard sniffers running, you may accidentally leak sensitive infomation.
Contribute
Bug reports, patches, ideas are welcome.
Or, if you like Wrench, you may also donate some money to me with Paypal:
https://pledgie.com/campaigns/32699
or with alipay: