Gaming using Computer Vision

A python bot used to play a game using computer vision and image processing techniques. The bot is able to play the game without any human intervention. It managed to make me a millionaire in the game overnight!

Here's a screen recording of the game play:

activity.mp4

Access a comprehensive guide in this article: https://paulonteri.com/thoughts/play-game-with-computer-vision.

Disclaimer - this content is for educational purposes only!

Why?

I've been playing strategy + city building + simulation? games like TownsMen 6, Clash of the Clans, SimCity for the last 10 years.

On trying out City Island 5 I found it mildly irritating that my collectables could not accumulate while I was outside the game. I might have had the best businesses, strategy, etc but I had to be in the game to ensure I collect the cash/keys/gold overtime. For example, if my bakery makes €100 per minute I would only earn €100 after leaving the game and coming back 24 hours later.

This became especially tiresome while trying to accumulate €5,000,000 required to buy the island shown below. This would take me roughly two weeks of gameplay if I don't spend any money - it's not worth it.

How it works

1. Capture the live game feed

I needed a way to capture the live game feed.

The easiest way to to capture an in-game screenshot and pass it to the next steps in the script.

2. Identify the valuables in the screenshot

We need a way to detect a valuable in the game's feed and then return its coordinates.

OpenCv's Template Matching algorithms are perfect for this.

They are used for searching and finding the location of a template image (like a valuable) in a larger image (like the game's feed). It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of the input image under the template image. Several comparison methods are implemented in OpenCV. (You can check docs for more details). We use it in the method: cv2.matchTemplate(... )

To achieve this, I needed the template images. I took screenshots by hand and then cropped off the cash, star and key:

3. Collect the valuables by clicking on them

Once we have the coordinates of an item we can try to click on it.

pyautogui's .click(x,y) function works like magic for this. It clicks the screen on the coordinates x and y where our valuable is lying. Learn more about it here.

click_cash.mp4

click_key.mp4

click_star.mp4

4. Close any popups that may appear

Our clicks above may result in popups when we are being given a reward, levelling up, etc.

We need to close it before attempting to collect valuables again. We use the same logic used in finding and clicking on valuables.

To achieve this, I needed the template images for the popups' close buttons so that they can be clicked. I took screenshots by hand and then cropped off the various close buttons: