ChatGPT vs Stockfish: Run an AI Chess Match

Updated 2 min read

Large language models lack internal spatial geometry, making rule-compliant chess difficult without explicit boundaries. This guide is for developers and AI hobbyists looking to build a structured pipeline combining statistical language models with deterministic engines. You will create a referee script that enforces legal moves and orchestrates an automated match between ChatGPT and Stockfish.

Watch the video on YouTube

Before you start

  • Ubuntu 24.04 or any Debian-based Linux environment
  • Python 3.10 or newer
  • An OpenAI API key with access to gpt-4o-mini

Understanding the Matchup

Stockfish and ChatGPT approach the chessboard with completely different computational paradigms. Stockfish performs alpha-beta minimax search augmented by efficiently updatable neural networks to evaluate millions of positions each second. It adheres strictly to board geometry and legal boundaries through deterministic rule sets.

ChatGPT operates strictly by predicting the next token based on text patterns seen during pretraining. It maintains no continuous internal coordinate system or spatial board awareness. Without external validation, language models inevitably suggest illegal piece hops or invalid king maneuvers, requiring an impartial referee script to manage the game state.

Installing Stockfish and Python Dependencies

Preparing the host environment requires pulling core system utilities, the dedicated chess engine, and the required Python modules. Linux distributions frequently carry older versions of Stockfish in standard repositories, making manual binary downloads preferable for modern performance features.

Run the package installation and binary extraction commands listed below to position the executable in the system path. Once the binary is configured, set up an isolated virtual environment and install the required Python packages to interface with the engine and the language model API.

TerminalNot verified in our test run
# Install system prerequisites
sudo apt update && sudo apt install -y python3 python3-pip python3-venv wget tar
# Download and install the official modern Stockfish x86-64 binary
wget -q https://github.com/official-stockfish/Stockfish/releases/latest/download/stockfish-ubuntu-x86-64-avx2.tar && tar -xf stockfish-ubuntu-x86-64-avx2.tar && sudo mv stockfish/stockfish-ubuntu-x86-64-avx2 /usr/local/bin/stockfish && rm -rf stockfish*
# Verify Stockfish installation via UCI protocol
stockfish uci | head -n 3
# Create Python virtual environment and install dependencies
python3 -m venv venv && source venv/bin/activate && pip install --upgrade openai python-chess
Output from our test run
$ sudo apt update && sudo apt install -y python3 python3-pip python3-venv wget tar
Get:1 http://deb.debian.org/debian stable InRelease [140 kB]
Get:2 http://deb.debian.org/debian stable-updates InRelease [47.3 kB]
Get:3 http://deb.debian.org/debian-security stable-security InRelease [43.4 kB]
Get:4 https://dl.google.com/linux/chrome-stable/deb stable InRelease [2548 B]
Get:5 https://deb.nodesource.com/node_22.x nodistro InRelease [12.1 kB]
Get:6 http://deb.debian.org/debian stable/main amd64 Packages [9678 kB]
Get:7 https://deb.nodesource.com/node_22.x nodistro/main amd64 Packages [12.8 kB]
Get:8 https://dl.google.com/linux/chrome-stable/deb stable/main amd64 Packages [1407 B]
Get:9 http://deb.debian.org/debian stable-updates/main amd64 Packages [1108 B]
Get:10 http://deb.debian.org/debian-security stable-security/main amd64 Packages [298 kB]
Fetched 10.2 MB in 2s (4142 kB/s)
27 packages can be upgraded. Run 'apt list --upgradable' to see them.
python3 is already the newest version (3.13.5-1).
python3-pip is already the newest version (25.1.1+dfsg-1).
python3-venv is already the newest version (3.13.5-1).
wget is already the newest version (1.25.0-2).
tar is already the newest version (1.35+dfsg-3.1).
tar set to manually installed.
Summary:
  Upgrading: 0, Installing: 0, Removing: 0, Not Upgrading: 27
$ wget -q https://github.com/official-stockfish/Stockfish/releases/latest/download/stockfish-ubuntu-x86-64-avx2.
tar && tar -xf stockfish-ubuntu-x86-64-avx2.tar && sudo mv stockfish/stockfish-ubuntu-x86-64-avx2 /usr/local/bin
/stockfish && rm -rf stockfish*
$ stockfish uci | head -n 3
bash: stockfish: command not found
$ python3 -m venv venv && source venv/bin/activate && pip install --upgrade openai python-chess

Building the Python Referee Script

The automated match relies on match.py to maintain single-source-of-truth board state using python-chess. For white moves, the script communicates directly with the local engine through standard interface calls, applying a defined skill ceiling and move duration.

When handling black moves, the referee supplies ChatGPT with the current board state in Forsyth-Edwards Notation along with a list of valid Universal Chess Interface moves. The script demands a structured JSON reply and enforces a retry loop with corrective prompts if the model returns an unparseable response or an illegal move.

match.py
import os
import json
import chess
import chess.engine
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
board = chess.Board()

stockfish_path = "/usr/local/bin/stockfish"
engine = chess.engine.SimpleEngine.popen_uci(stockfish_path)
engine.configure({"Skill Level": 3})

def get_llm_move(board, retries=3):
    legal_uci = [move.uci() for move in board.legal_moves]
    system_prompt = "You are playing chess as Black. Return only valid UCI moves."
    user_prompt = f"FEN: {board.fen()}\nLegal UCI moves: {', '.join(legal_uci)}\nRespond in JSON: {\"move\": \"<uci>\"}"
    
    for _ in range(retries):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_prompt}
            ],
            temperature=0.2
        )
        data = json.loads(response.choices[0].message.content)
        candidate = data.get("move", "").strip()
        if chess.Move.from_uci(candidate) in board.legal_moves:
            return chess.Move.from_uci(candidate)
        user_prompt += f"\n'{candidate}' is illegal. Choose strictly from: {', '.join(legal_uci)}"
    return None

print("Game started: Stockfish (White) vs ChatGPT (Black)")
while not board.is_game_over():
    if board.turn == chess.WHITE:
        result = engine.play(board, chess.engine.Limit(time=0.1))
        board.push(result.move)
        print(f"Stockfish played: {result.move.uci()}")
    else:
        move = get_llm_move(board)
        if move is None:
            print("ChatGPT forfeited due to repeated illegal moves.")
            break
        board.push(move)
        print(f"ChatGPT played: {move.uci()}")

print("Game over:", board.result())
engine.quit()

Executing the Automated Match

Executing the script requires exporting an active OpenAI API key into your shell session before invoking the interpreter. Once active inside the Python virtual environment, execute the main match file to initiate the automated game loop.

Stockfish outputs calculated turns almost instantaneously while ChatGPT processes the board string and valid candidates. With strict candidate filtering, the language model sustains extended games, though positional errors will steadily emerge against the engine.

TerminalNot verified in our test run
# Run the automated referee script
source venv/bin/activate && python3 match.py

Wrap-up

Combining structured LLM prompts with deterministic referee tooling enables robust experimentation against algorithmic systems. By offloading legal move enforcement to python-chess and structuring model outputs, you turn an autoregressive text generator into a viable participant against an engine. This pattern provides a solid foundation for evaluating spatial reasoning across various model architectures.

FAQ

Why does ChatGPT need a list of legal moves?

Because language models do not possess an internal 2D board representation, they frequently hallucinate illegal actions such as moving blocked pieces or walking into check. Supplying legal options constrains output selection.

Why use UCI notation instead of Standard Algebraic Notation?

The Universal Chess Interface standard uses simple coordinate strings like e2e4, which engines parse natively and reliably without the conversational ambiguities often found in algebraic notation.

How does Stockfish's architecture differ from an LLM?

Stockfish uses alpha-beta search trees paired with efficiently updatable neural networks optimized for evaluating board geometry, whereas ChatGPT generates moves purely through text-based token prediction.

Sources

Subscribe for more tested tutorials