LLM-Powered Game Development

A Unity-based game that leverages a local LLM (e.g., LLAMA) for NPC decision-making and dialogue, using a TCP communication layer between Unity (C#) and Python.

Important NOTE:

If the compile time/reloading domain time/enter play mode time are too long, then the issue is in your script, not in Unity. If certain scripts need to act in a certain way but are also causing these issues, the DomanReloadHelper.cs is an example of how you could build helper files for Unity.

---

Table of Contents

1. Features
2. Current Progress
3. To-Do
4. Future Improvements
5. Security Considerations
6. Setup
7. Notes & Pitfalls

---

Features

• Persistent TCP Connections with IPC handshake: Reuse the same client connections for multiple requests.
• LLM-Driven NPC Dialogue: NPC conversations are generated by the LLM.
• Custom Game Mechanics: Collision physics and movement systems built from scratch.
• Dynamic NPC Generation: Easily add new NPCs via scripting.
• Debug HTTP API (FastAPI): Alternative debugging interface (higher overhead).

---

Current Progress

• ✅ Using pipes for in-memory handshake before establishing TCP communication.
• ✅ Reliable connection management with delays and retry logic.
• ✅ TCP server implementation for Python ↔ Unity communication.
• ✅ NPC dialog integration with the LLM.
• ✅ Ports cleanly closed on application shutdown.
• ✅ Dynamic component injection for NPC prefabs.
• ✅ Custom movement and collision physics groundwork.

---

To-Do

1. Chunkify Data

   • Split large payloads into configurable-size chunks instead of fixed 1024 bytes.

2. Graceful Connection Handling

   • Avoid forcefully killing terminal connections.
   • Evaluate overhead of opening/closing vs. persistent sockets.
   • Consider storing client address globally and closing all on shutdown.

3. Secure TCP Layer

   • Encrypt payloads (TLS or custom crypto).

4. Protocol Refinement

   • Standardize request/response formats for LLM ↔ NPC communications.
   • Document message types and error codes.

5. Connection Reliability

   • Investigate startup race conditions causing first-connect failures.
   • Ensure sys.path and dependencies fully loaded before listening.

6. Embedded Python

   • The current implementation of invoking python process relies heavily on the user having python on their path and having the needed modules.
   • Embed python to privatise this a lot more.
   • Embedded python with python.Net increased compile time by a lot so investigate if it would still happen in this implementation.

---

Future Improvements

• Maintain active connections in a managed loop to avoid garbage collection.
• Auto-parse larger-than-buffer payloads reliably.
• Abstract connection setup inside request functions if client addresses must refresh per request.
• Improve FastAPI debug server while balancing throughput vs. overhead.
• Use a certificate for better authentication.

---

Security Considerations

• Ports exposed over local network are potential vulnerabilities.
• Forceful termination of sockets may lead to resource leaks.
• Conda environment quirks can break socket handling; prefer minimal environments.

---

Setup

1. Clone Repository:

   git clone <repo-url>
   cd <repo-folder>
   

Notes & Pitfalls

• Migrating to TLS/SSL might be a lot more challenging since the certificate generation only makes sense if they are dynamic and generated on a per-session basis. The current cryptography library for .NET in the Unity Engine I am using does not support this. My own approach through powershell did not work out either. Might need to find out if I even need this certificate.
• At the time of writing, do not have CONDA on your system. The system relies on python on Path but the code you write and interact with uses CONDA so basically CONDA ends up becoming a blanket that chokes you.