Project: Robotic Arm playing games

A way to have fun with your Ned2 Niryo Robot !

Table of Contents

• About the Project

  • Introduction

  • Project Structure

    • 1. Robotics
    • 2. Image Processing
    • 3. Strategies
    • 4. Graphics

  • Tech Stack

• Getting Started

  • Prerequisites
  • Run Locally

• Contact

About the Project

Introduction

As a student at Les Nancy Mines, my first-year project was the development of a scientific mediation showroom.

The aim is to spark interest in artificial intelligence and robotics among young people. To achieve this, I decided to create a workshop where visitors can play tic-tac-toe and Connect 4 against a robot arm.

Furthermore, it is a very comprehensive project which allows us to understand many concepts essential to the world of tomorrow.

Project Structure

The project is divided in 4 main parts that interact with each other :

1. Robotics
2. Image Processing
3. Strategies
4. Graphics

1. Robotics

The Robot used is Ned2 a six-axis collaborative robot, based on open-source technologies.

I control it with python pyniryo module, provided by the manufacturer, which is very convenient to start robotics.

The classes related to robot control are in the "Robot.py" files located in Morpion and Connect4 folders.

2. Image Processing

Old version

Before, Open-cv python library was used to process the images returned by the camera of the robot.

In the case of tic-tac-toe, the robot was detecting the shape of the pieces:

In the case of Connect 4, it was detecting their color:

The problem is that the accuracy of the detection was heavily depending on the lightning conditions

New version

To solve this problem, I opted for AI detection, using a FCNN trained on roughly 700 pictures. Its accuracy reached more than 99%, with highly undesirable lighting conditions, which is enough for this project.

This is a small sub-project that is detail in this repo

3. Strategies

To determine the next move, the robot can either send the current game grid to a minimax algorithm or to an AI model that will decide the best move to play.

The algorithm will explore all possible outcomes of the game to determine the best move.

The AI ​​will rely on prior training (during which it has played many games) to determine the best move. This is reinforcement learning

Note that the user can create its own custom AI model and train it.

The minimax algorithm is located in Morpion(resp. Connect4)/Minimax.

The AI algorithm is located in Morpion(resp. Connect4)/AI.

4. Graphics

I used kivy to make the GUI.

Several files are used to manage the interface:

• graphics.py : Game, strategy, and level selection menus will be found here
• ai_models_interface.py : All the graphics related to AI, such as model edit, training or selection.
• morpion(resp. connect4)Interface.py : GUI for playing against the robot. The purpose of this interface is to tell the robot when to play and to see if the piece detection is working correctly.
• game.py : GUI for playing against the computer. The purpose of this interface is to evaluate the power of an AI model by playing against it without needing the robot hardware.
• box_layout_with_action_bar.py : The actionbar which allows you to connect to the robot and to go to previous page.
• navigation_screen_manager.py : Manage the screens so that we can navigate between menus.

Tech Stack

Graphics

• Kivy

AI

• Tensorflow

Robot

• Ned2

Getting Started

Prerequisites

Install Python 3.10

Run the following commands required to setup the environment.

pip install tensorflow==2.13.0

pip install torch

pip install torchvision

pip install pygame

pip install kivy[full]

pip install matplotlib

pip install pyniryo==1.1.2

Run Locally

Clone the project

  git clone https://github.com/SamS709/ned_project.git

Go to the project directory

  cd ned_project

Start the application

  python main.py

Contact

Sami LEROUX - sami.lerouxpro@gmail.com

Project Link: https://github.com/SamS709/ned_project/tree/develop