Build a Gesture-Based Math Solver with AI & Streamlit

In the realm of computer vision and artificial intelligence, combining hand gestures with AI models unlocks exciting possibilities for human-computer interaction. This exploration details a captivating project: building a math gesture program leveraging the strengths of CVZone for hand detection, a drawing interface, and the power of Google's Gemini AI model for solving mathematical problems. Imagine drawing a math equation with your hand and receiving the solution in real-time! We will also use Streamlit for the app's interface.

The Core Idea of Math With Gestures

The goal is to create an interactive, intuitive math-solving application that lets users input math equations through simple hand gestures. This project combines various technologies to achieve real-time feedback and a seamless user experience. The program uses the CVZone library for hand detection and tracking, allowing the user to ‘draw’ mathematical equations in the air. Then, the magic happens: an AI model, leveraging the capabilities of Google's Gemini, interprets the drawing and computes the solution. Finally, Streamlit is employed to create an interface. This interface visually presents the problem and the solution, making the entire process engaging and user-friendly.

This gesture-based math program could be useful for online math teaching, for example. One of the things that makes this method useful is the ability to see real-time results.

Breaking Down the Project: Key Components

To make development manageable, the project is divided into several key components:

• Hand Detection: This component relies on CVZone to precisely identify and track the user's hand and finger positions. The goal is to accurately Translate hand movements into interpretable drawing data.
• Drawing Implementation: This aspect translates hand movement data into visual drawings on the screen, allowing the user to 'write' the equation in the air.
• AI Model Integration (Gemini): The core intelligence lies in the integration with the Google Gemini AI model. This AI will analyze the drawing, recognize the mathematical equation, solve it, and provide a clear explanation of the solution.
• Streamlit Interface: Streamlit creates the user-facing app that houses the video feed, allows drawing input, and displays the AI’s output. This carefully orchestrated combination of technologies forms the foundation of an interactive and educational mathematical tool.

Tools and Technologies required to run the Math Gesture Program

To successfully create this project, you’ll need to be familiar with the following tools and technologies:

• Python: The primary programming language used to build the application. Make sure you have it installed and configured correctly.
• CVZone: A Computer Vision library is crucial for hand detection. Installation is straightforward using pip install cvzone.
• Google Gemini API: Obtain a free API key from the Google AI for Developers platform to integrate the AI model.
• Streamlit: Streamlit allows you to create an interactive web application. Install it by using pip install streamlit.
• NumPy: Numerical Python is used for drawing the mathematical operations. Use pip install numpy.
• PIL(pillow) This image processing package is required to transfer images to the AI model in a format it can read. Install with pip install pillow.

• An IDE (e.g., PyCharm): An IDE can make coding a more streamlined and enjoyable experience. The project was created using PyCharm in the Tutorial. Download the Community edition to get started for free.

  Key Libraries:

  

Make sure that each of these applications is set up before proceeding.

Project Setup

The journey to building a math gesture solver begins with creating a dedicated space for your project. Start by creating a directory, perhaps named “GestureWithMath,” to house all your code and resources. Now, within this folder, the main.py file will house all the key components of this application.

Next you'll import all the different libraries required to make the project work.

import cvzone
import cv2
from cvzone.HandTrackingModule import HandDetector
import numpy as np
from PIL import Image
import streamlit as st

Next, enable webpage to extend into full width.

st.set_page_config(layout="wide")
st.image("MathGestures.png")

Now let’s configure and test the webcam feed.

#setting up webcam
cap = cv2.VideoCapture(1)
cap.set(3, 1280)
cap.set(4, 720)

detector = HandDetector(detectionCon=0.7, maxHands=1)

Finally, test your code to make sure the webcam feed is working properly.

Function-Based Refactoring for Modularity

Let’s encapsulate the complex hand tracking and math solving into smaller blocks using functions. To accomplish this, follow the steps below:

1. Convert the base code into a function:

   def getHandInfo(img):
   hands, img = detector.findHands(img)
   fingers=[]
   lmList=[]
   
   if hands:
   hand1 = hands[0]
   lmList=hand1["lmList"]
   fingers = detector.fingersUp(hand1)
   
   return fingers, lmList
   else:
   return None, None

2. Now call these in the appropriate functions.
3. Also refactor the Streamlit display code into two columns to keep the UI elegant. This promotes reusability and makes the code clearer. To do this, create 2 columns for your page, one for displaying the webcam and the other for displaying the solution.

Integrating Gemini AI for Dynamic Math Solving

The last part of the project is to configure the AI Key and integrate the Generative AI model, with code like this:

import google.generativeai as genai
genai.configure(api_key= "AIzaSyAu7w2TMd4kIAIB-RDMh8yvmrF80q8JpQk")
model = genai.GenerativeModel('gemini-1.5-flash')

if fingers == [1, 1, 1, 1, 0]:
 """ We need to PIL image here for Generative Model"""
 pil_image = Image.fromarray(canvas)
 response = model.generate_content("Solve this Math Problem", pil_image)

 output_text = response.text
 st.subheader(output_text)

   if canvas is None:
 canvas = np.zeros_like(img)
"""
The above code sets up all the parts you need for the AI chatbot and math solver. Now the AI chatbot should be able to guess the answer.

Getting Started: A Step-by-Step Guide

Follow these steps to get the most out of your new math solver:

• Check the 'Run' check box to start the webcam feed.
• Draw the math problem on the whiteboard, using your finger or a stylus. Make sure all parts of the equations are easy to distinguish for the best results.
• Make your AI requests by holding your palm upward with all fingers extended.
• Acknowledge the AI's result by taking your hand back down, before trying again with a new problem.

Understanding the Free Tier

The best part about this project is that we are using the completely free tier of the AI features. This is great for most learners, however here is what the tier is limited to:

• Requests per Minute (RPM): 15 requests per minute, meaning the AI can process 15 problems every minute.
• Tokens per Minute (TPM): 1 million tokens per minute. This parameter is a bit complicated but for now the short version is 1M tokens are more than enough to operate a smaller project like this one.
• Requests per Day (RPD): 1500. The best part: If you somehow go over the requests, you do not incur any charge. The application will simply stop working. This makes experimentation very safe!
• Gemini 1.5 Flash: This project will be using the flash AI model. For most users, this is more than enough to have the model work.

Core Functionality at a Glance

Here’s a snapshot of the solver’s capabilities:

• Hand Detection: Reliably detects and tracks the user's hand, ensuring accurate drawing input.
• Drawing: The AI will perform line drawing from your hand into a canvas.
• Image Interpretation: Using AI models, the drawing will be interpreted to find different mathematical symbols and characters in the drawing.
• Gemini AI Integration: AI analysis recognizes written math problems, offering steps toward the solution.
• Streamlit Interface: Clean and interactive web UI displaying results in real-time.

Transforming Education: Real-World Use Cases

This project unlocks possibilities for:

• Interactive Learning Environments: Students can solve equations with gestures, receiving immediate feedback and guidance.
• Assistive Technology: Provide a tool for individuals to interact with math despite physical limitations
• Innovative Math Tutoring: Design adaptive learning systems responding to a student’s drawn input.
• Real-Time demonstrations of mathematical topics: You can now demo problems by simply writing them on the screen and getting a step by step solution! This project shows just how well AI, computer vision and gesture recognition can work together. To accomplish this, follow the documentation to the letter.