tylerswink.me

Tag: python

  • Making an AI-Assistant to help me learn Physics


    The Problem: Studying Physics Is Hard When You Just Get Answers

    I often find myself exploring subjects I find academically interesting even if they don’t have personal relevance to my own professional work. Problems, and their solutions, are a valuable asset, but you can often “spoil” yourself  by looking at the solution too early. I wanted a way to bridge this divide by given hints based off the solutions while not giving the whole game away.

    That’s why I built Physics AI Tutor — a web-based AI assistant that acts as a Socratic study partner. You feed it your textbook material and problem sets, ask it questions, and instead of handing you the answer, it guides you with hints and questions to help you reason through the problem yourself.

    How It Works

    The tutor is built around three core ideas:

    1. Your content, your curriculum. Upload your own textbook chapters, lecture notes, or problem sets as PDF or text files. The system extracts the text, splits it into chunks, and stores it in a local vector database for fast retrieval.
    2. Semantic search for relevant context. When you ask a question, the app uses sentence embeddings (via sentence-transformers) and FAISS to find the most relevant passages from your uploaded material. These are provided to the AI as reference — so its hints are grounded in what you’re actually studying.
    3. Socratic prompting. The AI is instructed to never reveal the full solution. Instead, it gives one hint at a time, asks guiding questions (“What forces are acting on the block?” or “What conservation law applies here?”), and gradually increases specificity if you’re stuck.

    The Tech Stack

    The project is a lightweight Python web app that runs locally on your machine:

    • FastAPI — serves the web UI and handles file uploads and chat requests
    • GPT-4o via GitHub Models — powers the tutoring conversation (free with a GitHub Copilot subscription). You could easily re-engineer it to use a model of your choosing.
    • FAISS + sentence-transformers — local vector search for retrieving relevant textbook passages
    • PyPDF — extracts text from uploaded PDF files
    • MathJax — renders LaTeX equations in the browser so formulas display correctly

    The entire app is self-contained — no external databases, no cloud storage. Your textbook content stays on your machine.

    A Typical Study Session

    Here’s what using it looks like in practice:

    1. Start the server and open localhost:8000 in your browser
    2. Upload a chapter on electrostatics from your textbook PDF
    3. Type: “A point charge of +3μC is placed 0.5m from a -2μC charge. What is the force between them?”
    4. The tutor responds: “Good question! What law describes the force between two point charges? What quantities do you need?”
    5. You reply: “Coulomb’s law — I need the charges and the distance”
    6. The tutor: “Exactly! Now, can you write out Coulomb’s law and plug in the values? Remember to pay attention to the sign of the force — what does it tell you about the direction?”

    The AI draws on the textbook passages you uploaded to make its hints specific and relevant, but it never just hands you the formula with numbers filled in.

    Why Not Just Use ChatGPT?

    You absolutely can ask ChatGPT physics questions. But there are two key differences with this approach:

    • It’s tuned to teach, not answer. General-purpose chatbots default to giving complete solutions. This tutor is specifically prompted to hold back and make you think. It takes deliberate prompt engineering to maintain that behavior consistently across a conversation.
    • It knows your textbook. By uploading your specific course material, the hints align with the notation, methods, and examples your chosen learning material uses — not some generic explanation from the internet.

    Getting Started

    The project is open source and runs locally with Python. You’ll need:

    • Python 3.11+
    • Programatic access to an LLM of your choosing (API key)

    Setup takes about five minutes:

    git clone <your-repo-url>
cd Physics-AI-Assist
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

    Add your GitHub/API token to a .env file, then run:

    uvicorn app:app --reload

    Open http://localhost:8000 and start uploading your study materials.

    What’s Next

    This is a first version. Some ideas for future improvements:

    • Problem tracking — log which problems you’ve worked through and how many hints you needed
    • Diagram support — let the tutor generate or reference free-body diagrams and circuit schematics
    • Quiz mode — have the tutor generate practice problems from your uploaded material

    If you’re interested in taking a look, you can find this on my GitHub.

  • Foundry MCP – Using AI to make being a Dungeon Master a little easier

    I imagine the overlap in the venn diagram between IT professionals and “giant nerds” is pretty significant. I’m certainly both, and one of my favorite hobbies is partaking in a good old-fashioned D&D 5e campaign. Over the years, my group has taken a liking to Foundry VTT, an amazing engine for hosting tabletop RPGs of all varieties. With some carefully curated modules, you can automate a ton of functionality to remove the toil of bookkeeping and focus on the fun bits: namely roleplaying and combat.

    A screenshot of a Foundry VTT game session with various modules enabled
    A screenshot of Foundry, albeit one with many modules. (If you also play OldSchool RuneScape, bonus points to you, reader.)

    I’m usually the DM of our group. Some might consider this torture, but I actually quite enjoy the task of designing a story and world to facilitate it. That said, the work can be admittedly laborious — especially for a homebrew campaign. So many custom items, spells, and actors to create.

    In an attempt to ease my suffering, I noticed that some people were selling Foundry AI tools that could hook up models to your world. I was initially tempted to spend the money, but unfortunately they all followed the trend of requiring a subscription. So after much self-convincing, I set about making my own.

    And that’s exactly what I built. Foundry MCP is a local-only MCP (Model Context Protocol) server that connects any AI client — Claude, Cursor, or anything that speaks MCP — directly to a live Foundry VTT world. As of the writing of this post, it exposes 67 tools across 10 domains.

    Design Decisions

    There were a few requirements I wanted this tool to meet:

    Local-only by design. While there are advantages to running things in the cloud, my use case only required it to run locally on my LAN. Your campaign data never leaves your machine. Future versions could potentially expand this to run via a domain in the cloud.

    Discovery-first tool design. Tools are designed so the AI naturally discovers what it needs. It calls list-actors to find character IDs, then get-actor for details, then roll-attack with the right weapon. This mirrors how a human DM navigates Foundry.

    Bidirectional sync. The Foundry module pushes data to the gateway, and the gateway pushes commands back. The AI always has a fresh snapshot of the world without polling Foundry directly.

    System-aware. The D&D 5e tools know about proficiency, skill codes, and ability modifiers. When the AI rolls a Stealth check, it resolves the same way a player clicking the skill button would.

    Architecture

    The architecture is three layers, all running locally:

    1. A Foundry module (foundry-api-bridge) runs inside the browser. It connects via WebSocket to the gateway and pushes world data — actors, journals, scenes, items — every 30 seconds. It also listens for commands and executes them using Foundry’s API.
    2. A Python gateway (FastAPI + uvicorn) sits in the middle. It stores the latest world state in memory and forwards commands to Foundry through the WebSocket. This is the single source of truth the AI talks to.
    3. A Python MCP server exposes 67 tools over the Model Context Protocol. Any MCP-compatible AI client can connect via stdio or HTTP and start interacting with the game world.

    67 Tools Across 10 Domains

    The tool surface covers nearly everything a DM does:

    • Actors — List, search, create, update, and delete characters and NPCs. Pull full character sheets with HP, AC, abilities, skills, and inventory.
    • Combat — Create encounters, add combatants, roll initiative (individual or group), advance turns, mark defeated, end combat. The full encounter lifecycle.
    • D&D 5e Rolls — Skill checks, saving throws, ability checks, attack rolls, damage rolls. All 18 skills, all 6 abilities, with proficiency baked in. Results post directly to Foundry’s chat log so players see them.
    • Items — Full CRUD on actor inventory and world items. Use potions, cast spells, activate items with Midi-QOL targeting support.
    • Journals — Create, read, update, and delete journal entries and pages. Organize into folders. Great for AI-generated session notes or lore.
    • Tokens & Scenes — Place, move, and update tokens on the map. List and switch active scenes.
    • Compendiums — Browse, search, and pull official content. Import monsters, spells, and items directly from SRD packs.
    • Effects & Conditions — Toggle conditions like blinded, poisoned, or prone. Apply custom effects with stat modifications.

    What This Actually Looks Like

    Below is an example where I use the Foundry MCP to create an actor:

    Asking an AI to create an actor via Foundry MCP
    The AI's response showing the created actor details

    Of course, we want to verify this was actually created in Foundry — and sure enough, there’s our Gnome Child actor:

    The Gnome Child actor as it appears in Foundry VTT

    Anyone familiar with D&D knows a true Level 20 character would likely be much more powerful, but even with the need to tweak things, this is still a significant time-saver when creating custom characters. Below is an example of a Count Draynor boss I made for my own campaign, complete with custom abilities:

    Count Draynor custom boss character sheet in Foundry VTT

    What’s Next

    I’m actively developing Foundry MCP. The tool count keeps growing as I find new things I want the AI to handle — more granular token control, deeper system integration, and scene manipulation are all on the roadmap. The project is MIT licensed, so if you’re a Foundry VTT user who wants to experiment with AI-assisted gameplay — whether that’s solo adventures, automated encounter management, or just having a tireless assistant for session prep — feel free to check it out. You can the repo on Github here