{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 10: Auditing AI-Generated Numerical Code\n", "\n", "The AI use appendix on your assignments asks what the tool got wrong and how you caught it.\n", "Today we practice the second half: catching it.\n", "\n", "The three functions below were written with an AI coding assistant from the prompts shown, and then seeded with mistakes of the kind these tools often make.\n", "Each function has at least one mistake.\n", "Every one of them runs without errors and returns numbers that look reasonable.\n", "\n", "Your goal is not only to find the mistakes, but to find **tests that would have exposed them if no one had told you they were there**.\n", "Work with a partner, and keep an audit log as you go (template at the end)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## An auditing checklist\n", "\n", "1. **Known answer.** Run the code on a problem you can solve exactly.\n", "2. **Invariants.** Check normalization, symmetries, conservation laws, and detailed balance.\n", "3. **Limits.** Check $\\Delta x \\to 0$, $T \\to 0$ and $T \\to \\infty$, and more samples.\n", "4. **Bookkeeping.** What is counted, stored, and averaged: every step, or only some?\n", "5. **Edges.** Look at boundaries, signs, indices, and off-by-one errors.\n", "6. **One fix at a time.** Re-test after every fix: a second mistake can hide behind the first." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import itertools\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Eigenstates of a 1D Hamiltonian (about 15 minutes)\n", "\n", "The prompt given to the assistant:\n", "\n", "> Write a Python function that returns the k lowest energy eigenvalues and the normalized eigenfunctions of $H = -\\frac{1}{2}\\frac{d^2}{dx^2} + V(x)$ on a grid of $N_D + 1$ points from $-L/2$ to $L/2$, using finite differences." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def lowest_states(V, L=8.0, ND=600, k=4):\n", " # Grid and spacing\n", " x = np.linspace(-L / 2, L / 2, ND + 1)\n", " dx = x[1] - x[0]\n", "\n", " # Finite-difference Hamiltonian: kinetic term from the three-point stencil, plus V on the diagonal\n", " main = 1.0 / dx**2 + V(x)\n", " off = -0.5 / dx**2 * np.ones(ND)\n", " H = np.diag(main) + np.diag(off, 1) + np.diag(off, -1)\n", "\n", " # Solve the eigenvalue problem and keep the lowest k states\n", " E, v = np.linalg.eig(H)\n", " return E[:k], v[:, :k], x" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "E, phi, x = lowest_states(lambda x: 0.5 * x**2)\n", "print(\"E =\", E)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tasks**\n", "\n", "1. What should `E` be for the harmonic oscillator, $V(x) = \\frac12 x^2$? Does it match?\n", "2. The function promises normalized eigenfunctions. Check $\\int |\\phi_n(x)|^2\\, dx$.\n", "3. Find and fix the mistakes, then run tests 1 and 2 again." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your tests and your fixed version here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Sampling a positive variable (about 25 minutes)\n", "\n", "The prompt given to the assistant:\n", "\n", "> Write a Metropolis sampler for a positive variable $x$ with probability density $p(x) \\propto e^{-U(x)}$. Since $x$ must stay positive, propose multiplicative steps $x' = x\\, e^{\\sigma z}$ with $z$ a standard normal random number. Return the chain and the acceptance rate." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def metropolis_positive(U, x0=1.0, sigma=0.5, n_steps=200_000, seed=0):\n", " rng = np.random.default_rng(seed)\n", " x = x0\n", " samples = []\n", " accepted = 0\n", " for _ in range(n_steps):\n", " # Multiplicative proposal keeps x positive\n", " x_new = x * np.exp(sigma * rng.normal())\n", " # Metropolis acceptance\n", " if rng.random() < np.exp(U(x) - U(x_new)):\n", " x = x_new\n", " samples.append(x)\n", " accepted += 1\n", " return np.array(samples), accepted / n_steps" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We test it on $p(x) \\propto x\\, e^{-x}$ for $x > 0$ (a Gamma distribution), i.e., $U(x) = x - \\ln x$, for which $\\langle x\\rangle = 2$ and $\\mathrm{Var}(x) = 2$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "samples, acceptance = metropolis_positive(lambda x: x - np.log(x))\n", "print(f\"{len(samples)} samples, = {samples.mean():.3f}, Var(x) = {samples.var():.3f}, acceptance = {acceptance:.2f}\")\n", "\n", "xs = np.linspace(0.01, 10, 400)\n", "plt.hist(samples, bins=100, range=(0, 10), density=True, alpha=0.6, label=\"sampler\")\n", "plt.plot(xs, xs * np.exp(-xs), \"k\", label=r\"target $x\\,e^{-x}$\")\n", "plt.xlabel(\"$x$\")\n", "plt.ylabel(\"probability density\")\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tasks**\n", "\n", "1. Compare $\\langle x\\rangle$, $\\mathrm{Var}(x)$, and the histogram with the exact answers. How many samples did the chain return, and how many steps did it take?\n", "2. As in today's lecture, this proposal is not symmetric. The density of $x' = x\\,e^{\\sigma z}$ is\n", " $$q(x'|x) = \\frac{1}{x'\\sigma\\sqrt{2\\pi}}\\exp\\left[-\\frac{(\\ln x' - \\ln x)^2}{2\\sigma^2}\\right].$$\n", " Compute the ratio $q(x|x')/q(x'|x)$. What should the acceptance probability be?\n", "3. Fix the mistakes one at a time, re-running the test after each. How large is the error from each mistake on its own? Which one would you have noticed without knowing the exact answer?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your tests and your fixed version here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: The 2D Ising model (about 25 minutes)\n", "\n", "The prompt given to the assistant:\n", "\n", "> Write a Metropolis simulation of the 2D Ising model with $J = k_\\mathrm{B} = 1$ and periodic boundary conditions on an $L \\times L$ lattice. Use single-spin flips, and return the absolute magnetization per spin and the energy per spin after each sweep.\n", "\n", "The energy function below was written separately; you may treat it as correct." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def energy(s):\n", " # E = -sum over nearest-neighbor pairs s_i s_j, each pair counted once (periodic boundaries)\n", " return -np.sum(s * (np.roll(s, 1, axis=0) + np.roll(s, 1, axis=1)))\n", "\n", "\n", "def ising_metropolis(L=3, T=2.0, n_sweeps=20_000, seed=0):\n", " rng = np.random.default_rng(seed)\n", " s = rng.choice([-1, 1], size=(L, L))\n", " m_series, e_series = [], []\n", " for sweep in range(n_sweeps):\n", " for _ in range(L * L):\n", " i, j = rng.integers(0, L, size=2)\n", " s[i, j] *= -1 # trial flip\n", " neighbors = s[(i + 1) % L, j] + s[i - 1, j] + s[i, (j + 1) % L] + s[i, j - 1]\n", " dE = -s[i, j] * neighbors # energy change of the flip\n", " if dE > 0 and rng.random() > np.exp(-dE / T):\n", " pass # rejected\n", " m_series.append(abs(s.mean()))\n", " e_series.append(energy(s) / (L * L))\n", " return np.array(m_series), np.array(e_series)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "m, e = ising_metropolis(L=3, T=2.0)\n", "burn = 1000\n", "print(f\"L = 3, T = 2: <|m|> = {m[burn:].mean():.3f}, = {e[burn:].mean():.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tasks**\n", "\n", "1. A $3 \\times 3$ lattice has only $2^9 = 512$ configurations, so you can compute $\\langle |m|\\rangle$ and $\\langle e\\rangle$ exactly by summing over all of them with Boltzmann weights. Write that function and compare.\n", " (Hint: `itertools.product([-1, 1], repeat=9)` loops over all configurations.)\n", "2. Compare with your exact answers at other temperatures, including very high and very low $T$. Which temperature does the simulation seem to be at?\n", "3. Fix the mistakes one at a time, re-testing after each. What does each fix change?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your exact enumeration, tests, and fixed version here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 4: Ask an AI to review the code (about 10 minutes)\n", "\n", "Give one of the original functions above to an AI assistant and ask it to review the code for errors, without telling it what is wrong.\n", "\n", "- Which mistakes does it find?\n", "- Does it miss any, or report problems that are not there?\n", "- Would its review have been enough without your tests?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Audit log\n", "\n", "Fill in one row per mistake. We will compare logs at the end of the lab; the same kind of record belongs in the AI use appendix of your assignments.\n", "\n", "| Function | Symptom | Test that exposed it | Fix | Would you have caught it without that test? |\n", "|---|---|---|---|---|\n", "| | | | | |\n", "| | | | | |\n", "| | | | | |" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 2 }