Paperbase

Overview

Paperbase has one supported local workflow:

1. Copy the repo-root .env.example to .env.
2. Run make setup.
3. Run make doctor.
4. Run make dev.

The repo is intentionally opinionated: one canonical env file at the repo root, a small repo-root make surface with make test-smoke reserved for PR CI, and vp required for frontend builds. Git hooks run the full verification path before push; PR CI runs a cheaper smoke gate.

Prerequisites

• Python 3.11+
• PostgreSQL with pgvector and pg_trgm available locally
• vp installed with curl -fsSL https://vite.plus | bash

Local setup

Environment file

The only supported local env file is .env.example copied to .env at the repo root.

cp .env.example .env

The sections in that file map to deployment targets like this:

Optional browser-account settings hang off the API env surface:

• PAPERBASE_MV37_* powers login through the Paperbase OAuth start route, which creates PKCE state and redirects to the configured MV37 issuer (http://127.0.0.1:8081/realms/mv37 locally, https://auth.mv37.org/realms/mv37 in production).
• PAPERBASE_GITHUB_* powers the optional in-app GitHub connection flow and project repo sync.
• PAPERBASE_ADMIN_EMAILS is a comma-separated allowlist for the signed-in admin jobs workspace.
• PAPERBASE_WORKER_STATUS_URL lets the API read worker /health and /metrics so the admin jobs table can surface stalled runs.
• PAPERBASE_FETCH_CODE_CROSSLINKS_AT_INDEX controls the default-on Hugging Face paper-page lookup that prefers an exact arXiv-id githubRepo match over raw text extraction. Turn it off for pure text-only harvests.

Logged-in users can also store Anthropic keys, Modal sandbox credentials, a signed-in-only light/dark appearance preference, and a weekly papers review email preference.

Make commands

make help shows the supported local interface.

• make setup installs the Python packages, chat dependencies, repo hooks, and a starter .env.
• make doctor verifies vp, Python, Postgres, pgvector, ports, and the repo env.
• make dev initializes the local database and storage, then runs the API, worker, MCP bridge, and chat dev server. The API auto-reloads authored backend code changes without watching generated storage or frontend static output.
• make test-smoke runs the PR-oriented CI smoke gate: lint, chat checks/tests/build, a focused backend subset, and plugin tests.
• make test runs ruff check, vp check, a frontend build, backend tests, and plugin tests.
• make profile runs the deterministic hotpath profiling flow used for performance tracking.
• make benchmark BENCHMARK=iq-1 BENCHMARK_ARGS="--verbose" runs a repo-native benchmark through the supported benchmark workflow.
• make clean stops local services and removes generated logs, caches, and frontend output.

Reset the local stack

make reset-local stops the local stack, wipes the local database and storage, and recreates the latest schema. The lower-level admin CLI exposes the same operation as:

.venv/bin/paperbase-dev flush-local

Local architecture

Components

• paperbase/ contains the backend package and both runtime entrypoints.
• paperbase-plugin/ contains the MCP bridge used by Codex and Claude Code.
• chat/ contains the React frontend.
• paperbase/src/paperbase/static/ is generated build output, not source.

paperbase-api and paperbase-worker are the same codebase and image with different entrypoints: paperbase-api serves the FastAPI application, and paperbase-worker runs the background job loop.

make dev runs all of these locally:

• API at http://127.0.0.1:8080
• chat dev server at http://127.0.0.1:5173
• MCP bridge at http://127.0.0.1:8090/mcp

API, worker, and plugin logs go to .run/. Backend route and handler changes reload into paperbase-api automatically without a manual restart. Generated paper storage and packaged frontend static files are excluded from the API reload watcher so long-running streams are not interrupted by harvested artifacts or chat builds.

Browser chat

The browser chat experience is project-based: projects are persisted in the backend and own chat sessions. Creating a new project is local-first; GitHub sync is optional and can be enabled in Settings when users want repository backup, imports, or external version control.

• Each project can start paper explorations from a command-palette style search over paper title or arXiv id.
• Paper explorations and PDF modals support text highlights and pen-drawn region selections with cropped image context for grounded chat follow-ups.
• Project home can generate a cached presentation with a React/D3 visual, Markdown summary, optional public share page, and an optional ElevenLabs two-speaker audio walkthrough when the user has saved an ElevenLabs API key.
• Python and notebook files open as the single active project resource in a Monaco-backed high-contrast editor view. Notebooks are converted with Jupytext to py:percent source for editing and running, then regenerated back to .ipynb when saved to GitHub.
• Project chats can invoke a project-scoped coding tool surface against repo-relative workspace text files: list, read, grep, write, patch, move, delete, git-diff, exec, test, lint, and run.
• Research turns expose a single restricted paperbase_shell literature tool for search, results, ls, find, cat, head, lines, grep, map, read-only metadata sql, and ask-image over virtual paper paths. The MCP bridge exposes the same shell plus operational job/index tools.
• Save stores the current project file locally by default; when GitHub sync is enabled it creates a single Paperbase-managed GitHub commit. Runsaves first and then executes the full saved project workspace through the user’s Modal sandbox credentials with full mode by default.
• @-mentioning a paper in the chat composer sends the paper’s id with the request; if the row is not yet at index_status='indexed', the send path synchronously fetches and indexes it from arXiv before the chat turn runs, with mention_indexing_* SSE events surfacing fetch progress on the streaming endpoint.
• User-facing paper surfaces only expose fully indexed rows. Search, mention suggestions, related-paper lookups, paper detail reads, and artifact reads all require index_status='indexed'.

Frontend output

The frontend bundle served by Paperbase is built from chat/ into paperbase/src/paperbase/static/.

• That directory is generated and ignored by Git.
• make test builds it before backend tests because the API serves those files.
• make test-smoke also builds it so PR CI catches frontend bundle breakage.
• If you need the built shell without the dev server, run cd chat && vp build.

Admin CLI

The lower-level admin CLI is .venv/bin/paperbase-dev. It exposes maintenance operations that should not be part of the repo-root make surface.

Worker jobs

.venv/bin/paperbase-dev jobs list-dead
.venv/bin/paperbase-dev jobs retry <job_id>
.venv/bin/paperbase-dev jobs cancel <job_id>

Job responses include both raw status and server-computed effective_status. A running job is stalled when its worker lease expires without a heartbeat; pending jobs cancel immediately, while running jobs move through cooperative cancellation.

Historical harvests

Queue historical arXiv backfills with:

.venv/bin/paperbase-dev harvest \
  --categories cs.LG,cs.CL \
  --start 2015-01-01 \
  --end 2020-12-31 \
  --chunk-months 3 \
  --limit-per-window 1000

--limit-per-window caps each category-window chunk, not the whole category across the full date range. Harvest chunks run below interactive indexing priority so on-demand index_paper jobs can still run during a backfill.

Reindex embeddings

After changing the embedding model in a deployed environment, queue a full paper/chunk vector rewrite with:

.venv/bin/paperbase-dev reindex-embeddings

Code URL backfill

.venv/bin/paperbase-dev backfill-code-urls --dry-run
.venv/bin/paperbase-dev backfill-code-urls

The backfill uses Hugging Face paper search’s exact arXiv-id match before falling back to the in-text GitHub ranker.

Benchmarks

Deterministic hotpath and test-time artifacts live under benchmarks/. benchmarks/impact-scorecard-spec.md defines the scientific measurement spec for indexing quality, retrieval quality, and harness lift with vs without Paperbase.

make benchmark BENCHMARK=iq-1 BENCHMARK_ARGS="--verbose"

.venv/bin/paperbase-dev benchmark list
.venv/bin/paperbase-dev benchmark run iq-1 --verbose

The first implemented scorecard slice is a current-snapshot indexing census that writes benchmarks/out/iq1.current.json and benchmarks/out/iq1.current.md. .venv/bin/paperbase-dev benchmark iq-1 --verbose remains as a compatibility alias for the direct IQ-1 runner. Benchmark runs use a rich-backed live terminal UI during local dev and automatically fall back to plain text when the richer UI is unavailable.

MCP plugin

paperbase-plugin/ is the MCP bridge used by Codex and Claude Code. It exposes the same restricted paperbase_shell literature tool that the in-browser research turns use, plus operational job and index tools.

When make dev is running, the bridge listens at http://127.0.0.1:8090/mcp.

CI

• Pull requests run make test-smoke.
• Pushes to main and manual verify dispatches run full make test.
• Verify and profile workflows cancel superseded in-progress runs on the same PR or branch.
• Profiling runs only on main pushes or manual dispatch.

Source

The repository is open at github.com/mv37-org/paperbase. For use cases, feedback, or rough edges, email v@mv37.org.