Build a Research-Paper Recommender with Semantic Scholar + Claude: An End-to-End Streamlit Tutorial

Image: Semantic Scholar Academic Graph API product page, used for editorial coverage of the API taught in this tutorial.

What you’ll build

A research-paper recommender that takes a paper title or URL, looks the paper up on Semantic Scholar, asks the Recommendations API for related work, and hands each result to Claude Sonnet 4.6 to summarise in three sentences. The output is a clean ranked Markdown list (title, authors, venue, year, link, summary) rendered inside a Streamlit web app you can run on a laptop or deploy free to Streamlit Community Cloud.

The whole project sits in a single app.py file of roughly 180 lines plus a small requirements.txt. By the end, the aggregated source consensus on the workflow (paper search via the Graph API, similarity via the Recommendations API, summarisation via the Claude Messages API) will be wired together and running locally at http://localhost:8501.

Three APIs do the heavy lifting. The Semantic Scholar Recommendations API returns up to 500 similar papers for any paper ID, DOI, or arXiv ID, sorted by relevance. ¹ The Semantic Scholar Graph API search endpoint turns a free-text query (a title, partial title, or topic phrase) into a paper ID. ² The Anthropic Messages API powers the per-result summarisation with Claude Sonnet 4.6, listed at $3 per million input tokens and $15 per million output tokens. ³

What you’ll need

• Python 3.10 or later. The Anthropic SDK and current Streamlit releases both target 3.10+.
• A Semantic Scholar account is optional. Per the Academic Graph API product page, unauthenticated requests share a 1000-requests-per-second pool, while an API key gets an introductory dedicated 1 RPS. ⁴ The tutorial works without a key; budget for occasional throttling if you batch through many papers.
• An Anthropic API key. Sign in at console.anthropic.com, mint a key, and load $5 of credit. A walkthrough run of this tutorial (one paper lookup, ten recommendations, ten three-sentence summaries) costs under $0.05 at Sonnet 4.6 rates. ³
• About 45 minutes for copy-paste; longer if you read every line.

No background in vector search, embeddings, or RAG is required. The Recommendations API does the similarity step server-side; Claude is used purely for natural-language summarisation per result, not for retrieval.

Step 1: Install the packages

Create a project directory and a virtual environment:

mkdir paper-recommender && cd paper-recommender
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

Write a requirements.txt:

streamlit>=1.36
anthropic>=0.40
requests>=2.32

Install:

pip install -r requirements.txt

Three packages, no LangChain, no vector store. Per the Anthropic SDK’s PyPI listing, the anthropic package is the official first-party client. ⁵ The requests library handles the two HTTP calls to Semantic Scholar: the search and the recommendation lookup. ⁶ Streamlit provides the UI primitives (text input, button, status callout, Markdown rendering) without writing HTML. ⁷

Image: Streamlit documentation home, used for editorial coverage of the UI framework powering the recommender.

Step 2: Set the API key

Streamlit reads secrets from .streamlit/secrets.toml in development and from the Community Cloud dashboard in production. Create the file:

mkdir -p .streamlit

# .streamlit/secrets.toml
ANTHROPIC_API_KEY = "sk-ant-..."
# Optional, leave blank to use the shared unauthenticated pool
SEMANTIC_SCHOLAR_API_KEY = ""

Add .streamlit/secrets.toml to .gitignore before any commit. The Anthropic key is reachable from Python as st.secrets["ANTHROPIC_API_KEY"]; Streamlit Community Cloud surfaces the same dict in production after you paste the secrets into the app dashboard. ⁸

Step 3: Resolve a user’s input to a Semantic Scholar paper ID

Users will paste anything: a paper title, an arXiv URL, a DOI, or a Semantic Scholar URL. The Graph API’s /graph/v1/paper/search endpoint accepts a free-text query parameter and returns matching papers; the Recommendations API accepts paper IDs in several formats, including bare Semantic Scholar IDs, arXiv:NNNN.NNNNN, and doi:10.NNNN/.... ²

Create app.py and start with the resolver:

import re
from typing import Optional
import requests

S2_BASE = "https://api.semanticscholar.org"
SEARCH_FIELDS = "paperId,title,authors,year,venue,externalIds"


def normalise_paper_input(user_input: str) -> Optional[str]:
    """Convert pasted text into a Semantic Scholar-acceptable ID.

    Accepts: arXiv URLs, arXiv IDs, DOIs, S2 paper URLs, raw S2 IDs,
    or free-text titles (returns None so the caller falls back to
    search).
    """
    text = user_input.strip()
    if not text:
        return None

    # arXiv URL or bare arXiv ID
    arxiv_match = re.search(
        r"(?:arxiv\.org/(?:abs|pdf)/)?(\d{4}\.\d{4,5})(?:v\d+)?",
        text,
        re.IGNORECASE,
    )
    if arxiv_match:
        return f"arXiv:{arxiv_match.group(1)}"

    # DOI (with or without doi.org prefix)
    doi_match = re.search(r"(10\.\d{4,9}/[\w./()\-:;]+)", text)
    if doi_match:
        return f"doi:{doi_match.group(1).rstrip('.')}"

    # Semantic Scholar URL or 40-char paper ID
    s2_match = re.search(r"([0-9a-f]{40})", text)
    if s2_match:
        return s2_match.group(1)

    # Free text — caller will hit the search endpoint
    return None


def search_paper(query: str, api_key: str = "") -> Optional[dict]:
    """Hit the Graph API search endpoint, return the top hit."""
    headers = {"x-api-key": api_key} if api_key else {}
    response = requests.get(
        f"{S2_BASE}/graph/v1/paper/search",
        params={"query": query, "limit": 1, "fields": SEARCH_FIELDS},
        headers=headers,
        timeout=20,
    )
    response.raise_for_status()
    data = response.json().get("data") or []
    return data[0] if data else None

A few notes. The arXiv regex is deliberately loose: it accepts a bare 2401.12345 typed into the box just as readily as a full https://arxiv.org/abs/2401.12345v3 URL. The Semantic Scholar 40-character paper ID is a SHA-1 hex string; the regex picks it out of any pasted URL like semanticscholar.org/paper/<id>. Per the Recommendations API docs, all three ID formats (raw S2 IDs, arXiv:NNNN.NNNNN, and doi:10.NNNN/...) are accepted interchangeably on the recommendations endpoint, so the resolver normalises to whichever the user actually pasted. ¹

Image: Semantic Scholar API tutorial, used for editorial coverage of the endpoint patterns covered in this section.

Step 4: Call the Recommendations API

The recommendations endpoint is GET /recommendations/v1/papers/forpaper/{paper_id}. It accepts limit (1–500) and a fields comma-separated list naming the columns to return. Add this to app.py:

REC_FIELDS = "paperId,title,authors,year,venue,abstract,url,externalIds"


def fetch_recommendations(
    paper_id: str,
    limit: int = 10,
    api_key: str = "",
) -> list[dict]:
    """Return up to `limit` similar papers, sorted by relevance."""
    headers = {"x-api-key": api_key} if api_key else {}
    response = requests.get(
        f"{S2_BASE}/recommendations/v1/papers/forpaper/{paper_id}",
        params={"limit": limit, "fields": REC_FIELDS},
        headers=headers,
        timeout=30,
    )
    response.raise_for_status()
    return response.json().get("recommendedPapers", [])

That is the entire similarity step: one HTTP GET. Semantic Scholar runs the actual nearest-neighbour search server-side over its Specter-style paper embeddings; the API tutorial notes that results are returned in descending order of similarity, so the response order is the ranking. ⁹ Asking for limit=10 is a sensible default for an interactive UI; production batch jobs can ask for up to 500. ¹

The abstract field will be present for most papers but is occasionally None, since Semantic Scholar can’t redistribute every publisher’s abstract text. Handle the missing case in the summariser.

Step 5: Summarise each result with Claude

Now hand each recommendation to Claude Sonnet 4.6 for a three-sentence reader-facing summary. Add to app.py:

import anthropic

CLAUDE_MODEL = "claude-sonnet-4-6"

SUMMARY_PROMPT = """You are summarising a research paper for a working ML engineer browsing a recommendation list.

Write exactly three sentences:
1. What the paper does (the contribution).
2. The method or mechanism, in plain language.
3. Why a reader of the seed paper would care.

Do not include the title or authors — those are rendered separately.
Do not start with "This paper" — vary the opening.
If the abstract is empty, say so and stop.

Paper title: {title}
Authors: {authors}
Venue: {venue} ({year})
Abstract:
{abstract}"""


def summarise(paper: dict, client: anthropic.Anthropic) -> str:
    """Three-sentence summary of one recommendation."""
    authors = ", ".join(
        a.get("name", "") for a in (paper.get("authors") or [])[:5]
    ) or "Unknown"
    prompt = SUMMARY_PROMPT.format(
        title=paper.get("title") or "Untitled",
        authors=authors,
        venue=paper.get("venue") or "Preprint",
        year=paper.get("year") or "n.d.",
        abstract=paper.get("abstract") or "(no abstract available)",
    )
    message = client.messages.create(
        model=CLAUDE_MODEL,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}],
    )
    return message.content[0].text.strip()

Two design choices worth flagging. The summary prompt asks Claude to vary the opening sentence, because a common cadence-tell on LLM-generated lists is “This paper proposes…” repeated ten times. Steering the model away from that with one explicit instruction costs nothing. Truncating the author list to five is a UX choice: long author lists from physics or bio papers (a hundred-plus collaborators) clutter the rendered Markdown without adding signal.

Per the Anthropic Messages API reference, the model field accepts the dateless claude-sonnet-4-6 alias as a pinned snapshot. ¹⁰ The Models overview page confirms this is the current generally-available Sonnet release as of May 2026, ahead of Sonnet 4.5 and Sonnet 4. ³

Image: Anthropic Messages API reference, used for editorial coverage of the API method called in the summariser.

Step 6: Wire it to a Streamlit UI

The final block of app.py adds the user interface. Streamlit reruns the script top-to-bottom on every interaction; cache the Anthropic client so it isn’t reinstantiated per click.

import streamlit as st

st.set_page_config(
    page_title="Paper Recommender", page_icon=":bookmark_tabs:"
)
st.title("Research-Paper Recommender")
st.caption(
    "Paste a paper title, arXiv URL, DOI, or Semantic Scholar URL. "
    "The app fetches up to ten similar papers and summarises each with Claude."
)


@st.cache_resource
def get_anthropic_client() -> anthropic.Anthropic:
    return anthropic.Anthropic(api_key=st.secrets["ANTHROPIC_API_KEY"])


user_input = st.text_input(
    "Paper title or URL",
    placeholder="e.g. 'Attention Is All You Need' or arxiv.org/abs/1706.03762",
)
limit = st.slider("Number of recommendations", 3, 20, 10)

if st.button("Find similar papers", type="primary") and user_input:
    s2_key = st.secrets.get("SEMANTIC_SCHOLAR_API_KEY", "")
    client = get_anthropic_client()

    with st.status("Resolving paper...", expanded=True) as status:
        paper_id = normalise_paper_input(user_input)
        if paper_id is None:
            st.write("No ID detected — searching by title.")
            hit = search_paper(user_input, api_key=s2_key)
            if hit is None:
                status.update(label="No match found.", state="error")
                st.stop()
            paper_id = hit["paperId"]
            st.write(f"Matched: **{hit['title']}** ({hit.get('year')})")
        else:
            st.write(f"Using ID: `{paper_id}`")

        st.write("Fetching recommendations...")
        recs = fetch_recommendations(paper_id, limit=limit, api_key=s2_key)
        status.update(
            label=f"Got {len(recs)} recommendations. Summarising...",
            state="running",
        )

        lines: list[str] = []
        progress = st.progress(0.0)
        for i, paper in enumerate(recs, start=1):
            summary = summarise(paper, client)
            authors = ", ".join(
                a.get("name", "") for a in (paper.get("authors") or [])[:3]
            ) or "Unknown"
            url = paper.get("url") or (
                f"https://www.semanticscholar.org/paper/{paper['paperId']}"
            )
            lines.append(
                f"### {i}. [{paper.get('title')}]({url})\n"
                f"*{authors} — {paper.get('venue') or 'Preprint'} "
                f"({paper.get('year') or 'n.d.'})*\n\n"
                f"{summary}\n"
            )
            progress.progress(i / len(recs))
        status.update(label="Done.", state="complete")

    st.markdown("\n".join(lines))

The st.cache_resource decorator memoises the Anthropic client across reruns, so a user clicking the button five times in a row doesn’t pay the SDK initialisation cost five times. Streamlit’s documentation flags cache_resource as the right primitive for non-serialisable global objects like SDK clients and database connections. ⁷ The st.status callout gives users visible progress through the three phases (resolve, fetch, summarise); the st.progress bar ticks per summary so a 10-paper run feels responsive even though Claude takes a few seconds per call.

Image: Streamlit st.text_input reference, used for editorial coverage of the input widget used in the UI.

Step 7: Run it

streamlit run app.py

Streamlit opens http://localhost:8501 in a browser. Paste “Attention Is All You Need” into the box, set the slider to 10, click the button. Within roughly 15 to 20 seconds (one search call, one recommendations call, ten Claude calls in series), a ranked list of ten linked papers with three-sentence summaries renders below the form.

For an arXiv test case, paste arxiv.org/abs/2005.11401 (the original RAG paper) and watch the recommender surface retrieval-augmented generation follow-up work. For a DOI test case, paste 10.18653/v1/2020.acl-main.207 and the same ID-detection branch routes correctly.

Step 8: Deploy free on Streamlit Community Cloud

Per Streamlit’s deployment guide, Community Cloud takes a GitHub repository containing app.py and requirements.txt, builds a container, and serves the app at a streamlit.app subdomain at no cost. ⁸

Three steps:

1. Push the project to a public or private GitHub repo. Verify .streamlit/secrets.toml is in .gitignore.
2. At share.streamlit.io, click “New app”, point it at the repo and the app.py file.
3. In the app’s Settings → Secrets pane, paste the same two keys (ANTHROPIC_API_KEY, optionally SEMANTIC_SCHOLAR_API_KEY) in TOML format.

The first build takes a few minutes; subsequent pushes to the connected branch redeploy automatically. Be aware that Community Cloud apps sleep after inactivity and cold-start on first visit, which adds a few seconds before the first interaction. For higher traffic, the deployment guide notes a paid Teams tier with dedicated resources. ⁸

Where to take it from here

A few extensions worth exploring once the base app is running.

• Parallel summarisation. The current loop is sequential; ten Claude calls in series is fine for ten papers but uncomfortable for fifty. Wrap the per-paper summarise() call in asyncio.gather with anthropic.AsyncAnthropic and the wall-clock time drops sharply.
• Citation graph traversal. The Graph API also exposes /graph/v1/paper/{paper_id}/citations and /references; chain them to surface papers that cite the seed AND are recommended, which often catches survey-style work the recommender alone misses.
• Caching the recommendations. Wrap fetch_recommendations in @st.cache_data(ttl=3600) so re-entering the same paper within an hour doesn’t re-bill Semantic Scholar, useful when iterating on the summary prompt.
• A multi-paper “for a list of papers” mode. Per the Ai2 announcement, the Recommendations API also exposes POST /recommendations/v1/papers with a JSON body listing positive and negative examples, the natural next step for users who want recommendations grounded in their reading history rather than one seed paper. ¹¹

Total project surface: two files (app.py, requirements.txt), three external APIs (Semantic Scholar Graph, Semantic Scholar Recommendations, Anthropic Messages), zero infrastructure to provision. The whole thing fits into a 45-minute coffee break and costs pennies per recommendation list.