Build a Slack-Bot LLM Moderator With Claude: End-to-End Python Tutorial (May 2026)

Image: Anthropic API Messages documentation, used for editorial coverage of the Claude API surface this tutorial calls.

What you’ll build

A Slack bot that listens on a watched channel, sends every new message text to Claude for a four-way safety classification (harassment / spam / off-topic / OK), and DMs the original sender a private moderation note when the classifier flags the message. The bot runs locally during development behind an ngrok tunnel for Slack’s Events API to reach, then deploys to Fly.io as a long-running web service.

By the end you will have:

• A Slack app registered with the correct OAuth scopes and Events API subscriptions, per the Slack API documentation on Events API and OAuth scope reference.
• A Bolt for Python web app handling the message.channels event from Slack and the Messages API request to Anthropic.
• A working ngrok tunnel for local development that survives Slack’s request-signature verification.
• A Fly.io deployment carrying both Slack signing-secret and Anthropic API key as Fly secrets.

Skills assumed: Python 3.11 or later, comfort with pip, basic async/await recognition, comfort installing CLI tools. You do not need prior Slack-app or Fly.io experience; the steps below are reproducible end to end.

Status check before you start (May 2026)

Three things worth verifying in your own browser before committing an afternoon to this tutorial.

First, Bolt for Python is the Slack-maintained framework for Python Slack apps. The framework’s getting-started documentation at tools.slack.dev/bolt-python walks through the exact OAuth + Events API + signing-secret flow this tutorial uses ¹ . Verify the current install command and the canonical Python version requirement on that page before installing, since the package floor moves periodically.

Second, Slack’s Events API is the load-bearing surface for message-listening bots. Per the Events API documentation, your app subscribes to specific event types (here, message.channels) and Slack POSTs a JSON payload to your request URL each time the event fires ² . Slack requires the request URL to verify ownership via a one-time challenge handshake and then verify every subsequent request via an HMAC signature derived from your app’s signing secret. Bolt for Python handles both flows automatically, but the request URL has to be publicly reachable, which is why ngrok or an equivalent tunnel is required for local development.

Third, the Anthropic Messages API is the classification surface. Per the Messages API reference at docs.claude.com/en/api/messages, you POST a JSON body containing the model name, max_tokens, and a messages array, with the model name drawn from Anthropic’s current model catalogue ³ . Frontier-LLM model names drift on ~60-day cadences; do not assume the example identifier in the code below is still current on the day you build. Open the model catalogue in your browser and substitute the current entry.

Step 1: register the Slack app

Open api.slack.com/apps in your browser and click “Create New App”, then “From scratch”. Name it something honest (claude-moderator-dev works) and pick the workspace you’ll test in.

The next three sub-steps configure the app’s permissions and event subscriptions.

OAuth scopes

Under “OAuth & Permissions”, add the following bot-token scopes (the Slack OAuth scopes reference catalogues every scope name and its capability) ⁴ :

• channels:history: read messages in public channels the bot is added to.
• chat:write: post messages and DMs as the bot.
• im:write: open a direct-message channel with a user before sending a DM.
• users:read: resolve user IDs to display names for the moderation-note body.

The four scopes above are the minimum surface the moderator needs. Add only what the app actually uses; Slack’s permission model is least-privilege by design.

Event subscriptions

Under “Event Subscriptions”, flip the toggle to “On”. Leave the Request URL empty for now; you’ll fill it in after the ngrok tunnel is live. Under “Subscribe to bot events”, add the message.channels event ⁵ .

The message.channels event fires for every new message posted in a public channel the bot is a member of. The event payload includes the text, the channel ID, the user ID, and the message timestamp: everything the classifier needs.

Install to workspace

Under “Install App”, click “Install to Workspace”. Approve the permissions prompt; Slack hands you back a Bot User OAuth Token starting with xoxb-. Copy it and also copy the Signing Secret from the “Basic Information” page. Both go into a local .env file in the next step, never into source control.

Image: Anthropic API getting-started documentation, used for editorial coverage of the Claude API surface this tutorial calls.

Step 2: scaffold the Bolt for Python app

Create a working directory and install dependencies:

mkdir claude-slack-moderator && cd claude-slack-moderator
python3 -m venv .venv
source .venv/bin/activate
pip install "slack-bolt>=1.20" anthropic python-dotenv

Create a .env file in the project root with the three secrets:

SLACK_BOT_TOKEN=xoxb-...
SLACK_SIGNING_SECRET=...
ANTHROPIC_API_KEY=sk-ant-...

Add .env to .gitignore immediately. Leaking a Slack bot token gives the leaker full posting privileges on the workspace channels the bot can see; leaking the Anthropic key gives them billable API access on your account.

Now the application code. Save the following as app.py:

# app.py
import os
import logging

from dotenv import load_dotenv
from slack_bolt import App
from slack_bolt.adapter.flask import SlackRequestHandler
from flask import Flask, request
from anthropic import Anthropic

load_dotenv()
logging.basicConfig(level=logging.INFO)

WATCHED_CHANNEL = os.environ.get("WATCHED_CHANNEL_ID", "")

bolt_app = App(
    token=os.environ["SLACK_BOT_TOKEN"],
    signing_secret=os.environ["SLACK_SIGNING_SECRET"],
)
anthropic_client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
flask_app = Flask(__name__)
handler = SlackRequestHandler(bolt_app)


@flask_app.route("/slack/events", methods=["POST"])
def slack_events():
    return handler.handle(request)

The SlackRequestHandler adapter lets Bolt for Python run under Flask, which makes the eventual Fly.io deploy straightforward, since Fly’s default Procfile-style entry point is a long-running web process listening on $PORT. Per the Bolt for Python documentation, the adapter handles Slack’s signing-secret verification and event-payload parsing transparently ⁶ .

Step 3: wire the message listener and the Claude classifier

Append the listener and the classifier function to app.py:

CLASSIFIER_SYSTEM_PROMPT = """You are a Slack channel moderator.
Classify the user message into exactly one category:
- harassment: targeted insults, slurs, threats, or doxxing.
- spam: promotional links, repeated copy-paste, or off-topic
  advertising.
- off-topic: unrelated to the channel's stated purpose.
- ok: none of the above.

Respond with ONLY the category name in lowercase, nothing else."""


def classify_message(text: str) -> str:
    response = anthropic_client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=16,
        system=CLASSIFIER_SYSTEM_PROMPT,
        messages=[{"role": "user", "content": text}],
    )
    label = response.content[0].text.strip().lower()
    if label not in {"harassment", "spam", "off-topic", "ok"}:
        return "ok"
    return label


@bolt_app.event("message")
def handle_message(event, client, logger):
    if event.get("subtype") is not None:
        return
    if event.get("bot_id"):
        return
    if WATCHED_CHANNEL and event.get("channel") != WATCHED_CHANNEL:
        return

    text = event.get("text", "")
    user_id = event.get("user", "")
    if not text or not user_id:
        return

    try:
        label = classify_message(text)
    except Exception as exc:
        logger.exception("classifier call failed: %s", exc)
        return

    if label == "ok":
        return

    note = (
        f"Heads-up: your recent message was flagged as *{label}* "
        "by the channel's automated moderator. This is a private "
        "note, no public action has been taken. If you think the "
        "flag is wrong, please reply to a channel admin."
    )
    try:
        dm = client.conversations_open(users=user_id)
        channel_id = dm["channel"]["id"]
        client.chat_postMessage(channel=channel_id, text=note)
    except Exception as exc:
        logger.exception("DM post failed: %s", exc)

Two design choices worth flagging:

• The system prompt locks the output shape. Asking for “exactly one category” and validating against a known set means a classifier hallucination (Claude returning a long explanation, an unrelated word, or markdown formatting) defaults to ok rather than to a flagged false-positive. Per the Anthropic Messages API documentation, output discipline is the prompt-engineering surface; the API does not enforce categorical output natively ⁷ .
• The bot ignores its own messages and subtype events. Slack’s message.channels event fires on edits, deletions, channel-join events, and bot postings too. Filtering on subtype is not None and bot_id truthy keeps the classifier from running on its own moderation notes (which would loop).

Add the Flask entry point at the bottom of app.py:

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "3000"))
    flask_app.run(host="0.0.0.0", port=port)

Step 4: run locally behind ngrok

Slack’s Events API requires a publicly reachable HTTPS URL. ngrok creates a tunnel from a *.ngrok-free.app (or paid custom) URL to your local port; per the ngrok getting-started documentation, the free tier carries a randomly assigned subdomain that rotates on every restart, while paid tiers carry a reserved domain ⁸ .

In one terminal, run the Flask app:

python app.py

In a second terminal, start the tunnel:

ngrok http 3000

ngrok prints a forwarding URL like https://abc1-203-0-113-42.ngrok-free.app. Copy that URL.

Back in the Slack app dashboard, paste https://YOUR-NGROK-HOST/slack/events into the Event Subscriptions Request URL field. Slack sends a one-time url_verification challenge; Bolt for Python’s request handler responds to it automatically. The page should show “Verified” within a few seconds.

Invite the bot into a test channel (/invite @claude-moderator-dev from inside the channel). Set WATCHED_CHANNEL_ID in your .env to that channel’s ID; Slack channel URLs end in /archives/CHANNEL_ID. Restart python app.py. Post a clearly off-topic message in the channel; within a second or two the bot DMs you a moderation note.

Image: ngrok getting-started documentation, used for editorial coverage of the local-development tunnel used in this step.

Step 5: deploy to Fly.io

Once the bot works locally, move it to a host that doesn’t depend on your laptop being awake. Fly.io’s launch documentation walks through the fly launch command, which scaffolds a fly.toml and a default Dockerfile ⁹ .

Install flyctl per the Fly.io install instructions, then authenticate:

fly auth signup   # or `fly auth login` if you already have an account

Add a requirements.txt to the project root:

slack-bolt>=1.20
anthropic
python-dotenv
flask
gunicorn

Add a Dockerfile:

FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

ENV PORT=8080
EXPOSE 8080

CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "2", "app:flask_app"]

Fly.io’s Dockerfile-based deploys read the Dockerfile from the project root and build the image on Fly’s builder ¹⁰ .

Initialise the Fly app:

fly launch --no-deploy

Pick a region close to your Slack workspace’s user base; the prompt accepts default values for everything else. The command writes a fly.toml and reserves the app name.

Set the secrets. Per the Fly.io secrets documentation, secrets injected via fly secrets set appear as environment variables inside the running container ¹¹ :

fly secrets set \
    SLACK_BOT_TOKEN=xoxb-... \
    SLACK_SIGNING_SECRET=... \
    ANTHROPIC_API_KEY=sk-ant-... \
    WATCHED_CHANNEL_ID=C0123456789

Deploy:

fly deploy

Fly builds the image, ships it, and prints the public URL (typically https://APP-NAME.fly.dev where APP-NAME is the name you chose during fly launch). Update the Slack app’s Event Subscriptions Request URL to https://APP-NAME.fly.dev/slack/events. Slack re-verifies the URL on save.

Image: Fly.io launch documentation, used for editorial coverage of the production deployment step.

Image: Fly.io secrets management documentation, used for editorial coverage of the secrets-handling step in the deploy.

Step 6: verify end-to-end

A few checks worth running against the deployed bot:

• Post an OK message. No DM should arrive; the classifier returns ok and the listener short-circuits.
• Post a clearly off-topic message in the watched channel. The bot DMs you within a second or two.
• Check Fly.io logs. fly logs streams the container’s stdout; you should see an INFO line per inbound event and any classifier-call exceptions if the Anthropic API throttles or errors.
• Check Anthropic console usage. Token spend for each classification call appears on the Anthropic console; the prompt is short and max_tokens=16 caps the response, so each call is small. Per Anthropic’s Messages API reference, you pay for input and output tokens separately, so the system prompt counts on every call ¹² .

Known limitations and where this is honestly weak

A few caveats the build should acknowledge rather than paper over:

• Cost scales with channel volume. Every channel message becomes one Anthropic API call. For a low-volume channel (say, a few hundred messages per day) the spend is negligible; for a high-volume channel running into thousands of messages per day, a cheap regex pre-filter for obvious spam patterns before the Claude call is worth adding. The aggregated developer-forum consensus on building LLM-classifier pipelines treats the regex-prefilter as standard practice.
• The four-way classification is a starting point. Production moderators typically distinguish severity tiers (warning vs. action) and confidence levels. The Anthropic Messages API supports structured outputs via system-prompt discipline, but for high-stakes moderation, fine-tuning a smaller model on labelled in-domain examples generally outperforms zero-shot prompting per the model-choice guidance on the Anthropic docs.
• DM-as-feedback assumes the user reads DMs. Some workspaces have notification settings that suppress bot DMs. A workspace admin should be informed if the bot is intended to influence behaviour at scale; transparent disclosure of the moderation pipeline to channel members is a baseline community-management practice.
• No public action is taken. This bot only DMs the sender. Adding channels:manage scope and chat:delete would let the bot delete flagged messages, but auto-deletion without a clear appeals path tends to read as heavy-handed. Per the Bolt for Python message-listening documentation, the scope surface to delete is straightforward; the editorial choice is whether to wield it ¹³ .

What to read next

The single Slack workspace + single watched channel pattern this tutorial walks is the simplest possible deployment. Two natural extensions:

• Per-channel classifier configuration. Store a channel-to-prompt mapping (in a small database, or in environment variables) so the moderator can be configured for different channel cultures without redeploying.
• Audit-log persistence. Write every classification decision (timestamp, channel, user, label, message excerpt) to a small SQLite or Postgres database for retrospective review. Fly.io supports both volumes and managed Postgres on the same control plane as the app deploy.