PR: gate the public hashtag page behind login (scrapers were hammering it)

Summary #

yttrx’s /tags/<hashtag> page has been a recurring scraping target — prior mitigations included UA/ASN blocking and nginx per-IP rate-limiting with a fail2ban escalation, neither of which fully closed the gap. This patch closes it: it puts the HTML hashtag page behind a login wall. The tag timeline API, the tag RSS feed, and ActivityPub federation are deliberately left untouched.

Patch: 163f96cee...protect-tags-behind-auth.patch — 3 commits against Mastodon v4.6.3, waffle2k/mastodon@protect-tags-behind-auth.

Why the HTML page, and not the whole thing #

Before writing any code, I pulled nginx access logs (current + rotated, ~2 weeks) and split /tags/* and /api/v1/timelines/tag/* traffic by path shape and User-Agent:

The abuse was concentrated on the HTML page. Gating the API too would have punished real app users for scraper traffic they had nothing to do with, so the fix is scoped to just that one surface.

The fix #

TagsController (app/controllers/tags_controller.rb) gains one before_action:

before_action :authenticate_user!, if: -> { limited_federation_mode? || gate_html_request? }

# Anything that isn't the RSS feed or the ActivityPub JSON collection is
# the HTML page -- including ambiguous/wildcard Accept headers (bare
# curl and most naive scrapers send `Accept: */*` with no extension),
# which resolve to neither :json nor :rss and would slip past a plain
# `request.format == :html` check.
def gate_html_request?
  !%i(json rss).include?(request.format&.to_sym)
end

The first iteration used request.format == :html, which only matches an explicit Accept: text/html header — plain curl and most naive scrapers send Accept: */* with no extension, which resolves to neither :html, :json, nor :rss, so it slid right past the gate. The commit history in the patch documents catching this in testing and fixing it. An earlier iteration also reused Mastodon’s built-in local_topic_feed_access/remote_topic_feed_access setting, but that setting is shared with the (unrelated) trending-links timeline, so it was dropped in favor of a dedicated, unconditional gate on just this controller.

How to apply and deploy #

This instance runs a custom from-source image (yttrx-mastodon-birdui) with the Mastodon Bird UI themes baked in, since the stock runtime image can’t precompile assets. Applying the patch means rebuilding that image from the patched branch instead of stock Mastodon:

# 1. Pull the patched source instead of stock mastodon/mastodon
git clone --depth 1 --branch protect-tags-behind-auth \
  https://github.com/waffle2k/mastodon.git /root/birdui-build/mastodon

# 2. Re-apply the Bird UI theme layer (stock theme stays default; opt-in)
printf 'n\n' | bash /root/birdui-build/mastodon-bird-ui-nightly/scripts/install-to-mastodon.sh \
  --path /root/birdui-build/mastodon --variations

# 3. Build
docker build -t yttrx-mastodon-birdui:v4.6.3-tags-auth3 /root/birdui-build/mastodon

# 4. Point web + sidekiq at the new image and recreate them
sed -i 's/image: yttrx-mastodon-birdui:v4.6.3.*/image: yttrx-mastodon-birdui:v4.6.3-tags-auth3/' \
  ~mastodon/live/docker-compose.yml
cd ~mastodon/live && docker compose up -d web sidekiq

No database migration, no nginx stop, no version bump — just an image swap and a container recreate.

Testing #

Verified against the live instance immediately after each deploy, both the naive-scraper case (bare curl, Accept: */*) and a browser-realistic Accept header, plus every surface that must stay open:

The bare-curl row is the one that mattered: the first deploy passed the browser-header case and failed the bare-curl case silently (200 instead of 302), which is exactly the traffic pattern this patch exists to catch. Caught by testing both, not just one.

Confirming it in the logs #

tags-attack-watch (an internal ops script, deployed at /usr/local/bin/tags-attack-watch on mammut) now splits its status-code breakdown in two, since “200” means different things on each surface now:

== HTML page /tags/<name> status (302 = gated correctly, 200 = getting through) ==
     42 302
      3 403

== RSS + API status (200 expected here -- open by design) ==
     18 200

A 200 majority in the first block would mean the gate reverted or isn’t loading; a 200 majority in the second block is just the API and RSS working as intended. Run it with --watch for a live 10-second-refresh view during/after a deploy.

Side effects (and what stayed out of scope) #

• Anonymous hashtag browsing via the web page is gone — this also means the page drops out of search-engine indexing and link-preview unfurling for hashtag URLs, and removes a “browse before you sign up” path. Accepted trade-off given the abuse volume.
• Tag RSS and the tag timeline API remain fully open, on purpose — the log analysis showed they weren’t the abuse vector, and gating them would have broken the official app, third-party clients, and RSS readers for no anti-scraping benefit.
• Profile pages, profile RSS, the public timeline, and ActivityPub federation are all unaffected — none of them share code or settings with this controller.
• Setting.local_topic_feed_access/remote_topic_feed_access (Mastodon’s own admin-facing “topic feed access” setting) were touched in an earlier iteration and then reverted — they also gate the unrelated trending-links timeline, so this patch doesn’t use them at all.