commit 9cc01a2127afae7d1ef7ee7ae141341e3e61b566
parent f7a75b7f26583b457f2d518f662d5aec66daea5d
Author: Amit Dutta <mail@amit.is-a.dev>
Date: Sat, 10 Oct 2026 20:33:32 +0530
Merge pull request #41 from notamitgamer/link-prefetch
Make navigation fast: link prefetching, cached + gzipped pages, fewer storage calls
Diffstat:
15 files changed, 782 insertions(+), 42 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -10,7 +10,7 @@ permissions:
jobs:
checks:
- name: Lint and smoke test
+ name: Lint, tests and smoke test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
@@ -29,6 +29,18 @@ jobs:
- name: Byte-compile
run: python -m compileall -q app
+ - uses: actions/setup-node@v4
+ with:
+ node-version: "22"
+
+ - name: Prefetch script unit tests
+ run: node --test tests/*.test.js
+
+ - name: Server fast-path tests (HTML caching + gzip, skipped HEAD, single-flight listings)
+ env:
+ HF_BUCKET_ID: ci/placeholder
+ run: python tests/test_server_fastpaths.py
+
- name: Smoke test (app imports, static pages render)
env:
HF_BUCKET_ID: ci/placeholder
@@ -42,7 +54,7 @@ jobs:
m.repo_stats = no_stats
c = TestClient(m.app)
- for path in ["/new", "/static/manifest.json"]:
+ for path in ["/new", "/static/manifest.json", "/static/prefetch.js"]:
r = c.get(path)
assert r.status_code == 200, (path, r.status_code)
print(path, r.status_code)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -35,10 +35,13 @@ CI runs these on every pull request; please run them locally first:
pip install ruff
ruff check --select E9,F63,F7,F82 app # syntax errors and undefined names
python -m compileall -q app
+node --test tests/*.test.js # unit tests for the browser-side prefetcher (Node 20+)
+HF_BUCKET_ID=ci/x python tests/test_server_fastpaths.py # offline checks for the caching/gzip/listing fast paths
docker build -t cdnv2 . # the image must still build
```
-There is no full test suite yet. If you add logic that can be tested without network access (parsing,
+There is no full Python test suite yet; `tests/test_server_fastpaths.py` is a plain script (no pytest)
+that runs against a fake bucket. If you add logic that can be tested without network access (parsing,
validation, rate limiting), adding a test alongside it is very much appreciated.
## Pull requests
diff --git a/README.md b/README.md
@@ -23,6 +23,8 @@ Live instance: <https://cdn.amit.is-a.dev> · License: [MIT](LICENSE)
- **Abuse protection** — per-IP rate limits, video-upload blocking (by extension and by file contents),
and optional VPN/proxy blocking.
- **Stats** — `/stats` shows file counts and storage used.
+- **Fast navigation** — linked pages are prefetched in the background and cached/gzipped, so a click
+ is usually answered from the browser's own cache (details below).
- **PWA** — installable, with a service worker that deliberately caches nothing, so listings are
never stale.
@@ -129,11 +131,48 @@ Default limits per client IP: 200 MB/hour for file uploads, 50 MB/hour for URL u
per minute / 50 per hour for the shortener. GitHub sync has no rate limit but accepts at most a 200 MB
compressed archive. Video files are refused.
+## Fast navigation
+
+Moving around is meant to feel close to a static site. Three things work together:
+
+1. **Link prefetching.** `app/static/prefetch.js` (loaded from `<head>` on the public pages, not the
+ admin dashboard) fetches the pages linked from the current one as soon as the browser is idle.
+2. **Browser-cached, gzipped pages.** Rendered pages are identical for everyone, so they are sent with
+ `Cache-Control: private, max-age=30` and gzip (`app/fast_html.py`). A prefetched page therefore sits
+ in the browser's cache and the click is answered with no network at all; a page that wasn't
+ prefetched in time still arrives about 5x smaller (a typical folder listing is ~60 KB of HTML, ~12 KB
+ gzipped).
+3. **Fewer storage round trips.** Opening a folder used to cost a wasted `HEAD` request plus the
+ listing. When the parent folder's listing is cached (it is, because that's the page you clicked
+ from), the `HEAD` is skipped, and simultaneous requests for one folder share a single Hugging Face call.
+
+**Freshness trade-off:** because pages are cached for 30 s in the browser (on top of the server's 60 s
+listing cache), a file you just uploaded can take up to a minute or two to show in a folder you
+already had open. A hard refresh (Ctrl/Cmd+Shift+R) always fetches fresh.
+
+### What gets prefetched
+
+The prefetcher is deliberately conservative:
+
+- Same-origin links only, and never `/api/*`, `/admin`, `/mask/*`, `/pack/*`, `/static/*`, `/raw/*`,
+ downloads, `target`/`download` links, or anything inside `data-turbo="false"`.
+- At most **16 pages per page view**, **3 at a time**. Folder and file links on screen go first,
+ navbar and footer links last. Only while the tab is visible.
+- When you click, in-flight prefetches are cancelled, except the one for the page you're opening.
+- Disabled on Save-Data, 2g and `prefers-reduced-data`; reduced to 6 pages / 1 at a time on 3g.
+
+To turn it off for a page, add `<meta name="cdn-prefetch" content="off">`; to exclude a link or a
+whole section, add `data-no-prefetch`. To change the limits, edit `DEFAULTS` at the top of the file
+(keep `ttlMs` below the 30 s cache lifetime in `app/fast_html.py`). For debugging, run
+`__cdnPrefetch.stats()` in the browser console. Prefetches carry an `X-Purpose: prefetch` request
+header, and they don't show up in the Recent/Visited history, which is recorded by the page itself.
+
## Project layout
```
app/
main.py routes, uploads, zip packing, GitHub sync endpoint
+ fast_html.py browser-cache + gzip headers for rendered pages
storage.py Hugging Face bucket access, listing, caching, search
upload_guard.py upload checks: video blocking, VPN detection, client IP
admin.py /admin dashboard, bans, GitHub allowlist (encrypted state)
@@ -141,7 +180,8 @@ app/
shortener.py URL shortener
stats.py /stats
templates/ Jinja2 pages (index.html renders most page types)
- static/ icons, manifest, service worker
+ static/ icons, manifest, service worker, prefetch.js
+tests/ prefetcher unit tests (node) + offline server fast-path checks (python)
Dockerfile · render.yaml · start.sh
```
diff --git a/app/fast_html.py b/app/fast_html.py
@@ -0,0 +1,53 @@
+"""Make rendered HTML pages cheap to reuse.
+
+Every UI page is rendered from a template and is identical for everybody (it only carries global
+stats and URLs), so two things are safe and make navigation much faster:
+
+* ``Cache-Control: private, max-age=30``: the browser keeps the page for 30 seconds. The link
+ prefetcher (static/prefetch.js) fetches pages in the background, and a click on one of them is then
+ answered from the browser's own cache with no network at all. ``private`` keeps shared caches out of it.
+* gzip: the pages are 20-90 KB of HTML with inlined CSS/JS and compress about 5x, which matters most on
+ slow or high-latency connections.
+
+Only successful GET responses built through ``TemplateResponse`` are touched. File downloads, the API,
+redirects, errors and the admin dashboard never go through here.
+"""
+
+import gzip
+
+from fastapi.templating import Jinja2Templates
+from starlette.requests import Request
+
+HTML_MAX_AGE = 30 # seconds; keep in step with `ttlMs` in static/prefetch.js
+MIN_GZIP_BYTES = 1024
+GZIP_LEVEL = 5
+
+
+def optimise_html(request: Request | None, response):
+ """Add caching and compression headers to a rendered page (in place) and return it."""
+ if request is None or request.method != "GET" or response.status_code != 200:
+ return response
+
+ if "cache-control" not in response.headers:
+ response.headers["Cache-Control"] = f"private, max-age={HTML_MAX_AGE}"
+
+ # The body now depends on Accept-Encoding, so caches must key on it.
+ response.headers.append("Vary", "Accept-Encoding")
+
+ body = response.body
+ accepts_gzip = "gzip" in request.headers.get("accept-encoding", "").lower()
+ if accepts_gzip and "content-encoding" not in response.headers and len(body) >= MIN_GZIP_BYTES:
+ compressed = gzip.compress(body, GZIP_LEVEL)
+ response.body = compressed
+ response.headers["Content-Encoding"] = "gzip"
+ response.headers["Content-Length"] = str(len(compressed))
+ return response
+
+
+class FastTemplates(Jinja2Templates):
+ """Jinja2Templates whose responses are cached and compressed (see module docstring)."""
+
+ def TemplateResponse(self, *args, **kwargs): # noqa: N802 (matches the parent's name)
+ response = super().TemplateResponse(*args, **kwargs)
+ request = args[0] if args and isinstance(args[0], Request) else kwargs.get("request")
+ return optimise_html(request, response)
diff --git a/app/main.py b/app/main.py
@@ -17,11 +17,11 @@ from pathlib import Path
from pydantic import BaseModel
from fastapi import FastAPI, Request, File, UploadFile, HTTPException, Form
from fastapi.responses import StreamingResponse, HTMLResponse, PlainTextResponse, RedirectResponse, FileResponse
-from fastapi.templating import Jinja2Templates
+from .fast_html import FastTemplates
from .storage import (
is_file,
- get_file_info,
+ get_path_info,
list_directory,
list_files_recursive,
upload_temp_file,
@@ -44,23 +44,26 @@ from .admin import router as admin_router, record_upload, is_banned, gh_check, g
app = FastAPI()
app.include_router(admin_router)
app.include_router(stats_router)
-templates = Jinja2Templates(directory="app/templates")
+templates = FastTemplates(directory="app/templates")
STATIC_DIR = Path(__file__).parent / "static"
_NO_STORE_FILES = {"manifest.json", "sw.js"}
+_REVALIDATE_FILES = {"prefetch.js"}
@app.get("/static/{filename}")
async def static_no_cache_root(filename: str):
- if filename not in _NO_STORE_FILES:
+ if filename in _NO_STORE_FILES:
+ cache_control = "no-store, no-cache, must-revalidate, max-age=0"
+ elif filename in _REVALIDATE_FILES:
+ # Short cache: full page loads don't re-download it, and a deploy still shows up within minutes.
+ cache_control = "public, max-age=300"
+ else:
raise HTTPException(status_code=404)
file_path = STATIC_DIR / filename
if not file_path.is_file():
raise HTTPException(status_code=404)
- return FileResponse(
- file_path,
- headers={"Cache-Control": "no-store, no-cache, must-revalidate, max-age=0"},
- )
+ return FileResponse(file_path, headers={"Cache-Control": cache_control})
@app.get("/static/icons/{filename}")
@@ -1008,7 +1011,7 @@ async def serve(request: Request, path: str):
)
if clean_path:
- info = await get_file_info(clean_path)
+ info = await get_path_info(clean_path)
if info["exists"]:
filename = clean_path.split("/")[-1]
diff --git a/app/static/prefetch.js b/app/static/prefetch.js
@@ -0,0 +1,255 @@
+/*!
+ * cdnV2 link prefetcher.
+ *
+ * Once a page has settled, quietly fetches the pages it links to (folders, file pages, the navbar
+ * destinations) so the server's listing cache is already warm when you click. The expensive part of a
+ * click is the cold Hugging Face lookup behind a folder, and that cache only lives for 60 seconds, so
+ * this re-warms it just ahead of use.
+ *
+ * Deliberately conservative:
+ * - same-origin links only; never /api/*, /admin, /mask/*, /pack/*, /static/*, /raw/*, downloads,
+ * links with target/download, or anything inside data-turbo="false" / data-no-prefetch
+ * - at most 16 pages per page view, 3 at a time, page content before navbar links, visible links
+ * first, and only while the tab is visible
+ * - idle time only; cancelled the moment you navigate so it never competes with a real click
+ * - skipped entirely on Save-Data, 2g connections and prefers-reduced-data
+ *
+ * Opt out: <meta name="cdn-prefetch" content="off"> on a page, or data-no-prefetch on a link/container.
+ * Works with Turbo (re-runs on turbo:load) and without it. Loaded once from <head>; safe to include twice.
+ */
+(function (global) {
+ 'use strict';
+
+ // ttlMs stays below the pages' Cache-Control max-age (app/fast_html.py, 30 s): a page this old is
+ // re-fetched, one younger is still in the browser's cache and needs no work.
+ const DEFAULTS = { maxLinks: 16, concurrency: 3, ttlMs: 25000 };
+
+ // First path segments that belong to the app itself, not to the CDN's content. Anything under them is
+ // an API, a redirect, an admin page or a download, so it must never be fetched speculatively.
+ const RESERVED_SEGMENTS = new Set(['api', 'admin', 'mask', 'pack', 'static', 'raw']);
+
+ function pathAllowed(pathname) {
+ if (pathname === '/favicon.ico') return false;
+ const first = pathname.split('/')[1] || '';
+ return !RESERVED_SEGMENTS.has(first);
+ }
+
+ // Returns the normalised URL to prefetch for a link, or null if it should be left alone.
+ // `link` is plain data ({href, target, download, noPrefetch, turboOff}) and `base` a URL for the
+ // current page, so this stays testable without a DOM.
+ function candidateUrl(link, base) {
+ if (link.download || link.noPrefetch || link.turboOff) return null;
+ if (link.target && link.target !== '_self') return null;
+ let url;
+ try {
+ url = new URL(link.href, base);
+ } catch (e) {
+ return null;
+ }
+ if (url.protocol !== 'http:' && url.protocol !== 'https:') return null;
+ if (url.origin !== base.origin) return null;
+ if (!pathAllowed(url.pathname)) return null;
+ if (url.pathname === base.pathname && url.search === base.search) return null; // this page
+ url.hash = '';
+ return url.href;
+ }
+
+ // A small bounded work queue. `fetchFn(url, signal)` must return a promise that settles when the
+ // response has been fully read (the server only finishes warming its cache once it has answered).
+ function createPrefetcher(opts) {
+ const cfg = Object.assign({}, DEFAULTS, opts.config);
+ const now = opts.now || Date.now;
+ const seen = new Map(); // url -> when it last finished; stops repeat fetches inside the cache TTL
+ const inflight = new Map(); // url -> AbortController, so one request can be cancelled on its own
+ let queue = [];
+ let queued = new Set();
+ let active = 0;
+ let used = 0; // pages accepted for the current page view (the budget)
+ let paused = false;
+
+ function fresh(url) {
+ const t = seen.get(url);
+ return t !== undefined && now() - t < cfg.ttlMs;
+ }
+
+ function run(url) {
+ active++;
+ const ctl = new AbortController();
+ inflight.set(url, ctl);
+ Promise.resolve()
+ .then(() => opts.fetchFn(url, ctl.signal))
+ .then(
+ () => seen.set(url, now()),
+ // A failed fetch is remembered too, so a flaky or 404 link is not retried in a loop.
+ () => { if (!ctl.signal.aborted) seen.set(url, now()); }
+ )
+ .then(() => {
+ active--;
+ inflight.delete(url);
+ pump();
+ });
+ }
+
+ function pump() {
+ while (!paused && active < cfg.concurrency && queue.length) run(queue.shift());
+ }
+
+ return {
+ // Start a new page view: fresh budget, drop anything still waiting. Finished work is remembered.
+ newPage() {
+ queue = [];
+ queued = new Set();
+ used = 0;
+ },
+ remaining() {
+ return cfg.maxLinks - used;
+ },
+ enqueue(urls) {
+ let added = 0;
+ for (const url of urls) {
+ if (used >= cfg.maxLinks) break;
+ if (queued.has(url) || inflight.has(url) || fresh(url)) continue;
+ queue.push(url);
+ queued.add(url);
+ used++;
+ added++;
+ }
+ pump();
+ return added;
+ },
+ // Cancel in-flight requests and forget the queue (called when navigating away). A request for
+ // `keepUrl` (the page being opened) is left running: it is exactly what the click needs, and the
+ // browser can hand its response to the navigation instead of starting over.
+ abort(keepUrl) {
+ for (const [url, ctl] of inflight) if (url !== keepUrl) ctl.abort();
+ queue = [];
+ queued = new Set();
+ },
+ setPaused(value) {
+ paused = !!value;
+ if (!paused) pump();
+ },
+ stats() {
+ return { queued: queue.length, active, remembered: seen.size, used };
+ },
+ };
+ }
+
+ function shouldSkipDevice(win) {
+ const conn = win.navigator && (win.navigator.connection || win.navigator.mozConnection);
+ if (conn && (conn.saveData || /2g$/.test(conn.effectiveType || ''))) return true;
+ if (win.matchMedia && win.matchMedia('(prefers-reduced-data: reduce)').matches) return true;
+ return false;
+ }
+
+ function boot(win) {
+ const doc = win.document;
+ if (!doc || win.__cdnPrefetch || typeof win.fetch !== 'function') return;
+ if (shouldSkipDevice(win)) return;
+
+ const conn = win.navigator && win.navigator.connection;
+ const config = conn && conn.effectiveType === '3g' ? { maxLinks: 6, concurrency: 1 } : {};
+
+ const pf = createPrefetcher({
+ config,
+ fetchFn: (url, signal) =>
+ win
+ .fetch(url, {
+ credentials: 'same-origin',
+ signal,
+ priority: 'low', // ignored by browsers that don't support it
+ headers: { Accept: 'text/html', 'X-Purpose': 'prefetch' },
+ })
+ .then((res) => res.arrayBuffer()),
+ });
+ win.__cdnPrefetch = pf; // handy for debugging: __cdnPrefetch.stats()
+
+ function scan(freshPage) {
+ if (doc.querySelector('meta[name="cdn-prefetch"][content="off"]')) return;
+ if (freshPage) pf.newPage();
+ if (pf.remaining() <= 0) return;
+
+ const base = new URL(win.location.href);
+ const viewport = win.innerHeight || 800;
+ const found = [];
+ const dedupe = new Set();
+
+ doc.querySelectorAll('a[href]').forEach((a) => {
+ if (a.getClientRects().length === 0) return; // hidden: filtered-out rows, collapsed menus
+ const url = candidateUrl(
+ {
+ href: a.href,
+ target: a.getAttribute('target'),
+ download: a.hasAttribute('download'),
+ noPrefetch: !!a.closest('[data-no-prefetch]'),
+ turboOff: !!a.closest('[data-turbo="false"]'),
+ },
+ base
+ );
+ if (!url || dedupe.has(url)) return;
+ dedupe.add(url);
+ const r = a.getBoundingClientRect();
+ // Page content first (on-screen links, plus a screenful below, ahead of the rest), then site
+ // chrome (navbar, footer). Those pages are cheap for the server, so they only get what's left.
+ const chrome = !!a.closest('header, nav, footer');
+ found.push({ url, rank: chrome ? 2 : r.bottom > 0 && r.top < viewport * 1.5 ? 0 : 1 });
+ });
+
+ found.sort((x, y) => x.rank - y.rank);
+ pf.enqueue(found.map((f) => f.url));
+ }
+
+ // Wait for the document to be parsed and the browser to go idle before touching the network.
+ let pending = false;
+ let pendingFresh = false;
+ function request(freshPage) {
+ pendingFresh = pendingFresh || freshPage;
+ if (pending) return;
+ pending = true;
+ const go = () => {
+ const run = () => {
+ pending = false;
+ const f = pendingFresh;
+ pendingFresh = false;
+ scan(f);
+ };
+ if (win.requestIdleCallback) win.requestIdleCallback(run, { timeout: 500 });
+ else win.setTimeout(run, 100);
+ };
+ // The HTML is enough to know the links; don't wait for every image and script to finish.
+ if (doc.readyState !== 'loading') go();
+ else doc.addEventListener('DOMContentLoaded', go, { once: true });
+ }
+
+ request(true);
+ doc.addEventListener('turbo:load', () => request(true));
+
+ // Pages that build their list with JS (search results, history) add links after load.
+ let timer = 0;
+ new win.MutationObserver(() => {
+ win.clearTimeout(timer);
+ timer = win.setTimeout(() => request(false), 800);
+ }).observe(doc.documentElement, { childList: true, subtree: true });
+
+ pf.setPaused(doc.hidden);
+ doc.addEventListener('visibilitychange', () => pf.setPaused(doc.hidden));
+
+ // Navigating: stop prefetching so the real request gets the connection to itself.
+ doc.addEventListener('turbo:before-visit', (event) => {
+ let keep = null;
+ try {
+ const u = new URL(event.detail.url);
+ u.hash = '';
+ keep = u.href;
+ } catch (e) { /* no usable URL: cancel everything */ }
+ pf.abort(keep);
+ });
+ win.addEventListener('pagehide', () => pf.abort());
+ }
+
+ if (typeof module === 'object' && module.exports) {
+ module.exports = { DEFAULTS, pathAllowed, candidateUrl, createPrefetcher, boot };
+ } else {
+ boot(global);
+ }
+})(typeof window !== 'undefined' ? window : globalThis);
diff --git a/app/stats.py b/app/stats.py
@@ -14,12 +14,12 @@ from datetime import datetime, timezone
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
-from fastapi.templating import Jinja2Templates
+from .fast_html import FastTemplates
from . import storage
router = APIRouter()
-templates = Jinja2Templates(directory="app/templates")
+templates = FastTemplates(directory="app/templates")
STATS_TTL = 600
_cache: dict = {"at": 0.0, "data": None}
diff --git a/app/storage.py b/app/storage.py
@@ -116,6 +116,22 @@ async def get_file_info(path: str) -> dict:
_set_cache(cache_key, result)
return result
+async def get_path_info(path: str) -> dict:
+ """Like get_file_info(), for deciding whether a URL is a file page or a folder listing.
+
+ Opening a folder used to cost a wasted HEAD request (a folder isn't a file, so it 404s) before the
+ listing was fetched. When the parent folder's listing is already cached, which it normally is
+ because that is the page you clicked from, it already says whether `path` is a file or a folder
+ (and a file's size), so the HEAD is skipped. Anything not found there falls back to get_file_info().
+ """
+ parent = path.rpartition("/")[0]
+ for entry in _get_cache(f"list_dir_{parent}") or ():
+ if entry["path"] == path:
+ if entry["is_dir"]:
+ return {"exists": False, "size": None, "content_type": None}
+ return {"exists": True, "size": entry.get("size"), "content_type": None}
+ return await get_file_info(path)
+
def _under(path: str, items: list) -> list:
"""Keep only entries inside `path` (defensive: guards against prefix over-match
such as 'docs' also matching 'docs-old/...')."""
@@ -149,34 +165,42 @@ async def list_directory(path: str):
if cached is not None:
return cached
- try:
- items = await asyncio.to_thread(_fetch_tree, path)
- except Exception as e:
- print(f"[storage] list_repo_tree failed for {path!r}: {e}")
- return []
-
- files_and_folders = []
- for item in items:
- if _is_hidden_path(item.path):
- continue
+ # One fetch per folder at a time: when a prefetch and a real click ask for the same folder
+ # together, the second waits for the first instead of repeating the Hugging Face call.
+ async with _get_key_lock(cache_key):
+ cached = _get_cache(cache_key)
+ if cached is not None:
+ return cached
- name = item.path.split("/")[-1]
- is_dir = not hasattr(item, "size")
- size = getattr(item, "size", 0)
-
- size_str = format_size(size) if not is_dir else "-"
-
- files_and_folders.append({
- "name": name,
- "path": item.path,
- "is_dir": is_dir,
- "size_str": size_str
- })
-
- files_and_folders.sort(key=lambda x: (not x["is_dir"], x["name"].lower()))
-
- _set_cache(cache_key, files_and_folders)
- return files_and_folders
+ try:
+ items = await asyncio.to_thread(_fetch_tree, path)
+ except Exception as e:
+ print(f"[storage] list_repo_tree failed for {path!r}: {e}")
+ return []
+
+ files_and_folders = []
+ for item in items:
+ if _is_hidden_path(item.path):
+ continue
+
+ name = item.path.split("/")[-1]
+ is_dir = not hasattr(item, "size")
+ size = getattr(item, "size", 0)
+
+ size_str = format_size(size) if not is_dir else "-"
+
+ files_and_folders.append({
+ "name": name,
+ "path": item.path,
+ "is_dir": is_dir,
+ "size": size,
+ "size_str": size_str
+ })
+
+ files_and_folders.sort(key=lambda x: (not x["is_dir"], x["name"].lower()))
+
+ _set_cache(cache_key, files_and_folders)
+ return files_and_folders
async def list_files_recursive(path: str):
try:
diff --git a/app/templates/documentation.html b/app/templates/documentation.html
@@ -52,6 +52,7 @@
.footnote{font-size:.82rem;color:var(--muted);margin-top:3rem;border-top:1px solid var(--border);padding-top:1.2rem;}
footer{margin-top:4rem;padding-top:1.5rem;border-top:1px solid var(--border);font-size:.85rem;color:var(--muted);}
</style>
+ <script src="/static/prefetch.js" defer></script>
</head>
<body>
{% include "_navbar.html" %}
diff --git a/app/templates/gh_sync_docs.html b/app/templates/gh_sync_docs.html
@@ -120,6 +120,7 @@
color: var(--muted);
}
</style>
+ <script src="/static/prefetch.js" defer></script>
</head>
<body>
{% include "_navbar.html" %}
diff --git a/app/templates/index.html b/app/templates/index.html
@@ -27,6 +27,7 @@
<meta name="twitter:card" content="summary">
<script type="module" src="https://cdn.jsdelivr.net/npm/@hotwired/turbo@8.0.4/dist/turbo.es2017-esm.js"></script>
+ <script src="/static/prefetch.js" defer></script>
<style>
:root {
diff --git a/app/templates/stats.html b/app/templates/stats.html
@@ -73,6 +73,7 @@
}
:root { --c0: #4c8dff; --c1: #3fb950; --c2: #d29922; --c3: #f85149; --c4: #a371f7; --c5: #39c5cf; --c6: #db61a2; --c7: #8b949e; }
</style>
+ <script src="/static/prefetch.js" defer></script>
</head>
<body>
{% include "_navbar.html" %}
diff --git a/app/templates/url_shortener.html b/app/templates/url_shortener.html
@@ -15,6 +15,7 @@
<meta name="theme-color" content="#0d1117" media="(prefers-color-scheme: dark)">
<meta name="color-scheme" content="light dark">
<script type="module" src="https://cdn.jsdelivr.net/npm/@hotwired/turbo@8.0.4/dist/turbo.es2017-esm.js"></script>
+ <script src="/static/prefetch.js" defer></script>
<style>
:root {
/* Legacy names mapped onto the shared --ui- tokens (_styles.html) */
diff --git a/tests/prefetch.test.js b/tests/prefetch.test.js
@@ -0,0 +1,175 @@
+// Run with: node --test tests/*.test.js
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const { DEFAULTS, pathAllowed, candidateUrl, createPrefetcher } = require('../app/static/prefetch.js');
+
+const BASE = new URL('https://cdn.example.com/photos/');
+const link = (href, extra = {}) => ({ href, ...extra });
+const pick = (href, extra) => candidateUrl(link(href, extra), BASE);
+
+test('allows folders, files, and top-level pages', () => {
+ assert.equal(pick('/uploads/'), 'https://cdn.example.com/uploads/');
+ assert.equal(pick('/uploads/report.pdf'), 'https://cdn.example.com/uploads/report.pdf');
+ assert.equal(pick('/find'), 'https://cdn.example.com/find');
+ assert.equal(pick('/documentation'), 'https://cdn.example.com/documentation');
+ assert.equal(pick('sub/dir'), 'https://cdn.example.com/photos/sub/dir'); // relative
+});
+
+test('never prefetches the app’s own endpoints, downloads or redirects', () => {
+ for (const p of ['/api/get/uploads/big.zip', '/api/pack/uploads', '/admin', '/admin/x',
+ '/mask/abc123', '/pack/batch1', '/static/sw.js', '/raw/uploads/a.png',
+ '/favicon.ico']) {
+ assert.equal(pick(p), null, p);
+ }
+});
+
+test('reserved names only match whole path segments', () => {
+ assert.equal(pathAllowed('/administrator/'), true);
+ assert.equal(pathAllowed('/apikeys/'), true);
+ assert.equal(pathAllowed('/api'), false);
+ assert.equal(pathAllowed('/'), true);
+});
+
+test('skips other origins and non-http schemes', () => {
+ assert.equal(pick('https://raw.cdn.example.com/uploads/a.png'), null);
+ assert.equal(pick('https://example.org/'), null);
+ assert.equal(pick('mailto:a@b.c'), null);
+ assert.equal(pick('javascript:void(0)'), null);
+});
+
+test('respects opt-outs on the link', () => {
+ assert.equal(pick('/uploads/', { download: true }), null);
+ assert.equal(pick('/uploads/', { target: '_blank' }), null);
+ assert.equal(pick('/uploads/', { target: '_self' }), 'https://cdn.example.com/uploads/');
+ assert.equal(pick('/uploads/', { turboOff: true }), null);
+ assert.equal(pick('/uploads/', { noPrefetch: true }), null);
+});
+
+test('skips the current page, strips hashes', () => {
+ assert.equal(pick('/photos/'), null);
+ assert.equal(pick('/photos/#top'), null);
+ assert.equal(pick('/uploads/#x'), 'https://cdn.example.com/uploads/');
+ assert.equal(pick('/photos/?q=1'), 'https://cdn.example.com/photos/?q=1');
+});
+
+// --- queue ---------------------------------------------------------------------------------------
+
+// A fetch whose responses the test finishes by hand, so we can observe concurrency.
+function manualFetch() {
+ const calls = [];
+ const fn = (url, signal) =>
+ new Promise((resolve, reject) => {
+ const call = { url, signal, resolve, reject, aborted: false };
+ signal.addEventListener('abort', () => { call.aborted = true; reject(new Error('aborted')); });
+ calls.push(call);
+ });
+ return { fn, calls };
+}
+const tick = () => new Promise((r) => setImmediate(r));
+const urls = (n) => Array.from({ length: n }, (_, i) => `https://cdn.example.com/f${i}/`);
+
+test('runs at most `concurrency` fetches at once and drains the queue', async () => {
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 2, maxLinks: 10 } });
+ pf.enqueue(urls(5));
+ await tick();
+ assert.equal(calls.length, 2);
+ calls[0].resolve();
+ await tick();
+ assert.equal(calls.length, 3);
+ calls.forEach((c) => c.resolve());
+ await tick(); await tick();
+ assert.equal(calls.length, 5);
+});
+
+test('stops accepting links after maxLinks per page view, and newPage resets the budget', async () => {
+ const { fn } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { maxLinks: 3 } });
+ assert.equal(pf.enqueue(urls(10)), 3);
+ assert.equal(pf.remaining(), 0);
+ assert.equal(pf.enqueue(urls(10)), 0);
+ pf.newPage();
+ assert.equal(pf.remaining(), 3);
+});
+
+test('does not refetch inside the TTL, does after it', async () => {
+ let t = 1000;
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, now: () => t, config: { ttlMs: 50000, maxLinks: 10 } });
+ pf.enqueue(urls(1));
+ await tick();
+ calls[0].resolve();
+ await tick();
+ pf.newPage();
+ assert.equal(pf.enqueue(urls(1)), 0); // still fresh
+ t += 60000;
+ assert.equal(pf.enqueue(urls(1)), 1); // expired
+});
+
+test('does not queue a URL that is already in flight', async () => {
+ const { fn } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { maxLinks: 10 } });
+ pf.enqueue(urls(1));
+ await tick();
+ pf.newPage();
+ assert.equal(pf.enqueue(urls(1)), 0);
+});
+
+test('abort cancels in-flight requests and clears the queue', async () => {
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 1, maxLinks: 10 } });
+ pf.enqueue(urls(4));
+ await tick();
+ pf.abort();
+ await tick();
+ assert.equal(calls[0].aborted, true);
+ assert.equal(calls.length, 1); // nothing else started
+ assert.equal(pf.stats().queued, 0);
+ // an aborted fetch is not remembered as done, so it can be tried again later
+ pf.newPage();
+ assert.equal(pf.enqueue(urls(1)), 1);
+});
+
+test('a failing fetch does not stall the queue and is not retried in a loop', async () => {
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 1, maxLinks: 10 } });
+ pf.enqueue(urls(2));
+ await tick();
+ calls[0].reject(new Error('network'));
+ await tick(); await tick();
+ assert.equal(calls.length, 2); // moved on to the next one
+ pf.newPage();
+ assert.equal(pf.enqueue(urls(1)), 0); // failed URL remembered for the TTL
+});
+
+test('paused queue holds work until resumed', async () => {
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 2, maxLinks: 10 } });
+ pf.setPaused(true);
+ pf.enqueue(urls(3));
+ await tick();
+ assert.equal(calls.length, 0);
+ pf.setPaused(false);
+ await tick();
+ assert.equal(calls.length, 2);
+});
+
+test('abort(keepUrl) leaves the page being opened running and cancels the rest', async () => {
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 3, maxLinks: 10 } });
+ const [a, b, c] = urls(3);
+ pf.enqueue([a, b, c, ...urls(6).slice(3)]);
+ await tick();
+ pf.abort(b);
+ await tick();
+ assert.deepEqual(calls.map((x) => x.aborted), [true, false, true]);
+ assert.equal(pf.stats().queued, 0);
+ calls[1].resolve(); // the kept request finishes normally and is remembered
+ await tick(); await tick();
+ pf.newPage();
+ assert.equal(pf.enqueue([b]), 0);
+});
+
+test('prefetch TTL is shorter than the pages\' browser-cache lifetime (30 s)', () => {
+ assert.ok(DEFAULTS.ttlMs < 30000);
+});
diff --git a/tests/test_server_fastpaths.py b/tests/test_server_fastpaths.py
@@ -0,0 +1,170 @@
+"""Offline checks for the navigation fast paths (no network, no Hugging Face).
+
+Run from the repo root: HF_BUCKET_ID=ci/placeholder python tests/test_server_fastpaths.py
+
+Covers: HTML caching + gzip (app/fast_html.py), skipping the wasted HEAD request when the parent
+folder's listing is cached (storage.get_path_info), and one Hugging Face call per folder even when
+several requests ask at once (storage.list_directory).
+"""
+
+import asyncio
+import gzip
+import os
+import sys
+
+os.environ.setdefault("HF_BUCKET_ID", "ci/placeholder")
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from fastapi.testclient import TestClient # noqa: E402
+from starlette.requests import Request # noqa: E402
+from starlette.responses import HTMLResponse # noqa: E402
+
+import app.main as m # noqa: E402
+from app import storage # noqa: E402
+from app.fast_html import HTML_MAX_AGE, optimise_html # noqa: E402
+
+# A tiny fake bucket. Directories have no `size` attribute, like the real tree items.
+FILES = {
+ "top.txt": 7,
+ "bsc/README.md": 3000,
+ "bsc/semester_1/notes.txt": 12,
+ "bsc/semester_1/deep/a.bin": 5,
+ "_logs/hidden.log": 1,
+}
+calls = {"tree": 0, "head": 0}
+
+
+class Dir:
+ def __init__(self, path):
+ self.path = path
+
+
+class File:
+ def __init__(self, path, size):
+ self.path, self.size = path, size
+
+
+def fake_fetch_tree(path):
+ calls["tree"] += 1
+ path = path.strip("/")
+ prefix = path + "/" if path else ""
+ out = {}
+ for fp, size in FILES.items():
+ if not fp.startswith(prefix):
+ continue
+ rest = fp[len(prefix):]
+ if "/" in rest:
+ d = prefix + rest.split("/")[0]
+ out[d] = Dir(d)
+ else:
+ out[fp] = File(fp, size)
+ return list(out.values())
+
+
+async def fake_get_file_info(path):
+ calls["head"] += 1
+ return {"exists": path in FILES, "size": FILES.get(path), "content_type": None}
+
+
+async def no_stats():
+ return None
+
+
+storage._fetch_tree = fake_fetch_tree
+storage.get_file_info = fake_get_file_info
+m.repo_stats = no_stats
+
+
+def reset():
+ storage._cache.clear()
+ calls["tree"] = calls["head"] = 0
+
+
+def check(cond, msg):
+ if not cond:
+ raise AssertionError(msg)
+ print("ok -", msg)
+
+
+# --- storage ----------------------------------------------------------------------------------------
+
+reset()
+loop = asyncio.new_event_loop()
+
+async def burst():
+ return await asyncio.gather(*[storage.list_directory("bsc") for _ in range(6)])
+
+results = loop.run_until_complete(burst())
+check(calls["tree"] == 1, "six simultaneous listings of one folder make a single Hugging Face call")
+check(all(r == results[0] for r in results), "...and all six get the same listing")
+names = [(e["name"], e["is_dir"]) for e in results[0]]
+check(names == [("semester_1", True), ("README.md", False)], "listing is folders-first, hidden paths excluded")
+check(results[0][1]["size"] == 3000 and results[0][1]["size_str"] == "2.93 KB", "entries keep the raw size too")
+
+reset()
+loop.run_until_complete(storage.list_directory("bsc"))
+calls["head"] = 0
+info = loop.run_until_complete(storage.get_path_info("bsc/semester_1"))
+check(info["exists"] is False and calls["head"] == 0, "folder is recognised from the parent listing, no HEAD")
+info = loop.run_until_complete(storage.get_path_info("bsc/README.md"))
+check(info["exists"] is True and info["size"] == 3000 and calls["head"] == 0, "file + size come from the parent listing, no HEAD")
+info = loop.run_until_complete(storage.get_path_info("bsc/missing"))
+check(info["exists"] is False and calls["head"] == 1, "a path the parent doesn't list falls back to the HEAD check")
+reset()
+info = loop.run_until_complete(storage.get_path_info("bsc/README.md"))
+check(info["exists"] is True and calls["head"] == 1, "with no cached parent listing it falls back to the HEAD check")
+loop.close()
+
+# --- HTML optimisation -------------------------------------------------------------------------------
+
+def req(method="GET", accept="gzip"):
+ return Request({"type": "http", "method": method, "headers": [(b"accept-encoding", accept.encode())]})
+
+big = "<html>" + "x" * 5000 + "</html>"
+r = optimise_html(req(), HTMLResponse(big))
+check(r.headers["content-encoding"] == "gzip" and gzip.decompress(r.body).decode() == big, "gzip round-trips")
+check(int(r.headers["content-length"]) == len(r.body) < len(big), "content-length matches the compressed body")
+check(r.headers["cache-control"] == f"private, max-age={HTML_MAX_AGE}", "pages are cached privately")
+check("accept-encoding" in r.headers["vary"].lower(), "Vary: Accept-Encoding is set")
+r = optimise_html(req(accept="identity"), HTMLResponse(big))
+check("content-encoding" not in r.headers and r.body.decode() == big, "no gzip unless the client accepts it")
+r = optimise_html(req(), HTMLResponse("tiny"))
+check("content-encoding" not in r.headers, "tiny bodies are left uncompressed")
+r = optimise_html(req(), HTMLResponse(big, status_code=404))
+check("cache-control" not in r.headers and "content-encoding" not in r.headers, "non-200 responses are untouched")
+r = optimise_html(req(method="POST"), HTMLResponse(big))
+check("cache-control" not in r.headers, "non-GET requests are untouched")
+r = HTMLResponse(big, headers={"Cache-Control": "no-store"})
+check(optimise_html(req(), r).headers["cache-control"] == "no-store", "an explicit Cache-Control is respected")
+
+# --- end to end through the real app -----------------------------------------------------------------
+
+reset()
+c = TestClient(m.app)
+gz = {"Accept-Encoding": "gzip"}
+
+r = c.get("/bsc", headers=gz)
+check(r.status_code == 200 and r.headers.get("content-encoding") == "gzip", "/bsc listing is served gzipped")
+check("semester_1" in r.text and "README.md" in r.text, "...and decodes to the real page")
+check(r.headers["cache-control"] == f"private, max-age={HTML_MAX_AGE}", "...with the browser-cache header")
+
+calls["head"] = 0
+r = c.get("/bsc/semester_1", headers=gz)
+check(r.status_code == 200 and "notes.txt" in r.text and calls["head"] == 0, "opening a folder from its parent makes no HEAD request")
+r = c.get("/bsc/README.md", headers=gz)
+check(r.status_code == 200 and "2.93 KB" in r.text and calls["head"] == 0, "opening a file from its parent makes no HEAD request")
+
+r = c.get("/bsc", headers={"Accept-Encoding": "identity"})
+check(r.status_code == 200 and "content-encoding" not in r.headers, "identity clients get plain HTML")
+
+for path in ["/new", "/find", "/history", "/shorten", "/documentation", "/gh-sync"]:
+ r = c.get(path, headers=gz)
+ check(r.status_code == 200 and r.headers.get("content-encoding") == "gzip"
+ and "max-age" in r.headers["cache-control"], f"{path} is cached and gzipped")
+
+r = c.get("/no/such/folder", headers=gz)
+check(r.status_code == 404 and "max-age" not in r.headers.get("cache-control", ""), "404 pages are not cached")
+r = c.get("/static/sw.js", headers=gz)
+check(r.headers["cache-control"].startswith("no-store") and "content-encoding" not in r.headers, "service worker is still no-store and untouched")
+
+print("\nall server fast-path checks passed")
