commit 18e6f713c28a3ea5b602ea75f82e12d320076d8a
parent 0a8d2e1dfebbf7172ff0300fa153bc742a3279b1
Author: bot-for-notamitgamer[bot] <bot-for-notamitgamer[bot]@users.noreply.github.com>
Date: Sat, 10 Oct 2026 14:02:45 +0000
Make navigation feel instant: cached + gzipped pages, fewer storage calls
Prefetching only warmed the server, so every click still paid a full
round trip plus an uncompressed 60-90 KB page. Now a prefetched page is
actually reusable by the browser.
- fast_html.py: rendered pages (identical for everyone) get
Cache-Control: private, max-age=30 and gzip (~5x smaller). Only 200 GET
TemplateResponses are touched; downloads, API, redirects, errors and the
admin dashboard are not.
- storage: opening a folder no longer makes a wasted HEAD request when the
parent folder's listing is cached (get_path_info), and concurrent
requests for one folder share a single Hugging Face call.
- prefetch.js: start once the document is parsed instead of waiting for
'load'; 3 at a time; TTL below the page cache lifetime; a click no longer
cancels the in-flight request for the page being opened.
- tests: offline server fast-path checks (fake bucket) run in CI; extra
prefetcher tests.
- docs: README "Fast navigation" section, including the freshness trade-off.
Diffstat:
10 files changed, 371 insertions(+), 68 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -36,6 +36,11 @@ jobs:
- name: Prefetch script unit tests
run: node --test tests/*.test.js
+ - name: Server fast-path tests (HTML caching + gzip, skipped HEAD, single-flight listings)
+ env:
+ HF_BUCKET_ID: ci/placeholder
+ run: python tests/test_server_fastpaths.py
+
- name: Smoke test (app imports, static pages render)
env:
HF_BUCKET_ID: ci/placeholder
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -36,11 +36,13 @@ pip install ruff
ruff check --select E9,F63,F7,F82 app # syntax errors and undefined names
python -m compileall -q app
node --test tests/*.test.js # unit tests for the browser-side prefetcher (Node 20+)
+HF_BUCKET_ID=ci/x python tests/test_server_fastpaths.py # offline checks for the caching/gzip/listing fast paths
docker build -t cdnv2 . # the image must still build
```
-The Python side has no test suite yet. If you add logic that can be tested without network access
-(parsing, validation, rate limiting), adding a test alongside it is very much appreciated.
+There is no full Python test suite yet; `tests/test_server_fastpaths.py` is a plain script (no pytest)
+that runs against a fake bucket. If you add logic that can be tested without network access (parsing,
+validation, rate limiting), adding a test alongside it is very much appreciated.
## Pull requests
diff --git a/README.md b/README.md
@@ -23,8 +23,8 @@ Live instance: <https://cdn.amit.is-a.dev> · License: [MIT](LICENSE)
- **Abuse protection** — per-IP rate limits, video-upload blocking (by extension and by file contents),
and optional VPN/proxy blocking.
- **Stats** — `/stats` shows file counts and storage used.
-- **Link prefetching** — pages you can navigate to from the current one are fetched quietly in the
- background, so the server's listing cache is warm by the time you click (details below).
+- **Fast navigation** — linked pages are prefetched in the background and cached/gzipped, so a click
+ is usually answered from the browser's own cache (details below).
- **PWA** — installable, with a service worker that deliberately caches nothing, so listings are
never stale.
@@ -131,30 +131,48 @@ Default limits per client IP: 200 MB/hour for file uploads, 50 MB/hour for URL u
per minute / 50 per hour for the shortener. GitHub sync has no rate limit but accepts at most a 200 MB
compressed archive. Video files are refused.
-## Link prefetching
+## Fast navigation
-`app/static/prefetch.js` (loaded from `<head>` on the public pages, not the admin dashboard) fetches the pages linked from the
-current one once the browser is idle. The slow part of opening a folder is the first Hugging Face
-lookup behind it, which the server then caches for 60 seconds; prefetching pays that cost ahead of the
-click. It is deliberately conservative:
+Moving around is meant to feel close to a static site. Three things work together:
+
+1. **Link prefetching.** `app/static/prefetch.js` (loaded from `<head>` on the public pages, not the
+ admin dashboard) fetches the pages linked from the current one as soon as the browser is idle.
+2. **Browser-cached, gzipped pages.** Rendered pages are identical for everyone, so they are sent with
+ `Cache-Control: private, max-age=30` and gzip (`app/fast_html.py`). A prefetched page therefore sits
+ in the browser's cache and the click is answered with no network at all; a page that wasn't
+ prefetched in time still arrives about 5x smaller (a typical folder listing is ~60 KB of HTML, ~12 KB
+ gzipped).
+3. **Fewer storage round trips.** Opening a folder used to cost a wasted `HEAD` request plus the
+ listing. When the parent folder's listing is cached (it is, because that's the page you clicked
+ from), the `HEAD` is skipped, and simultaneous requests for one folder share a single Hugging Face call.
+
+**Freshness trade-off:** because pages are cached for 30 s in the browser (on top of the server's 60 s
+listing cache), a file you just uploaded can take up to a minute or two to show in a folder you
+already had open. A hard refresh (Ctrl/Cmd+Shift+R) always fetches fresh.
+
+### What gets prefetched
+
+The prefetcher is deliberately conservative:
- Same-origin links only, and never `/api/*`, `/admin`, `/mask/*`, `/pack/*`, `/static/*`, `/raw/*`,
downloads, `target`/`download` links, or anything inside `data-turbo="false"`.
-- At most **16 pages per page view**, **2 at a time**. Folder and file links on screen go first,
+- At most **16 pages per page view**, **3 at a time**. Folder and file links on screen go first,
navbar and footer links last. Only while the tab is visible.
-- Cancelled as soon as you navigate, so it never competes with a real click.
+- When you click, in-flight prefetches are cancelled, except the one for the page you're opening.
- Disabled on Save-Data, 2g and `prefers-reduced-data`; reduced to 6 pages / 1 at a time on 3g.
To turn it off for a page, add `<meta name="cdn-prefetch" content="off">`; to exclude a link or a
-whole section, add `data-no-prefetch`. To change the limits, edit `DEFAULTS` at the top of the file. For
-debugging, run `__cdnPrefetch.stats()` in the browser console. Prefetches carry an `X-Purpose: prefetch`
-request header, and they don't show up in the Recent/Visited history, which is recorded by the page itself.
+whole section, add `data-no-prefetch`. To change the limits, edit `DEFAULTS` at the top of the file
+(keep `ttlMs` below the 30 s cache lifetime in `app/fast_html.py`). For debugging, run
+`__cdnPrefetch.stats()` in the browser console. Prefetches carry an `X-Purpose: prefetch` request
+header, and they don't show up in the Recent/Visited history, which is recorded by the page itself.
## Project layout
```
app/
main.py routes, uploads, zip packing, GitHub sync endpoint
+ fast_html.py browser-cache + gzip headers for rendered pages
storage.py Hugging Face bucket access, listing, caching, search
upload_guard.py upload checks: video blocking, VPN detection, client IP
admin.py /admin dashboard, bans, GitHub allowlist (encrypted state)
@@ -163,7 +181,7 @@ app/
stats.py /stats
templates/ Jinja2 pages (index.html renders most page types)
static/ icons, manifest, service worker, prefetch.js
-tests/ node --test tests/*.test.js (prefetcher unit tests)
+tests/ prefetcher unit tests (node) + offline server fast-path checks (python)
Dockerfile · render.yaml · start.sh
```
diff --git a/app/fast_html.py b/app/fast_html.py
@@ -0,0 +1,53 @@
+"""Make rendered HTML pages cheap to reuse.
+
+Every UI page is rendered from a template and is identical for everybody (it only carries global
+stats and URLs), so two things are safe and make navigation much faster:
+
+* ``Cache-Control: private, max-age=30``: the browser keeps the page for 30 seconds. The link
+ prefetcher (static/prefetch.js) fetches pages in the background, and a click on one of them is then
+ answered from the browser's own cache with no network at all. ``private`` keeps shared caches out of it.
+* gzip: the pages are 20-90 KB of HTML with inlined CSS/JS and compress about 5x, which matters most on
+ slow or high-latency connections.
+
+Only successful GET responses built through ``TemplateResponse`` are touched. File downloads, the API,
+redirects, errors and the admin dashboard never go through here.
+"""
+
+import gzip
+
+from fastapi.templating import Jinja2Templates
+from starlette.requests import Request
+
+HTML_MAX_AGE = 30 # seconds; keep in step with `ttlMs` in static/prefetch.js
+MIN_GZIP_BYTES = 1024
+GZIP_LEVEL = 5
+
+
+def optimise_html(request: Request | None, response):
+ """Add caching and compression headers to a rendered page (in place) and return it."""
+ if request is None or request.method != "GET" or response.status_code != 200:
+ return response
+
+ if "cache-control" not in response.headers:
+ response.headers["Cache-Control"] = f"private, max-age={HTML_MAX_AGE}"
+
+ # The body now depends on Accept-Encoding, so caches must key on it.
+ response.headers.append("Vary", "Accept-Encoding")
+
+ body = response.body
+ accepts_gzip = "gzip" in request.headers.get("accept-encoding", "").lower()
+ if accepts_gzip and "content-encoding" not in response.headers and len(body) >= MIN_GZIP_BYTES:
+ compressed = gzip.compress(body, GZIP_LEVEL)
+ response.body = compressed
+ response.headers["Content-Encoding"] = "gzip"
+ response.headers["Content-Length"] = str(len(compressed))
+ return response
+
+
+class FastTemplates(Jinja2Templates):
+ """Jinja2Templates whose responses are cached and compressed (see module docstring)."""
+
+ def TemplateResponse(self, *args, **kwargs): # noqa: N802 (matches the parent's name)
+ response = super().TemplateResponse(*args, **kwargs)
+ request = args[0] if args and isinstance(args[0], Request) else kwargs.get("request")
+ return optimise_html(request, response)
diff --git a/app/main.py b/app/main.py
@@ -17,11 +17,11 @@ from pathlib import Path
from pydantic import BaseModel
from fastapi import FastAPI, Request, File, UploadFile, HTTPException, Form
from fastapi.responses import StreamingResponse, HTMLResponse, PlainTextResponse, RedirectResponse, FileResponse
-from fastapi.templating import Jinja2Templates
+from .fast_html import FastTemplates
from .storage import (
is_file,
- get_file_info,
+ get_path_info,
list_directory,
list_files_recursive,
upload_temp_file,
@@ -44,7 +44,7 @@ from .admin import router as admin_router, record_upload, is_banned, gh_check, g
app = FastAPI()
app.include_router(admin_router)
app.include_router(stats_router)
-templates = Jinja2Templates(directory="app/templates")
+templates = FastTemplates(directory="app/templates")
STATIC_DIR = Path(__file__).parent / "static"
_NO_STORE_FILES = {"manifest.json", "sw.js"}
@@ -1011,7 +1011,7 @@ async def serve(request: Request, path: str):
)
if clean_path:
- info = await get_file_info(clean_path)
+ info = await get_path_info(clean_path)
if info["exists"]:
filename = clean_path.split("/")[-1]
diff --git a/app/static/prefetch.js b/app/static/prefetch.js
@@ -9,7 +9,7 @@
* Deliberately conservative:
* - same-origin links only; never /api/*, /admin, /mask/*, /pack/*, /static/*, /raw/*, downloads,
* links with target/download, or anything inside data-turbo="false" / data-no-prefetch
- * - at most 16 pages per page view, 2 at a time, page content before navbar links, visible links
+ * - at most 16 pages per page view, 3 at a time, page content before navbar links, visible links
* first, and only while the tab is visible
* - idle time only; cancelled the moment you navigate so it never competes with a real click
* - skipped entirely on Save-Data, 2g connections and prefers-reduced-data
@@ -20,7 +20,9 @@
(function (global) {
'use strict';
- const DEFAULTS = { maxLinks: 16, concurrency: 2, ttlMs: 50000 };
+ // ttlMs stays below the pages' Cache-Control max-age (app/fast_html.py, 30 s): a page this old is
+ // re-fetched, one younger is still in the browser's cache and needs no work.
+ const DEFAULTS = { maxLinks: 16, concurrency: 3, ttlMs: 25000 };
// First path segments that belong to the app itself, not to the CDN's content. Anything under them is
// an API, a redirect, an admin page or a download, so it must never be fetched speculatively.
@@ -58,13 +60,12 @@
const cfg = Object.assign({}, DEFAULTS, opts.config);
const now = opts.now || Date.now;
const seen = new Map(); // url -> when it last finished; stops repeat fetches inside the cache TTL
- const inflight = new Set();
+ const inflight = new Map(); // url -> AbortController, so one request can be cancelled on its own
let queue = [];
let queued = new Set();
let active = 0;
let used = 0; // pages accepted for the current page view (the budget)
let paused = false;
- let controller = new AbortController();
function fresh(url) {
const t = seen.get(url);
@@ -73,14 +74,14 @@
function run(url) {
active++;
- inflight.add(url);
- const signal = controller.signal;
+ const ctl = new AbortController();
+ inflight.set(url, ctl);
Promise.resolve()
- .then(() => opts.fetchFn(url, signal))
+ .then(() => opts.fetchFn(url, ctl.signal))
.then(
() => seen.set(url, now()),
// A failed fetch is remembered too, so a flaky or 404 link is not retried in a loop.
- () => { if (!signal.aborted) seen.set(url, now()); }
+ () => { if (!ctl.signal.aborted) seen.set(url, now()); }
)
.then(() => {
active--;
@@ -116,10 +117,11 @@
pump();
return added;
},
- // Cancel in-flight requests and forget the queue (called when navigating away).
- abort() {
- controller.abort();
- controller = new AbortController();
+ // Cancel in-flight requests and forget the queue (called when navigating away). A request for
+ // `keepUrl` (the page being opened) is left running: it is exactly what the click needs, and the
+ // browser can hand its response to the navigation instead of starting over.
+ abort(keepUrl) {
+ for (const [url, ctl] of inflight) if (url !== keepUrl) ctl.abort();
queue = [];
queued = new Set();
},
@@ -197,7 +199,7 @@
pf.enqueue(found.map((f) => f.url));
}
- // Wait for the page to finish loading and the browser to go idle before touching the network.
+ // Wait for the document to be parsed and the browser to go idle before touching the network.
let pending = false;
let pendingFresh = false;
function request(freshPage) {
@@ -211,11 +213,12 @@
pendingFresh = false;
scan(f);
};
- if (win.requestIdleCallback) win.requestIdleCallback(run, { timeout: 2000 });
- else win.setTimeout(run, 300);
+ if (win.requestIdleCallback) win.requestIdleCallback(run, { timeout: 500 });
+ else win.setTimeout(run, 100);
};
- if (doc.readyState === 'complete') go();
- else win.addEventListener('load', go, { once: true });
+ // The HTML is enough to know the links; don't wait for every image and script to finish.
+ if (doc.readyState !== 'loading') go();
+ else doc.addEventListener('DOMContentLoaded', go, { once: true });
}
request(true);
@@ -232,7 +235,15 @@
doc.addEventListener('visibilitychange', () => pf.setPaused(doc.hidden));
// Navigating: stop prefetching so the real request gets the connection to itself.
- doc.addEventListener('turbo:before-visit', () => pf.abort());
+ doc.addEventListener('turbo:before-visit', (event) => {
+ let keep = null;
+ try {
+ const u = new URL(event.detail.url);
+ u.hash = '';
+ keep = u.href;
+ } catch (e) { /* no usable URL: cancel everything */ }
+ pf.abort(keep);
+ });
win.addEventListener('pagehide', () => pf.abort());
}
diff --git a/app/stats.py b/app/stats.py
@@ -14,12 +14,12 @@ from datetime import datetime, timezone
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
-from fastapi.templating import Jinja2Templates
+from .fast_html import FastTemplates
from . import storage
router = APIRouter()
-templates = Jinja2Templates(directory="app/templates")
+templates = FastTemplates(directory="app/templates")
STATS_TTL = 600
_cache: dict = {"at": 0.0, "data": None}
diff --git a/app/storage.py b/app/storage.py
@@ -116,6 +116,22 @@ async def get_file_info(path: str) -> dict:
_set_cache(cache_key, result)
return result
+async def get_path_info(path: str) -> dict:
+ """Like get_file_info(), for deciding whether a URL is a file page or a folder listing.
+
+ Opening a folder used to cost a wasted HEAD request (a folder isn't a file, so it 404s) before the
+ listing was fetched. When the parent folder's listing is already cached, which it normally is
+ because that is the page you clicked from, it already says whether `path` is a file or a folder
+ (and a file's size), so the HEAD is skipped. Anything not found there falls back to get_file_info().
+ """
+ parent = path.rpartition("/")[0]
+ for entry in _get_cache(f"list_dir_{parent}") or ():
+ if entry["path"] == path:
+ if entry["is_dir"]:
+ return {"exists": False, "size": None, "content_type": None}
+ return {"exists": True, "size": entry.get("size"), "content_type": None}
+ return await get_file_info(path)
+
def _under(path: str, items: list) -> list:
"""Keep only entries inside `path` (defensive: guards against prefix over-match
such as 'docs' also matching 'docs-old/...')."""
@@ -149,34 +165,42 @@ async def list_directory(path: str):
if cached is not None:
return cached
- try:
- items = await asyncio.to_thread(_fetch_tree, path)
- except Exception as e:
- print(f"[storage] list_repo_tree failed for {path!r}: {e}")
- return []
-
- files_and_folders = []
- for item in items:
- if _is_hidden_path(item.path):
- continue
+ # One fetch per folder at a time: when a prefetch and a real click ask for the same folder
+ # together, the second waits for the first instead of repeating the Hugging Face call.
+ async with _get_key_lock(cache_key):
+ cached = _get_cache(cache_key)
+ if cached is not None:
+ return cached
- name = item.path.split("/")[-1]
- is_dir = not hasattr(item, "size")
- size = getattr(item, "size", 0)
-
- size_str = format_size(size) if not is_dir else "-"
-
- files_and_folders.append({
- "name": name,
- "path": item.path,
- "is_dir": is_dir,
- "size_str": size_str
- })
-
- files_and_folders.sort(key=lambda x: (not x["is_dir"], x["name"].lower()))
-
- _set_cache(cache_key, files_and_folders)
- return files_and_folders
+ try:
+ items = await asyncio.to_thread(_fetch_tree, path)
+ except Exception as e:
+ print(f"[storage] list_repo_tree failed for {path!r}: {e}")
+ return []
+
+ files_and_folders = []
+ for item in items:
+ if _is_hidden_path(item.path):
+ continue
+
+ name = item.path.split("/")[-1]
+ is_dir = not hasattr(item, "size")
+ size = getattr(item, "size", 0)
+
+ size_str = format_size(size) if not is_dir else "-"
+
+ files_and_folders.append({
+ "name": name,
+ "path": item.path,
+ "is_dir": is_dir,
+ "size": size,
+ "size_str": size_str
+ })
+
+ files_and_folders.sort(key=lambda x: (not x["is_dir"], x["name"].lower()))
+
+ _set_cache(cache_key, files_and_folders)
+ return files_and_folders
async def list_files_recursive(path: str):
try:
diff --git a/tests/prefetch.test.js b/tests/prefetch.test.js
@@ -1,7 +1,7 @@
// Run with: node --test tests/*.test.js
const test = require('node:test');
const assert = require('node:assert/strict');
-const { pathAllowed, candidateUrl, createPrefetcher } = require('../app/static/prefetch.js');
+const { DEFAULTS, pathAllowed, candidateUrl, createPrefetcher } = require('../app/static/prefetch.js');
const BASE = new URL('https://cdn.example.com/photos/');
const link = (href, extra = {}) => ({ href, ...extra });
@@ -144,7 +144,7 @@ test('a failing fetch does not stall the queue and is not retried in a loop', as
test('paused queue holds work until resumed', async () => {
const { fn, calls } = manualFetch();
- const pf = createPrefetcher({ fetchFn: fn, config: { maxLinks: 10 } });
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 2, maxLinks: 10 } });
pf.setPaused(true);
pf.enqueue(urls(3));
await tick();
@@ -153,3 +153,23 @@ test('paused queue holds work until resumed', async () => {
await tick();
assert.equal(calls.length, 2);
});
+
+test('abort(keepUrl) leaves the page being opened running and cancels the rest', async () => {
+ const { fn, calls } = manualFetch();
+ const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 3, maxLinks: 10 } });
+ const [a, b, c] = urls(3);
+ pf.enqueue([a, b, c, ...urls(6).slice(3)]);
+ await tick();
+ pf.abort(b);
+ await tick();
+ assert.deepEqual(calls.map((x) => x.aborted), [true, false, true]);
+ assert.equal(pf.stats().queued, 0);
+ calls[1].resolve(); // the kept request finishes normally and is remembered
+ await tick(); await tick();
+ pf.newPage();
+ assert.equal(pf.enqueue([b]), 0);
+});
+
+test('prefetch TTL is shorter than the pages\' browser-cache lifetime (30 s)', () => {
+ assert.ok(DEFAULTS.ttlMs < 30000);
+});
diff --git a/tests/test_server_fastpaths.py b/tests/test_server_fastpaths.py
@@ -0,0 +1,170 @@
+"""Offline checks for the navigation fast paths (no network, no Hugging Face).
+
+Run from the repo root: HF_BUCKET_ID=ci/placeholder python tests/test_server_fastpaths.py
+
+Covers: HTML caching + gzip (app/fast_html.py), skipping the wasted HEAD request when the parent
+folder's listing is cached (storage.get_path_info), and one Hugging Face call per folder even when
+several requests ask at once (storage.list_directory).
+"""
+
+import asyncio
+import gzip
+import os
+import sys
+
+os.environ.setdefault("HF_BUCKET_ID", "ci/placeholder")
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from fastapi.testclient import TestClient # noqa: E402
+from starlette.requests import Request # noqa: E402
+from starlette.responses import HTMLResponse # noqa: E402
+
+import app.main as m # noqa: E402
+from app import storage # noqa: E402
+from app.fast_html import HTML_MAX_AGE, optimise_html # noqa: E402
+
+# A tiny fake bucket. Directories have no `size` attribute, like the real tree items.
+FILES = {
+ "top.txt": 7,
+ "bsc/README.md": 3000,
+ "bsc/semester_1/notes.txt": 12,
+ "bsc/semester_1/deep/a.bin": 5,
+ "_logs/hidden.log": 1,
+}
+calls = {"tree": 0, "head": 0}
+
+
+class Dir:
+ def __init__(self, path):
+ self.path = path
+
+
+class File:
+ def __init__(self, path, size):
+ self.path, self.size = path, size
+
+
+def fake_fetch_tree(path):
+ calls["tree"] += 1
+ path = path.strip("/")
+ prefix = path + "/" if path else ""
+ out = {}
+ for fp, size in FILES.items():
+ if not fp.startswith(prefix):
+ continue
+ rest = fp[len(prefix):]
+ if "/" in rest:
+ d = prefix + rest.split("/")[0]
+ out[d] = Dir(d)
+ else:
+ out[fp] = File(fp, size)
+ return list(out.values())
+
+
+async def fake_get_file_info(path):
+ calls["head"] += 1
+ return {"exists": path in FILES, "size": FILES.get(path), "content_type": None}
+
+
+async def no_stats():
+ return None
+
+
+storage._fetch_tree = fake_fetch_tree
+storage.get_file_info = fake_get_file_info
+m.repo_stats = no_stats
+
+
+def reset():
+ storage._cache.clear()
+ calls["tree"] = calls["head"] = 0
+
+
+def check(cond, msg):
+ if not cond:
+ raise AssertionError(msg)
+ print("ok -", msg)
+
+
+# --- storage ----------------------------------------------------------------------------------------
+
+reset()
+loop = asyncio.new_event_loop()
+
+async def burst():
+ return await asyncio.gather(*[storage.list_directory("bsc") for _ in range(6)])
+
+results = loop.run_until_complete(burst())
+check(calls["tree"] == 1, "six simultaneous listings of one folder make a single Hugging Face call")
+check(all(r == results[0] for r in results), "...and all six get the same listing")
+names = [(e["name"], e["is_dir"]) for e in results[0]]
+check(names == [("semester_1", True), ("README.md", False)], "listing is folders-first, hidden paths excluded")
+check(results[0][1]["size"] == 3000 and results[0][1]["size_str"] == "2.93 KB", "entries keep the raw size too")
+
+reset()
+loop.run_until_complete(storage.list_directory("bsc"))
+calls["head"] = 0
+info = loop.run_until_complete(storage.get_path_info("bsc/semester_1"))
+check(info["exists"] is False and calls["head"] == 0, "folder is recognised from the parent listing, no HEAD")
+info = loop.run_until_complete(storage.get_path_info("bsc/README.md"))
+check(info["exists"] is True and info["size"] == 3000 and calls["head"] == 0, "file + size come from the parent listing, no HEAD")
+info = loop.run_until_complete(storage.get_path_info("bsc/missing"))
+check(info["exists"] is False and calls["head"] == 1, "a path the parent doesn't list falls back to the HEAD check")
+reset()
+info = loop.run_until_complete(storage.get_path_info("bsc/README.md"))
+check(info["exists"] is True and calls["head"] == 1, "with no cached parent listing it falls back to the HEAD check")
+loop.close()
+
+# --- HTML optimisation -------------------------------------------------------------------------------
+
+def req(method="GET", accept="gzip"):
+ return Request({"type": "http", "method": method, "headers": [(b"accept-encoding", accept.encode())]})
+
+big = "<html>" + "x" * 5000 + "</html>"
+r = optimise_html(req(), HTMLResponse(big))
+check(r.headers["content-encoding"] == "gzip" and gzip.decompress(r.body).decode() == big, "gzip round-trips")
+check(int(r.headers["content-length"]) == len(r.body) < len(big), "content-length matches the compressed body")
+check(r.headers["cache-control"] == f"private, max-age={HTML_MAX_AGE}", "pages are cached privately")
+check("accept-encoding" in r.headers["vary"].lower(), "Vary: Accept-Encoding is set")
+r = optimise_html(req(accept="identity"), HTMLResponse(big))
+check("content-encoding" not in r.headers and r.body.decode() == big, "no gzip unless the client accepts it")
+r = optimise_html(req(), HTMLResponse("tiny"))
+check("content-encoding" not in r.headers, "tiny bodies are left uncompressed")
+r = optimise_html(req(), HTMLResponse(big, status_code=404))
+check("cache-control" not in r.headers and "content-encoding" not in r.headers, "non-200 responses are untouched")
+r = optimise_html(req(method="POST"), HTMLResponse(big))
+check("cache-control" not in r.headers, "non-GET requests are untouched")
+r = HTMLResponse(big, headers={"Cache-Control": "no-store"})
+check(optimise_html(req(), r).headers["cache-control"] == "no-store", "an explicit Cache-Control is respected")
+
+# --- end to end through the real app -----------------------------------------------------------------
+
+reset()
+c = TestClient(m.app)
+gz = {"Accept-Encoding": "gzip"}
+
+r = c.get("/bsc", headers=gz)
+check(r.status_code == 200 and r.headers.get("content-encoding") == "gzip", "/bsc listing is served gzipped")
+check("semester_1" in r.text and "README.md" in r.text, "...and decodes to the real page")
+check(r.headers["cache-control"] == f"private, max-age={HTML_MAX_AGE}", "...with the browser-cache header")
+
+calls["head"] = 0
+r = c.get("/bsc/semester_1", headers=gz)
+check(r.status_code == 200 and "notes.txt" in r.text and calls["head"] == 0, "opening a folder from its parent makes no HEAD request")
+r = c.get("/bsc/README.md", headers=gz)
+check(r.status_code == 200 and "2.93 KB" in r.text and calls["head"] == 0, "opening a file from its parent makes no HEAD request")
+
+r = c.get("/bsc", headers={"Accept-Encoding": "identity"})
+check(r.status_code == 200 and "content-encoding" not in r.headers, "identity clients get plain HTML")
+
+for path in ["/new", "/find", "/history", "/shorten", "/documentation", "/gh-sync"]:
+ r = c.get(path, headers=gz)
+ check(r.status_code == 200 and r.headers.get("content-encoding") == "gzip"
+ and "max-age" in r.headers["cache-control"], f"{path} is cached and gzipped")
+
+r = c.get("/no/such/folder", headers=gz)
+check(r.status_code == 404 and "max-age" not in r.headers.get("cache-control", ""), "404 pages are not cached")
+r = c.get("/static/sw.js", headers=gz)
+check(r.headers["cache-control"].startswith("no-store") and "content-encoding" not in r.headers, "service worker is still no-store and untouched")
+
+print("\nall server fast-path checks passed")
