cdnV2

git clone https://github.com/notamitgamer/cdnV2.git
· rebuilt hourlyOpen on GitHub
commit 0a8d2e1dfebbf7172ff0300fa153bc742a3279b1
parent f228efdf93625a133dab093dfb7fa53fe012a4c4
Author: bot-for-notamitgamer[bot] <bot-for-notamitgamer[bot]@users.noreply.github.com>
Date:   Sat, 10 Oct 2026 05:55:08 +0000

Add background link prefetching

Once a page is idle, fetch the pages it links to so the server's 60 s
listing cache is already warm when the user clicks. The slow part of
opening a folder is the cold Hugging Face lookup behind it.

- app/static/prefetch.js: conservative prefetcher. Same-origin links
  only; never /api, /admin, /mask, /pack, /static, /raw, downloads,
  target/download links or data-turbo="false" content. Max 16 pages per
  page view, 2 at a time, page content before navbar links, on-screen
  links first, only while the tab is visible, cancelled on navigation.
  Skipped on Save-Data, 2g and prefers-reduced-data (reduced on 3g).
  Works with Turbo (re-runs on turbo:load) and without it. Opt out with
  <meta name="cdn-prefetch" content="off"> or data-no-prefetch.
- Serve it from /static/ (allowlisted, short max-age) and load it from
  the head of the public page templates.
- tests/prefetch.test.js: 13 dependency-free unit tests (node --test),
  run in CI; CI smoke test also checks /static/prefetch.js.
- Document it in README and CONTRIBUTING.

Diffstat:
M.github/workflows/ci.yml | 11+++++++++--
MCONTRIBUTING.md | 5+++--
MREADME.md | 24+++++++++++++++++++++++-
Mapp/main.py | 13++++++++-----
Aapp/static/prefetch.js | 244+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mapp/templates/documentation.html | 1+
Mapp/templates/gh_sync_docs.html | 1+
Mapp/templates/index.html | 1+
Mapp/templates/stats.html | 1+
Mapp/templates/url_shortener.html | 1+
Atests/prefetch.test.js | 155+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
11 files changed, 447 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml @@ -10,7 +10,7 @@ permissions: jobs: checks: - name: Lint and smoke test + name: Lint, tests and smoke test runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 @@ -29,6 +29,13 @@ jobs: - name: Byte-compile run: python -m compileall -q app + - uses: actions/setup-node@v4 + with: + node-version: "22" + + - name: Prefetch script unit tests + run: node --test tests/*.test.js + - name: Smoke test (app imports, static pages render) env: HF_BUCKET_ID: ci/placeholder @@ -42,7 +49,7 @@ jobs: m.repo_stats = no_stats c = TestClient(m.app) - for path in ["/new", "/static/manifest.json"]: + for path in ["/new", "/static/manifest.json", "/static/prefetch.js"]: r = c.get(path) assert r.status_code == 200, (path, r.status_code) print(path, r.status_code) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md @@ -35,11 +35,12 @@ CI runs these on every pull request; please run them locally first: pip install ruff ruff check --select E9,F63,F7,F82 app # syntax errors and undefined names python -m compileall -q app +node --test tests/*.test.js # unit tests for the browser-side prefetcher (Node 20+) docker build -t cdnv2 . # the image must still build ``` -There is no full test suite yet. If you add logic that can be tested without network access (parsing, -validation, rate limiting), adding a test alongside it is very much appreciated. +The Python side has no test suite yet. If you add logic that can be tested without network access +(parsing, validation, rate limiting), adding a test alongside it is very much appreciated. ## Pull requests diff --git a/README.md b/README.md @@ -23,6 +23,8 @@ Live instance: <https://cdn.amit.is-a.dev> · License: [MIT](LICENSE) - **Abuse protection** — per-IP rate limits, video-upload blocking (by extension and by file contents), and optional VPN/proxy blocking. - **Stats** — `/stats` shows file counts and storage used. +- **Link prefetching** — pages you can navigate to from the current one are fetched quietly in the + background, so the server's listing cache is warm by the time you click (details below). - **PWA** — installable, with a service worker that deliberately caches nothing, so listings are never stale. @@ -129,6 +131,25 @@ Default limits per client IP: 200 MB/hour for file uploads, 50 MB/hour for URL u per minute / 50 per hour for the shortener. GitHub sync has no rate limit but accepts at most a 200 MB compressed archive. Video files are refused. +## Link prefetching + +`app/static/prefetch.js` (loaded from `<head>` on the public pages, not the admin dashboard) fetches the pages linked from the +current one once the browser is idle. The slow part of opening a folder is the first Hugging Face +lookup behind it, which the server then caches for 60 seconds; prefetching pays that cost ahead of the +click. It is deliberately conservative: + +- Same-origin links only, and never `/api/*`, `/admin`, `/mask/*`, `/pack/*`, `/static/*`, `/raw/*`, + downloads, `target`/`download` links, or anything inside `data-turbo="false"`. +- At most **16 pages per page view**, **2 at a time**. Folder and file links on screen go first, + navbar and footer links last. Only while the tab is visible. +- Cancelled as soon as you navigate, so it never competes with a real click. +- Disabled on Save-Data, 2g and `prefers-reduced-data`; reduced to 6 pages / 1 at a time on 3g. + +To turn it off for a page, add `<meta name="cdn-prefetch" content="off">`; to exclude a link or a +whole section, add `data-no-prefetch`. To change the limits, edit `DEFAULTS` at the top of the file. For +debugging, run `__cdnPrefetch.stats()` in the browser console. Prefetches carry an `X-Purpose: prefetch` +request header, and they don't show up in the Recent/Visited history, which is recorded by the page itself. + ## Project layout ``` @@ -141,7 +162,8 @@ app/ shortener.py URL shortener stats.py /stats templates/ Jinja2 pages (index.html renders most page types) - static/ icons, manifest, service worker + static/ icons, manifest, service worker, prefetch.js +tests/ node --test tests/*.test.js (prefetcher unit tests) Dockerfile · render.yaml · start.sh ``` diff --git a/app/main.py b/app/main.py @@ -48,19 +48,22 @@ templates = Jinja2Templates(directory="app/templates") STATIC_DIR = Path(__file__).parent / "static" _NO_STORE_FILES = {"manifest.json", "sw.js"} +_REVALIDATE_FILES = {"prefetch.js"} @app.get("/static/{filename}") async def static_no_cache_root(filename: str): - if filename not in _NO_STORE_FILES: + if filename in _NO_STORE_FILES: + cache_control = "no-store, no-cache, must-revalidate, max-age=0" + elif filename in _REVALIDATE_FILES: + # Short cache: full page loads don't re-download it, and a deploy still shows up within minutes. + cache_control = "public, max-age=300" + else: raise HTTPException(status_code=404) file_path = STATIC_DIR / filename if not file_path.is_file(): raise HTTPException(status_code=404) - return FileResponse( - file_path, - headers={"Cache-Control": "no-store, no-cache, must-revalidate, max-age=0"}, - ) + return FileResponse(file_path, headers={"Cache-Control": cache_control}) @app.get("/static/icons/{filename}") diff --git a/app/static/prefetch.js b/app/static/prefetch.js @@ -0,0 +1,244 @@ +/*! + * cdnV2 link prefetcher. + * + * Once a page has settled, quietly fetches the pages it links to (folders, file pages, the navbar + * destinations) so the server's listing cache is already warm when you click. The expensive part of a + * click is the cold Hugging Face lookup behind a folder, and that cache only lives for 60 seconds, so + * this re-warms it just ahead of use. + * + * Deliberately conservative: + * - same-origin links only; never /api/*, /admin, /mask/*, /pack/*, /static/*, /raw/*, downloads, + * links with target/download, or anything inside data-turbo="false" / data-no-prefetch + * - at most 16 pages per page view, 2 at a time, page content before navbar links, visible links + * first, and only while the tab is visible + * - idle time only; cancelled the moment you navigate so it never competes with a real click + * - skipped entirely on Save-Data, 2g connections and prefers-reduced-data + * + * Opt out: <meta name="cdn-prefetch" content="off"> on a page, or data-no-prefetch on a link/container. + * Works with Turbo (re-runs on turbo:load) and without it. Loaded once from <head>; safe to include twice. + */ +(function (global) { + 'use strict'; + + const DEFAULTS = { maxLinks: 16, concurrency: 2, ttlMs: 50000 }; + + // First path segments that belong to the app itself, not to the CDN's content. Anything under them is + // an API, a redirect, an admin page or a download, so it must never be fetched speculatively. + const RESERVED_SEGMENTS = new Set(['api', 'admin', 'mask', 'pack', 'static', 'raw']); + + function pathAllowed(pathname) { + if (pathname === '/favicon.ico') return false; + const first = pathname.split('/')[1] || ''; + return !RESERVED_SEGMENTS.has(first); + } + + // Returns the normalised URL to prefetch for a link, or null if it should be left alone. + // `link` is plain data ({href, target, download, noPrefetch, turboOff}) and `base` a URL for the + // current page, so this stays testable without a DOM. + function candidateUrl(link, base) { + if (link.download || link.noPrefetch || link.turboOff) return null; + if (link.target && link.target !== '_self') return null; + let url; + try { + url = new URL(link.href, base); + } catch (e) { + return null; + } + if (url.protocol !== 'http:' && url.protocol !== 'https:') return null; + if (url.origin !== base.origin) return null; + if (!pathAllowed(url.pathname)) return null; + if (url.pathname === base.pathname && url.search === base.search) return null; // this page + url.hash = ''; + return url.href; + } + + // A small bounded work queue. `fetchFn(url, signal)` must return a promise that settles when the + // response has been fully read (the server only finishes warming its cache once it has answered). + function createPrefetcher(opts) { + const cfg = Object.assign({}, DEFAULTS, opts.config); + const now = opts.now || Date.now; + const seen = new Map(); // url -> when it last finished; stops repeat fetches inside the cache TTL + const inflight = new Set(); + let queue = []; + let queued = new Set(); + let active = 0; + let used = 0; // pages accepted for the current page view (the budget) + let paused = false; + let controller = new AbortController(); + + function fresh(url) { + const t = seen.get(url); + return t !== undefined && now() - t < cfg.ttlMs; + } + + function run(url) { + active++; + inflight.add(url); + const signal = controller.signal; + Promise.resolve() + .then(() => opts.fetchFn(url, signal)) + .then( + () => seen.set(url, now()), + // A failed fetch is remembered too, so a flaky or 404 link is not retried in a loop. + () => { if (!signal.aborted) seen.set(url, now()); } + ) + .then(() => { + active--; + inflight.delete(url); + pump(); + }); + } + + function pump() { + while (!paused && active < cfg.concurrency && queue.length) run(queue.shift()); + } + + return { + // Start a new page view: fresh budget, drop anything still waiting. Finished work is remembered. + newPage() { + queue = []; + queued = new Set(); + used = 0; + }, + remaining() { + return cfg.maxLinks - used; + }, + enqueue(urls) { + let added = 0; + for (const url of urls) { + if (used >= cfg.maxLinks) break; + if (queued.has(url) || inflight.has(url) || fresh(url)) continue; + queue.push(url); + queued.add(url); + used++; + added++; + } + pump(); + return added; + }, + // Cancel in-flight requests and forget the queue (called when navigating away). + abort() { + controller.abort(); + controller = new AbortController(); + queue = []; + queued = new Set(); + }, + setPaused(value) { + paused = !!value; + if (!paused) pump(); + }, + stats() { + return { queued: queue.length, active, remembered: seen.size, used }; + }, + }; + } + + function shouldSkipDevice(win) { + const conn = win.navigator && (win.navigator.connection || win.navigator.mozConnection); + if (conn && (conn.saveData || /2g$/.test(conn.effectiveType || ''))) return true; + if (win.matchMedia && win.matchMedia('(prefers-reduced-data: reduce)').matches) return true; + return false; + } + + function boot(win) { + const doc = win.document; + if (!doc || win.__cdnPrefetch || typeof win.fetch !== 'function') return; + if (shouldSkipDevice(win)) return; + + const conn = win.navigator && win.navigator.connection; + const config = conn && conn.effectiveType === '3g' ? { maxLinks: 6, concurrency: 1 } : {}; + + const pf = createPrefetcher({ + config, + fetchFn: (url, signal) => + win + .fetch(url, { + credentials: 'same-origin', + signal, + priority: 'low', // ignored by browsers that don't support it + headers: { Accept: 'text/html', 'X-Purpose': 'prefetch' }, + }) + .then((res) => res.arrayBuffer()), + }); + win.__cdnPrefetch = pf; // handy for debugging: __cdnPrefetch.stats() + + function scan(freshPage) { + if (doc.querySelector('meta[name="cdn-prefetch"][content="off"]')) return; + if (freshPage) pf.newPage(); + if (pf.remaining() <= 0) return; + + const base = new URL(win.location.href); + const viewport = win.innerHeight || 800; + const found = []; + const dedupe = new Set(); + + doc.querySelectorAll('a[href]').forEach((a) => { + if (a.getClientRects().length === 0) return; // hidden: filtered-out rows, collapsed menus + const url = candidateUrl( + { + href: a.href, + target: a.getAttribute('target'), + download: a.hasAttribute('download'), + noPrefetch: !!a.closest('[data-no-prefetch]'), + turboOff: !!a.closest('[data-turbo="false"]'), + }, + base + ); + if (!url || dedupe.has(url)) return; + dedupe.add(url); + const r = a.getBoundingClientRect(); + // Page content first (on-screen links, plus a screenful below, ahead of the rest), then site + // chrome (navbar, footer). Those pages are cheap for the server, so they only get what's left. + const chrome = !!a.closest('header, nav, footer'); + found.push({ url, rank: chrome ? 2 : r.bottom > 0 && r.top < viewport * 1.5 ? 0 : 1 }); + }); + + found.sort((x, y) => x.rank - y.rank); + pf.enqueue(found.map((f) => f.url)); + } + + // Wait for the page to finish loading and the browser to go idle before touching the network. + let pending = false; + let pendingFresh = false; + function request(freshPage) { + pendingFresh = pendingFresh || freshPage; + if (pending) return; + pending = true; + const go = () => { + const run = () => { + pending = false; + const f = pendingFresh; + pendingFresh = false; + scan(f); + }; + if (win.requestIdleCallback) win.requestIdleCallback(run, { timeout: 2000 }); + else win.setTimeout(run, 300); + }; + if (doc.readyState === 'complete') go(); + else win.addEventListener('load', go, { once: true }); + } + + request(true); + doc.addEventListener('turbo:load', () => request(true)); + + // Pages that build their list with JS (search results, history) add links after load. + let timer = 0; + new win.MutationObserver(() => { + win.clearTimeout(timer); + timer = win.setTimeout(() => request(false), 800); + }).observe(doc.documentElement, { childList: true, subtree: true }); + + pf.setPaused(doc.hidden); + doc.addEventListener('visibilitychange', () => pf.setPaused(doc.hidden)); + + // Navigating: stop prefetching so the real request gets the connection to itself. + doc.addEventListener('turbo:before-visit', () => pf.abort()); + win.addEventListener('pagehide', () => pf.abort()); + } + + if (typeof module === 'object' && module.exports) { + module.exports = { DEFAULTS, pathAllowed, candidateUrl, createPrefetcher, boot }; + } else { + boot(global); + } +})(typeof window !== 'undefined' ? window : globalThis); diff --git a/app/templates/documentation.html b/app/templates/documentation.html @@ -52,6 +52,7 @@ .footnote{font-size:.82rem;color:var(--muted);margin-top:3rem;border-top:1px solid var(--border);padding-top:1.2rem;} footer{margin-top:4rem;padding-top:1.5rem;border-top:1px solid var(--border);font-size:.85rem;color:var(--muted);} </style> + <script src="/static/prefetch.js" defer></script> </head> <body> {% include "_navbar.html" %} diff --git a/app/templates/gh_sync_docs.html b/app/templates/gh_sync_docs.html @@ -120,6 +120,7 @@ color: var(--muted); } </style> + <script src="/static/prefetch.js" defer></script> </head> <body> {% include "_navbar.html" %} diff --git a/app/templates/index.html b/app/templates/index.html @@ -27,6 +27,7 @@ <meta name="twitter:card" content="summary"> <script type="module" src="https://cdn.jsdelivr.net/npm/@hotwired/turbo@8.0.4/dist/turbo.es2017-esm.js"></script> + <script src="/static/prefetch.js" defer></script> <style> :root { diff --git a/app/templates/stats.html b/app/templates/stats.html @@ -73,6 +73,7 @@ } :root { --c0: #4c8dff; --c1: #3fb950; --c2: #d29922; --c3: #f85149; --c4: #a371f7; --c5: #39c5cf; --c6: #db61a2; --c7: #8b949e; } </style> + <script src="/static/prefetch.js" defer></script> </head> <body> {% include "_navbar.html" %} diff --git a/app/templates/url_shortener.html b/app/templates/url_shortener.html @@ -15,6 +15,7 @@ <meta name="theme-color" content="#0d1117" media="(prefers-color-scheme: dark)"> <meta name="color-scheme" content="light dark"> <script type="module" src="https://cdn.jsdelivr.net/npm/@hotwired/turbo@8.0.4/dist/turbo.es2017-esm.js"></script> + <script src="/static/prefetch.js" defer></script> <style> :root { /* Legacy names mapped onto the shared --ui- tokens (_styles.html) */ diff --git a/tests/prefetch.test.js b/tests/prefetch.test.js @@ -0,0 +1,155 @@ +// Run with: node --test tests/*.test.js +const test = require('node:test'); +const assert = require('node:assert/strict'); +const { pathAllowed, candidateUrl, createPrefetcher } = require('../app/static/prefetch.js'); + +const BASE = new URL('https://cdn.example.com/photos/'); +const link = (href, extra = {}) => ({ href, ...extra }); +const pick = (href, extra) => candidateUrl(link(href, extra), BASE); + +test('allows folders, files, and top-level pages', () => { + assert.equal(pick('/uploads/'), 'https://cdn.example.com/uploads/'); + assert.equal(pick('/uploads/report.pdf'), 'https://cdn.example.com/uploads/report.pdf'); + assert.equal(pick('/find'), 'https://cdn.example.com/find'); + assert.equal(pick('/documentation'), 'https://cdn.example.com/documentation'); + assert.equal(pick('sub/dir'), 'https://cdn.example.com/photos/sub/dir'); // relative +}); + +test('never prefetches the app’s own endpoints, downloads or redirects', () => { + for (const p of ['/api/get/uploads/big.zip', '/api/pack/uploads', '/admin', '/admin/x', + '/mask/abc123', '/pack/batch1', '/static/sw.js', '/raw/uploads/a.png', + '/favicon.ico']) { + assert.equal(pick(p), null, p); + } +}); + +test('reserved names only match whole path segments', () => { + assert.equal(pathAllowed('/administrator/'), true); + assert.equal(pathAllowed('/apikeys/'), true); + assert.equal(pathAllowed('/api'), false); + assert.equal(pathAllowed('/'), true); +}); + +test('skips other origins and non-http schemes', () => { + assert.equal(pick('https://raw.cdn.example.com/uploads/a.png'), null); + assert.equal(pick('https://example.org/'), null); + assert.equal(pick('mailto:a@b.c'), null); + assert.equal(pick('javascript:void(0)'), null); +}); + +test('respects opt-outs on the link', () => { + assert.equal(pick('/uploads/', { download: true }), null); + assert.equal(pick('/uploads/', { target: '_blank' }), null); + assert.equal(pick('/uploads/', { target: '_self' }), 'https://cdn.example.com/uploads/'); + assert.equal(pick('/uploads/', { turboOff: true }), null); + assert.equal(pick('/uploads/', { noPrefetch: true }), null); +}); + +test('skips the current page, strips hashes', () => { + assert.equal(pick('/photos/'), null); + assert.equal(pick('/photos/#top'), null); + assert.equal(pick('/uploads/#x'), 'https://cdn.example.com/uploads/'); + assert.equal(pick('/photos/?q=1'), 'https://cdn.example.com/photos/?q=1'); +}); + +// --- queue --------------------------------------------------------------------------------------- + +// A fetch whose responses the test finishes by hand, so we can observe concurrency. +function manualFetch() { + const calls = []; + const fn = (url, signal) => + new Promise((resolve, reject) => { + const call = { url, signal, resolve, reject, aborted: false }; + signal.addEventListener('abort', () => { call.aborted = true; reject(new Error('aborted')); }); + calls.push(call); + }); + return { fn, calls }; +} +const tick = () => new Promise((r) => setImmediate(r)); +const urls = (n) => Array.from({ length: n }, (_, i) => `https://cdn.example.com/f${i}/`); + +test('runs at most `concurrency` fetches at once and drains the queue', async () => { + const { fn, calls } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 2, maxLinks: 10 } }); + pf.enqueue(urls(5)); + await tick(); + assert.equal(calls.length, 2); + calls[0].resolve(); + await tick(); + assert.equal(calls.length, 3); + calls.forEach((c) => c.resolve()); + await tick(); await tick(); + assert.equal(calls.length, 5); +}); + +test('stops accepting links after maxLinks per page view, and newPage resets the budget', async () => { + const { fn } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, config: { maxLinks: 3 } }); + assert.equal(pf.enqueue(urls(10)), 3); + assert.equal(pf.remaining(), 0); + assert.equal(pf.enqueue(urls(10)), 0); + pf.newPage(); + assert.equal(pf.remaining(), 3); +}); + +test('does not refetch inside the TTL, does after it', async () => { + let t = 1000; + const { fn, calls } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, now: () => t, config: { ttlMs: 50000, maxLinks: 10 } }); + pf.enqueue(urls(1)); + await tick(); + calls[0].resolve(); + await tick(); + pf.newPage(); + assert.equal(pf.enqueue(urls(1)), 0); // still fresh + t += 60000; + assert.equal(pf.enqueue(urls(1)), 1); // expired +}); + +test('does not queue a URL that is already in flight', async () => { + const { fn } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, config: { maxLinks: 10 } }); + pf.enqueue(urls(1)); + await tick(); + pf.newPage(); + assert.equal(pf.enqueue(urls(1)), 0); +}); + +test('abort cancels in-flight requests and clears the queue', async () => { + const { fn, calls } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 1, maxLinks: 10 } }); + pf.enqueue(urls(4)); + await tick(); + pf.abort(); + await tick(); + assert.equal(calls[0].aborted, true); + assert.equal(calls.length, 1); // nothing else started + assert.equal(pf.stats().queued, 0); + // an aborted fetch is not remembered as done, so it can be tried again later + pf.newPage(); + assert.equal(pf.enqueue(urls(1)), 1); +}); + +test('a failing fetch does not stall the queue and is not retried in a loop', async () => { + const { fn, calls } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, config: { concurrency: 1, maxLinks: 10 } }); + pf.enqueue(urls(2)); + await tick(); + calls[0].reject(new Error('network')); + await tick(); await tick(); + assert.equal(calls.length, 2); // moved on to the next one + pf.newPage(); + assert.equal(pf.enqueue(urls(1)), 0); // failed URL remembered for the TTL +}); + +test('paused queue holds work until resumed', async () => { + const { fn, calls } = manualFetch(); + const pf = createPrefetcher({ fetchFn: fn, config: { maxLinks: 10 } }); + pf.setPaused(true); + pf.enqueue(urls(3)); + await tick(); + assert.equal(calls.length, 0); + pf.setPaused(false); + await tick(); + assert.equal(calls.length, 2); +});