diff --git a/AGENTS.md b/AGENTS.md index 100b157..3ec32bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,15 +1,49 @@ -# AGENTS — AO3 Mirror +# AGENTS — AO3 Mirror v5 This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror. ## Project Overview -AO3 Mirror is a high-availability reverse proxy for archiveofourown.org, designed to restore access for Chinese users. It bypasses Cloudflare's bot protection using TLS fingerprint impersonation (curl_cffi), a tiered WebShare proxy pool (726 proxies, 72%+ CF bypass rate), cookie-aware session management, and user-browser CF challenge solving. +AO3 Mirror v5 is a high-performance reverse proxy for archiveofourown.org, redesigned from the ground up with sticky sessions, Service Worker client-side caching, and a full 5000-proxy P2C pool — matching the go3 architecture with modern Python tooling. - **Domain**: agento3.miscs.dev (behind Cloudflare CDN) - **Target**: archiveofourown.org - **Stack**: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy -- **QPS**: 720 (cache) | 270 (direct proxy) +- **Proxy Pool**: 5000 WebShare proxies, P2C selection, state machine (healthy/unstable/blocked/probing) +- **Client**: Service Worker for static cache + offline fallback +- **Architecture Reference**: https://tomorin.cv/AO3-CN/go3 (go3 v2 branch) + +## Architecture v5 + +``` +User Browser + │ ao3_sessid_proxy cookie = sticky proxy + │ + ├─ Service Worker (sw.js) + │ ├─ Cache API — static assets cached in browser + │ ├─ Navigation intercept — offline mirror picker + │ └─ Request passthrough — for non-static, non-nav + │ + └─ HTTPS → Caddy → 2× uvicorn workers (8081-8082) + │ + ├─ Proxy Pool (5000, P2C selection) + │ ├─ State machine: healthy/unstable/blocked/probing + │ ├─ Concurrency limit: asyncio.Semaphore(400) + │ └─ Passive probing: 10s interval, 10 concurrency + │ + ├─ Sticky Session (ao3_sessid_proxy) + │ └─ User pinned to one proxy for session duration + │ + ├─ Service Worker Deploy (/sw.js, /sw-YYYYMMDD.js) + │ └─ HTML injection: auto-register SW on page load + │ + ├─ CF Challenge → User Browser + │ └─ All proxies blocked → rewrite challenge → browser solves + │ + ├─ Static Cache (server-side LRU + disk MD5) + │ + └─ Local routes: /stats, /health, /metrics +``` ## Build Commands @@ -26,85 +60,60 @@ sudo systemctl restart ao3-daemon # Restart Caddy only sudo systemctl reload caddy -# Scan proxy pool (MUST run in foreground!) -cd /home/ubuntu/ao3-mirror && python3 scripts/scan_proxies_cffi.py - -# Syntax check +# Syntax check all Python files python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py + +# Deploy Service Worker update +# 1. Edit static/sw.js +# 2. Update SERVICE_WORKER_DATE in app.py +# 3. Restart workers ``` -## Architecture - -``` -User → Cloudflare CDN (agento3.miscs.dev) → Caddy :443 (60s, round-robin) - → 2× uvicorn workers (8081-8082) [cpu-pinned, uvloop] - → Tiered Proxy Pool: - Fast (50): POST/login → 8s timeout, 1 retry - Main (676): GET/browse → 15s timeout, 2 retries - → Cookie-aware sessions: per-proxy cf_clearance persistence - → TLS impersonation: safari15_5, safari17_0, chrome123, chrome124 - → archiveofourown.org - -CF Challenge Solving (v4): - 1. All proxies hit CF challenge → generate challenge_token - 2. Rewrite challenge page → forward to user's browser - 3. User browser executes CF JS → challenge solved - 4. cf_clearance captured → saved to proxy cookie jar - 5. Original request retried → content delivered -``` - -### Core Files +### Core Files (v5) | File | Purpose | |------|---------| -| `app.py` | FastAPI backend v4 — proxy handler, stats, CF challenge solving | -| `proxy_pool.py` | Tiered async proxy pool — cookie jar, weighted selection, sampling | -| `ao3_fetcher.py` | Async fetcher — CF detection, smart retry, cookie injection | +| `app.py` | FastAPI backend v5 — proxy handler, sticky sessions, SW deploy, stats | +| `proxy_pool.py` | P2C proxy pool v5 — 5000 proxies, state machine, semaphore limiter | +| `ao3_fetcher.py` | Single-request fetcher v5 — no internal retry, CF detection | | `cache.py` | Per-worker LRU cache (5000 entries, path-differentiated TTL) | | `stats.py` | In-memory stats with batch SQLite flush every 60s | | `url_rewriter.py` | URL/header rewriting (ao3 → mirror domain) | -| `scripts/daemon.py` | Passive worker supervision (systemd, 30s check, 3-strike restart) | -| `scripts/scan_proxies_cffi.py` | Proxy scanner with TLS fingerprint fallback | -| `start.sh` | Startup script — kills old workers, starts new, reloads Caddy | -| `Caddyfile` | Caddy config — TLS, round-robin, 60s timeouts | +| `static/sw.js` | Service Worker — client-side cache, offline fallback | +| `Caddyfile` | Caddy config — SW routes, round-robin, 60s timeouts | +| `scripts/daemon.py` | Passive worker supervision (systemd) | +| `start.sh` | Startup — kills old workers, starts new, reloads Caddy | ## Security Baseline -- No secrets in code — proxy credentials are in `/home/ubuntu/proxy.txt` -- Worker processes bound to 127.0.0.1 only (not exposed publicly) -- Cloudflare CDN terminates TLS and provides DDoS protection at edge +- No secrets in code — proxy credentials in `/home/ubuntu/proxy.txt` +- Workers bound to 127.0.0.1 only +- Cloudflare CDN terminates TLS at edge - CORS headers restrict cross-origin access -- stats/health/metrics endpoints are read-only, no mutation -- All proxied content is user-facing; no admin endpoints exposed +- stats/health/metrics are read-only ## Engine Guidance - Complex multi-file changes, architecture evolution → Hermes (here) - Quick targeted fixes, single-file changes → Hermes - Deploy, monitor, notify, schedule → Hermes -- Proxy scanning, proxy pool refresh → Hermes cron (30min) +- Proxy pool refresh → Hermes cron (30min) - Not sure? Start with Hermes — everything runs via Hermes ## Monitoring -- Health endpoint: https://agento3.miscs.dev/health -- Stats dashboard: https://agento3.miscs.dev/stats -- Metrics (Prometheus): https://agento3.miscs.dev/metrics -- Worker logs: /home/ubuntu/ao3-mirror/worker-0.log, worker-1.log -- Daemon log: /home/ubuntu/ao3-mirror/daemon.log -- Systemd: `systemctl status ao3-daemon`, `journalctl -u ao3-daemon` +- Health: https://agento3.miscs.dev/health +- Stats: https://agento3.miscs.dev/stats +- Metrics: https://agento3.miscs.dev/metrics +- SW: https://agento3.miscs.dev/sw.js → /sw-YYYYMMDD.js -## Deployment +## CTO Goal (Persistent Focus) -- Single server (current host) -- Caddy manages Let's Encrypt TLS certs on agento3.miscs.dev -- CF CDN fronts the domain (Orange Cloud = on) -- Proxy pool: WebShare static residential proxies, refreshed every 30min via Hermes cron -- No CI/CD — manual deploy via start.sh +Manage akiba/agento3 as CTO. Triage issues hourly, implement top priority, get founder approval before merging. Never ship without YES. ## Commit Conventions - One commit per meaningful change -- Python files only (no binaries, no .pyc, no logs, no .env) -- SOUL.md updated to reflect architectural changes +- Python files only (no binaries, .pyc, logs, .env) +- SOUL.md updated for architectural changes - ao3-mirror skill updated when workflows change diff --git a/Caddyfile b/Caddyfile index 55f70d8..1cdba97 100644 --- a/Caddyfile +++ b/Caddyfile @@ -1,5 +1,5 @@ -# AO3 Mirror - Caddy 配置 -# 前端负载均衡 + TLS 终止 + Cloudflare CDN 集成 +# AO3 Mirror v5 - Caddy 配置 +# 前端负载均衡 + TLS 终止 + Cloudflare CDN 集成 + Service Worker agento3.miscs.dev { # 全局头 @@ -9,6 +9,7 @@ agento3.miscs.dev { X-Frame-Options "SAMEORIGIN" X-XSS-Protection "1; mode=block" Referrer-Policy "strict-origin-when-cross-origin" + Content-Security-Policy "default-src 'self'; script-src 'self' https://cdn.jsdelivr.net; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; object-src 'none'" -Server -X-Powered-By } @@ -26,6 +27,60 @@ agento3.miscs.dev { # 压缩 encode gzip + # Service Worker routes — versioned + redirect + route /sw-* { + reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 { + lb_policy round_robin + health_uri /health + health_interval 10s + health_timeout 5s + transport http { + read_timeout 60s + write_timeout 60s + dial_timeout 5s + } + } + header { + Cache-Control "public, max-age=86400, immutable" + } + } + + # SW redirect + route /sw.js { + reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 { + lb_policy round_robin + health_uri /health + health_interval 10s + health_timeout 5s + transport http { + read_timeout 60s + write_timeout 60s + dial_timeout 5s + } + } + header { + Cache-Control "no-store" + } + } + + # Mirror domains JSON + route /mirror-domains.json { + reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 { + lb_policy round_robin + health_uri /health + health_interval 10s + health_timeout 5s + transport http { + read_timeout 60s + write_timeout 60s + dial_timeout 5s + } + } + header { + Cache-Control "public, max-age=3600" + } + } + # 后端负载均衡 (轮询) - 2 workers on 2-core machine reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 { lb_policy round_robin @@ -34,7 +89,7 @@ agento3.miscs.dev { health_interval 10s health_timeout 5s - # 超时配置 + # 超时配置 — 代理可能需要更长时间 transport http { read_timeout 30s write_timeout 30s @@ -55,6 +110,7 @@ stats.agento3.miscs.dev { X-Content-Type-Options "nosniff" X-Frame-Options "SAMEORIGIN" Referrer-Policy "strict-origin-when-cross-origin" + Content-Security-Policy "default-src 'self'; script-src 'self' https://cdn.jsdelivr.net; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; object-src 'none'" -Server -X-Powered-By } @@ -72,41 +128,67 @@ stats.agento3.miscs.dev { # 压缩 encode gzip - # 路由规则 - 只暴露 /stats 和 /metrics + # 路由规则 - 只暴露 /stats, /metrics, /health, /sw route { - # /stats 页面 - 代理到后端 + # /stats 页面 reverse_proxy /stats* 127.0.0.1:8081 127.0.0.1:8082 { lb_policy round_robin - health_uri /health health_interval 10s health_timeout 5s - transport http { - read_timeout 30s - write_timeout 30s + read_timeout 60s + write_timeout 60s dial_timeout 5s } - header_up X-Forwarded-For {remote_host} header_up X-Forwarded-Proto {scheme} header_up X-Real-IP {remote_host} } - # /metrics 指标 - 代理到后端 + # /metrics 指标 reverse_proxy /metrics* 127.0.0.1:8081 127.0.0.1:8082 { lb_policy round_robin - health_uri /health health_interval 10s health_timeout 5s - transport http { - read_timeout 30s - write_timeout 30s + read_timeout 60s + write_timeout 60s dial_timeout 5s } + header_up X-Forwarded-For {remote_host} + header_up X-Forwarded-Proto {scheme} + header_up X-Real-IP {remote_host} + } + # Service Worker + reverse_proxy /sw* 127.0.0.1:8081 127.0.0.1:8082 { + lb_policy round_robin + health_uri /health + health_interval 10s + health_timeout 5s + transport http { + read_timeout 60s + write_timeout 60s + dial_timeout 5s + } + header_up X-Forwarded-For {remote_host} + header_up X-Forwarded-Proto {scheme} + header_up X-Real-IP {remote_host} + } + + # Mirror domains + reverse_proxy /mirror-domains* 127.0.0.1:8081 127.0.0.1:8082 { + lb_policy round_robin + health_uri /health + health_interval 10s + health_timeout 5s + transport http { + read_timeout 60s + write_timeout 60s + dial_timeout 5s + } header_up X-Forwarded-For {remote_host} header_up X-Forwarded-Proto {scheme} header_up X-Real-IP {remote_host} diff --git a/ao3_fetcher.py b/ao3_fetcher.py index 5af87ec..acebf56 100644 --- a/ao3_fetcher.py +++ b/ao3_fetcher.py @@ -1,25 +1,22 @@ """ -异步 AO3 内容抓取器 v4 — Cookie 感知 + CF 挑战检测 + 智能重试 +异步 AO3 内容抓取器 v5 — 单次代理请求 + CF 挑战检测 -v4 vs v3: -- CF 挑战检测:不再把 403/503 一律当失败 -- Cookie 注入:请求时自动携带 proxy 级别的 cookies(cf_clearance 等) -- Cookie 保存:成功请求后自动提取 Set-Cookie 并保存到 proxy cookie jar -- 智能重试:挑战时优先用带 cookie 的代理,无 cookie 则换代理 -- 挑战页面透传:所有重试都遇到挑战时,返回挑战 HTML 让用户浏览器求解 +v5 vs v4: +- 去掉内部重试循环(重试逻辑迁移到 app.py 的 P2C 选择中) +- 简化 API:直接接受 ProxySession 对象 +- 保留 CF 挑战检测 + 状态机反馈 """ import asyncio import logging import time from typing import Optional -from proxy_pool import get_proxy_pool, ProxySession +from proxy_pool import ProxySession logger = logging.getLogger("ao3-fetcher") # ─── CF Challenge Detection ─────────────────────────────────────────────── -# Markers in response body that indicate a Cloudflare challenge page CF_CHALLENGE_MARKERS = [ b'/cdn-cgi/challenge-platform', b'cf-challenge-running', @@ -28,29 +25,24 @@ CF_CHALLENGE_MARKERS = [ b'challenge-platform', b'cf-turnstile', b'cf_chl_', - # Sometimes CF just returns "Just a moment..." without JS markers b'Checking your browser', b'Just a moment...', ] def is_cf_challenge(status: int, body: bytes, headers: dict) -> bool: - """Detect if response is a Cloudflare challenge page (not a real error).""" + """Detect if response is a Cloudflare challenge page.""" if status not in (403, 503, 429): return False - # Quick check: CF always sets Server header on challenge pages server = headers.get("Server", headers.get("server", "")) if "cloudflare" not in server.lower(): - # Check body for challenge markers for marker in CF_CHALLENGE_MARKERS: if marker in body: return True return False - # Server: cloudflare + non-200 status = likely challenge for marker in CF_CHALLENGE_MARKERS: if marker in body: return True - # If Server is cloudflare and status is 403/503, it's almost certainly a challenge if status in (403, 503): return True return False @@ -69,31 +61,29 @@ CHROME_HEADERS = { "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36", + "User-Agent": ( + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " + "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + ), "DNT": "1", } API_HEADERS = { "Accept": "*/*", "Accept-Language": "en-US,en;q=0.9", - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36", + "User-Agent": ( + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " + "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + ), } -# ─── Timeout config ─────────────────────────────────────────────────────── - -FAST_REQUEST_TIMEOUT = 8 # login/register/signup +# Timeout config +FAST_REQUEST_TIMEOUT = 8 NORMAL_REQUEST_TIMEOUT = 15 -CONNECT_TIMEOUT = 5 -MAX_RETRIES = 2 -FAST_MAX_RETRIES = 1 FAST_PATHS = { - "/users/login", - "/users/sign_up", - "/users/new", - "/invitation_requests", - "/token_dispenser.json", - "/user_sessions", + "/users/login", "/users/sign_up", "/users/new", + "/invitation_requests", "/token_dispenser.json", "/user_sessions", } @@ -106,200 +96,97 @@ def _is_fast_path(url: str) -> bool: return False -def _now() -> float: - try: - return asyncio.get_running_loop().time() - except RuntimeError: - return time.time() - - def _merge_cookies(proxy_cookies: str, user_cookies: str) -> str: - """Merge proxy-level cookies (cf_clearance) with user cookies. User cookies take precedence.""" if not proxy_cookies and not user_cookies: return "" if not proxy_cookies: return user_cookies if not user_cookies: return proxy_cookies - # User cookies take priority (they contain session auth) - # But put proxy cookies first so user cookies can override return f"{proxy_cookies}; {user_cookies}" +# ─── Main fetch function ────────────────────────────────────────────────── + async def fetch_url( url: str, + proxy: Optional[ProxySession] = None, method: str = "GET", headers: Optional[dict] = None, body: Optional[bytes] = None, cookies: Optional[dict] = None, is_api: bool = False, - preferred_proxy: Optional[str] = None, # v4: proxy host:port for challenge affinity ) -> dict: """ - Async fetch from AO3 through proxy pool with cookie-aware session reuse. + Single async fetch through a specific proxy. No internal retry loop. - Priority-based: - - Login/register paths → fast pool + 8s timeout + 1 retry - - Normal paths → cookie-preferring proxies + 15s timeout + 2 retries + Args: + url: Target AO3 URL + proxy: ProxySession to use (must already be selected by app.py) + method: HTTP method + headers: Request headers + body: Request body (POST/PUT/PATCH) + cookies: User cookies dict + is_api: Use API headers instead of Chrome headers - v4 improvements: - - CF challenge detection: don't treat challenge as proxy failure - - Cookie injection: auto-send saved cf_clearance per proxy - - Cookie saving: auto-extract Set-Cookie on success - - Challenge body returned for user-browser solving - - Returns: { - "status": 200, - "headers": {...}, - "body": b"...", - "cookies": {...}, - "success": True/False, - "error": "...", - "elapsed": 0.5, - "proxy_host": "p.webshare.io:10296", - # v4: - "is_challenge": False, - "challenge_body": b"" | None, # CF challenge HTML for user-browser solving - "challenge_proxy": "" | None, # which proxy got the challenge - } + Returns: + dict with success/status/headers/body/cookies/elapsed/is_challenge etc. """ - pool = get_proxy_pool() - base_headers = API_HEADERS.copy() if is_api else CHROME_HEADERS.copy() + if proxy is None: + return { + "status": 0, "headers": {}, "body": b"", "cookies": {}, + "headers_raw": [], + "success": False, "error": "No proxy provided", + "elapsed": 0, "proxy_host": None, + "is_challenge": False, "challenge_body": None, "challenge_proxy": None, + } + host_port = proxy.host_port + start_time = time.time() + + base_headers = API_HEADERS.copy() if is_api else CHROME_HEADERS.copy() if headers: - for h in ["Cookie", "Referer", "Content-Type", "X-Requested-With", "Accept", - "Origin", "X-CSRF-Token", "Authorization"]: + for h in ["Cookie", "Content-Type", "X-Requested-With", + "Accept", "X-CSRF-Token", "Authorization"]: if h in headers: base_headers[h] = headers[h] + # Rewrite Origin and Referer: mirror domain → AO3 domain + # AO3 CSRF checks Origin against archiveofourown.org + if "Origin" in headers: + origin = headers["Origin"] + origin = origin.replace("agento3.miscs.dev", "archiveofourown.org") + base_headers["Origin"] = origin + if "Referer" in headers: + referer = headers["Referer"] + referer = referer.replace("agento3.miscs.dev", "archiveofourown.org") + base_headers["Referer"] = referer + + # Filter out OUR cookies — proxy is transparent, only AO3 cookies should reach AO3 + FORWARD_BLOCKED = {"ao3_sessid_proxy", "__cf_bm", "_cfuvid", "_cf_token"} user_cookie_str = "" if cookies: - user_cookie_str = "; ".join(f"{k}={v}" for k, v in cookies.items()) - - # Determine priority level - is_fast = _is_fast_path(url) or method in ("POST", "PUT", "PATCH") - request_timeout = FAST_REQUEST_TIMEOUT if is_fast else NORMAL_REQUEST_TIMEOUT - max_retries = FAST_MAX_RETRIES if is_fast else MAX_RETRIES - - last_error = None - last_challenge_body = None - last_challenge_proxy = None - seen_proxies = set() # Don't retry with same proxy - - # v4: if preferred_proxy is set (from challenge affinity), use it first - if preferred_proxy: - try: - target_proxy = None - for p in pool._proxies: - if p.host_port == preferred_proxy: - target_proxy = p - break - if target_proxy and target_proxy.is_available: - result = await _try_proxy( - target_proxy, url, method, body, base_headers, user_cookie_str, - request_timeout, is_fast - ) - if result["success"]: - # preferred proxy worked — save cookies - target_proxy.save_cookies(result.get("headers", {})) - return result - if result.get("is_challenge"): - last_challenge_body = result.get("challenge_body") - last_challenge_proxy = target_proxy.host_port - seen_proxies.add(target_proxy.host_port) - except Exception: - pass - - for attempt in range(max_retries): - # v4: prefer proxies with cookies, fallback to any available - if attempt == 0: - # First attempt: use proxy with cookies if available - proxy = pool.get_proxy_with_cookies() - if not proxy: - proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy() - elif attempt == 1 and last_challenge_body: - # Second attempt after challenge: try another proxy with cookies - proxy = pool.get_proxy_with_cookies() - # Make sure it's not the same one - if proxy and proxy.host_port in seen_proxies: - proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy() - if not proxy: - proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy() - else: - proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy() - - if not proxy or proxy.host_port in seen_proxies: - # Find an unseen proxy - for _ in range(5): - p = pool.get_proxy() - if p and p.host_port not in seen_proxies: - proxy = p - break - if not proxy or proxy.host_port in seen_proxies: - continue - - seen_proxies.add(proxy.host_port) - - result = await _try_proxy( - proxy, url, method, body, base_headers, user_cookie_str, - request_timeout, is_fast + user_cookie_str = "; ".join( + f"{k}={v}" for k, v in cookies.items() + if k not in FORWARD_BLOCKED ) - if result["success"]: - # Save response cookies to proxy cookie jar for future requests - proxy.save_cookies(result.get("headers", {})) - return result - - if result.get("is_challenge"): - last_challenge_body = result.get("challenge_body") - last_challenge_proxy = proxy.host_port - last_error = result.get("error", "CF_CHALLENGE") - # Don't break — try another proxy - continue - - # Real failure (connection error, timeout) - last_error = result.get("error", "UNKNOWN") - - # All retries exhausted - return { - "status": 0, "headers": {}, "body": b"", "cookies": {}, - "success": False, - "error": f"All retries failed: {last_error}", - "elapsed": 0, "proxy_host": None, - "is_challenge": last_challenge_body is not None, - "challenge_body": last_challenge_body, - "challenge_proxy": last_challenge_proxy, - } - - -async def _try_proxy( - proxy: ProxySession, - url: str, - method: str, - body: Optional[bytes], - base_headers: dict, - user_cookie_str: str, - request_timeout: int, - is_fast: bool, -) -> dict: - """Try a single request through one proxy. Returns result dict.""" - host_port = f"{proxy.host}:{proxy.port}" - start_time = _now() + is_fast = _is_fast_path(url) or method in ("POST", "PUT", "PATCH") + request_timeout = FAST_REQUEST_TIMEOUT if is_fast else NORMAL_REQUEST_TIMEOUT try: session = await proxy.get_session() - - # v4: Build per-request headers with proxy cookies - # IMPORTANT: use headers= parameter (never session.headers.update() — race condition!) request_headers = base_headers.copy() - # Inject proxy-level cookies (cf_clearance, etc.) + # Inject proxy-level cookies (cf_clearance) proxy_cookie_str = proxy.get_cookie_header() cookie_str = _merge_cookies(proxy_cookie_str, user_cookie_str) if cookie_str: request_headers["Cookie"] = cookie_str # Execute request + request_cookie_preview = cookie_str[:500] + "..." if len(cookie_str) > 500 else cookie_str + logger.info(f"[FETCH] {method} {url[:80]} full_cookie={request_cookie_preview}") if method == "GET": resp = await session.get(url, timeout=request_timeout, headers=request_headers) elif method == "POST": @@ -310,89 +197,79 @@ async def _try_proxy( resp = await session.request(method, url, data=body, timeout=request_timeout, headers=request_headers) - elapsed = _now() - start_time + elapsed = time.time() - start_time status = resp.status_code resp_body = resp.content - resp_headers = dict(resp.headers) + resp_headers = dict(resp.headers) # For Content-Type/Location lookups + resp_headers_raw = list(resp.headers.multi_items()) # Preserves ALL Set-Cookie values + + # Debug: log response Set-Cookie count + set_cookie_count = sum(1 for k, v in resp_headers_raw if k.lower() == "set-cookie") + resp_status = resp.status_code + body_preview = resp_body[:200].decode('utf-8', errors='replace') + logger.info(f"[FETCH_RESP] {method} {url[:60]} status={resp_status} set_cookies={set_cookie_count} body={body_preview}") + resp_cookies = {} if hasattr(resp, "cookies"): for k, v in resp.cookies.items(): resp_cookies[k] = v - # v4: Check for CF challenge + # CF challenge detection if is_cf_challenge(status, resp_body, resp_headers): - proxy.mark_challenged() - logger.warning(f"[CF_CHALLENGE] {host_port} -> {url[:60]}: {status} ({elapsed:.1f}s)") + logger.warning(f"[CF_CHALLENGE] {host_port} -> {url[:60]}: {status}") return { - "status": status, - "headers": resp_headers, - "body": resp_body, + "status": status, "headers": resp_headers, "body": resp_body, + "headers_raw": resp_headers_raw, "cookies": resp_cookies, - "success": False, - "error": f"CF_CHALLENGE_{status}", - "elapsed": elapsed, - "proxy_host": host_port, - "is_challenge": True, - "challenge_body": resp_body, + "success": False, "error": f"CF_CHALLENGE_{status}", + "elapsed": elapsed, "proxy_host": host_port, + "is_challenge": True, "challenge_body": resp_body, "challenge_proxy": host_port, } - # Success — any 2xx-4xx is proxied through + # Success if 200 <= status < 500: - proxy.mark_success(elapsed) - logger.debug(f"{host_port} -> {url[:60]}: {status} ({elapsed:.2f}s)") - - # v4: Save response cookies to proxy + sc_from_dict = resp_headers.get("Set-Cookie", resp_headers.get("set-cookie", "NONE")) + logger.info(f"[SAVE_COOKIES] proxy={host_port} set_cookie_dict={sc_from_dict[:120]}...") proxy.save_cookies(resp_headers) - return { - "status": status, - "headers": resp_headers, - "body": resp_body, + "status": status, "headers": resp_headers, "body": resp_body, + "headers_raw": resp_headers_raw, "cookies": resp_cookies, "success": True, - "elapsed": elapsed, - "proxy_host": host_port, - "is_challenge": False, - "challenge_body": None, - "challenge_proxy": None, + "elapsed": elapsed, "proxy_host": host_port, + "is_challenge": False, "challenge_body": None, "challenge_proxy": None, } - # True error status (5xx) - proxy.mark_failure() - logger.warning(f"Attempt: {host_port} -> {url[:60]}: {status}") + # Error status + logger.warning(f"{host_port} -> {url[:60]}: {status}") return { - "status": status, - "headers": resp_headers, - "body": resp_body, + "status": status, "headers": resp_headers, "body": resp_body, + "headers_raw": resp_headers_raw, "cookies": resp_cookies, - "success": False, - "error": f"HTTP_{status}", - "elapsed": elapsed, - "proxy_host": host_port, - "is_challenge": False, - "challenge_body": None, - "challenge_proxy": None, + "success": False, "error": f"HTTP_{status}", + "elapsed": elapsed, "proxy_host": host_port, + "is_challenge": False, "challenge_body": None, "challenge_proxy": None, } except asyncio.TimeoutError: - proxy.mark_failure() - elapsed = _now() - start_time - logger.warning(f"TIMEOUT: {host_port} -> {url[:60]} ({elapsed:.1f}s, timeout={request_timeout}s)") + elapsed = time.time() - start_time + logger.warning(f"TIMEOUT: {host_port} -> {url[:60]}") return { "status": 0, "headers": {}, "body": b"", "cookies": {}, + "headers_raw": [], "success": False, "error": "TIMEOUT", "elapsed": elapsed, "proxy_host": host_port, "is_challenge": False, "challenge_body": None, "challenge_proxy": None, } except Exception as e: - proxy.mark_failure() - elapsed = _now() - start_time + elapsed = time.time() - start_time err_str = str(e)[:120] - logger.debug(f"ERROR: {host_port} -> {e} ({elapsed:.1f}s)") + logger.debug(f"ERROR: {host_port} -> {err_str}") return { "status": 0, "headers": {}, "body": b"", "cookies": {}, + "headers_raw": [], "success": False, "error": err_str, "elapsed": elapsed, "proxy_host": host_port, "is_challenge": False, "challenge_body": None, "challenge_proxy": None, diff --git a/app.py b/app.py index e0909e7..6307822 100644 --- a/app.py +++ b/app.py @@ -1,13 +1,17 @@ """ -AO3 反代后端 v4 — Cookie 感知 + CF 挑战用户浏览器求解 +AO3 反代后端 v5 — 粘性会话 + Service Worker + P2C 代理池 + 并发限流 -v4 vs v3: -- Challenge token 映射:用户浏览器求解 CF 挑战时,保证同一 proxy 亲和性 -- 挑战页面透传:所有代理都遇到 CF 挑战时,把挑战页发给用户浏览器求解 -- Cookie 流:成功响应自动保存 proxy cookie,后续请求自动携带 -- 统计面板展示 cookie-aware 代理数 +v5 vs v4: +- 粘性会话: ao3_sessid_proxy cookie (对标 go3) +- Service Worker: /sw.js, /sw-YYYYMMDD.js for client-side caching +- HTML 注入: 自动注入 SW 注册脚本 + 镜像域名 +- P2C 代理选择: Power of Two Choices (对标 go3) +- 并发限流: asyncio.Semaphore(400) (Webshare 上限) +- 全量 5000 代理: 不再按端口过滤 +- 状态机: healthy/unstable/blocked/probing """ import hashlib +import html import json import logging import os @@ -23,9 +27,9 @@ import uvicorn sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from proxy_pool import get_proxy_pool -from ao3_fetcher import fetch_url, is_cf_challenge -from url_rewriter import rewrite_body, rewrite_response_headers, needs_rewrite, MIRROR_DOMAIN +from proxy_pool import get_proxy_pool, MAX_CONCURRENT_REQUESTS, OUTAGE_THRESHOLD +from ao3_fetcher import fetch_url, is_cf_challenge, FAST_REQUEST_TIMEOUT, NORMAL_REQUEST_TIMEOUT +from url_rewriter import rewrite_body, rewrite_response_headers, rewrite_response_headers_raw, needs_rewrite, MIRROR_DOMAIN from cache import get_cache, get_ttl_for_path from stats import get_stats_collector @@ -39,13 +43,15 @@ logger = logging.getLogger("ao3-backend") AO3_BASE = "https://archiveofourown.org" MIRROR_HOST = MIRROR_DOMAIN +# Service Worker version date (change on SW updates) +SERVICE_WORKER_DATE = "20260623" + LOCAL_PATHS = {"/stats", "/health", "/metrics", "/favicon.ico", "/robots.txt"} -# ─── Challenge Token Map (v4) ────────────────────────────────────────────── -# Maps challenge_token → (method, ao3_url, headers, body, cookies, proxy_host, expires) +# ─── Challenge Token Map (v4 retained) ────────────────────────────────────── _challenge_map: dict[str, tuple] = {} _challenge_lock = threading.Lock() -CHALLENGE_TOKEN_TTL = 120 # 2 minutes for the user's browser to solve the challenge +CHALLENGE_TOKEN_TTL = 120 def _make_challenge_token() -> str: @@ -55,7 +61,6 @@ def _make_challenge_token() -> str: def _store_challenge(token: str, method: str, ao3_url: str, headers: dict, body: bytes, cookies: dict, proxy_host: str): with _challenge_lock: - # Clean expired tokens now = time.time() expired = [k for k, v in _challenge_map.items() if v[6] < now] for k in expired: @@ -71,16 +76,38 @@ def _get_challenge(token: str) -> tuple | None: del _challenge_map[token] return entry if entry: - del _challenge_map[token] # expired + del _challenge_map[token] return None +# ─── Sticky Session ───────────────────────────────────────────────────────── + +STICKY_COOKIE_NAME = "ao3_sessid_proxy" + + +def get_sticky_proxy_idx(request: Request) -> tuple[int, bool]: + """Get sticky proxy index from cookie. Returns (idx, had_sticky).""" + c = request.cookies.get(STICKY_COOKIE_NAME) + if c is not None: + try: + idx = int(c) + pool = get_proxy_pool() + if 0 <= idx < len(pool.proxies): + return idx, True + except (ValueError, TypeError): + pass + # No sticky cookie — random + import random + pool = get_proxy_pool() + return random.randint(0, max(len(pool.proxies) - 1, 0)), False + + # ─── App ─────────────────────────────────────────────────────────────────── app = FastAPI( title="AO3 Mirror", description="AO3 reverse proxy mirror for Chinese users", - version="4.0.0", + version="5.0.0", docs_url=None, redoc_url=None, ) @@ -106,31 +133,51 @@ def build_ao3_url(path: str, query: str = "") -> str: # ─── Challenge page URL rewriting ────────────────────────────────────────── def _rewrite_challenge_page(body: bytes, proxy_host: str, token: str) -> bytes: - """Rewrite CF challenge page so all URLs go through mirror, and tag with challenge token.""" - import re - - # Basic AO3 domain rewrite + """Rewrite CF challenge page — inject retry UI + auto cookie.""" + # Rewrite domain body = body.replace(b"archiveofourown.org", MIRROR_DOMAIN.encode()) + # Rewrite CF challenge paths — our CF CDN would intercept /cdn-cgi/, + # so we remap to /__cf/ which our proxy handler forwards to AO3 + body = body.replace(b"/cdn-cgi/", b"/__cf/") + body = body.replace(b"cdn-cgi/challenge", b"__cf/challenge") - # Rewrite /cdn-cgi/ challenge endpoints to go through mirror - # These are CF's internal challenge platform URLs - body = body.replace( - b"/cdn-cgi/challenge-platform", - f"/cdn-cgi/challenge-platform".encode() + # Inject cookie + retry overlay + retry_overlay = ( + f''.encode() + + b'' + + b'
' + + 'AO3 Mirror - CF challenge solving in background. '.encode() + + 'Click to retry after a few seconds.'.encode() + + b'
' ) - - # Inject a marker meta tag so we can detect challenge pages coming back - marker = f''.encode() - marker_cookie = ( - f'' - ).encode() - body = body.replace(b"", b"" + marker + marker_cookie, 1) - if b"" not in body: - body = b"" + marker + marker_cookie + b"" + body - + body = body.replace(b" bytes: + """Inject SW registration script into HTML, before .""" + import re + sw_script = ( + b'' + ) + # Match with optional leading whitespace + match = re.search(rb'\s*', html_body) + if match: + pos = match.start() + return html_body[:pos] + sw_script + html_body[pos:] + # Fallback: insert at beginning if no head tag + return sw_script + html_body + + # ─── Response header helpers ─────────────────────────────────────────────── def filter_response_headers(headers: dict) -> dict: @@ -149,7 +196,7 @@ def filter_response_headers(headers: dict) -> dict: if key.lower().startswith("cf-"): continue result[key] = value - result["X-Proxy"] = "AO3-Mirror/4.0" + result["X-Proxy"] = "AO3-Mirror/5.0" result["X-Cache"] = "MISS" result["Cache-Control"] = "public, max-age=60, s-maxage=60" return result @@ -169,15 +216,12 @@ def add_cors(response: Response): async def health(): pool = get_proxy_pool() stats = pool.get_stats() - resp = JSONResponse({ + return JSONResponse({ "status": "ok", "timestamp": time.time(), - "version": "4.0.0", + "version": "5.0.0", "proxy_pool": stats, }) - resp.headers["Cache-Control"] = "no-store, no-cache, must-revalidate" - resp.headers["Pragma"] = "no-cache" - return resp @app.get("/robots.txt") @@ -187,6 +231,43 @@ async def robots(): ) +# ─── Service Worker Routes ───────────────────────────────────────────────── + +@app.get("/sw.js") +async def service_worker_redirect(): + """Redirect to versioned SW file.""" + return RedirectResponse(url=f"/sw-{SERVICE_WORKER_DATE}.js", status_code=302) + + +@app.get("/sw-{date}.js") +async def service_worker(date: str): + """Serve versioned Service Worker.""" + sw_path = os.path.join(os.path.dirname(__file__), "static", "sw.js") + try: + with open(sw_path, "rb") as f: + sw_content = f.read() + except FileNotFoundError: + return Response(status_code=404) + + return Response( + content=sw_content, + media_type="application/javascript", + headers={ + "Cache-Control": "public, max-age=86400, immutable", + "Service-Worker-Allowed": "/", + }, + ) + + +@app.get("/mirror-domains.json") +async def mirror_domains(): + """JSON endpoint for SW cross-domain awareness.""" + return JSONResponse({ + "domains": [MIRROR_DOMAIN], + "primary": MIRROR_DOMAIN, + }) + + # ─── Stats Dashboard ─────────────────────────────────────────────────────── @app.get("/stats", response_class=HTMLResponse) @@ -213,7 +294,7 @@ async def stats_page(): - AO3 Mirror - 统计面板 v4 + AO3 Mirror - 统计面板 v5
-

🛡 AO3 Mirror 统计面板 v4

+

🛡 AO3 Mirror 统计面板 v5

@@ -275,14 +357,14 @@ async def stats_page():
P99: {stats['p99_elapsed_ms']} ms
-
代理池
-
{proxy_stats['alive']}/{proxy_stats['total']}
-
可用: {proxy_stats['available']} | 受损: {proxy_stats['dead']} | Banned: {proxy_stats['banned']}
+
代理池 ({proxy_stats['total']})
+
{proxy_stats['healthy']}/{proxy_stats['total']}
+
不稳定: {proxy_stats['unstable']} | 封禁: {proxy_stats['blocked']} | 探测: {proxy_stats['probing']}
Cookie 代理
-
{proxy_stats['with_cookies']}/{proxy_stats['alive']}
-
已持有 cf_clearance
+
{proxy_stats['with_cookies']}/{proxy_stats['total']}
+
inflight: {proxy_stats['total_inflight']}
本地缓存
@@ -306,7 +388,7 @@ async def stats_page():

热门路径

- {''.join(f'' for p in stats['top_paths'][:15])} + {''.join(f'' for p in stats['top_paths'][:15])}
路径请求数成功平均延迟
{p["path"]}{p["requests"]:,}0.8 else "badge-red"}">{round(p["successful"]/max(p["requests"],1)*100)}%{p["avg_elapsed"]*1000:.0f}ms
{html.escape(p["path"])}{p["requests"]:,}0.8 else "badge-red"}">{round(p["successful"]/max(p["requests"],1)*100)}%{p["avg_elapsed"]*1000:.0f}ms
@@ -394,11 +476,13 @@ async def metrics(): f'ao3_mirror_avg_elapsed_ms {stats["avg_elapsed_ms"]}', "# HELP ao3_mirror_proxy_pool Proxy pool status", "# TYPE ao3_mirror_proxy_pool gauge", - f'ao3_mirror_proxy_alive {proxy_stats["alive"]}', - f'ao3_mirror_proxy_dead {proxy_stats["dead"]}', - f'ao3_mirror_proxy_banned {proxy_stats["banned"]}', - f'ao3_mirror_proxy_available {proxy_stats["available"]}', + f'ao3_mirror_proxy_total {proxy_stats["total"]}', + f'ao3_mirror_proxy_healthy {proxy_stats["healthy"]}', + f'ao3_mirror_proxy_unstable {proxy_stats["unstable"]}', + f'ao3_mirror_proxy_blocked {proxy_stats["blocked"]}', + f'ao3_mirror_proxy_probing {proxy_stats["probing"]}', f'ao3_mirror_proxy_with_cookies {proxy_stats["with_cookies"]}', + f'ao3_mirror_proxy_inflight {proxy_stats["total_inflight"]}', "# HELP ao3_mirror_cache_size Current cache size", "# TYPE ao3_mirror_cache_size gauge", f'ao3_mirror_cache_size {cache_stats["size"]}', @@ -406,11 +490,11 @@ async def metrics(): return PlainTextResponse("\n".join(lines)) -# ─── Proxy Core (v4) ────────────────────────────────────────────────────── +# ─── Proxy handler (v5) ────────────────────────────────────────────────── @app.api_route("/{path:path}", methods=["GET", "POST", "HEAD", "OPTIONS", "PUT", "DELETE", "PATCH"]) async def proxy_handler(request: Request, path: str): - """Main proxy handler — with cookie-aware fetching and CF challenge solving.""" + """Main proxy handler — sticky sessions + P2C selection + semaphore-limited.""" start_time = time.time() client_ip = get_client_ip(request) @@ -418,88 +502,93 @@ async def proxy_handler(request: Request, path: str): path = "" full_path = f"/{path}" if path else "/" - if full_path in LOCAL_PATHS or full_path.startswith("/stats") or full_path.startswith("/health"): + if full_path in LOCAL_PATHS or full_path.startswith("/sw"): return JSONResponse({"error": "Not found"}, status_code=404) + # Remap /__cf/ → /cdn-cgi/ (CF challenge paths bypass our CDN) + if full_path.startswith("/__cf/"): + path = "cdn-cgi" + path[4:] + full_path = "/" + path + if request.method == "OPTIONS": resp = Response() add_cors(resp) return resp - # ── v4: Challenge token check ────────────────────────────────────────── - # If this request has a _cf_token cookie, it's part of a challenge resolution flow + pool = get_proxy_pool() + query_string = request.url.query + ao3_url = build_ao3_url(f"/{path}" if path else "/", query_string) + + # ── v4: Challenge token check ──────────────────────────────────────── cf_token = request.cookies.get("_cf_token") or request.query_params.get("_cf_token") - preferred_proxy = None if cf_token: challenge_entry = _get_challenge(cf_token) if challenge_entry: orig_method, orig_url, orig_headers, orig_body, orig_cookies, proxy_host, _ = challenge_entry - preferred_proxy = proxy_host - logger.info(f"Challenge resolution: using proxy {proxy_host} for {orig_url[:60]}") - # Re-fetch the original request with the challenge proxy + logger.info(f"Challenge resolution: using proxy {proxy_host}") + challenge_proxy_obj = pool.get_by_addr(proxy_host) result = await fetch_url( url=orig_url, method=orig_method, + proxy=challenge_proxy_obj, headers=dict(request.headers), body=await request.body() if orig_method in ("POST", "PUT", "PATCH") else None, cookies=orig_cookies, is_api="/api/" in orig_url, - preferred_proxy=preferred_proxy, ) - if result["success"]: - # Challenge solved! Proxy now has cf_clearance. Return content. elapsed = time.time() - start_time rewritten_body = rewrite_body(result["body"], - result["headers"].get("Content-Type", "")) + result["headers"].get("Content-Type", result["headers"].get("content-type", ""))) + content_type = result["headers"].get("Content-Type", result["headers"].get("content-type", "")) + if "text/html" in content_type: + rewritten_body = inject_service_worker(rewritten_body) rewritten_headers = rewrite_response_headers(result["headers"]) final_headers = filter_response_headers(rewritten_headers) + final_headers.pop("Set-Cookie", None) + final_headers.pop("set-cookie", None) final_headers["X-Cache"] = "MISS" final_headers["X-CF-Status"] = "solved" - # Set cf_clearance as a cookie on the mirror domain too - # so subsequent requests benefit resp = Response(content=rewritten_body, status_code=result["status"], headers=final_headers) - # Forward any Set-Cookie from AO3 (includes cf_clearance) - for k, v in result["headers"].items(): - if k.lower() == "set-cookie": - # Rewrite domain - v_rewritten = v.replace("domain=archiveofourown.org", - f"domain={MIRROR_DOMAIN}") - v_rewritten = v_rewritten.replace("domain=.archiveofourown.org", - f"domain=.{MIRROR_DOMAIN}") - resp.headers.add("Set-Cookie", v_rewritten) - # Clean up challenge token cookie + # Append ALL Set-Cookie from raw (preserves multi-value) + # Filter: strip AO3 CF cookies + SKIP_CF = {"__cf_bm", "_cfuvid"} + ao3_headers_raw = result.get("headers_raw", []) + if ao3_headers_raw: + rewritten_raw = rewrite_response_headers_raw(ao3_headers_raw) + for key, value in rewritten_raw: + if key.lower() == "set-cookie": + cookie_name = value.split("=")[0].split(";")[0].strip() + if cookie_name in SKIP_CF: + continue + resp.headers.append("Set-Cookie", value) + resp.delete_cookie("_cf_token", path="/") collector = get_stats_collector() collector.log_request(method=orig_method, path=full_path, status=result["status"], elapsed=elapsed, - cached=False, proxy_host=preferred_proxy, + cached=False, proxy_host=proxy_host, client_ip=client_ip) add_cors(resp) return resp - # ── Normal request flow ──────────────────────────────────────────────── - - query_string = request.url.query - ao3_url = build_ao3_url(f"/{path}" if path else "/", query_string) - - # Check cache + # ── Static cache check (static assets only) ──────────────────── + from cache import is_static_path cache = get_cache() - cached = cache.get(ao3_url, dict(request.headers)) - if cached: - body, resp_headers, status = cached - elapsed = time.time() - start_time - collector = get_stats_collector() - collector.log_request(method=request.method, path=full_path, status=status, - elapsed=elapsed, cached=True, client_ip=client_ip) - headers = filter_response_headers(resp_headers) - headers["X-Cache"] = "HIT" - return Response(content=body, status_code=status, headers=headers) + if request.method == "GET" and is_static_path(full_path): + cached = cache.get(ao3_url) + if cached: + body, resp_headers, status = cached + elapsed = time.time() - start_time + collector = get_stats_collector() + collector.log_request(method=request.method, path=full_path, status=status, + elapsed=elapsed, cached=True, client_ip=client_ip) + return Response(content=body, status_code=status, headers=resp_headers) - # Prepare request + # ── Prepare request ────────────────────────────────────────────────── method = request.method client_headers = dict(request.headers) req_body = None @@ -514,101 +603,167 @@ async def proxy_handler(request: Request, path: str): k, v = pair.split("=", 1) cookies[k.strip()] = v.strip() - # Forward to AO3 - result = await fetch_url( - url=ao3_url, method=method, - headers=client_headers, body=req_body, cookies=cookies, - is_api="/api/" in full_path or full_path.startswith("/api/"), - preferred_proxy=preferred_proxy, - ) + # ── v5: Sticky proxy + P2C selection with semaphore ────────────────── + sticky_idx, had_sticky = get_sticky_proxy_idx(request) + final_proxy_idx = sticky_idx + tried_idxs: set[int] = set() + + import random + is_fast = method in ("POST", "PUT", "PATCH") or "/users/" in full_path + request_timeout = FAST_REQUEST_TIMEOUT if is_fast else NORMAL_REQUEST_TIMEOUT + + # Try with sticky first, then P2C fallback + has_session = bool(cookies) # Stateful request: has user cookies + session_cookie_val = cookies.get("_otwarchive_session", "")[:80] + logger.info(f"Request {method} {full_path} | cookies={list(cookies.keys())} | session={session_cookie_val}... | has_session={has_session}") + async with pool.concurrency_limiter: + # First attempt: sticky proxy + result = None + sticky_proxy = pool.sticky_select(sticky_idx, tried_idxs) + if sticky_proxy: + sticky_proxy.inflight += 1 + tried_idxs.add(sticky_proxy.idx) + try: + result = await fetch_url( + url=ao3_url, method=method, + proxy=sticky_proxy, + headers=client_headers, body=req_body, cookies=cookies, + is_api="/api/" in full_path, + ) + if result["success"]: + final_proxy_idx = sticky_proxy.idx + sticky_proxy.mark_success(result.get("elapsed", 0)) + sticky_proxy.inflight -= 1 + elif result.get("is_challenge"): + sticky_proxy.mark_challenged() + sticky_proxy.inflight -= 1 + else: + status = result.get("status", 0) + if status in (403, 525): + sticky_proxy.mark_blocked() + else: + sticky_proxy.mark_failure(is_network=True) + sticky_proxy.inflight -= 1 + result = None # Force retry + except Exception: + if sticky_proxy: + sticky_proxy.mark_failure(is_network=True) + sticky_proxy.inflight -= 1 + result = None + + # Retry loop with P2C — but NEVER for stateful requests + # If the sticky proxy failed on a request with cookies, switching + # proxies breaks CSRF/session affinity. Forward CF challenge instead. + if (not result or not result["success"]) and not has_session: + max_retries = 5 + for attempt in range(max_retries): + if attempt > 0: + await _sleep(0.1 * (2 ** min(attempt, 3))) + proxy = pool.p2c_select(exclude_idxs=tried_idxs) + if not proxy: + break + proxy.inflight += 1 + tried_idxs.add(proxy.idx) + try: + result = await fetch_url( + url=ao3_url, method=method, + proxy=proxy, + headers=client_headers, body=req_body, cookies=cookies, + is_api="/api/" in full_path, + ) + if result["success"]: + final_proxy_idx = proxy.idx + proxy.mark_success(result.get("elapsed", 0)) + proxy.inflight -= 1 + break + elif result.get("is_challenge"): + proxy.mark_challenged() + proxy.inflight -= 1 + else: + status = result.get("status", 0) + if status in (403, 525): + proxy.mark_blocked() + elif status > 0: + proxy.mark_failure(is_network=False) + else: + proxy.mark_failure(is_network=True) + proxy.inflight -= 1 + result = None + except Exception: + proxy.mark_failure(is_network=True) + proxy.inflight -= 1 + result = None elapsed = time.time() - start_time - # ── v4: CF Challenge handling ────────────────────────────────────────── - if result.get("is_challenge") and result.get("challenge_body"): - challenge_proxy = result.get("challenge_proxy", "unknown") - challenge_body = result["challenge_body"] + # ── All proxies exhausted — outage ─────────────────────────────────── + if not result or not result["success"]: + if result and result.get("is_challenge") and result.get("challenge_body"): + challenge_proxy = result.get("challenge_proxy", "unknown") + token = _make_challenge_token() + _store_challenge(token, method, ao3_url, client_headers, + req_body or b"", cookies, challenge_proxy) + rewritten_challenge = _rewrite_challenge_page( + result["challenge_body"], challenge_proxy, token + ) + logger.warning(f"CF Challenge via {challenge_proxy}, sending to browser") + collector = get_stats_collector() + collector.log_request(method=method, path=full_path, status=503, + elapsed=elapsed, cached=False, + proxy_host=challenge_proxy, client_ip=client_ip) + resp = HTMLResponse( + content=rewritten_challenge, + status_code=200, # 200 so browser renders and executes CF JS + headers={ + "X-CF-Challenge": "true", + "X-CF-Challenge-Proxy": challenge_proxy, + }, + ) + resp.set_cookie( + key="_cf_token", value=token, + path="/", max_age=CHALLENGE_TOKEN_TTL, + httponly=False, samesite="lax", + ) + resp.delete_cookie("cf_clearance", path="/") + add_cors(resp) + return resp - # Generate challenge token for proxy affinity - token = _make_challenge_token() - _store_challenge(token, method, ao3_url, client_headers, - req_body or b"", cookies, challenge_proxy) + # Check if we hit outage threshold + if pool.pool_unavailable_ratio() >= OUTAGE_THRESHOLD: + logger.error(f"OUTAGE: {pool.pool_unavailable_ratio():.1%} proxies unavailable") + return HTMLResponse( + content=_outage_page(), status_code=503 + ) - # Rewrite challenge page for user's browser - rewritten_challenge = _rewrite_challenge_page(challenge_body, challenge_proxy, token) - - logger.warning( - f"CF Challenge detected via {challenge_proxy} for {ao3_url[:80]}. " - f"Sending challenge to user browser (token={token[:8]}...)" - ) - - collector = get_stats_collector() - collector.log_request(method=method, path=full_path, status=503, - elapsed=elapsed, cached=False, - proxy_host=challenge_proxy, client_ip=client_ip) - - # Return challenge page with 503 status + token cookie - resp = HTMLResponse( - content=rewritten_challenge, - status_code=503, - headers={ - "X-CF-Challenge": "true", - "X-CF-Challenge-Proxy": challenge_proxy, - "Retry-After": "5", - }, - ) - resp.set_cookie( - key="_cf_token", value=token, - path="/", max_age=CHALLENGE_TOKEN_TTL, - httponly=False, # JS needs to read it - samesite="lax", - ) - resp.delete_cookie("cf_clearance", path="/") # Clear stale clearance - add_cors(resp) - return resp - - # ── Standard failure ─────────────────────────────────────────────────── - if not result["success"]: - logger.error(f"Failed to fetch {ao3_url[:80]}: {result.get('error', 'unknown')}") + logger.error(f"Failed: {ao3_url[:80]}: all retries exhausted") collector = get_stats_collector() collector.log_request(method=method, path=full_path, status=502, elapsed=elapsed, cached=False, client_ip=client_ip) + return HTMLResponse(content=_transient_error_page(), status_code=502) - error_html = f""" - -AO3 Mirror - 暂时不可用 - - -
-

🔄 正在尝试连接 AO3

-

镜像站正在尝试通过代理重新连接 AO3 服务器。

-

请稍后刷新页面重试。

- 刷新页面 -
- -""" - return HTMLResponse(content=error_html, status_code=502) - - # ── Success ──────────────────────────────────────────────────────────── + # ── Success ────────────────────────────────────────────────────────── ao3_status = result["status"] ao3_headers = result.get("headers", {}) + ao3_headers_raw = result.get("headers_raw", []) raw_body = result.get("body", b"") - content_type = ao3_headers.get("Content-Type", "") + content_type = ao3_headers.get("Content-Type", ao3_headers.get("content-type", "")) rewritten_body = rewrite_body(raw_body, content_type) + + # Inject Service Worker into HTML responses + if "text/html" in content_type: + rewritten_body = inject_service_worker(rewritten_body) + rewritten_headers = rewrite_response_headers(ao3_headers) final_headers = filter_response_headers(rewritten_headers) + # Remove Set-Cookie from dict — multi-value Set-Cookie from raw below + final_headers.pop("Set-Cookie", None) + final_headers.pop("set-cookie", None) final_headers["X-Cache"] = "MISS" - # Cache successful GET responses - if method == "GET" and 200 <= ao3_status < 400: - ttl = get_ttl_for_path(full_path) - cache.set(ao3_url, rewritten_body, rewritten_headers, ao3_status, ttl=ttl) + # Cache successful static GET responses (disk MD5 cache, like go3) + if method == "GET" and ao3_status == 200 and is_static_path(full_path): + cache.set(ao3_url, rewritten_body, rewritten_headers, ao3_status) # Handle redirects if 300 <= ao3_status < 400 and "location" in rewritten_headers: @@ -624,27 +779,94 @@ h1 {{ color: #990000; }} client_ip=client_ip) resp = Response(content=rewritten_body, status_code=ao3_status, headers=final_headers) - # Forward Set-Cookie from AO3 (user session cookies + cf_clearance) - for k, v in ao3_headers.items(): - if k.lower() == "set-cookie": - v_rewritten = v.replace("domain=archiveofourown.org", - f"domain={MIRROR_DOMAIN}") - v_rewritten = v_rewritten.replace("domain=.archiveofourown.org", - f"domain=.{MIRROR_DOMAIN}") - resp.headers.add("Set-Cookie", v_rewritten) + + # Append ALL Set-Cookie headers from raw (preserves multi-value — pitfall #18a) + # Filter: strip AO3 CF cookies — our own CF edge handles these + SKIP_COOKIES = {"__cf_bm", "_cfuvid"} + if ao3_headers_raw: + rewritten_raw = rewrite_response_headers_raw(ao3_headers_raw) + for key, value in rewritten_raw: + if key.lower() == "set-cookie": + cookie_name = value.split("=")[0].split(";")[0].strip() + if cookie_name in SKIP_COOKIES: + continue + logger.info(f"[SET_COOKIE_FWD] {value[:120]}...") + resp.headers.append("Set-Cookie", value) + + # Set sticky session cookie + resp.set_cookie( + key=STICKY_COOKIE_NAME, + value=str(final_proxy_idx), + path="/", + httponly=True, + samesite="lax", + ) + add_cors(resp) return resp +# ─── Error pages ─────────────────────────────────────────────────────────── + +def _outage_page() -> str: + return """ + +AO3 Mirror - 暂时不可用 + + +
+

⚠️ AO3 暂时不可用

+

所有代理目前都无法连接 AO3。可能 AO3 正在维护或大规模封锁。

+

请稍后重试。如果持续出现,请切换到数据流量。

+ 刷新页面 +
+ +""" + + +def _transient_error_page() -> str: + return """ + +AO3 Mirror - 连接失败 + + +
+

🔄 正在尝试连接 AO3

+

镜像站正在通过代理重新连接 AO3 服务器。

+

请稍后刷新页面重试。

+ 刷新页面 +
+ +""" + + +# ─── Helper ──────────────────────────────────────────────────────────────── + +async def _sleep(seconds: float): + import asyncio + await asyncio.sleep(seconds) + + # ─── Startup / Shutdown ──────────────────────────────────────────────────── @app.on_event("startup") async def startup(): - logger.info("AO3 Mirror backend v4 starting up...") - get_proxy_pool() + logger.info("AO3 Mirror backend v5 starting up...") + pool = get_proxy_pool() get_cache() get_stats_collector() - logger.info("AO3 Mirror backend v4 started (cookie-aware + CF challenge solving)") + logger.info(f"AO3 Mirror backend v5 started ({len(pool.proxies)} proxies, " + f"P2C selection, sticky sessions, SW deploy)") @app.on_event("shutdown") @@ -660,7 +882,7 @@ async def shutdown(): if __name__ == "__main__": port = int(os.environ.get("PORT", "8080")) workers = int(os.environ.get("WORKERS", "4")) - logger.info(f"Starting AO3 Mirror v4 on port {port} with {workers} workers") + logger.info(f"Starting AO3 Mirror v5 on port {port} with {workers} workers") uvicorn.run( "app:app", diff --git a/cache.py b/cache.py index c65cbfd..c06a0f0 100644 --- a/cache.py +++ b/cache.py @@ -1,123 +1,164 @@ """ -简单高效的缓存层 -- 内存 LRU 缓存,每个 worker 独立 -- 短 TTL 避免内容过时 -- 针对不同路径设置不同 TTL -""" +静态资源磁盘缓存 v5 — 对标 go3 +只缓存明确是静态资源的路径(CSS/JS/图片/字体)。 +动态 HTML / API / 登录流程 零缓存 — 直接透传。 +缓存文件以 MD5(URL) 命名,存储在 cache/ 目录。 + +go3 参考: staticExtensions map + MD5 cache +""" import hashlib +import logging +import os import threading import time -from collections import OrderedDict from typing import Optional +logger = logging.getLogger("ao3-cache") -class LRUCache: - """Thread-safe LRU cache with TTL support.""" +CACHE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "cache") - def __init__(self, capacity: int = 2000, default_ttl: int = 30): - self.capacity = capacity - self.default_ttl = default_ttl - self._cache: OrderedDict[str, tuple[float, bytes, dict, int]] = OrderedDict() - # (expiry_time, body, headers, status) - self._lock = threading.RLock() +# Only cache these extensions — everything else passes through +STATIC_EXTENSIONS = { + ".css", ".js", ".jpg", ".jpeg", ".png", ".gif", ".ico", + ".woff", ".woff2", ".ttf", ".svg", ".webp", +} - def _make_key(self, url: str, headers: Optional[dict] = None) -> str: - """Generate cache key from URL and key headers.""" - # Use URL + relevant headers - accept = "" - if headers: - accept = headers.get("Accept-Encoding", "") - raw = f"{url}|{accept}" - return hashlib.md5(raw.encode()).hexdigest() +# Path prefixes that are always static +STATIC_PATH_PREFIXES = ( + "/stylesheets/", "/javascripts/", "/images/", "/media/", + "/skins/", "/assets/", "/favicon.ico", +) - def get(self, url: str, headers: Optional[dict] = None) -> Optional[tuple[bytes, dict, int]]: - """Get cached response. Returns (body, headers, status) or None.""" - key = self._make_key(url, headers) - with self._lock: - if key not in self._cache: - return None - expiry, body, resp_headers, status = self._cache[key] - if time.time() > expiry: - del self._cache[key] - return None - # Move to end (most recently used) - self._cache.move_to_end(key) - return (body, resp_headers, status) +# Cache TTL: 7 days for static assets (like go3's static_cache_ttl_seconds) +STATIC_TTL = 604800 # 7 days - def set(self, url: str, body: bytes, headers: dict, status: int, - ttl: Optional[int] = None, request_headers: Optional[dict] = None): - """Store response in cache.""" - key = self._make_key(url, request_headers) - t = ttl if ttl is not None else self.default_ttl - expiry = time.time() + t - with self._lock: - self._cache[key] = (expiry, body, headers, status) - self._cache.move_to_end(key) - if len(self._cache) > self.capacity: - self._cache.popitem(last=False) +def is_static_path(path: str) -> bool: + """Check if a URL path points to a static asset.""" + path_lower = path.lower() + for ext in STATIC_EXTENSIONS: + if path_lower.endswith(ext): + return True + for prefix in STATIC_PATH_PREFIXES: + if path_lower.startswith(prefix): + return True + return False - def invalidate(self, url: str, headers: Optional[dict] = None): - """Remove a specific URL from cache.""" - key = self._make_key(url, headers) - with self._lock: - self._cache.pop(key, None) - def clear(self): - with self._lock: - self._cache.clear() +class StaticCache: + """Disk-based static file cache. No LRU, no TTL on dynamic content.""" - @property - def size(self) -> int: - with self._lock: - return len(self._cache) + def __init__(self): + os.makedirs(CACHE_DIR, exist_ok=True) + self._lock = threading.Lock() + self._hits = 0 + self._misses = 0 + + def _cache_path(self, url: str) -> str: + h = hashlib.md5(url.encode()).hexdigest() + return os.path.join(CACHE_DIR, h) + + def get(self, url: str) -> Optional[tuple[bytes, dict, int]]: + """Get cached static file. Returns (body, headers, status) or None.""" + if not is_static_path(url): + return None + path = self._cache_path(url) + try: + with self._lock: + if os.path.exists(path): + mtime = os.path.getmtime(path) + if time.time() - mtime < STATIC_TTL: + with open(path, "rb") as f: + body = f.read() + self._hits += 1 + # Minimal headers for static content + headers = { + "Content-Type": _guess_content_type(url), + "Cache-Control": f"public, max-age={STATIC_TTL}, immutable", + } + return (body, headers, 200) + else: + os.remove(path) + except Exception: + pass + self._misses += 1 + return None + + def set(self, url: str, body: bytes, headers: dict, status: int): + """Store static file on disk.""" + if not is_static_path(url): + return + if status != 200: + return + path = self._cache_path(url) + try: + with self._lock: + with open(path, "wb") as f: + f.write(body) + except Exception as e: + logger.debug(f"Cache write failed: {e}") def get_stats(self) -> dict: with self._lock: - return { - "size": len(self._cache), - "capacity": self.capacity, - "usage_pct": round(len(self._cache) / self.capacity * 100, 1) if self.capacity else 0, - } + try: + files = os.listdir(CACHE_DIR) + total_size = sum( + os.path.getsize(os.path.join(CACHE_DIR, f)) + for f in files + if os.path.isfile(os.path.join(CACHE_DIR, f)) + ) + except Exception: + files = [] + total_size = 0 + return { + "size": len(files), + "capacity": "unlimited", + "usage_pct": round(total_size / (1024 * 1024), 1), + "hits": self._hits, + "misses": self._misses, + } -# TTL 策略:不同路径不同缓存时间 -PATH_TTL = { - "/": 30, # 首页 30s - "/works": 60, # 作品列表 60s - "/chapters": 120, # 章节内容 120s - "/series": 60, # 系列 60s - "/collections": 60, - "/tags": 60, - "/users": 30, - "/pseuds": 30, - "/bookmarks": 60, - "/skins": 300, # CSS 皮肤缓存 5 分钟 - "/stylesheets": 300, - "/images": 600, # 图片缓存 10 分钟 - "/media": 600, - "/javascripts": 300, - "/api": 15, # API 响应 15s - "/external_links": 30, - # Default: 30s -} +def _guess_content_type(url: str) -> str: + url_lower = url.lower() + if url_lower.endswith(".css"): + return "text/css; charset=utf-8" + if url_lower.endswith(".js"): + return "application/javascript; charset=utf-8" + if url_lower.endswith(".png"): + return "image/png" + if url_lower.endswith(".jpg") or url_lower.endswith(".jpeg"): + return "image/jpeg" + if url_lower.endswith(".gif"): + return "image/gif" + if url_lower.endswith(".svg"): + return "image/svg+xml" + if url_lower.endswith(".ico"): + return "image/x-icon" + if url_lower.endswith(".woff"): + return "font/woff" + if url_lower.endswith(".woff2"): + return "font/woff2" + if url_lower.endswith(".ttf"): + return "font/ttf" + if url_lower.endswith(".webp"): + return "image/webp" + return "application/octet-stream" def get_ttl_for_path(path: str) -> int: - """Determine cache TTL based on URL path.""" - for prefix, ttl in PATH_TTL.items(): - if path.startswith(prefix): - return ttl - return 30 # default + """Only used for static path TTL. Returns 0 for dynamic paths.""" + return STATIC_TTL if is_static_path(path) else 0 -# 全局缓存实例 -_cache: Optional[LRUCache] = None +# ─── Singleton ────────────────────────────────────────────────────────────── + +_cache: Optional[StaticCache] = None -def get_cache() -> LRUCache: +def get_cache() -> StaticCache: global _cache if _cache is None: - _cache = LRUCache(capacity=5000, default_ttl=30) + _cache = StaticCache() return _cache diff --git a/cache/0b56e3783141862ec51eaed10f75076a b/cache/0b56e3783141862ec51eaed10f75076a new file mode 100644 index 0000000..47480a6 --- /dev/null +++ b/cache/0b56e3783141862ec51eaed10f75076a @@ -0,0 +1,8 @@ +/*! + * JavaScript Cookie v2.2.1 + * https://github.com/js-cookie/js-cookie + * + * Copyright 2006, 2015 Klaus Hartl & Fagner Brack + * Released under the MIT license + */ +!function(e){var n;if("function"==typeof define&&define.amd&&(define(e),n=!0),"object"==typeof exports&&(module.exports=e(),n=!0),!n){var t=window.Cookies,o=window.Cookies=e();o.noConflict=function(){return window.Cookies=t,o}}}(function(){function e(){for(var e=0,n={};e .module { + width: 48.5% +} + +.logged-in .splash > div:nth-of-type(odd) { + margin-left: 0; + margin-right: 1.5%; +} + +.logged-in .splash > div:nth-of-type(even) { + margin-left: 1.5%; + margin-right: 0; +} + +/* 18 zone searchbrowse */ + +form.filters { + width: auto; + min-width: 23%; + max-width: 24%; +} + +.filters fieldset { + margin-right: 0; +} + +form.filters dl { + margin-left: 0.25em; + margin-right: 0.25em; +} + +/* 21 userstuff */ + +#workskin { + margin: auto 1.5%; +} diff --git a/cache/17342e82684c267ba4f1e531738eb513 b/cache/17342e82684c267ba4f1e531738eb513 new file mode 100644 index 0000000..aa21dc5 Binary files /dev/null and b/cache/17342e82684c267ba4f1e531738eb513 differ diff --git a/cache/2f7d84701e69c2978451a4693dbf6b35 b/cache/2f7d84701e69c2978451a4693dbf6b35 new file mode 100644 index 0000000..35a2b24 Binary files /dev/null and b/cache/2f7d84701e69c2978451a4693dbf6b35 differ diff --git a/cache/306b4df560405cad691843c852971d0a b/cache/306b4df560405cad691843c852971d0a new file mode 100644 index 0000000..4260581 --- /dev/null +++ b/cache/306b4df560405cad691843c852971d0a @@ -0,0 +1 @@ +function setupFilterToggles(){var e=$j(".filters").find("dt.filter-toggle");e.each(function(){var e=$j(this).next().attr("id");$j(this).wrapInner('')}),$j("dt.tags button").on("click",function(){"false"==$j(this).attr("aria-expanded")?$j(this).attr("aria-expanded","true"):$j(this).attr("aria-expanded","false")})}function showFilters(){var e=$j(".filters").find("dd.expandable");e.each(function(e,t){var a=$j(t).find("input").filter('[value]:not([value=""])'),r=$j(t).attr("id"),n=$j("#toggle_"+r),i=$j('[aria-controls="'+r+'"]');a.each(function(e,a){$j(a).is(':checked, [type="text"]')&&($j(t).removeClass("hidden"),$j(n).removeClass("collapsed").addClass("expanded"),$j(i).attr("aria-expanded","true"))})})}function setupNarrowScreenFilters(){var e=$j("form.filters"),t=$j("#outer"),a=$j("#go_to_filters"),r=$j("#leave_filters");a.click(function(a){a.preventDefault(),e.removeClass("narrow-hidden"),t.addClass("filtering"),e.find(":focusable").first().focus(),e.trap()}),r.click(function(r){r.preventDefault(),t.removeClass("filtering"),e.addClass("narrow-hidden"),a.focus()})}$j(document).ready(function(){setupFilterToggles(),showFilters(),setupNarrowScreenFilters()}); diff --git a/cache/331a4a4d15181598c366618812c89376 b/cache/331a4a4d15181598c366618812c89376 new file mode 100644 index 0000000..5b3ae4d --- /dev/null +++ b/cache/331a4a4d15181598c366618812c89376 @@ -0,0 +1,76 @@ +/* MEDIA: speech */ + +.landmark { + opacity: 1; + height: auto; + width: auto; + font-size: 100%; + line-height: 0; + color: black; +} + +a.tag { + speak: no-punctuation; +} + +em { + voice-stress: moderate; +} + +strong, .warnings, .caution { + voice-stress: strong; +} + +cite { + voice-rate: 95%; +} + +acronym { + speak: spell-out; +} + +abbr { + content: attr(title); +} + +code, kbd, tt, samp { + voice-rate: 95%; +} + +.userstuff { + speak-numeral: continuous; +} + +.meta { + voice-rate: 110%; +} + +.blurb { + voice-rate: 105%; +} + +.userstuff h1, .userstuff h2.title { + voice-rate: 90%; + pause: 5ms 10ms; +} + +.chapter { + pause: 10ms 10ms; +} + +.userstuff dt { + voice-balance: leftwards; +} + +.userstuff dd { + voice-balance: rightwards; +} + +.navigation { + voice-stress: moderate; + speak: no-punctuation; +} + +.splash .browse li a:before { + speak: none; +} diff --git a/cache/34a22b671f9a37f8e9c84a7871acf424 b/cache/34a22b671f9a37f8e9c84a7871acf424 new file mode 100644 index 0000000..e017f95 --- /dev/null +++ b/cache/34a22b671f9a37f8e9c84a7871acf424 @@ -0,0 +1 @@ +!function(o){"use strict";var n="[data-toggle=dropdown]",t=function(n){var t=o(n).on("click.dropdown.data-api",this.toggle);o("html").on("click.dropdown.data-api",(function(){t.parent().removeClass("open")}))};function e(){o(n).each((function(){d(o(this)).removeClass("open")}))}function d(n){var t,e=n.attr("data-target");return e||(e=(e=n.attr("href"))&&/#/.test(e)&&e.replace(/.*(?=#[^\s]*$)/,"")),(t=e&&o(e))&&t.length||(t=n.parent()),t}t.prototype={constructor:t,toggle:function(n){var t,r,i=o(this);if(!i.is(".disabled, :disabled"))return r=(t=d(i)).hasClass("open"),e(),r?(t.children("ul").hide(),i.blur()):(t.toggleClass("open").children("ul").removeAttr("style"),i.focus()),i.focus(),!1},keydown:function(t){var e,r,i,a,s;if(/(38|40|27)/.test(t.keyCode)&&(e=o(this),t.preventDefault(),t.stopPropagation(),!e.is(".disabled, :disabled"))){if(!(a=(i=d(e)).hasClass("open"))||a&&27==t.keyCode)return 27==t.which&&i.find(n).focus(),e.click();(r=o("ul.menu li:not(.divider):visible a",i)).length&&(s=r.index(r.filter(":focus")),38==t.keyCode&&s>0&&s--,40==t.keyCode&&s +All rights reserved. + +Official repository: https://github.com/julienw/jquery-trap-input +License is there: https://github.com/julienw/jquery-trap-input/blob/master/LICENSE +This is version 1.2.0. +*/(function(e,t){function r(e){if(e.keyCode===9){var t=!!e.shiftKey;if(i(this,e.target,t)){e.preventDefault();e.stopPropagation()}}}function i(e,t,n){var r=a(e),i=t,s,o,u,f;do{s=r.index(i);o=s+1;u=s-1;f=r.length-1;switch(s){case-1:return false;case 0:u=f;break;case f:o=0;break}if(n){o=u}i=r.get(o);if(!i||i===t){return true}try{i.focus()}catch(l){return true}}while(r.length>1&&t===t.ownerDocument.activeElement);return true}function s(){return this.tabIndex>0}function o(){return!this.tabIndex}function u(e,t){return e.t-t.t||e.i-t.i}function a(t){var n=e(t);var r=[],i=0;h.enable&&h.enable();n.find("a[href], link[href], [draggable=true], [contenteditable=true], :input:enabled, [tabindex=0]").filter(":visible").filter(o).each(function(e,t){r.push({v:t,t:0,i:i++})});n.find("[tabindex]").filter(":visible").filter(s).each(function(e,t){r.push({v:t,t:t.tabIndex,i:i++})});h.disable&&h.disable();r=e.map(r.sort(u),function(e){return e.v});return e(r)}function f(){this.keydown(r);this.data(n,true);return this}function l(){this.unbind("keydown",r);this.removeData(n);return this}function c(){return!!this.data(n)}var n="trap.isTrapping";e.fn.extend({trap:f,untrap:l,isTrapping:c});var h={};if(e.find.find&&e.find.attr!==e.attr){(function(){function i(e){var r=e.getAttributeNode(n);return r&&r.specified?parseInt(r.value,10):t}function s(){r[n]=r.tabIndex=i}function o(){delete r[n];delete r.tabIndex}var n="tabindex";var r=e.expr.attrHandle;h={enable:s,disable:o}})()}})(jQuery); \ No newline at end of file diff --git a/cache/36f64a27d9f23f25aeaee0dba47e0872 b/cache/36f64a27d9f23f25aeaee0dba47e0872 new file mode 100644 index 0000000..06f3845 --- /dev/null +++ b/cache/36f64a27d9f23f25aeaee0dba47e0872 @@ -0,0 +1,7 @@ +/** + * Copyright (c) 2007 Ariel Flesler - aflesler ○ gmail • com | https://github.com/flesler + * Licensed under MIT + * @author Ariel Flesler + * @version 2.1.2 + */ +;(function(f){"use strict";"function"===typeof define&&define.amd?define(["jquery"],f):"undefined"!==typeof module&&module.exports?module.exports=f(require("jquery")):f(jQuery)})(function($){"use strict";function n(a){return!a.nodeName||-1!==$.inArray(a.nodeName.toLowerCase(),["iframe","#document","html","body"])}function h(a){return $.isFunction(a)||$.isPlainObject(a)?a:{top:a,left:a}}var p=$.scrollTo=function(a,d,b){return $(window).scrollTo(a,d,b)};p.defaults={axis:"xy",duration:0,limit:!0};$.fn.scrollTo=function(a,d,b){"object"=== typeof d&&(b=d,d=0);"function"===typeof b&&(b={onAfter:b});"max"===a&&(a=9E9);b=$.extend({},p.defaults,b);d=d||b.duration;var u=b.queue&&1=f[g]?0:Math.min(f[g],n));!a&&1 25 || list.data("force-contract")) { + list.hide(); + $(this).show(); + } else { + // show the shuffle and contract button only + $(this).nextAll(".shuffle").show(); + $(this).next(".contract").show(); + } + + // set up click event to expand the list + $(this).click(function(event){ + list = $($(this).data("action-target")); + list.show(); + + // show the contract & shuffle buttons and hide us + $(this).next(".contract").show(); + $(this).nextAll(".shuffle").show(); + $(this).hide(); + }); + }); + + $(".contract").each(function(){ + $(this).click(function(event){ + // hide the list when clicked + list = $($(this).data("action-target")); + list.hide(); + + // show the expand and shuffle buttons and hide us + $(this).prev(".expand").show(); + $(this).nextAll(".shuffle").hide(); + $(this).hide(); + }); + }); + + $(".shuffle").each(function(){ + // shuffle the list's children when clicked + $(this).click(function(event){ + list = $($(this).data("action-target")); + list.children().shuffle(); + }); + }); + + $(".expand_all").each(function(){ + target = "." + $(this).data("target-class"); + $(this).click(function(event) { + $(this).closest(target).find(".expand").click(); + }); + }); + + $(".contract_all").each(function(){ + target = "." + $(this).data("target-class"); + $(this).click(function(event) { + $(this).closest(target).find(".contract").click(); + }); + }); +}); + +// check all or none within the parent fieldset, optionally with a string to match on the id attribute of the checkboxes +// stored in the "data-checkbox-id-filter" attribute on the all/none links. +// allow for some flexibility by checking the next and previous fieldset if the checkboxes aren't in this one +jQuery(function($){ + $('.check_all').each(function(){ + $(this).click(function(event){ + var filter = $(this).data('checkbox-id-filter'); + var checkboxes; + if (filter) { + checkboxes = $(this).closest('fieldset').find('input[id*="' + filter + '"][type="checkbox"]'); + } else { + checkboxes = $(this).closest("fieldset").find(':checkbox'); + if (checkboxes.length == 0) { + checkboxes = $(this).closest("fieldset").next().find(':checkbox'); + if (checkboxes.length == 0) { + checkboxes = $(this).closest("fieldset").prev().find(':checkbox'); + } + } + } + checkboxes.prop('checked', true); + event.preventDefault(); + }); + }); + + $('.check_none').each(function(){ + $(this).click(function(event){ + var filter = $(this).data('checkbox-id-filter'); + var checkboxes; + if (filter) { + checkboxes = $(this).closest('fieldset').find('input[id*="' + filter + '"][type="checkbox"]'); + } else { + checkboxes = $(this).closest("fieldset").find(':checkbox'); + if (checkboxes.length == 0) { + checkboxes = $(this).closest("fieldset").next().find(':checkbox'); + if (checkboxes.length == 0) { + checkboxes = $(this).closest("fieldset").prev().find(':checkbox'); + } + } + } + checkboxes.prop('checked', false); + event.preventDefault(); + }); + }); +}); + +// Set up open and close toggles for a given object +// Typical setup (this will leave the toggled item open for users without javascript but hide the controls from them): +// +//
+// foo! +// +//
+// +// Notes: +// - The open button CANNOT be inside the toggled div, the close button can be (but doesn't have to be) +// - You can have multiple open and close buttons for the same div since those are labeled with classes +// - You don't have to use div and a, those are just examples. Anything you put the toggled and _open/_close classes on will work. +// - If you want the toggled item not to be visible to users without JavaScript by default, add the class "hidden" to the toggled item as well. +// (and you can then add an alternative link for them using