akiba/agento3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Security Audit] Caddy timeout at exact backend limit causes intermittent 504 errors #4

New Issue
Closed
opened 2026-06-29 04:03:29 -05:00 by akiba · 7 comments
akiba commented 2026-06-29 04:03:29 -05:00
Owner
Copy Link

Severity: MEDIUM

Description

The Caddy reverse proxy read_timeout/write_timeout (30s) equals the backend's maximum fetch time (15s timeout x 2 retries = 30s), creating a race condition that causes intermittent 504 Gateway Timeout errors for slow requests.

Affected File

  • Caddyfile lines 39-40: read_timeout 30s, write_timeout 30s

Details

  • Backend normal path: 15s timeout, up to 2 retries = 30s maximum
  • Backend fast path: 8s timeout, up to 1 retry = 8s maximum
  • Caddy timeout: exactly 30s

When a request takes exactly 30s, the race between Caddy timeout and backend completion causes 504 errors. Per the project's own documented best practice, Caddy timeouts must be strictly greater than the backend's maximum.

Impact

Users experience intermittent 504 errors on slow pages (large works, search results), degrading availability and reliability.

Fix

Increase Caddy read_timeout and write_timeout to 60s in the Caddyfile to provide headroom above the 30s maximum backend time.

## Severity: MEDIUM ### Description The Caddy reverse proxy read_timeout/write_timeout (30s) equals the backend's maximum fetch time (15s timeout x 2 retries = 30s), creating a race condition that causes intermittent 504 Gateway Timeout errors for slow requests. ### Affected File - Caddyfile lines 39-40: read_timeout 30s, write_timeout 30s ### Details - Backend normal path: 15s timeout, up to 2 retries = 30s maximum - Backend fast path: 8s timeout, up to 1 retry = 8s maximum - Caddy timeout: exactly 30s When a request takes exactly 30s, the race between Caddy timeout and backend completion causes 504 errors. Per the project's own documented best practice, Caddy timeouts must be strictly greater than the backend's maximum. ### Impact Users experience intermittent 504 errors on slow pages (large works, search results), degrading availability and reliability. ### Fix Increase Caddy read_timeout and write_timeout to 60s in the Caddyfile to provide headroom above the 30s maximum backend time.
akiba added the bug label 2026-06-29 04:03:29 -05:00
akiba self-assigned this 2026-06-29 05:01:20 -05:00
akiba commented 2026-06-29 05:01:39 -05:00
Author
Owner
Copy Link

Auto-Triage Report for Issue #4

Status: Open
Existing Labels: bug (correct)
Assignee: @akiba (newly assigned)

Summary: [Security Audit] Caddy timeout at exact backend limit causes intermittent 504 errors. Severity: MEDIUM.

Assessment: Valid bug. Caddy read_timeout/write_timeout (30s) exactly equals the backend's max fetch time (15s timeout x 2 retries = 30s), creating a race condition. The documented best practice is that Caddy timeouts must be strictly greater than the backend's maximum.

Next Steps:

  1. Increase Caddy read_timeout and write_timeout to 60s in Caddyfile (lines 39-40)
  2. Deploy and verify 504s are eliminated
  3. Confirm with a slow-request test that 30s+ requests complete successfully
## Auto-Triage Report for Issue #4 **Status**: Open **Existing Labels**: bug (correct) **Assignee**: @akiba (newly assigned) **Summary**: [Security Audit] Caddy timeout at exact backend limit causes intermittent 504 errors. Severity: MEDIUM. **Assessment**: Valid bug. Caddy read_timeout/write_timeout (30s) exactly equals the backend's max fetch time (15s timeout x 2 retries = 30s), creating a race condition. The documented best practice is that Caddy timeouts must be strictly greater than the backend's maximum. **Next Steps**: 1. Increase Caddy read_timeout and write_timeout to 60s in Caddyfile (lines 39-40) 2. Deploy and verify 504s are eliminated 3. Confirm with a slow-request test that 30s+ requests complete successfully
akiba commented 2026-06-29 08:02:29 -05:00
Author
Owner
Copy Link

Triage Summary

Status: 🟡 Confirmed — Race condition causing intermittent 504 errors
Severity: MEDIUM
Priority: 🟠 High — affects user-facing reliability

Assessment

This is a legitimate race condition. Caddy's read_timeout and write_timeout are both set to 30s, which equals the backend's maximum fetch time (15s timeout × 2 retries = 30s). When a request takes exactly 30s, Caddy and the backend race — if Caddy wins, the user gets a 504 even though the backend completed successfully.

Recommended Next Steps

  1. Increase Caddy read_timeout and write_timeout to 60s in the Caddyfile (lines 39-40)
  2. This provides a 2× safety margin above the 30s maximum backend time
  3. Deploy and monitor 504 rates to confirm resolution

Triage Actions Taken

  • Labels: bug (already set) ✅
  • Assigned to: @akiba (already assigned) ✅
  • Fix is in Caddyfile — simple config change
## Triage Summary **Status:** 🟡 Confirmed — Race condition causing intermittent 504 errors **Severity:** MEDIUM **Priority:** 🟠 High — affects user-facing reliability ### Assessment This is a legitimate race condition. Caddy's `read_timeout` and `write_timeout` are both set to 30s, which equals the backend's maximum fetch time (15s timeout × 2 retries = 30s). When a request takes exactly 30s, Caddy and the backend race — if Caddy wins, the user gets a 504 even though the backend completed successfully. ### Recommended Next Steps 1. Increase Caddy `read_timeout` and `write_timeout` to 60s in the Caddyfile (lines 39-40) 2. This provides a 2× safety margin above the 30s maximum backend time 3. Deploy and monitor 504 rates to confirm resolution ### Triage Actions Taken - Labels: `bug` (already set) ✅ - Assigned to: @akiba (already assigned) ✅ - Fix is in Caddyfile — simple config change
akiba commented 2026-06-29 18:03:06 -05:00
Author
Owner
Copy Link

🤖 Triage Summary

Classification: Bug (MEDIUM severity)
Labels: bug ✅
Assignee: akiba ✅

Assessment: Real configuration bug — Caddy's read/write timeouts (30s) exactly equal the backend's maximum fetch time (15s × 2 retries = 30s), creating a race condition that causes intermittent 504 Gateway Timeout errors.

Recommended Next Steps:

  1. Increase Caddy read_timeout and write_timeout from 30s to 60s in the Caddyfile (lines 39-40)
  2. Deploy and verify — this is a low-risk, one-line config change
  3. Consider adding a comment in the Caddyfile documenting the relationship between backend timeout and proxy timeout

Note: Per the project's own best practice, Caddy timeouts must be strictly greater than the backend's maximum to avoid this race condition.

## 🤖 Triage Summary **Classification:** Bug (MEDIUM severity) **Labels:** bug ✅ **Assignee:** akiba ✅ **Assessment:** Real configuration bug — Caddy's read/write timeouts (30s) exactly equal the backend's maximum fetch time (15s × 2 retries = 30s), creating a race condition that causes intermittent 504 Gateway Timeout errors. **Recommended Next Steps:** 1. Increase Caddy read_timeout and write_timeout from 30s to 60s in the Caddyfile (lines 39-40) 2. Deploy and verify — this is a low-risk, one-line config change 3. Consider adding a comment in the Caddyfile documenting the relationship between backend timeout and proxy timeout **Note:** Per the project's own best practice, Caddy timeouts must be strictly greater than the backend's maximum to avoid this race condition.
akiba commented 2026-06-30 00:02:07 -05:00
Author
Owner
Copy Link

🔄 Follow-up Triage (2026-06-30)

Status: Still open. Labels (bug) and assignee (akiba) remain correct.

Action needed: Increase Caddy read_timeout/write_timeout from 30s → 60s in Caddyfile lines 39-40. Simple config change. No new comments or activity since last triage.

## 🔄 Follow-up Triage (2026-06-30) **Status:** Still open. Labels (`bug`) and assignee (akiba) remain correct. **Action needed:** Increase Caddy `read_timeout`/`write_timeout` from 30s → 60s in Caddyfile lines 39-40. Simple config change. No new comments or activity since last triage.
akiba commented 2026-06-30 04:02:01 -05:00
Author
Owner
Copy Link

🤖 Triage Summary

Status: Confirmed — legitimate MEDIUM severity bug.
Labels: bug ✅
Assignee: akiba ✅

Next Steps

  1. Increase read_timeout and write_timeout from 30s to 60s in Caddyfile lines 39-40
  2. Test with slow requests (near 30s response time) to verify the 504 errors are resolved
  3. Consider adding a brief comment explaining the timeout-buffer rationale

Severity Assessment

Correctly assessed as Medium. This causes intermittent availability degradation for slow endpoints. The fix is low-risk and should be straightforward.

## 🤖 Triage Summary **Status:** Confirmed — legitimate MEDIUM severity bug. **Labels:** bug ✅ **Assignee:** akiba ✅ ### Next Steps 1. Increase `read_timeout` and `write_timeout` from 30s to **60s** in Caddyfile lines 39-40 2. Test with slow requests (near 30s response time) to verify the 504 errors are resolved 3. Consider adding a brief comment explaining the timeout-buffer rationale ### Severity Assessment Correctly assessed as **Medium**. This causes intermittent availability degradation for slow endpoints. The fix is low-risk and should be straightforward.
akiba commented 2026-06-30 08:01:47 -05:00
Author
Owner
Copy Link

Triage Assessment: MEDIUM severity

This is a valid race-condition bug. When both retries are consumed, the backend takes exactly 30s — matching Caddy's read/write timeout and causing intermittent 504s.

Labels: bug ✅
Assignee: akiba ✅

Recommended next steps:

  • Increase Caddy timeout to 45s or 60s in Caddyfile (lines 39-40)
  • Alternatively, reduce backend max retries or add a small grace buffer in Caddy
  • Test with slow requests to confirm the fix eliminates the race
  • Priority: MEDIUM — should be addressed within current sprint
**Triage Assessment: MEDIUM severity** This is a valid race-condition bug. When both retries are consumed, the backend takes exactly 30s — matching Caddy's read/write timeout and causing intermittent 504s. **Labels:** bug ✅ **Assignee:** akiba ✅ **Recommended next steps:** - Increase Caddy timeout to 45s or 60s in Caddyfile (lines 39-40) - Alternatively, reduce backend max retries or add a small grace buffer in Caddy - Test with slow requests to confirm the fix eliminates the race - Priority: MEDIUM — should be addressed within current sprint
akiba commented 2026-06-30 09:04:44 -05:00
Author
Owner
Copy Link

FIXED — Commit 122c408

Fix: Changed all read_timeout 30s → read_timeout 60s and write_timeout 30s → write_timeout 60s in Caddyfile (9 blocks: SW routes, main proxy, stats subdomain, metrics, etc.).

This eliminates the race condition where Caddy's 30s timeout exactly matched the backend's max 30s (15s × 2 retries). With 60s, even if both retries are consumed, Caddy has a 30s safety buffer.

Deployed via start.sh — Caddy reloaded.

**FIXED** — Commit 122c408 Fix: Changed all `read_timeout 30s` → `read_timeout 60s` and `write_timeout 30s` → `write_timeout 60s` in Caddyfile (9 blocks: SW routes, main proxy, stats subdomain, metrics, etc.). This eliminates the race condition where Caddy's 30s timeout exactly matched the backend's max 30s (15s × 2 retries). With 60s, even if both retries are consumed, Caddy has a 30s safety buffer. Deployed via start.sh — Caddy reloaded.
akiba closed this issue 2026-06-30 09:04:44 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
priority:high
question
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
akiba
No Assignees akiba
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: akiba/agento3#4
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?