How it works

Every incoming HTTP request is intercepted and classified based on the User-Agent header:

• Bad bots (scrapers, crawlers, aggressive indexers) → blocked immediately with 403 Forbidden.
• Allowed bots (Googlebot, Bingbot, etc.) → during peak hours (04:00–17:00 UTC), subject to a very strict rate limit (default: 5 requests/minute) instead of being blocked. This allows legitimate crawlers to continue working at a reduced pace without impacting server performance.
• Authenticated users (non-bot) → pass through without any rate limiting.
• Public users / unrecognized bots → rate limited to 200 requests/minute per IP address. Exceeding this threshold returns 429 Too Many Requests.

Static assets (/web/static/, /web/assets/) are excluded from rate limiting to avoid penalizing legitimate crawlers fetching CSS/JS resources.

Configuration via robots.txt

The bot lists are configurable directly from the Odoo backend at Website → Configuration → robots.txt by adding special comment directives in the # custom # section:

# restrict-ip: allowed_bot = googlebot, bingbot, facebookexternalhit, keycdn, meta-externalagent
# restrict-ip: bad_bot = spider, crawler, scrapy, ahrefsbot, bytespider, mj12bot, amazonbot, dotbot, claudebot, semrushbot, petalbot
# restrict-ip: rate_limit = 200
# restrict-ip: bot_peak_rate_limit = 5
# restrict-ip: blocked_ip = 1.2.3.4, 5.6.7.8

All directives are optional. If not present, built-in defaults are used automatically. The blocked_ip directive accepts one or more IP addresses separated by commas. Any request from a blocked IP immediately receives 403 Forbidden, regardless of the User-Agent. Blocked IPs are logged at WARNING level, including the IP address and the User-Agent string. The configuration is cached using Odoo’s native ormcache mechanism and is invalidated automatically whenever the website registry is cleared (e.g., after saving changes in the Odoo backend).

Memory & Performance

• Rate limiting counters are stored in memory (no DB writes per request) for maximum performance.
• IP entries are automatically cleaned up when all their requests expire, preventing memory leaks in long-running workers with many unique visitors.
• Rate limit threshold is configurable via robots.txt directive (rate_limit); default is 200 req/min.
• Bot config is cached using Odoo’s native ormcache per website — invalidated automatically on registry clear.

Relationship with robots.txt

robots.txt is a voluntary convention — well-behaved bots respect it, aggressive bots ignore it. This module provides active, mandatory enforcement at the server level, complementing robots.txt:

Table of contents

Improvements

• Cleanup of expired IP entries now runs at the beginning of _dispatch (before bot detection and rate limiting) — ensures request_count dict is cleaned even when requests are rejected early during flood attacks.

Improvements

• Replaced global _last_cleanup_time reassignment with _cleanup_state dict mutation — no global keyword needed, cleaner and easier to test.

Improvements

• Replaced _request_total_count counter + global with time-based cleanup: runs every 60 seconds (_CLEANUP_INTERVAL_SECONDS) instead of every 500 requests — more predictable and avoids counter overflow.

Improvements

• Authenticated users always pass through without rate limiting (removed bot_status == "not_bot" condition — a logged-in user is never rate-limited regardless of user-agent).
• Periodic cleanup now runs for all public requests including allowed bots — previously skipped for allowed_bot requests.

Improvements

• Flood protection cap is now dynamic: max_stored = rate_limit * 2 instead of hardcoded 400 — avoids blocking legitimate IPs when rate_limit is configured above 200.

Improvements

• Flood protection: recent list is now capped at 400 entries (2× the default rate limit) to prevent a single IP from exhausting worker RAM with thousands of requests per second.

Improvements

• Static asset check (/web/static/, /web/assets/) moved to the very beginning of _dispatch, before bot detection and cache access — reduces overhead for the most frequent request type.

Improvements

• Eliminated double iteration over request_count per request: recent list is now built once and reused for both allowed-bot peak-hour throttling and standard rate limiting.

Improvements

• Allowed bots during peak hours now receive a strict rate limit (default: 5 req/min, configurable via # restrict-ip: bot_peak_rate_limit = 5) instead of a blanket 503 block — crawlers continue at reduced pace without impacting server performance.

Improvements

• Replaced manual in-memory cache (_bot_config_cache with TTL) with Odoo’s native ormcache per website — cache is invalidated automatically on registry clear instead of expiring after a fixed TTL.
• Removed cache_ttl directive from robots.txt configuration (no longer needed).

Improvements

• Bad bots now return 403 Forbidden instead of 404 Not Found — semantically correct and consistent with security best practices.
• Fixed datetime.now() called twice: now is computed once, threshold derived from it.

Improvements

• Periodic cleanup of expired IP entries (every 500 requests) instead of at every request — reduces CPU overhead on high-traffic sites.
• Fixed rate limiting bug: current request is now always counted, even when previous entries were all expired.

Improvements

• cache_ttl and rate_limit are now configurable via robots.txt directives (# restrict-ip: cache_ttl = 5 / # restrict-ip: rate_limit = 200); defaults remain 5 min and 200 req/min.

Features

• Bot detection and rate limiting via ir.http._dispatch: bad bots blocked (403), allowed bots throttled during peak hours (503 + Retry-After), public users limited to 200 req/min per IP (429).
• Bot lists configurable from robots.txt custom directives (# restrict-ip: allowed_bot = ... / # restrict-ip: bad_bot = ...).
• Memory leak fix: IP keys in request_count are deleted when all requests expire.

Improvements

• Replaced deprecated datetime.utcnow() with timezone-aware datetime.now(timezone.utc).
• Lowered rate limiting threshold from 1000 to 200 requests/minute.
• Bot detection logic moved from terrabit_agroamat into this generic module.

Tests

• Standard Odoo tests (TransactionCase) in tests/test_restrict_ip.py covering bot detection, cache TTL, rate limiting, peak hours, and memory leak fix.