Performance Optimization Beyond Django (Gunicorn, Nginx, MariaDB, Elasticsearch, Redis)

This guide covers performance work outside Django application code for OLDP-style deployments.

It focuses on:

gunicorn (Python app server)
nginx (reverse proxy / compression / static assets)
MariaDB (primary relational DB)
Elasticsearch (search backend)
Redis (cache backend)
docker compose examples for local/prod-like setups

This document is intended as a practical checklist and starting point, not a one-size-fits-all config. Benchmark with your own traffic patterns.

1. Optimization Priorities (Recommended Order)

Start here before tuning individual services:

Measure baseline latency and throughput per endpoint group (/, /case/, /law/, /api/*, /search/*)
Separate cold vs warm cache behavior
Confirm where time is spent:
- app CPU
- DB queries
- Elasticsearch queries
- network / proxy overhead
Tune one layer at a time
Re-measure after each change

2. Gunicorn (App Server)

Goals

Keep worker processes busy but not overloaded
Avoid memory bloat / worker stalls
Reduce tail latency under burst traffic

Recommended baseline settings

Use gthread or sync workers (start simple)
Preload app if startup cost is high and memory sharing helps (--preload)
Limit worker lifetime to mitigate leaks (--max-requests, --max-requests-jitter)
Set sane timeouts (avoid very high defaults masking issues)

Example command (good starting point)

gunicorn oldp.wsgi:application \
  --bind 0.0.0.0:8000 \
  --workers 3 \
  --threads 4 \
  --worker-class gthread \
  --timeout 30 \
  --graceful-timeout 30 \
  --keep-alive 5 \
  --max-requests 1000 \
  --max-requests-jitter 100 \
  --preload \
  --access-logfile - \
  --error-logfile -

Sizing guidance

CPU-bound endpoints: fewer workers, fewer threads
I/O-bound endpoints (DB/search/cache): moderate threads help
Start with:
- workers ~= CPU cores
- threads = 2..4
Increase gradually while monitoring:
- p95 latency
- memory usage per worker
- worker restarts / timeouts

Common mistakes

Too many workers causing DB connection pressure
Very high timeout hiding slow dependencies
No max-requests in long-running processes with slow memory growth

3. Nginx (Reverse Proxy, Compression, Static Files)

Goals

Terminate client connections efficiently
Compress text responses
Serve static files directly
Protect Gunicorn from slow clients

Recommended optimizations

Enable gzip (and brotli if available in your image)
Cache static assets aggressively (immutable for hashed files)
Proxy buffering on
Keepalive between nginx and gunicorn
Set request/response timeouts explicitly

Example Nginx config (OLDP-oriented)

server {
    listen 80;
    server_name _;

    client_max_body_size 20m;

    gzip on;
    gzip_comp_level 5;
    gzip_min_length 1024;
    gzip_types
        text/plain
        text/css
        text/javascript
        application/javascript
        application/json
        application/xml
        application/rss+xml
        image/svg+xml;
    gzip_vary on;

    location /static/ {
        alias /srv/oldp/static/;
        access_log off;
        expires 30d;
        add_header Cache-Control "public, max-age=2592000, immutable";
    }

    location /media/ {
        alias /srv/oldp/media/;
        expires 1d;
        add_header Cache-Control "public, max-age=86400";
    }

    location / {
        proxy_pass http://app:8000;
        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 5s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        proxy_buffering on;
        proxy_buffers 16 16k;
        proxy_busy_buffers_size 64k;

        proxy_set_header Connection "";
    }
}

Optional: Nginx micro-cache (anonymous only)

If you later want edge/proxy caching for anon traffic:

Cache only GET/HEAD
Bypass when auth/session cookies are present
Start with very small TTL (1-10s)

This is powerful but should be a follow-up after validating Django-side cache correctness.

4. MariaDB (Database)

Goals

Keep hot indexes/data in memory
Avoid expensive disk flush behavior that is too strict for your workload
Support concurrent reads from API/page traffic

High-impact areas

InnoDB buffer pool size
Connection limits / thread behavior
Slow query logging
Correct indexes (still the biggest win after memory tuning)

Recommended MariaDB baseline (`my.cnf` snippet)

[mysqld]
innodb_buffer_pool_size = 1G
innodb_buffer_pool_instances = 1
innodb_log_file_size = 256M
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 1
max_connections = 200
thread_cache_size = 50
table_open_cache = 2048

# Observability
slow_query_log = 1
long_query_time = 0.5
log_queries_not_using_indexes = 0

Notes

innodb_buffer_pool_size should usually be the largest memory allocation on a DB host/container.
For dedicated DB hosts, common targets are 50-70% of RAM.
In containers, ensure memory limits and DB tuning agree (don’t allocate a 2G buffer pool in a 1G-limited container).

Django/MariaDB interaction tips

Keep CONN_MAX_AGE reasonable (persistent connections help, but can pin too many DB connections if gunicorn workers are oversized)
Tune gunicorn worker count and DB max_connections together
Use slow query logs to drive schema/index changes before over-tuning MariaDB globals

5. Elasticsearch (Search Backend)

Goals

Stable low-latency search queries
Sufficient heap without over-allocation
Predictable indexing/refresh behavior

High-impact settings

JVM heap size (Xms, Xmx)
Index shard/replica counts (especially for single-node)
Refresh interval (during heavy indexing)
Host kernel setting: vm.max_map_count

Recommended single-node dev/staging baseline

environment:
  - discovery.type=single-node
  - xpack.security.enabled=false
  - "ES_JAVA_OPTS=-Xms1g -Xmx1g"

Host requirement (important)

On the host (not usually in compose), set:

sudo sysctl -w vm.max_map_count=262144

Persist in /etc/sysctl.conf (or platform equivalent).

Index tuning guidance (OLDP-like search)

Single node:
- number_of_replicas: 0
Avoid too many shards for modest datasets
During bulk indexing:
- increase refresh_interval temporarily (e.g. 30s)
- restore after indexing

Performance troubleshooting checklist

Heap pressure / GC pauses
Slow queries by field / analyzer mismatch
Oversharding
Disk I/O saturation
Search timeouts causing fallback behavior in the app

6. Redis (Cache Backend)

Goals

Fast shared cache for Django page/API caching
Predictable eviction behavior
Optional queue/session support separation

Recommended uses in this stack

Django cache backend (page/API/query fragments)
Optional rate limiting / locks
Avoid mixing critical and disposable data in one Redis DB without policy planning

Redis config basics

Set maxmemory
Choose an eviction policy appropriate for cache usage
- good default for cache-only Redis: allkeys-lru
Disable AOF if using Redis only as ephemeral cache (depends on ops policy)

Example redis.conf snippet:

maxmemory 512mb
maxmemory-policy allkeys-lru
save ""
appendonly no

Django cache integration (example)

If/when switching from file/locmem to Redis in production:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://redis:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        },
        "KEY_PREFIX": "oldp",
    }
}

Operational notes

Monitor:
- hit rate
- memory usage
- evictions
- latency spikes
Keep cache key cardinality under control (host/language/path variance is correct, but measure growth)

7. Docker Compose Example (Production-Like Baseline)

This is an example docker compose stack showing the service relationships and performance-focused defaults. Adapt paths, secrets, and image tags.

`compose.performance.yaml`

services:
  app:
    build: .
    command: >
      gunicorn oldp.wsgi:application
      --bind 0.0.0.0:8000
      --workers 3
      --threads 4
      --worker-class gthread
      --timeout 30
      --graceful-timeout 30
      --keep-alive 5
      --max-requests 1000
      --max-requests-jitter 100
      --preload
      --access-logfile -
      --error-logfile -
    env_file:
      - .env
    environment:
      DJANGO_SETTINGS_MODULE: oldp.settings
      DJANGO_CONFIGURATION: ProdConfiguration
      DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY}
      DATABASE_URL: mysql://oldp:${MARIADB_PASSWORD}@db/oldp
      DJANGO_ELASTICSEARCH_URL: http://search:9200/
      DJANGO_FILE_CACHE_LOCATION: /var/tmp/django_cache
      # Example: enable Redis cache if configured in settings
      # REDIS_URL: redis://redis:6379/1
    depends_on:
      db:
        condition: service_healthy
      search:
        condition: service_started
      redis:
        condition: service_started
    expose:
      - "8000"
    volumes:
      - static_data:/srv/oldp/static
      - media_data:/srv/oldp/media
      - django_cache:/var/tmp/django_cache
    restart: unless-stopped

  nginx:
    image: nginx:1.27-alpine
    depends_on:
      - app
    ports:
      - "80:80"
    volumes:
      - ./deployment/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
      - static_data:/srv/oldp/static:ro
      - media_data:/srv/oldp/media:ro
    restart: unless-stopped

  db:
    image: mariadb:lts
    environment:
      MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
      MARIADB_DATABASE: oldp
      MARIADB_USER: oldp
      MARIADB_PASSWORD: ${MARIADB_PASSWORD}
    command:
      - --innodb-buffer-pool-size=1G
      - --innodb-log-file-size=256M
      - --max-connections=200
      - --slow-query-log=1
      - --long-query-time=0.5
    volumes:
      - mariadb_data:/var/lib/mysql
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s
    restart: unless-stopped

  search:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.12
    environment:
      cluster.name: oldp
      discovery.type: single-node
      xpack.security.enabled: "false"
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es_data:/usr/share/elasticsearch/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    command: >
      redis-server
      --maxmemory 512mb
      --maxmemory-policy allkeys-lru
      --save ""
      --appendonly no
    volumes:
      - redis_data:/data
    restart: unless-stopped

volumes:
  mariadb_data:
  es_data:
  redis_data:
  static_data:
  media_data:
  django_cache:

8. Docker Compose Tuning Notes

Resource limits (recommended)

Set explicit CPU/memory limits (compose syntax depends on runtime/orchestrator support):

Prevent Elasticsearch heap from exceeding container memory
Prevent MariaDB OOM kills under burst load
Avoid noisy-neighbor behavior in shared hosts

Separate persistent vs ephemeral volumes

Persistent:
- MariaDB data
- Elasticsearch data
- Media uploads
Ephemeral / disposable:
- Django file cache (if used)
- Redis cache-only data (optional persistence)

Healthchecks and startup ordering

Use healthchecks for:

MariaDB
Elasticsearch (optional curl http://localhost:9200/_cluster/health)

Do not assume depends_on alone means “ready for traffic”.

9. Observability and Load Testing (Recommended)

What to monitor

Nginx:
- request rate
- p95/p99 latency
- upstream response times
- 4xx/5xx rates
Gunicorn:
- worker restarts/timeouts
- memory per worker
MariaDB:
- slow queries
- buffer pool hit rate
- connections in use
Elasticsearch:
- heap %
- GC time
- query latency
- rejected requests
Redis:
- memory
- evictions
- hit ratio

Load test examples

Anonymous API:

hey -z 30s -c 20 http://localhost/api/laws/

Frontend cached page:

hey -z 30s -c 20 http://localhost/law/gg/

Search endpoint (with ES available):

hey -z 30s -c 10 'http://localhost/search/?q=gg'

10. Practical Rollout Strategy

Move from runserver to gunicorn behind nginx
Add Redis for shared Django cache (if not already)
Tune MariaDB memory and enable slow query logs
Tune Elasticsearch heap and verify host vm.max_map_count
Benchmark
Revisit Django-level cache TTLs and variance after observing real hit rates

11. Checklist (Quick Reference)

[ ] Gunicorn workers/threads sized for available CPU/RAM
[ ] Gunicorn max-requests enabled
[ ] Nginx gzip enabled
[ ] Nginx serves static files directly
[ ] MariaDB slow query log enabled
[ ] MariaDB InnoDB buffer pool sized appropriately
[ ] Elasticsearch heap set explicitly
[ ] Host vm.max_map_count configured
[ ] Redis maxmemory + eviction policy set
[ ] Docker volumes split by persistence needs
[ ] Baseline and post-change benchmarks captured

Performance Optimization Beyond Django (Gunicorn, Nginx, MariaDB, Elasticsearch, Redis)

1. Optimization Priorities (Recommended Order)

2. Gunicorn (App Server)

Goals

Recommended baseline settings

Example command (good starting point)

Sizing guidance

Common mistakes

3. Nginx (Reverse Proxy, Compression, Static Files)

Goals

Recommended optimizations

Example Nginx config (OLDP-oriented)

Optional: Nginx micro-cache (anonymous only)

4. MariaDB (Database)

Goals

High-impact areas

Recommended MariaDB baseline (my.cnf snippet)

Notes

Django/MariaDB interaction tips

5. Elasticsearch (Search Backend)

Goals

High-impact settings

Recommended single-node dev/staging baseline

Host requirement (important)

Index tuning guidance (OLDP-like search)

Performance troubleshooting checklist

6. Redis (Cache Backend)

Goals

Recommended uses in this stack

Redis config basics

Django cache integration (example)

Operational notes

7. Docker Compose Example (Production-Like Baseline)

compose.performance.yaml

8. Docker Compose Tuning Notes

Resource limits (recommended)

Separate persistent vs ephemeral volumes

Healthchecks and startup ordering

9. Observability and Load Testing (Recommended)

What to monitor

Load test examples

10. Practical Rollout Strategy

11. Checklist (Quick Reference)

Recommended MariaDB baseline (`my.cnf` snippet)

`compose.performance.yaml`