Concurrency tuning

Included free on every install.

Your instance hands background work to a shared pool of workers, runs some work in parallel, such as cache warmup and dashboard queries, and can run the hooks that follow a save in the background. Concurrency tuning shows how busy each of these is and lets a super admin resize them without a restart.

Best for

• Explaining slow requests when the database is not the cause. See whether the pool has no free workers, or work is waiting for one.
• Finding work that never ran. dropped counts tasks whose caller gave up before a worker was free.
• Taking after-save hooks out of the request. Switch async hooks on and watch the queue.
• Sizing for a traffic event. Grow the pool for the day, then let a restart put it back.

Use something else when

• You want to know which endpoint is slow, or where its CPU time goes. Request profiling answers that.
• The instance itself is overloaded. Shedding load and adding replicas are on scale and tune.

How it works

Part	What it does	Default
Worker pool	Runs the background work that features hand off.	100 workers, 30 second task timeout
Parallel engine	Runs a batch of tasks side by side, such as cache warmup and dashboard queries.	4 at once, 30 second timeout
Async hooks	Runs after-save hooks on the pool instead of inside the request.	Off, 8 workers, a queue of 1024, 5 second timeout
Tracker	Counts the background tasks the instance runs, by name and by feature.

Each replica has its own set, and a change made over the API applies to the replica that received it until it restarts. There are no per-tenant limits.

Try it

You need an admin token in TOKEN. The quickstart shows how to get one.

1. Read the worker pool:

   curl http://localhost:3001/api/admin/debug/goroutines/pool \
     -H "Authorization: Bearer $TOKEN"

   { "size": 100, "active": 0, "waiting": 0, "completed": 3, "failed": 0, "dropped": 0, "avg_latency": "127.377427ms", "task_timeout": "30s" }

2. Read the parallel engine:

   curl http://localhost:3001/api/admin/debug/goroutines/parallel \
     -H "Authorization: Bearer $TOKEN"

   { "max_concurrent": 4, "timeout": "30s" }

3. Read the async hook queue:

   curl http://localhost:3001/api/admin/debug/goroutines/async-hooks \
     -H "Authorization: Bearer $TOKEN"

   { "enabled": false, "workers": 8, "queue_size": 1024, "timeout": "5s", "queued": 0, "running": 0, "overflow": 0, "dropped": 0 }

4. As a super admin, grow the pool. The answer is the new state:

   curl -X PUT http://localhost:3001/api/admin/debug/goroutines/pool \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"size": 200}'

5. In the admin console, open Insight > Observability. The Goroutine engine section shows the same views, with forms for the changes.

Read the numbers

Field	Means
Pool active, waiting	Tasks running now, and tasks waiting for a free worker. A steady waiting above 0 means the pool is too small.
Pool failed	Tasks that ran and failed. A bigger pool does not help.
Pool dropped	Tasks that never started, because their caller gave up before a worker was free. Read this before you raise the size.
Async hooks overflow	Hooks that found the queue full and ran inside the request instead.
Status by_owner	Live tasks and the oldest one's age, per feature. A count that only grows is a leak.

Change a setting

Writes need a super admin. The pool write needs size. In the other two, a field you leave out keeps its value. Durations are strings such as 30s or 5m. Each write answers the new state and is recorded in the audit log with the values before and after.

Request	Body	Allowed values
PUT /api/admin/debug/goroutines/pool	{"size": 200}	size from 1 to 4096.
PUT /api/admin/debug/goroutines/parallel	{"max_concurrent": 8, "timeout": "30s"}	max_concurrent from 1 to 1024. timeout above 0, at most 1h.
PUT /api/admin/debug/goroutines/async-hooks	{"workers": 8, "queue_size": 1024, "timeout": "5s", "enabled": true}	workers from 1 to 1024. queue_size from 1 to 65536. timeout above 0, at most 10m.

Growing the pool lets waiting work start at once. Shrinking it lets running work finish. Switching async hooks on or off applies from the next hook.

Beta

Concurrency tuning is in beta. A change made over the API is lost on restart, so set the matching environment variable too when it should last.

Settings

These set the starting values.

Variable	What it does	Default
GOROUTINE_ENGINE_ENABLED	Applies GOROUTINE_ENGINE_POOL_SIZE, lets ASYNC_HOOKS_ENABLED take effect, and runs cache warmup and shutdown in parallel.	false
GOROUTINE_ENGINE_POOL_SIZE	Workers in the pool. Must be positive. Read only when GOROUTINE_ENGINE_ENABLED=true.	100
ASYNC_HOOKS_ENABLED	Runs after-save hooks in the background from the start. Needs GOROUTINE_ENGINE_ENABLED=true.	false
ASYNC_HOOK_TIMEOUT	How long one async hook may run.	5s
DB_WARMUP_PARALLELISM	How many tasks the parallel engine runs at once. Must be positive.	4

A value that does not parse stops the boot with a message naming the variable, and so does a pool size or parallelism that is not positive. An ASYNC_HOOK_TIMEOUT of zero or less keeps the 5s default. Concurrency tuning runs on every install. If you set LYEVE_PLUGINS to choose which features start, include goroutine-engine in it. See licensing and tiers.

Routes

Method	Path	Who	Answers
GET	/api/admin/debug/goroutines/status	Admin	total, runtime_total, by_source, by_owner, max_goroutines, leak_threshold.
GET	/api/admin/debug/goroutines/pool	Admin	size, active, waiting, completed, failed, dropped, avg_latency, task_timeout.
GET	/api/admin/debug/goroutines/parallel	Admin	max_concurrent, timeout.
GET	/api/admin/debug/goroutines/async-hooks	Admin	enabled, workers, queue_size, timeout, queued, running, overflow, dropped.
PUT	/api/admin/debug/goroutines/pool	Super admin	Resize the pool.
PUT	/api/admin/debug/goroutines/parallel	Super admin	Change the parallel engine.
PUT	/api/admin/debug/goroutines/async-hooks	Super admin	Change async hooks.

Errors

Status	Message
400	invalid JSON, also sent for a field the route does not take.
409	async hooks need a worker pool, and this install runs without one
422	nothing to change: set max_concurrent or timeout (or workers, queue_size, timeout or enabled)
422	timeout must be a duration such as 30s
422	A value outside the allowed range. The message states the range.

Related

• Request profiling: goroutine counts, CPU profiles and the slowest endpoints.
• Scale and tune: load shedding, replicas and the connection pool.
• Troubleshooting: logs, request ids and tracing.