Task autoscaling¶
Choosing a good task limit for Stroom's stream processing is hard. Too many tasks and a node can become unresponsive or crash; too few and CPU sits idle. The right number varies through the day, and a static limit risks over-committing nodes during spikes such as backlog processing or several large searches.
A StroomTaskAutoscaler adjusts a Stroom job's task limit on each processing node to keep the node's CPU usage
inside a target range.
Requirements¶
- metrics-server, which provides Pod CPU usage.
- A CPU limit (
resources.limits.cpu) on each node set to be scaled. CPU usage is measured as a percentage of it, so node sets without one aren't scaled. - Database access from the operator Pod. Task limits are read and written directly in the Stroom database.
Create an autoscaler¶
apiVersion: stroom.gchq.github.io/v1
kind: StroomTaskAutoscaler
metadata:
name: dev
namespace: stroom
spec:
stroomClusterRef:
name: dev # namespace defaults to the autoscaler's
taskName: Data Processor
adjustmentIntervalMins: 1
metricsSlidingWindowMins: 5
minCpuPercent: 50
maxCpuPercent: 90
minTaskLimit: 1
maxTaskLimit: 8
stepAmount: 1
See also samples/operator-database/autoscaler.yaml.
Each autoscaler keeps its own per-node state, so a StroomCluster can have several, for example to scale
different jobs. Don't point two autoscalers at the same job on the same cluster, as each adjusts the task limit
without regard to the other.
| Field | Default | Description |
|---|---|---|
stroomClusterRef |
The cluster to scale. | |
taskName |
The Stroom job whose task limit is adjusted, usually Data Processor. Must match the name in Stroom's Jobs screen. |
|
adjustmentIntervalMins |
1 | How often each node's limit may change. |
metricsSlidingWindowMins |
1 | The window over which CPU usage is averaged. |
minCpuPercent |
50 | Below this, the limit is raised (subject to the rule below). |
maxCpuPercent |
90 | Above this, the limit is lowered. |
minTaskLimit, maxTaskLimit |
1, 20 | Bounds for the limit. |
stepAmount |
1 | How much the limit changes at each adjustment. |
How it works¶
The operator samples the CPU usage of the stroom-node container of every Pod in every node set that isn't
Frontend, about every 10 seconds, and keeps the samples in memory for up to an hour. Each time
adjustmentIntervalMins has passed for a node, it averages the samples from the last metricsSlidingWindowMins
and compares the result, as a percentage of the CPU limit, with the target range:
- Above
maxCpuPercent: the node's limit fortaskNameis lowered bystepAmount, but not belowminTaskLimit. - Below
minCpuPercent: the limit is raised bystepAmount, up tomaxTaskLimit, but only if the node is already running as many tasks as its limit allows. An idle node is left alone, so quiet periods don't push every node's limit up to the maximum. - Inside the range: nothing changes.
Each change is logged by the operator, for example
Updating task limit for node 'stroom-dev-node-data-0' from 4 to 5 due to CPU usage being at 42 percent.
Choosing values¶
adjustmentIntervalMins: often enough to react to load spikes, but not so often that each change has no time to take effect.metricsSlidingWindowMins: long enough that a momentary spike doesn't cut the limit, short enough that a sustained spike does.minCpuPercentandmaxCpuPercent: a fairly narrow range keeps the limit close to optimal.minTaskLimitandmaxTaskLimit: safety bounds. The number of CPUs in the limit is a reasonable starting point formaxTaskLimit. Be realistic: if a node can never reachmaxTaskLimitwithout overloading, the autoscaler may keep raising the limit until it does.
Notes¶
- Samples are held in the operator's memory, so they are lost if the operator restarts, and scaling resumes once enough new samples have been collected. In very large clusters, give the operator more memory; see Helm chart values.
- The autoscaler changes the same setting as the Jobs screen in Stroom, so manual changes to the job's task limit on a scaled node are overridden.
Disable autoscaling¶
Delete the autoscaler. Task limits stay at their last values.