Troubleshooting¶
The examples use the namespace stroom, the cluster dev and the node stroom-dev-node-data-0.
Where to look¶
The operator¶
The operator logs what it creates and updates, and any errors reconciling your resources:
Resources and events¶
kubectl -n stroom get stroomclusters,databaseservers,databasebackups,stroomtaskautoscalers
kubectl -n stroom get statefulsets,pods,pvc,ingresses -l stroom/cluster=dev
kubectl -n stroom describe pod stroom-dev-node-data-0 # probe failures, FailedPreStopHook, scheduling
kubectl -n stroom get events --sort-by=.lastTimestamp
Stroom's console log¶
kubectl -n stroom logs stroom-dev-node-data-0 -c stroom-node -f
kubectl -n stroom logs stroom-dev-node-data-0 -c stroom-node --previous # the container before its last restart
Log files in the node¶
These are on the node's PersistentVolumeClaim, so they survive restarts:
| File | Contents |
|---|---|
/stroom/logs/k8s/node-start.log |
The startup script: node and jobs enabled |
/stroom/logs/k8s/pre-stop.log |
The preStop drain: jobs and node disabled, remaining task counts |
/stroom/logs/k8s/api.log |
Failed Stroom API and token requests made by either script |
/stroom/logs/app/app.log |
Stroom application log |
/stroom/logs/access/access.log |
HTTP access log |
/stroom/logs/user/user.log |
User (audit) events |
/stroom/logs/migration/migration.log |
Database migration SQL |
The app, access and user logs are rolled every minute (and removed once shipped, if
log shipping is on). For example:
kubectl -n stroom exec stroom-dev-node-data-0 -c stroom-node -- tail -n 50 /stroom/logs/k8s/api.log
kubectl -n stroom exec stroom-dev-node-data-0 -c stroom-node -- ls -l /stroom/logs/app
Health checks¶
# Every health check, with details
kubectl -n stroom exec stroom-dev-node-data-0 -c stroom-node -- \
curl -s 'http://localhost:8081/stroomAdmin/healthcheck?pretty=true'
# Exactly what the liveness probe sees
kubectl -n stroom exec stroom-dev-node-data-0 -c stroom-node -- \
curl -s -w '\nHTTP %{http_code}\n' 'http://localhost:8081/stroomAdmin/filteredhealthcheck?allow=deadlocks'
Common problems¶
No StatefulSets are created¶
Check the operator log. Common causes:
DatabaseServer '...' could not be found:spec.databaseServerRef.serverRefnames aDatabaseServerthat doesn't exist in that namespace. The operator keeps retrying, with increasing back-off.DatabaseServer ... already claimed by StroomCluster ...: eachDatabaseServerserves only one cluster.- A Kubernetes validation error when updating a StatefulSet: you changed an immutable field. See Applying changes.
Nodes restart during startup¶
The startup probe ran out of time. Look at kubectl describe pod for Startup probe failed events, then:
/stroom/logs/k8s/api.log: token or API errors (see below).- The console log: is Stroom still migrating the database, or waiting for the bootstrap lock? If so, raise
startupProbeTimings.failureThreshold; see Health probes. - The console log: did Stroom fail to start, for example because of a configuration error or no database connection?
Authentication failures in api.log¶
Token request ... failed or responses of 401 or 403 mean the scripts can't authenticate:
- Is Stroom configured with
identityProviderType: EXTERNAL_IDP? The internal IdP isn't supported. See Identity provider. - Is the IdP reachable from the Pod?
- Are the client ID and secret in
spec.openIdcorrect, and does the client allow the client credentials grant? - Does the token's audience match what Stroom expects?
- Does the identity behind the token have permission in Stroom to manage nodes and jobs?
To test the token request from inside a node (this prints only the HTTP status, not the token):
kubectl -n stroom exec stroom-dev-node-data-0 -c stroom-node -- bash -c '
curl -s -o /dev/null -w "HTTP %{http_code}\n" -X POST \
-H "Content-Type: application/json" -H "Accept: text/plain" \
-d "{\"clientId\": \"$STROOM_OPERATOR_OPENID_CLIENT_ID\", \"clientSecret\": \"$STROOM_OPERATOR_OPENID_CLIENT_SECRET\"}" \
http://localhost:8080/api/authproxy/v1/fetchClientCredsToken'
Nodes are Running but not Ready¶
The readiness probe is failing. Run the filtered health check the probe uses (from kubectl describe pod, under
Readiness) and see which check is unhealthy. If you changed allow or deny, check that every name is
registered: an unknown name always fails the probe.
Nodes restart repeatedly after starting¶
The liveness probe is failing. If livenessProbeTimings.allow or deny contains a name Stroom doesn't register,
the probe can never pass; fix the name. Otherwise check the health check output for deadlocked threads, and the
previous container's log (--previous).
MySQL can't change the data directory owner¶
The DatabaseServer Pod restarts repeatedly, and its log ends with:
[ERROR] [MY-010174] [Server] Can't change data directory owner to mysql
[ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable.
This happens when the Kubernetes node's host has MySQL installed and AppArmor enabled, as on GitHub's Ubuntu
runners or a developer's machine running kind. AppArmor attaches the host's profile for /usr/sbin/mysqld to
mysqld in containers too, and the profile doesn't let the mysql/mysql-server image's mysqld, which runs as root,
take ownership of a new data directory. Unload the profile on the host:
To keep it unloaded after a reboot, also link it into /etc/apparmor.d/disable/.
A Pod is stuck terminating¶
It's probably draining. Check /stroom/logs/k8s/pre-stop.log for the remaining task count. The Pod is killed when
nodeTerminationPeriodSecs runs out. To stop it sooner, see Force deletion.
Deleting a StroomCluster or DatabaseServer hangs¶
StroomCluster: the operator is waiting for processing tasks to finish; the operator log names the nodes and task counts. The check runs once a minute.DatabaseServer: it waits for theStroomClusterusing it to be deleted.- Either: is the operator running? Its finalizers block deletion until it handles them.
The autoscaler does nothing¶
- Is metrics-server installed?
kubectl top pod -n stroomshould show usage. - Does the node set have
resources.limits.cpu? - Look for
PodMetrics not foundandCPU usage for Stroom nodein the operator log. - Limits are only raised on nodes already running as many tasks as their limit allows.
Containers are OOMKilled¶
Lower memoryOptions.maxPercentage or raise the memory limit. See
Resources and JVM memory.
The Ingress returns 404 or 502¶
- Does
ingress.classNamematch an IngressClass in the cluster? Always set it: the operator sends the value even when it's empty, and the API server may reject an Ingress with an empty class name (look for Ingress errors in the operator log). - With
spec.https, the Ingress must connect to the nodes over HTTPS; see Default annotations. - Are the nodes
Ready? Un-ready nodes are removed from the Service.