Skip to content

Rollout Failures#

Sometimes a build may fail with a rollout failure and it may appear like this 

Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...
error: deployment "nginx" exceeded its progress deadline

Analyse log output#

There is a large amount of information displayed after the error which is very useful to help try and figure out what has gone wrong.

An example of this output is as follows, lets break it down peice by peice.

##############################################
STEP Applying Deployments: Failed at 2024-09-16 15:57:00 (UTC)
The information below could be useful in helping debug what went wrong
##############################################
Rollout for nginx failed, tried to gather some startup logs of the containers, hope this helps debugging:
======== /tmp/kubectl-build-deploy/logs/container/nginx/nginx-476fb88445-f9cw7-php.log =========

======== /tmp/kubectl-build-deploy/logs/container/nginx/nginx-476fb88445-f9cw7-nginx.log =========

##############################################
If there is any additional information about the status of pods, it will be available here
POD/SERVICE NAME    STATUS  CONDITION   MESSAGE
nginx-476fb88445-f9cw7  Running PodReadyToStartContainers   
nginx-476fb88445-f9cw7  Running Initialized 
nginx-476fb88445-f9cw7  Running Ready   containers with unready status: [nginx]
nginx-476fb88445-f9cw7  Running ContainersReady containers with unready status: [nginx]
nginx-476fb88445-f9cw7  Running PodScheduled    
nginx-79d7ccf956-v4wxk  Running PodReadyToStartContainers   
nginx-79d7ccf956-v4wxk  Running Initialized 
nginx-79d7ccf956-v4wxk  Running Ready   
nginx-79d7ccf956-v4wxk  Running ContainersReady 
nginx-79d7ccf956-v4wxk  Running PodScheduled

Captured container logs#

The build will try to capture the output of the containers that failed to start. These can usually help pinpoint why the failure occured, and hopefully help you to resolve the issue. Some examples of how these appear with available and missing logs are shown below. Each application is different, so sometimes the log output may not be as clear to read.

Container logs are available#

In this example, the error says that the command the container is trying to run is not found. Running this locally should yield a similar result. You may need to check the entrypoint or ensure the application is installed correctly.

##############################################
Rollout for node failed, tried to gather some startup logs of the containers, hope this helps debugging:
Terminated
======== /tmp/kubectl-build-deploy/logs/container/node/node-87798f576-d6r6x-node.log =========
2024-07-23T11:51:28.320396391Z /bin/sh: [node,scripts/ignition.js,--start]: not found

##############################################

Container logs are empty#

In some cases like the example below, the containers may not present any logs.

##############################################
Rollout for nginx failed, tried to gather some startup logs of the containers, hope this helps debugging:
======== /tmp/kubectl-build-deploy/logs/container/nginx/nginx-476fb88445-f9cw7-php.log =========

======== /tmp/kubectl-build-deploy/logs/container/nginx/nginx-476fb88445-f9cw7-nginx.log =========

##############################################

These ones can be tricky to figure out what has gone wrong, but its likely one of the common reasons below.

Pod status#

The final section of the log describes the status of the pods in the environment. This is mostly informational, but it can help to identify the state at the time the failure occured

In this example, the nginx service is a Lagoon nginx-php service type. This means the pod contains both the nginx and php containers. 

##############################################
If there is any additional information about the status of pods, it will be available here
POD/SERVICE NAME    STATUS  CONDITION   MESSAGE
nginx-476fb88445-f9cw7  Running PodReadyToStartContainers   
nginx-476fb88445-f9cw7  Running Initialized 
nginx-476fb88445-f9cw7  Running Ready   containers with unready status: [nginx]
nginx-476fb88445-f9cw7  Running ContainersReady containers with unready status: [nginx]
nginx-476fb88445-f9cw7  Running PodScheduled    
nginx-79d7ccf956-v4wxk  Running PodReadyToStartContainers   
nginx-79d7ccf956-v4wxk  Running Initialized 
nginx-79d7ccf956-v4wxk  Running Ready   
nginx-79d7ccf956-v4wxk  Running ContainersReady 
nginx-79d7ccf956-v4wxk  Running PodScheduled

From this log we can see this line containers with unready status: [nginx]. The output shows that the nginx container is the only one presenting as unready in the pod.

In this scenario, it is very likely that the nginx container has something wrong with its configuration. Running the environment locally should help identify the issue in the container log output.

Common reasons#

  • Configuration is wrong and the application fails to start.
  • The application may be starting on one port, but the liveness/readiness probes are checking a different port, depends on the Lagoon service type used.
  • Possibly permission issue trying to access parts of the filesystem that are read only, or incorrect permissions.
  • Wrong command, or incorrect command structure in container.

In most cases you should be able to replicate the reason by running the environment locally. If that doesn't help it may be related to small differences between local and cloud, in these cases, reach out to support to help diagnose.

Contact Support#

If the output doesn't help point out what has gone wrong, and you aren't able to replicate the issue locally, you may need to contact support to help check what has gone wrong. Make sure you provide a link to the build log from the dashboard when submitting your request.