Common Issues

Celery Crashes with the following exception:

amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

This could be caused by long running tasks in Celery not acknowledging a task before RabbitMQ's default timeout. You can determine this by reviewing the results of the most recent tasks in the Celery logs. e.g. (note: stdout and stderr were clipped for brevity):

[2022-11-30 10:55:45,405: INFO/ForkPoolWorker-2] Task autopkg:run_recipe[b03cf2ba-9b50-4443-94ea-5f6570a0752e] succeeded in 3708.8848277970683s: {
    'event': 'recipe_run_dev',
    'event_id': None,
    'recipe_id': 'local.XcodeIDE',
    'success': False, 
    'stdout': 'Processing local.XcodeIDE... <...>',
    'stderr': '<...>'
}

Specifically, notice `succeeded in 3708.8848277970683s` (aka 61.8 minutes) while the default timeout is (30 minutes as of RabbitMQ v3.8.17). To prevent this, increase RabbitMQ's ACK time:

• Create a rabbitmq.conf in the proper directory for the method used to install RabbitMQ
• Set the `consumer_timeout` key to a value that works for your environment

You can confirm the configuration by running the following command:

• rabbitmqctl eval 'application:get_env(rabbit, consumer_timeout).'