> ## Documentation Index
> Fetch the complete documentation index at: https://docs.curator.interworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Queued Processes 

> View, cancel, and clear background jobs running in Curator from the Queued Processes backend list.

Curator runs many tasks in the background -- syncing users, warming caches, importing data, cleaning up old
records, and more. Each background task is recorded as a "queued process" so you can see what is currently
running, what is waiting to run, and what has errored out. The Queued Processes list is the place to monitor
and manage that queue.

## Viewing Queued Processes

1. Navigate to your Curator backend > Settings > Curator > Queued Processes.
2. The list displays each job with its type, status (for example, "Pending", "Running", "Complete", or an
   error status), and timestamps for when it was created and last updated.
3. Use the "Refresh List" button in the toolbar to reload the list and see the latest status.

## Clearing Queued Processes

The toolbar above the list provides three actions for removing jobs from the queue:

| Action | What it does |
| - | - |
| Cancel Selected | Deletes the jobs you have checked in the list. Disabled until at least one row is checked. |
| Clear Pending | Removes every job currently in the "Pending" status. |
| Clear Errors | Removes every job currently in an error status. |

Use "Cancel Selected" when you want to remove a small number of specific jobs without affecting the rest of
the queue. Use "Clear Pending" when pending jobs are stuck or piling up and you want a clean slate before the
next scheduled run. Use "Clear Errors" once you have reviewed the failures and no longer need the records for
troubleshooting.

Each button asks for confirmation before deleting. Removing a job only deletes the queue record -- it does not
undo any work the job already performed.

## Automatic Cleanup

Curator automatically prunes old queued process records once per day so the list does not grow indefinitely.
The daily cleanup task removes any job whose last update was more than three days ago, regardless of status.
Recent jobs -- including pending ones -- are left in place so that in-flight work is not interrupted.

If you want to remove jobs sooner than the three-day window, use the toolbar actions described above.

## Troubleshooting a Queue That Is Not Processing

A healthy queue empties quickly. Curator's scheduled task runs once a minute, picks up every job whose "Run After"
time has passed, and works through them one at a time. Pending jobs normally disappear within a minute or two of
being created. If the Pending count keeps growing, work through the steps below in order.

### Step 1: Confirm the scheduled task is firing

Every background job in Curator depends on a server-level scheduled task (cron on Linux, Task Scheduler on
Windows) that runs `artisan schedule:run` once a minute. If that task stops, nothing in the queue moves.

1. Open the backend Dashboard and find **Last Cron Run** in the **Curator Status** widget. It should read a minute
   or two ago. A warning icon appears once the value is more than 24 hours old, and the Dashboard also shows a
   "Curator's Job System (cron) is not correctly configured" warning.
2. If the timestamp is old or missing, fix the scheduled task before doing anything else. See
   [Linux Cron Troubleshooting](https://docs.curator.interworks.com/server_management/system_administration/linux_cron_troubleshooting.md) or
   [Windows Cron Troubleshooting](https://docs.curator.interworks.com/server_management/system_administration/windows_cron_troubleshooting.md).

Firing the **Portal** > **cron** API endpoint by hand (see "Test Cron" in the Linux guide) runs a single pass of
the schedule. A "success" response only means Curator received the request and finished that pass. It does not
prove the server task is firing on its own, and one pass processes about a minute of queued work at most. Use
the **Last Cron Run** timestamp, not the endpoint response, to judge whether the schedule is healthy.

### Step 2: Check whether one job is holding the queue

Only one queue worker runs at a time, and it processes jobs one after another. A single long job, such as a full
backup of a large webroot, a large user sync, or a big import, keeps every job behind it in Pending until it
finishes.

1. Look for a job whose status is not "Pending". A job the worker has picked up shows "Claimed", a step
   description such as "Step 1 of 3: Backing up the Curator filesystem.", or a percentage.
2. Watch its **Updated At** value across a few refreshes. If it keeps advancing, the job is still working. Leave
   it alone; the Pending jobs follow once it finishes. Some jobs, such as backups, only update their status when
   they move to the next step, so a long gap on its own does not mean the job is dead.
3. If the job appears stuck, open the Event Log (**Settings** > **Logs** > **Event Log**) and search for
   "Curator's queue shut down prematurely". That entry means the worker died while running the job, for example
   because it ran out of memory or was killed by the operating system, and it includes the underlying error. Fix
   the cause if the entry names one, then remove the dead job with **Cancel Selected**. Curator requeues import
   jobs on its own; every other job type keeps its last status until you remove it or the daily cleanup does. A
   dead job does not block the queue by itself, so the remaining Pending jobs continue on the next pass.

### Step 3: Wait out the queue lock

Curator uses a lock to stop two queue workers from running at the same time. When a worker is killed without
warning, the lock can outlive it, and every pass until it expires skips the queue even though the scheduled
task is firing. The lock expires on its own, normally 60 minutes after the run started.

If you cannot wait, you can release the lock yourself, but only once you have confirmed that the worker that
took it is no longer running. Use the checks in Step 2: the job it was running has stopped advancing its
**Updated At** value, and the Event Log holds a "Curator's queue shut down prematurely" entry for it. If either
check fails, the worker may still be working, so wait for the lock to expire instead.

Once you are sure the worker is dead, call the
[/portal/flushAllCache](https://docs.curator.interworks.com/curator_api/api_docs/curator_api.md#portalflushallcache) API endpoint. **Clear Cache**
in the backend does not release the lock; this endpoint does. It works on both Curator SaaS and On-Prem:

```text theme={null}
[your_domain]/api/v1/portal/flushAllCache?apikey=[your_api_key_here]
```

To generate the link with your key filled in, follow the
[Auto Generate API Links](https://docs.curator.interworks.com/curator_api/getting_started/curator_api_overview.md) instructions and set the dropdowns
to **Portal** and **flushAllCache**. The next scheduled pass, normally within a minute, starts a fresh worker.
The endpoint flushes the whole cache, so the next few page loads are slower while Curator rebuilds it and
compiles the theme CSS again.

> [!WARNING]
>
> Do not remove the lock while a worker is still active. The next scheduled pass would start a second worker
> beside it. The two workers never pick up the same job, because each job is claimed with a single atomic
> database update, but they do process different jobs at the same time, which is exactly what the lock exists
> to prevent. Two backups, for example, could then run at once.

### Step 4: Turn on Cron Debug Mode

If the schedule is firing and nothing above explains the backlog, turn on Cron Debug Mode to see what the
worker does on each pass.

1. Navigate to **Settings** > **Curator** > **Portal Settings** and open the **General** tab.
2. In the **Debug Settings** section, turn on **Cron Debug Mode** and click **Save**. The mode turns itself off
   after 24 hours unless you change **Expiration of Cron Debug Mode**.
3. Wait a couple of minutes, then open the Event Log and look for entries that begin with "CRON DEBUG MODE".

Each pass logs "Updating cron last run timestamp" when the schedule fires and "Running queued task" each time
the worker picks up a job. If the timestamp entries appear but no "Running queued task" entries follow, the
worker is not reaching the queue and the lock in Step 3 is the most likely cause. If a "Running queued task"
entry appears and nothing follows it, that job is the one holding the queue.

### Step 5: Clear the backlog

When a job fails with an error, Curator writes the full error to the Event Log as "Error while running queued
process" and, for most job types, removes the job from the list. Jobs that stay in the list with a status
beginning with "Error:" can be removed with **Clear Errors** once you have read the log.

If the queue is draining again but the backlog is too large to be useful, or the pending work is no longer
wanted, use **Clear Pending** to remove every Pending job and start fresh.

> [!WARNING]
>
> **Clear Pending** deletes every Pending job. Work that a schedule queued, such as a scheduled backup, a user
> sync, or cache warming, is queued again at its next scheduled time. Work that a person started, such as an
> import or a backup taken with **Take New Backup**, is not, so start it again once the queue is draining.

## Troubleshooting Backups That Stay Pending or Never Appear

Curator runs every full backup through this queue. A scheduled backup adds a "Curator Backup" job at the
configured time, and **Take New Backup** on the Backups page adds the same job immediately. If the queue is not
processing, the backup waits in Pending with everything else, so work through the section above first. Two
backups queued back to back run one after the other, never at the same time.

While a backup runs, its status moves through "Step 1 of 3: Backing up the Curator filesystem.", "Step 2 of 3:
Backing up the Curator database." and "Step 3 of 3: Packaging the full backup file." Step 1 can take a long time
on a large webroot. When the backup finishes, the job is removed from the list and the new file appears under
**Settings** > **Curator** > **Backups**.

If the job disappears from the list but no new backup appears, or the new file name ends in `_NO_DATABASE`, the
backup failed part way through. Curator writes the reason to the Event Log. The common causes are:

**`ERROR: Not enough space for full backup.`**

Free disk space on the server is smaller than the size of the last backup. Free up space or lower
**Backup Retention**, then take a new backup. See [Curator Backup](https://docs.curator.interworks.com/upgrading_migration/backups/curator_backup.md).

**`ERROR: Unable to backup Curator files: <path> is not writable.`**

The web-server user cannot write to the **Backup Location** (`<web root>/storage/temp` by default). Grant the
web-server user write permission to that folder, or point **Backup Location** at a folder it can write to.

**`Unable to find mysqldump executable. Check that it has been added to PATH environment variable.`**

`mysqldump` is not on the PATH of the account that runs the scheduled task. The backup still completes, but
without the database, and the file name ends in `_NO_DATABASE`. Install the MySQL client tools and add them to
the PATH of the account that runs `schedule:run` (the Task Scheduler account on Windows), then take a new
backup.

**`ERROR: Unable to dump Curator database during backup`**

`mysqldump` ran but failed, or produced a file smaller than 100 KB. The log entry lists the dump file path and
size. Confirm the database credentials in Curator's configuration work from the Curator server, and that the
installed `mysqldump` matches the database server version.

A backup whose name ends in `_NO_DATABASE` shows "Does not include a database export!" in the Backups list and
cannot restore the database. Fix the database cause and take a new backup.

On Curator SaaS, InterWorks manages the server, its disk space, and backup storage. If a scheduled backup stays
Pending after the queue is confirmed healthy, or the Event Log shows one of the errors above, contact InterWorks
support.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.