Understanding the Cross-Site Task Concurrency Limit

Cross-Site Task Concurrency Limit is a feature that prevents background tasks from deadlocking when multiple sites in a shared database attempt to run the same or conflicting tasks simultaneously. It provides two complementary mechanisms:
  • Cross-Site Concurrency Limit - controls how many instances of the same task can run simultaneously across all sites in a database
  • Cross-Site Task Exclusion - prevents conflicting tasks, such as Planning and Scheduling, from running simultaneously across different sites.

Who should use this

  • System administrators managing Mongoose environments with multiple sites sharing a single database.
  • Application administrators within Mongoose Application products like APS administrators configuring Planning, Scheduling, and APSResyncAll tasks that overlap and cause deadlocks.

Prerequisites

  • Multiple sites configured in a shared database
  • Access to the BGTaskDefinitions form

Key benefits

  • In environments with many sites sharing a single database, background tasks such as Planning, Scheduling, and APSResyncAll could run simultaneously across different sites, causing deadlocks. This feature adds cross-site task concurrency limits and cross-site task exclusion rules, allowing administrators to control how many instances of the same task can run across all sites and to prevent conflicting tasks from executing simultaneously across sites.
  • The added support for cross-site background task concurrency limits and task exclusion rules, administrators can now control how many instances of a background task run simultaneously across all sites sharing a database and define tasks that must not run simultaneously. When task conflicts occur, blocked tasks automatically retry after the conflicting task completes, helping to reduce deadlocks and improve task execution reliability in multi-site environments.
  • You will benefit from a more reliable background task execution across sites, with automatic prevention of task conflicts and deadlocks, reduced administrative overhead, and improved operational stability from this feature.

Concepts

Per-Site vs. Cross-Site

Scope Exisiting Behavior New behavior after Cross-Site Task Concurrency Limit has been added
Per-site MaxConcurrent limits task instances within one site Unchanged
Cross-site No mechanism existed MaxConcurrentCrossSite limits task instances across all sites
Per-site exclusion TaskExclusion prevents conflicting tasks on the same site Unchanged
Cross-site exclusion No mechanism existed CrossSiteFlag = 1 prevents conflicting tasks across all sites
Dequeue Time When the Taskman service pics up a task from the queue to process it

When each limit is enforced

Feature Enforcement point Effect when blocked
MaxConcurrentCrossSite Dequeue time - task stays READY until limit has bandwidth Task waits in queue; submission always succeeds
Cross-Site exclusion Submit time - same as per-site exclusions Direct submissions receive an error; scheduled tasks retry automatically

How both limits work together

Both features operate independently. A task must pass both checks to run:
  1. At submit time: no cross-site exclusion conflict exists.
  2. A dequeue tim: the cross-site concurrency limit has not been reached.

Configuring Cross-Site Concurrency Limits

Setting the limit

  1. Open the BGTaskDefinitions form.
  2. Locate the task you want to limit.
  3. Set the Max Concurrent Cross-Site to the desired value
    • -1 - No cross-site limit enforced.
    • 1 - Only one instance can run across all sites at a time.
    • 2 or higher - That many instances can run simultaneously across all sites.

Example configuration

To allow Planning to run two sites simultaneously but never more than three total:
Setting Value Effect
MaxConcurrent 2 Up to 2 task per site.
MaxConcurrentCrossSite 3 Never more than 3 tasks total across all sites.

Configuring Cross-Site Task Exclusions

Mutual Exclusion

Mutual exclusion means: Task A cannot run while Task B is running, and Task B cannot run while Task A is running. This requires exclusion rules in both directions

For N task in mutual exclusion, you need Nx(N-1) rules. For example, a three-way mutual exclusion between Planning, Scheduling, and APSResyncAll requires six rules.

Setting up Cross-Site exclusions

  1. Open the BGTaskDefinitions form.
  2. Select the task you want to configure.
  3. Click the Cross-Site Excluded Tasks... button.
  4. Add exclusion rules:
Here is an example from SyteLine setting up a three-way mutual exclusion between Planning, Scheduling, and APSResyncAll. These rules could be added in a Mongoose application.
Task name Excluded task Description
Planning Scheduling Planing cannot start while Scheduling is running on any site
Planning APSResyncAll Panning cannot start while APSResyncAll is running on any site
Scheduling Planning Scheduling cannot start while Planning is running on any site
Scheduling APSResyncAll Scheduling cannot start while APSResyncAll is running on any site
APSResyncAll Planning APSResyncAll cannot start while Planning is running on any site
APSResyncAll Scheduling APSResyncAll cannot start while Scheduling is running on any site

Per-Site vs. Cross-Site exclusion UI

The BGTaskDefinitions form has two separate buttons:
  • Exclude Tasks... - opens the per-site exclusions form (existing, unchanged).
  • Cross-Site Excluded Tasks... - opens the cross-site exclusion form (new).
Both forms have the same layout. The difference is scope.
  • Per-site exclusions only check tasks on the same site.
  • Cross-site exclusions check tasks on all sites in the database.

Behavior when a task is blocked

Scheduled tasks (Automatic retry)

When a scheduled task is blocked by a cross-site exclusion:

  • The parent tasks stays in WAITING status
  • The system retries on the next polling cycle
  • NextStartDate is not advanced, the task will keep retrying until the conflict clears
  • After the blocking task completes, the scheduled task starts on the next retry

No manual intervention is required

Direct user submissions

When a user directly submits a task that is blocked by a cross-site exclusion:
  • The submission is rejected with an error message listing the blocking task, for example:“The following tasks are already running which prevent task: Planning from being run at this time: Scheduling”
  • The user must wait for the conflicting task to complete and resubmit.

Example Scenarios

Scenario 1: Cross-Site Concurrency Limit

Configuration: MaxConcurrentCrossSite = 1 for Planning

Time Event Result
01:30 DALS Planning starts Runs (1 of 1 limit)
01:35 LA Planning submitted Stays in READY queue (limit reached)
01:40 DALS Planning finishes LA Planning dequeued and starts
01:45 LOND Planning submitted Starts immediately (nothing running)

Scenario 2: Cross-Site Task Exclusion

Configuration: Planning, Scheduling, and APSResyncAll in mutual cross-site exclusion

Time Event Result
01:30 DALS APSResyncAll starts Runs normally
01:30 LA Planning submitted (scheduled) Rejected - APSResyncAll is running. Parent stays in WAITING status
01:40 DALS APSREsyncAll finishes
01:40 LA Planning retries No conflict - starts running
01:42 LOND Scheduling is submitted Rejected - Planning is running cross-site. Retries.
01:45 LA Planning finishes
01:45 LOND Scheduling retries No conflict - starts running

Performance considerations

Both features are designed with zero cost when not configured:
  • Cross-Site Concurrency - The dequeue query only runs the cross-site check if at least one candidate task has MaxConcurrentCrossSite > 0.
  • Cross-Site Exclusions - The submit-time check only queries ActiveBGTasks_mst if the task has CrossSiteFlag = 1.
Note: There is no performance effect on environments that do not use these features.

Backward compatibility

Setting Default Effect
MaxConcurrentCrossSite -1 No cross-site limit, matches existing behavior.
CrossSiteFlag 0 Existing exclusion rules remain per-site only
Note: All existing configurations continue to work unchanged. The new features are opt-in.

Database changes

Object Change
BGTaskDefinitions table New column: MaxConcurrentCrossSite INT NOT NULL DEFAULT -1
TaskExclusion table New column: CrossSiteFlag FlagNyType NOT NULL DEFAULT 0
CheckTaskExclusionSp Updated to handle CrossSiteFlag = 1 against ActiveBGTasks_mst
CLM_GetBGTasksToProcess3Sp Updated to enforce cross-site concurrency limit at dequeue