What Breaks When 100 AI Agent Tasks Run at Once?

An AI assistant can work well when one person is trying it. They ask a question, wait for the answer and move on to the next task.

Now imagine the whole team using it on Monday morning. Some people are researching a topic. Others are searching company information or asking the assistant to create a ticket in another system. A few requests involve several of those steps.

The application accepts every request quickly. The screens all say work is in progress.

Five minutes later, some people are still waiting. The infrastructure dashboard looks healthy, and nobody can tell whether adding another worker would help.

This is the problem behind "Can it handle 100 tasks at once?" The answer depends on what those tasks do, where they wait and what counts as finished.

When I assess an agent workflow, I want to follow the work from the initial request to the result someone can use. A fast acknowledgement is one measurement along that path.

What do you mean by 100 tasks at once?

There are several different workloads hidden inside that phrase.

A platform that passes the first test has not necessarily passed the others.

For example, an API can return 202 Accepted for each request after saving it to a queue. If ten tasks can run at a time, most of that burst will initially wait. That can be a sensible design, provided the waiting is visible and the backlog drains at an acceptable rate.

The test needs to distinguish accepting work, processing it and delivering its result.

The scenarios below are a test plan, not measured results from a particular platform. The figure of 100 gives us a concrete workload to discuss.

Follow one task all the way through

A research task might involve a worker claiming the saved request, several model calls, retrieval, a tool request and a final document.

I would record at least these points in its lifecycle:

Request arrives
  -> Task is saved and acknowledged
  -> A worker starts execution
  -> Model and tool calls run
  -> The final result is saved
  -> The result is available to the user

That gives the team somewhere to start investigating. Long waits before execution suggest a different problem from long model calls. A result saved promptly but displayed much later points towards delivery or interface behaviour.

I would use the same task identifier across these records and separate identifiers for individual attempts. Otherwise, a retry can look like another completed task and inflate the numbers.

The queue can hide a capacity problem

A queue lets the application absorb a burst without trying to run everything immediately. It does not create more execution capacity.

Suppose, just for illustration, ten workers each complete one fixed task every ten seconds. With no other delays, that is about 60 completed tasks per minute. A burst of 100 takes ten waves, or roughly 100 seconds, to finish.

If 90 new tasks then arrive every minute, the backlog will keep growing even though every incoming request gets a quick response.

Real agent tasks vary much more than this example. The arithmetic still explains why I want to see how old queued work is, not just whether the queue is accepting new items.

For a business application, I would agree what happens as capacity runs out. The system might place work in a bounded queue, show an expected waiting state or decline new work with a clear explanation. Accepting more work than it can retain and process creates a promise the application cannot keep.

I would also test a short request arriving behind a large batch from another team or customer. Does it get a reasonable opportunity to run? Per-customer concurrency limits or fair scheduling can become useful when a shared queue lets one busy customer occupy everything.

Adding workers can move the bottleneck

More workers help when useful work is waiting for worker capacity. They can make another limit more visible.

Each active task needs memory, network connections and some access to the database. If a worker opens a database transaction and keeps it open while waiting for a model, increasing the worker count can leave the connection pool exhausted.

I would normally keep those transactions short: save the state needed to proceed, finish the transaction, make the external call and save its outcome in another transaction.

The work between those transactions still needs coordination. A lease is one way to give a worker temporary ownership of a task. If that ownership expires, a late worker must not overwrite the result of a newer attempt. Permissions may also need to be checked again before an external action.

Under load, I would watch database connection waits, lock waits, transaction duration, memory use and whether workers are making progress. A large worker count on its own is not evidence of useful capacity.

One agent task can create many provider requests

An agent may call a model to plan, retrieve information, call a tool, ask another specialist for help and then write an answer.

If each of 100 tasks needs six model calls, that is 600 calls before retries. The number is illustrative; the important detail is that incoming tasks and provider requests are different units.

Your own servers can be healthy while a model provider is limiting requests or tokens. A tool provider can impose a separate quota. Adding workers does not increase either allowance.

I would put limits around active work, model and tool calls per task, retries and spending. When a call needs to wait for capacity, the saved task should explain that wait. For longer delays, waiting work can be scheduled for later instead of occupying a scarce worker slot unnecessarily.

Retry behaviour needs particular attention. The application, orchestration layer and provider's client library may each retry. Their attempts can multiply unless the team understands the combined budget.

Backoff spaces retries out; jitter varies that delay so many tasks do not all retry together. Amazon's account of timeouts and retries explains why retries need limits when a dependency is already struggling. Provider-specific retry guidance still matters.

One task should not create two tickets

Imagine a tool successfully creates a support ticket, but its worker stops before saving the receipt. Another worker picks up the task and sees no saved result.

Repeating the call could create another ticket. Marking the task as failed could also mislead someone into starting the work again.

Load makes these awkward timings easier to encounter. The recovery contract needs to exist before the test starts: what can be repeated, what can be looked up and what must remain uncertain until reconciled?

I would terminate a test worker at selected points, delay a tool response and check both the saved task history and the fake provider's records. Counting a task as completed is not enough if it produced two external actions.

Retrying an AI Agent Is Not the Same as Retrying an API Call explores that recovery problem in more detail. Here, the question is whether those guarantees still hold while other tasks compete for resources.

Waiting and cancellation need honest states

Some tasks wait for a person to approve an action. Others wait for a provider quota to recover. Those are useful states, but they should not all appear as "Running."

The user needs to know whether work is queued, actively executing, awaiting input, completed or unable to continue. On returning after a disconnection, the application should recover the saved task and its result.

I would also check what cancellation means. Does it prevent further tool calls? Does it release capacity? An external request already in flight may still finish or incur a charge, so an immediate "Canceled" label is not proof that every operation stopped.

The interface and the execution record should agree about those boundaries. I covered the browser side in How to Track Background Job Progress in React and FastAPI.

How I would run the test

I would start with controlled model and tool responses, then introduce a limited set of representative real-provider checks. That makes it possible to identify application bottlenecks before mixing in changing quotas, model output and external network delays.

1. Set the workload and the acceptance criteria

Choose tasks representative of the product: a short lookup, a longer research request, an external action and a workflow that waits for approval.

Agree what an acceptable wait looks like, what outcomes are allowed and what evidence confirms the action happened. Use artificial test records and bounded workloads in a test environment.

2. Measure a repeatable baseline

Run a fixed task repeatedly at low load. Give the fake model and tools known response times and known results.

Record time to acceptance, queue wait, execution time and time until the result is usable. This gives the team something meaningful to compare with the busier runs.

3. Send the burst, then account for every task

Submit 100 distinct tasks and retain every accepted task ID.

Follow that group until each task is completed, failed, canceled, waiting for an explicit reason or still outstanding at the test deadline. Report unfinished work as unfinished. A completion-time average that quietly drops the slowest pending tasks gives a misleading picture.

Then watch how long the queue takes to drain. Briefly accepting the burst and recovering from it are separate results.

4. Keep new work arriving

A burst test ends. Business demand usually continues.

Gradually increase the arrival rate while earlier tasks are still running. k6's open workload model is useful here because new iterations can start independently of earlier ones finishing.

With a fixed population of simulated users that each wait for a result, the test can slow its own submissions as the application slows. That may be a useful model of a particular user journey, but it does not establish how the system handles an independent stream of incoming work.

For an asynchronous API, retain a separate completion observer or status collector. Fast 202 responses must not become the only result. Also check the load generator's own capacity: k6 can report dropped iterations when it cannot start all the scheduled work.

5. Mix customers, introduce faults and check recovery

Combine short and long tasks across test customers. Cancel some, let others wait for approval and introduce bounded faults such as rate limits or worker termination.

Check that one customer's batch does not indefinitely block another, that waiting work releases the resources it should, and that late results cannot overwrite a newer attempt.

Finally, use selected real-provider runs to check the assumptions behind the fakes and the quality of representative outputs. A correct execution record and a useful answer are separate things to evaluate. Controlled responses alone cannot establish how well a model handles the actual business task.

What I would put in the report

I want a report that connects system measurements to the experience of the team using the application.

For timing measurements, I would show the slower end as well as the middle. A p95 of 30 seconds means 95% of the measured observations took no more than 30 seconds; it says nothing about work excluded from that measurement. Unfinished tasks therefore need their own count and age.

The report should name the workload, test duration, worker limits, provider configuration and resources. It should also say what was simulated and what used a real provider. That makes the result useful when someone later changes a model, adds a tool or brings another department onto the platform.

The decision the test should support

There is no single component that always breaks first. The test should show where this application's useful capacity ends and how it behaves beyond that point.

The next change might be more worker capacity, shorter database transactions, fairer scheduling, a provider-budget adjustment or better recovery. Those are different investments. Measurement helps choose between them.

For a team moving beyond an AI demo, I would want a clear answer to this: when more people use the application, can we account for their work, deliver results within an agreed range and recover without losing or repeating actions?

That is the kind of delivery question I address when building internal applications and agent workflows, alongside the interface, integrations and operational controls the team needs to use them.