Parallel Processing in Workflows

Parallel Processing

IdentityFlow allows you to process multiple items or execute several distinct tasks concurrently within a workflow. This is particularly useful for bulk operations or when independent branches of logic can run simultaneously, significantly improving overall workflow performance.

Example: Bulk Item Processing

export default defineWorkflow('bulk-processing', async (flow) => {
  const { items } = flow.params;

  // Use Promise.all to wait for all concurrent tasks to complete
  const results = await Promise.all(
    items.map((item) =>
      // Each flow.do() call within the map can execute in parallel
      flow.do(`process item ${item.id}`, async () => {
        return processItem(item); // Your custom function to process an individual item
      }),
    ),
  );

  // The 'results' array will contain the outcome of each processed item
  return results;
});

Considerations for Parallel Processing:

• Error Handling: When using Promise.all, if any of the parallel tasks reject, Promise.all itself will reject immediately. Consider using Promise.allSettled if you need to know the outcome of all tasks, even if some fail. See Error Handling for more strategies.
• Resource Limits: Be mindful of external resource limits (e.g., database connections, API rate limits) when running many tasks in parallel. See Resource Management.
• Idempotency: Ensure that concurrently running tasks are idempotent if there’s a chance of retries.

Parallel processing is a powerful feature for optimizing workflows that involve multiple independent operations.

See Sub-Workflows for another way to manage complex processes.