Observability

The record IdentityFlow keeps for each instance answers most questions: the instance page shows its state, the Audit tab its events, and GraphQL the same data for your tools.

This page shows where to read what happened to a workflow run, so you can decide what to monitor. First step: open the instance page, /instances/<id>, of a run you want to check.

The program exports metrics and traces only when you ask for them with OTEL_* variables; Prometheus then scrapes a separate port. See Export metrics and traces.

Where to read what happened

For a single run, the record IdentityFlow keeps for every instance answers most questions:

Current state and steps. The instance page, /instances/<id>, shows the status and each step with its status and times. For operators, the Diagnostics section of the operations page counts errored steps and shows the instance’s message. See Operate running workflows.
Ordered events. The Audit tab lists every recorded event with the account that caused it. See Audit trail.
The same data through GraphQL. The operations, business and audit views return it to your own tools; see the GraphQL API reference.
Operational metrics. The Operations page shows the completion rate, the error rate and the number of finished instances.

Export metrics and traces

Without OTEL_* exporter settings the program exports nothing, opens no extra port and starts no OpenTelemetry SDK. Set at least one exporter in the environment the program starts in:

Prometheus. OTEL_METRICS_EXPORTER=prometheus serves metrics on a separate listener at http://<host>:9464/metrics, in the Prometheus text format. The public HTTP port does not serve them.
OpenTelemetry (OTLP). OTEL_METRICS_EXPORTER=otlp and OTEL_TRACES_EXPORTER=otlp send metrics and traces to the collector at OTEL_EXPORTER_OTLP_ENDPOINT, by default http://localhost:4318, as http/protobuf. Metrics go out every 60 seconds (OTEL_METRIC_EXPORT_INTERVAL in milliseconds).

export OTEL_METRICS_EXPORTER=prometheus

export OTEL_METRICS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.example.com:4318

Exporters you leave unset stay off. OTEL_SDK_DISABLED=true turns everything off even when exporters are set. The variables may also come from the .env files the program reads. How to scrape the endpoint in Kubernetes is in Collect metrics.

Variable	Default	Purpose
`OTEL_METRICS_EXPORTER`	none	`prometheus` or `otlp`, or both separated by a comma.
`OTEL_TRACES_EXPORTER`	none	`otlp`.
`OTEL_EXPORTER_PROMETHEUS_PORT`	`9464`	Port of the metrics listener.
`OTEL_EXPORTER_PROMETHEUS_ENDPOINT`	`/metrics`	Path of the metrics endpoint.
`OTEL_EXPORTER_PROMETHEUS_HOST`	all interfaces	Address of the metrics listener, such as `127.0.0.1` when Prometheus runs on the same host.
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4318`	Address of your OpenTelemetry collector.
`OTEL_SERVICE_NAME`	`identity-flow`	Service name on every metric and trace.
`OTEL_NODE_RESOURCE_DETECTORS`	`env,host,os,serviceinstance,container`	Which facts about the host and process are attached; see below.
`OTEL_SDK_DISABLED`	`false`	`true` exports nothing.

If the metrics port is already taken, the log shows Error: listen EADDRINUSE: address already in use <address>:9464 and the program keeps serving without the metrics listener.

What you get

Node.js runtime metrics: event loop delay and utilization (nodejs_eventloop_*), heap use (v8js_memory_*) and garbage collection time (v8js_gc_duration).
Outgoing HTTP requests: http_client_request_duration, for example for OIDC discovery, with method, status code, server address and port. Paths and query strings are not part of the metric.
Large principal sets: identity_flow_authorization_principal_count is a histogram of an account’s deduplicated principal count when it exceeds the review threshold of 5,000 principals. It appears after such a principal resolution, including one performed during authorization. Prometheus exposes its _bucket, _sum and _count series, aggregated across accounts and traces without per-observation labels. Provider and account IDs, the observed count, configured limits and trace ID remain in trace events and operator warnings for diagnosis.
Traces of the workflow engine’s work, such as scheduler runs and hook calls, and of outgoing HTTP requests.
target_info, one series that names the program: service_name is identity-flow, service_version the program version.

Incoming HTTP requests and PostgreSQL queries are neither measured nor traced.

Every metric and trace carries facts about the host: host name, operating system, an ID of the running program and, in a container, the container ID. Facts about the process (process ID, the program’s path and command line, and the operating-system account it runs as) are left out: a command line can carry secrets, and the metrics endpoint has no authentication. To add them, include process:

export OTEL_NODE_RESOURCE_DETECTORS=env,host,os,serviceinstance,container,process

A trace of an outgoing request records its URL with every query value replaced by REDACTED; the parameter names stay. The path is recorded as it is, so keep tokens out of URL paths. Traces of the workflow engine record the error message of a failed step; keep secrets out of error messages.

Server log

The program writes its log to standard output and standard error. Collect it with the process supervisor you run it under. The lines that show a normal start are listed in Install and run IdentityFlow. A start that fails ends with the reason, for example WF_DEFINITION_BUNDLE_CHANGED or WF_DATABASE_MIGRATION_MISMATCH.

To check from a monitoring system that the program is up, request /-/ready and expect 200 with the body ready. It answers 503 while the program stops.

Workflow Test diagnostics

When a Workflow Test fails, the message names the workflow, the step, the actor and the events in order, and leaves out parameters, dialog answers, event payloads, binding arguments and results, secrets, tokens and raw errors. You can paste it into an issue as it is. See Test workflows.

Your own instrumentation

If you log or trace from your own bindings or hooks, keep credentials and external payloads out of those records. What you write to the console ends up in the program’s log.