diff --git a/docs/ai/debugging.md b/docs/ai/debugging.md index 9369e2b97..60efd6bba 100644 --- a/docs/ai/debugging.md +++ b/docs/ai/debugging.md @@ -15,7 +15,7 @@ These behaviors are especially hard to diagnose in a complex or long-running age Durable workflows help by making it easier to **observe** the root cause of the failure, deterministically **reproduce** the failure, and **test or apply** fixes. Because workflows checkpoint the outcome of each step of your workflow, you can review these checkpoints to see the cause of the failure and audit every step that led to it. -For example, using the [DBOS Console dashboard](../production/workflow-management.md), you might see that your agent failed because of a validation error caused by a malformed structured output: +For example, using the [DBOS Console dashboard](../conductor/workflow-management.md), you might see that your agent failed because of a validation error caused by a malformed structured output: Failing Agent diff --git a/docs/architecture.md b/docs/architecture.md index 2daadd7bf..7d975f756 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -120,11 +120,11 @@ When operating DBOS durable workflows in production, we strongly recommend conne Conductor is the control plane for your durable workflows, providing: - [**High availability**](./production/workflow-recovery.md): In a distributed environment with many executors running durable workflows, Conductor automatically detects when the execution of a durable workflow is interrupted (for example, if its executor is restarted, interrupted, or crashes) and recovers the workflow to another healthy executor. -- [**Workflow and queue observability**](./production/workflow-management.md): Conductor provides dashboards of all active and past workflows and all queued tasks as well as real-time workflow visualization. -- [**Workflow and queue management**](./production/workflow-management.md): From the Conductor dashboard, you can pause any workflow execution, start any stopped or enqueued workflow, or restart any workflow from a specific step. This is useful for rapidly responding to incidents or debugging. -- [**Managed Retention Policies**](./production/retention.md): From the Conductor dashboard, manage how much workflow history each of your applications should retain and for how long to retain it. -- [**Autoscaling and version management**](./production/autoscaling.md): Conductor computes how many executors each version of your application needs from queue utilization, so autoscalers like KEDA can size a deployment per application version, drain old versions down to zero, and drive rollouts. -- [**Observability Integrations**](./production/metrics.md): Conductor exposes metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint, so you can monitor your DBOS applications in Datadog, Grafana, or any other tool that understands the OpenMetrics format. +- [**Workflow and queue observability**](./conductor/workflow-management.md): Conductor provides dashboards of all active and past workflows and all queued tasks as well as real-time workflow visualization. +- [**Workflow and queue management**](./conductor/workflow-management.md): From the Conductor dashboard, you can pause any workflow execution, start any stopped or enqueued workflow, or restart any workflow from a specific step. This is useful for rapidly responding to incidents or debugging. +- [**Managed Retention Policies**](./conductor/retention.md): From the Conductor dashboard, manage how much workflow history each of your applications should retain and for how long to retain it. +- [**Autoscaling and version management**](./conductor/autoscaling.md): Conductor computes how many executors each version of your application needs from queue utilization, so autoscalers like KEDA can size a deployment per application version, drain old versions down to zero, and drive rollouts. +- [**Observability Integrations**](./conductor/metrics.md): Conductor exposes metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint, so you can monitor your DBOS applications in Datadog, Grafana, or any other tool that understands the OpenMetrics format. Architecturally, Conductor looks like this: @@ -140,4 +140,4 @@ This architecture has two useful implications: 2. Conductor is **off your workflows orchestration path**. Conductor drives observability, recovery, and retention policies, and is never involved in workflow execution (unlike the external orchestrators of other workflow systems). If your application's connection to Conductor is interrupted, it will continue to operate normally, and any failed workflows will automatically be recovered as soon as the connection is restored. -For more information on Conductor, see [its docs](./production/conductor.md). +For more information on Conductor, see [its docs](./conductor/overview.md). diff --git a/docs/production/alerting.md b/docs/conductor/alerting.md similarity index 98% rename from docs/production/alerting.md rename to docs/conductor/alerting.md index e9f2540b0..e2f9bfb7b 100644 --- a/docs/production/alerting.md +++ b/docs/conductor/alerting.md @@ -1,10 +1,10 @@ --- -sidebar_position: 25 +sidebar_position: 50 title: Alerting toc_max_heading_level: 3 --- -If you are using [Conductor](./conductor.md), you can configure automatic alerts when certain failure conditions are met. +If you are using [Conductor](./overview.md), you can configure automatic alerts when certain failure conditions are met. You can configure alerts either in Conductor directly or on [Conductor-exported metrics](#metrics-based-alerts) using your existing observability stack. :::info diff --git a/docs/production/audit-logs.md b/docs/conductor/audit-logs.md similarity index 91% rename from docs/production/audit-logs.md rename to docs/conductor/audit-logs.md index cea0b2986..eeba160c7 100644 --- a/docs/production/audit-logs.md +++ b/docs/conductor/audit-logs.md @@ -1,10 +1,10 @@ --- -sidebar_position: 32 +sidebar_position: 80 title: Audit Logs toc_max_heading_level: 3 --- -If you are using [Conductor](./conductor.md), you can retrieve an **audit log** of the mutating operations performed against your organization: registering and deleting applications, managing workflows and schedules, creating and revoking API keys, changing roles and membership, and updating organization settings. +If you are using [Conductor](./overview.md), you can retrieve an **audit log** of the mutating operations performed against your organization: registering and deleting applications, managing workflows and schedules, creating and revoking API keys, changing roles and membership, and updating organization settings. The audit log is append-only and records who did what, when, from where, and whether the operation succeeded. :::info @@ -13,7 +13,7 @@ Audit logs require a [DBOS Enterprise](https://www.dbos.dev/dbos-pricing) plan. ## The Audit Logs Endpoint -Conductor exposes an organization's audit log through the [Conductor API](./conductor-api.md) at: +Conductor exposes an organization's audit log through the [Conductor API](./reference/conductor-api.md) at: ``` GET https://cloud.dbos.dev/conductor/v2/orgs/{orgName}/audit-logs @@ -34,7 +34,7 @@ curl -G https://cloud.dbos.dev/conductor/v2/orgs/my_org/audit-logs \ ``` :::note -Audit logs are an organization-level concept, so a [self-hosted Conductor](./hosting-conductor.md) running with authentication disabled does not register this operation and responds `404`. See [Self-hosted differences](./conductor-api.md#self-hosted-differences). +Audit logs are an organization-level concept, so a [self-hosted Conductor](./self-hosting/hosting-conductor.md) running with authentication disabled does not register this operation and responds `404`. See [Self-hosted differences](./reference/conductor-api.md#self-hosted-differences). ::: Entries are returned newest first (by emit time). diff --git a/docs/production/autoscaling.md b/docs/conductor/autoscaling.md similarity index 94% rename from docs/production/autoscaling.md rename to docs/conductor/autoscaling.md index e1cffdfd0..81826547a 100644 --- a/docs/production/autoscaling.md +++ b/docs/conductor/autoscaling.md @@ -1,10 +1,10 @@ --- -sidebar_position: 22 +sidebar_position: 60 title: Autoscaling and Version Management toc_max_heading_level: 3 --- -[Conductor](./conductor.md) lets you attach autoscaling policies to your applications. An autoscaling policy computes how many executors your application needs, per application version, to drain one of your application's queues. A common example is configuring a [KEDA](https://keda.sh/) ScaledObject to size your application deployments based on queue utilization. +[Conductor](./overview.md) lets you attach autoscaling policies to your applications. An autoscaling policy computes how many executors your application needs, per application version, to drain one of your application's queues. A common example is configuring a [KEDA](https://keda.sh/) ScaledObject to size your application deployments based on queue utilization. :::info Autoscaling requires a [DBOS Teams](https://www.dbos.dev/dbos-pricing) plan. @@ -15,7 +15,7 @@ To use policies: 1. **Attach an autoscaling policy** to an application, naming the queue whose backlog drives the executor count. 2. **Poll the desired executor count**, either one version at a time or for all active versions at once. -All endpoints on this page are part of the [Conductor API](./conductor-api.md); see that page for the base URL and authentication. +All endpoints on this page are part of the [Conductor API](./reference/conductor-api.md); see that page for the base URL and authentication. The examples below use `$CONDUCTOR` for the base URL and `$CONDUCTOR_KEY` for an [API key](./permissions.md). ## How It Works diff --git a/docs/conductor/distributed-recovery.md b/docs/conductor/distributed-recovery.md new file mode 100644 index 000000000..1d65de7cf --- /dev/null +++ b/docs/conductor/distributed-recovery.md @@ -0,0 +1,16 @@ +--- +sidebar_position: 30 +title: Distributed Recovery +--- + +If your application is connected to [DBOS Conductor](./overview.md), workflow recovery is automatic. +When Conductor detects that an executor is unhealthy, it automatically signals another executor to recover its workflows. + +When an executor disconnects from Conductor, its status is changed to `DISCONNECTED` while Conductor waits for it to reconnect. +If it has not reconnected after a certain period of time, its status is changed to `DEAD` and Conductor signals another executor to recover its workflows. +After recovery is confirmed, Conductor deletes its record of the executor. + +By default, the executor timeout is 60 seconds, so Conductor waits 60 seconds after an executor disconnects before recovering its workflows. +You can configure the executor timeout per application from the DBOS Console. + +Workflow Timeout Configuration diff --git a/docs/production/metrics.md b/docs/conductor/metrics.md similarity index 97% rename from docs/production/metrics.md rename to docs/conductor/metrics.md index a484ba36c..f3791a0d6 100644 --- a/docs/production/metrics.md +++ b/docs/conductor/metrics.md @@ -1,10 +1,10 @@ --- -sidebar_position: 24 +sidebar_position: 40 title: Metrics toc_max_heading_level: 3 --- -If you are using [Conductor](./conductor.md), you can scrape metrics about your applications' workflows, steps, and executors from a [Prometheus](https://prometheus.io/)-compatible endpoint. +If you are using [Conductor](./overview.md), you can scrape metrics about your applications' workflows, steps, and executors from a [Prometheus](https://prometheus.io/)-compatible endpoint. This lets you monitor your DBOS applications in Prometheus, Grafana, or any other tool that understands the [OpenMetrics](https://prometheus.io/docs/specs/om/open_metrics_spec/) format. :::info diff --git a/docs/production/conductor.md b/docs/conductor/overview.md similarity index 92% rename from docs/production/conductor.md rename to docs/conductor/overview.md index cf8fa1cee..d28056eb6 100644 --- a/docs/production/conductor.md +++ b/docs/conductor/overview.md @@ -1,18 +1,18 @@ --- -sidebar_position: 10 -title: DBOS Conductor +sidebar_position: 1 +title: DBOS Conductor Overview --- When operating DBOS durable workflows in production, we strongly recommend connecting your application to Conductor. Conductor is the control plane for your durable workflows, providing: -- [**High availability**](./workflow-recovery.md): In a distributed environment with many executors running durable workflows, Conductor automatically detects when a workflow is interrupted (for example, if its executor disconnects or crashes) and recovers the workflow to another healthy executor. +- [**High availability**](./distributed-recovery.md): In a distributed environment with many executors running durable workflows, Conductor automatically detects when a workflow is interrupted (for example, if its executor disconnects or crashes) and recovers the workflow to another healthy executor. - [**Workflow and queue observability**](./workflow-management.md): Conductor provides dashboards of all active and past workflows and all queued tasks as well as real-time workflow visualization. - [**Workflow and queue management**](./workflow-management.md): From the Conductor dashboard, you can pause any workflow execution, start any stopped or enqueued workflow, or restart any workflow from a specific step. This is useful for rapidly responding to incidents or debugging. - [**Managed Retention Policies**](./retention.md): From the Conductor dashboard, manage how much workflow history each of your applications should retain and for how long to retain it. - [**Autoscaling and version management**](./autoscaling.md): Conductor computes how many executors each version of your application needs from queue utilization, so autoscalers like KEDA can size a deployment per application version, drain old versions down to zero, and drive rollouts. - [**Observability Integrations**](./metrics.md): Conductor exposes metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint, so you can monitor your DBOS applications in Datadog, Grafana, or any other tool that understands the OpenMetrics format. -- [**Programmatic access**](./conductor-api.md): Conductor's workflow, queue, and schedule management is available over an OpenAPI-described HTTP API and from the [`dbosctl` command-line client](./dbosctl.md), so you can script incident response and wire Conductor into your own tooling. +- [**Programmatic access**](./reference/conductor-api.md): Conductor's workflow, queue, and schedule management is available over an OpenAPI-described HTTP API and from the [`dbosctl` command-line client](./reference/dbosctl.md), so you can script incident response and wire Conductor into your own tooling. Architecturally, Conductor is not part of your workflows orchestration path. If your connection to Conductor is interrupted, your applications will continue operating normally. @@ -22,7 +22,7 @@ Recovery, observability, and workflow management will automatically resume once ## Connecting To Conductor -To connect your application to Conductor, first register your application on the [DBOS console](https://console.dbos.dev). +To connect your application to Conductor, first register your application on the [DBOS Console](https://console.dbos.dev). **The name you register must match the name you give your application in its configuration.** Workflow List diff --git a/docs/production/permissions.md b/docs/conductor/permissions.md similarity index 92% rename from docs/production/permissions.md rename to docs/conductor/permissions.md index 16a241bfb..cd8ff49a5 100644 --- a/docs/production/permissions.md +++ b/docs/conductor/permissions.md @@ -1,12 +1,12 @@ --- -sidebar_position: 31 +sidebar_position: 70 title: Permissions and API Keys --- DBOS Conductor controls access to your organization's applications, workflows, and settings using **role-based access control (RBAC)** for users and **scoped API keys** for applications and automation. This page describes the permission model, the built-in and custom roles, and how to create and manage API keys. -You manage permissions and API keys from the [DBOS console](https://console.dbos.dev). +You manage permissions and API keys from the [DBOS Console](https://console.dbos.dev). ## Permissions @@ -76,7 +76,7 @@ Like a role, every API key carries a set of permissions. They can also be scoped to specific applications. API keys do not expire, but can be revoked at any time. -A key can be renamed after creation without changing its secret, from the console, with [`dbosctl api-key rename`](./dbosctl.md#dbosctl-api-key-rename), or through the [Conductor API](./conductor-api.md#roles-permissions-and-api-keys). +A key can be renamed after creation without changing its secret, from the console, with [`dbosctl api-key rename`](./reference/dbosctl.md#dbosctl-api-key-rename), or through the [Conductor API](./reference/conductor-api.md#roles-permissions-and-api-keys). ### Permissions and application scope @@ -89,7 +89,7 @@ For example, an API key with only `application.read` scoped to a single applicat ### Using an API key -Supply the key to your DBOS application to connect it to Conductor, as described in [Connecting to Conductor](./conductor.md#connecting-to-conductor). +Supply the key to your DBOS application to connect it to Conductor, as described in [Connecting to Conductor](./overview.md#connecting-to-conductor). You can also use an API key to authenticate HTTP calls to the Conductor API (for example the [metrics endpoint](./metrics.md)), passing the key as a bearer token: diff --git a/docs/conductor/reference/_category_.json b/docs/conductor/reference/_category_.json new file mode 100644 index 000000000..e67ccca21 --- /dev/null +++ b/docs/conductor/reference/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Reference", + "position": 100 +} diff --git a/docs/production/conductor-api.md b/docs/conductor/reference/conductor-api.md similarity index 87% rename from docs/production/conductor-api.md rename to docs/conductor/reference/conductor-api.md index a68ed3b04..a4f72de9a 100644 --- a/docs/production/conductor-api.md +++ b/docs/conductor/reference/conductor-api.md @@ -5,7 +5,7 @@ title: Conductor API Conductor is the control plane for your durable workflows, and this HTTP API is how you drive it programmatically: register applications with Conductor and tune their settings, search workflows, cancel or fork them, inspect queues and schedules, drive schedules, read metrics and audit logs, and manage members, roles, and API keys. -This is the Conductor half of the [DBOS console](https://console.dbos.dev) — what the console shows for an application connected to Conductor, whether that application runs on your own infrastructure or on DBOS Cloud. DBOS Cloud's own operations, such as [deploying an application](./dbos-cloud/deploying-to-cloud.md) or [provisioning a database](./dbos-cloud/database-management.md), are not part of this API; they have their own [CLI](./dbos-cloud/cloud-cli.md). +This is the Conductor half of the [DBOS Console](https://console.dbos.dev) — what the console shows for an application connected to Conductor, whether that application runs on your own infrastructure or on DBOS Cloud. DBOS Cloud's own operations, such as [deploying an application](./dbos-cloud/deploying-to-cloud.md) or [provisioning a database](./dbos-cloud/database-management.md), are not part of this API; they have their own [CLI](./dbos-cloud/cloud-cli.md). The API is described by an OpenAPI 3.1 specification generated directly from the running server, so it is never out of date with the deployment serving it. Both the console and the [`dbosctl` CLI](./dbosctl.md) drive Conductor through this API, using clients generated from that spec. @@ -13,8 +13,8 @@ The API is described by an OpenAPI 3.1 specification generated directly from the | Deployment | Base URL | | --- | --- | -| DBOS-managed Conductor | `https://cloud.dbos.dev/conductor` | -| [Self-hosted Conductor](./hosting-conductor.md) | `http://:8090` (port `8090` by default) | +| DBOS-hosted Conductor | `https://cloud.dbos.dev/conductor` | +| [Self-hosted Conductor](../self-hosting/hosting-conductor.md) | `http://:8090` (port `8090` by default) | Every path is relative to that base, so the full URL of an operation is, for example: @@ -28,7 +28,7 @@ The paths themselves are identical in both deployments; only the base differs. There are three ways to obtain the spec. -**From DBOS-managed Conductor.** The spec is served publicly (no authentication required) and reflects the currently deployed version: +**From DBOS-hosted Conductor.** The spec is served publicly (no authentication required) and reflects the currently deployed version: ```shell curl -O https://cloud.dbos.dev/conductor/v2/openapi.json @@ -40,7 +40,7 @@ If your toolchain does not yet support OpenAPI 3.1, request the 3.0 downgrade in curl -O https://cloud.dbos.dev/conductor/v2/openapi-3.0.json ``` -The spec served here is Conductor's own, with only its `servers` entry repointed at `/conductor` so that generated clients resolve paths correctly through DBOS Cloud. +The spec served here is Conductor's own, with only its `servers` entry repointed at `/conductor` so that generated clients resolve paths correctly through DBOS-hosted Conductor. **From a self-hosted Conductor.** The server mounts the spec and an interactive browser at its root, all unauthenticated: @@ -52,7 +52,7 @@ The spec served here is Conductor's own, with only its `servers` entry repointed | `/docs` | Interactive API browser | | `/schemas/*` | The JSON Schema documents referenced by the spec | -For example, with the Docker Compose setup from [Self-Hosting Conductor](./hosting-conductor.md), open `http://localhost:8090/docs` to explore the API in your browser. +For example, with the Docker Compose setup from the [Self-Hosting Guide](../self-hosting/hosting-conductor.md), open `http://localhost:8090/docs` to explore the API in your browser. **From the Conductor image.** Conductor's `openapi` subcommand prints the spec to stdout without connecting to a database or requiring any runtime configuration, which is convenient in CI and code generation pipelines. The image's entrypoint starts the server, so override it to reach the subcommand: @@ -87,7 +87,7 @@ Conductor accepts two kinds of token, distinguished by their prefix: Both are sent the same way; Conductor tells them apart by the `dbos_` prefix. -Authorization is enforced per operation using the permission model described in [Permissions and API Keys](./permissions.md) — a caller needs `application.read` to list workflows, `application.write` to cancel one, `organization.write` to manage members, and so on. An unauthenticated request returns `401`; an authenticated request lacking the required permission returns `403`. +Authorization is enforced per operation using the permission model described in [Permissions and API Keys](../permissions.md) — a caller needs `application.read` to list workflows, `application.write` to cancel one, `organization.write` to manage members, and so on. An unauthenticated request returns `401`; an authenticated request lacking the required permission returns `403`. ## Resource Naming @@ -184,7 +184,7 @@ The tables below are a map of the whole API. The generated spec is the authorita | Claim a domain | `POST /v2/orgs/{orgName}/domain-claims` | | Release a domain claim | `DELETE /v2/orgs/{orgName}/domain-claims/{domain}` | -A **domain claim** automatically adds users who register with an email at that domain to your organization. On DBOS-managed Conductor a claim takes effect only after DBOS approves it; on a self-hosted deployment it takes effect immediately. Claims apply to new registrations only: approving one never moves users who already have accounts, and releasing one never removes them. +A **domain claim** automatically adds users who register with an email at that domain to your organization. On DBOS-hosted Conductor a claim takes effect only after DBOS approves it; on a self-hosted deployment it takes effect immediately. Claims apply to new registrations only: approving one never moves users who already have accounts, and releasing one never removes them. ### Roles, permissions, and API keys @@ -209,7 +209,7 @@ Create an API key with an optional body scoping it to particular applications an } ``` -The response contains the key's secret. It is returned **once**, at creation, and cannot be retrieved afterwards. A key can be renamed afterwards with `PATCH /v2/orgs/{orgName}/tokens/{tokenName}` and a body of `{"newName": "..."}`; the secret itself never changes, so rotating it means deleting the key and creating a new one. See [Permissions and API Keys](./permissions.md) for the full list of permissions. +The response contains the key's secret. It is returned **once**, at creation, and cannot be retrieved afterwards. A key can be renamed afterwards with `PATCH /v2/orgs/{orgName}/tokens/{tokenName}` and a body of `{"newName": "..."}`; the secret itself never changes, so rotating it means deleting the key and creating a new one. See [Permissions and API Keys](../permissions.md) for the full list of permissions. ### Applications @@ -225,10 +225,10 @@ The response contains the key's secret. It is returned **once**, at creation, an | List executors | `GET /v2/orgs/{orgName}/apps/{appName}/executors` | | List metrics | `GET /v2/orgs/{orgName}/apps/{appName}/metrics` | -`PATCH .../apps/{appName}` is where an application's tuning settings live: the executor timeout, the global workflow timeout, the [workflow retention thresholds](./retention.md), and private mode. +`PATCH .../apps/{appName}` is where an application's tuning settings live: the executor timeout, the global workflow timeout, the [workflow retention thresholds](../retention.md), and private mode. :::info -`GET .../metrics` returns metrics for one application over a time window. If you want to scrape Conductor from Prometheus, Datadog, or Grafana, use the OpenMetrics endpoint described in [Metrics](./metrics.md) instead. +`GET .../metrics` returns metrics for one application over a time window. If you want to scrape Conductor from Prometheus, Datadog, or Grafana, use the OpenMetrics endpoint described in [Metrics](../metrics.md) instead. ::: ### Workflows @@ -255,7 +255,7 @@ The response contains the key's secret. It is returned **once**, at creation, an | Bulk delete | `POST .../workflows/bulk-delete` | | Bulk fork from failure | `POST .../workflows/bulk-fork-from-failure` | -The semantics of cancelling, resuming, and forking are described in [Workflow Management](./workflow-management.md). The bulk variants take an array of workflow IDs and apply the same operation to each, which is far cheaper than issuing the calls one at a time. **Export** and **import** move a workflow and its steps between deployments as a JSON document — useful for reproducing a production failure in a development environment. +The semantics of cancelling, resuming, and forking are described in [Workflow Management](../workflow-management.md). The bulk variants take an array of workflow IDs and apply the same operation to each, which is far cheaper than issuing the calls one at a time. **Export** and **import** move a workflow and its steps between deployments as a JSON document — useful for reproducing a production failure in a development environment. ### Queues @@ -274,7 +274,7 @@ The semantics of cancelling, resuming, and forking are described in [Workflow Ma | Desired executors, all versions | `GET /v2/orgs/{orgName}/apps/{appName}/autoscale` | | Desired executors, one version | `GET /v2/orgs/{orgName}/apps/{appName}/autoscale/versions/{version}` | -The policy names the queue whose backlog drives the executor count; the two `autoscale` operations return how many executors each application version needs right now. `{version}` is a registered version or `latest`. See [Autoscaling and Version Management](./autoscaling.md). +The policy names the queue whose backlog drives the executor count; the two `autoscale` operations return how many executors each application version needs right now. `{version}` is a registered version or `latest`. See [Autoscaling and Version Management](../autoscaling.md). ### Schedules @@ -298,11 +298,11 @@ The policy names the queue whose backlog drives the executor count; the two `aut | Delete alerting rule | `DELETE /v2/orgs/{orgName}/apps/{appName}/alerting-rules/{ruleId}` | | List audit logs | `GET /v2/orgs/{orgName}/audit-logs` | -Alerting rules are described in [Alerting](./alerting.md). Audit log listing accepts `startTime`, `endTime`, `operation`, `subject`, and `target` filters alongside `limit` and `offset`; see [Audit Logs](./audit-logs.md). +Alerting rules are described in [Alerting](../alerting.md). Audit log listing accepts `startTime`, `endTime`, `operation`, `subject`, and `target` filters alongside `limit` and `offset`; see [Audit Logs](../audit-logs.md). ## Self-Hosted Differences -A self-hosted Conductor can run with OIDC authentication enabled or with authentication disabled entirely (see [Self-Hosting Conductor](./hosting-conductor.md)). In no-auth mode there is no user identity and no multi-organization concept, so the operations that depend on them are **not registered at all** and respond `404`: +A self-hosted Conductor can run with OIDC authentication enabled or with authentication disabled entirely (see the [Self-Hosting Guide](../self-hosting/hosting-conductor.md)). In no-auth mode there is no user identity and no multi-organization concept, so the operations that depend on them are **not registered at all** and respond `404`: - every organization operation: `getOrg`, `updateOrg`, `joinOrg`, `generateSecret`, `listMembers`, `removeMember`, `listDomainClaims`, `requestDomainClaim`, and `releaseDomainClaim`; - every role operation: `listRoles`, `createRole`, `deleteRole`, `grantRole`; diff --git a/docs/production/dbos-cloud/_category_.json b/docs/conductor/reference/dbos-cloud/_category_.json similarity index 100% rename from docs/production/dbos-cloud/_category_.json rename to docs/conductor/reference/dbos-cloud/_category_.json diff --git a/docs/production/dbos-cloud/account-management.md b/docs/conductor/reference/dbos-cloud/account-management.md similarity index 100% rename from docs/production/dbos-cloud/account-management.md rename to docs/conductor/reference/dbos-cloud/account-management.md diff --git a/docs/production/dbos-cloud/application-management.md b/docs/conductor/reference/dbos-cloud/application-management.md similarity index 100% rename from docs/production/dbos-cloud/application-management.md rename to docs/conductor/reference/dbos-cloud/application-management.md diff --git a/docs/production/dbos-cloud/assets/cc-logs.png b/docs/conductor/reference/dbos-cloud/assets/cc-logs.png similarity index 100% rename from docs/production/dbos-cloud/assets/cc-logs.png rename to docs/conductor/reference/dbos-cloud/assets/cc-logs.png diff --git a/docs/production/dbos-cloud/assets/cc-orgs.png b/docs/conductor/reference/dbos-cloud/assets/cc-orgs.png similarity index 100% rename from docs/production/dbos-cloud/assets/cc-orgs.png rename to docs/conductor/reference/dbos-cloud/assets/cc-orgs.png diff --git a/docs/production/dbos-cloud/assets/cc-traces.png b/docs/conductor/reference/dbos-cloud/assets/cc-traces.png similarity index 100% rename from docs/production/dbos-cloud/assets/cc-traces.png rename to docs/conductor/reference/dbos-cloud/assets/cc-traces.png diff --git a/docs/production/dbos-cloud/assets/dash-debug-wf.png b/docs/conductor/reference/dbos-cloud/assets/dash-debug-wf.png similarity index 100% rename from docs/production/dbos-cloud/assets/dash-debug-wf.png rename to docs/conductor/reference/dbos-cloud/assets/dash-debug-wf.png diff --git a/docs/production/dbos-cloud/assets/execution-seconds.png b/docs/conductor/reference/dbos-cloud/assets/execution-seconds.png similarity index 100% rename from docs/production/dbos-cloud/assets/execution-seconds.png rename to docs/conductor/reference/dbos-cloud/assets/execution-seconds.png diff --git a/docs/production/dbos-cloud/assets/filters.png b/docs/conductor/reference/dbos-cloud/assets/filters.png similarity index 100% rename from docs/production/dbos-cloud/assets/filters.png rename to docs/conductor/reference/dbos-cloud/assets/filters.png diff --git a/docs/production/dbos-cloud/assets/log.png b/docs/conductor/reference/dbos-cloud/assets/log.png similarity index 100% rename from docs/production/dbos-cloud/assets/log.png rename to docs/conductor/reference/dbos-cloud/assets/log.png diff --git a/docs/production/dbos-cloud/assets/time_picker.png b/docs/conductor/reference/dbos-cloud/assets/time_picker.png similarity index 100% rename from docs/production/dbos-cloud/assets/time_picker.png rename to docs/conductor/reference/dbos-cloud/assets/time_picker.png diff --git a/docs/production/dbos-cloud/assets/timeseries.png b/docs/conductor/reference/dbos-cloud/assets/timeseries.png similarity index 100% rename from docs/production/dbos-cloud/assets/timeseries.png rename to docs/conductor/reference/dbos-cloud/assets/timeseries.png diff --git a/docs/production/dbos-cloud/assets/ttdbg-code-lens.png b/docs/conductor/reference/dbos-cloud/assets/ttdbg-code-lens.png similarity index 100% rename from docs/production/dbos-cloud/assets/ttdbg-code-lens.png rename to docs/conductor/reference/dbos-cloud/assets/ttdbg-code-lens.png diff --git a/docs/production/dbos-cloud/assets/ttdbg-debugging.png b/docs/conductor/reference/dbos-cloud/assets/ttdbg-debugging.png similarity index 100% rename from docs/production/dbos-cloud/assets/ttdbg-debugging.png rename to docs/conductor/reference/dbos-cloud/assets/ttdbg-debugging.png diff --git a/docs/production/dbos-cloud/assets/ttdbg-launch-proxy.png b/docs/conductor/reference/dbos-cloud/assets/ttdbg-launch-proxy.png similarity index 100% rename from docs/production/dbos-cloud/assets/ttdbg-launch-proxy.png rename to docs/conductor/reference/dbos-cloud/assets/ttdbg-launch-proxy.png diff --git a/docs/production/dbos-cloud/assets/ttdbg-wfid-manual.png b/docs/conductor/reference/dbos-cloud/assets/ttdbg-wfid-manual.png similarity index 100% rename from docs/production/dbos-cloud/assets/ttdbg-wfid-manual.png rename to docs/conductor/reference/dbos-cloud/assets/ttdbg-wfid-manual.png diff --git a/docs/production/dbos-cloud/assets/ttdbg-wfid-quick-pick.png b/docs/conductor/reference/dbos-cloud/assets/ttdbg-wfid-quick-pick.png similarity index 100% rename from docs/production/dbos-cloud/assets/ttdbg-wfid-quick-pick.png rename to docs/conductor/reference/dbos-cloud/assets/ttdbg-wfid-quick-pick.png diff --git a/docs/production/dbos-cloud/byod-management.md b/docs/conductor/reference/dbos-cloud/byod-management.md similarity index 100% rename from docs/production/dbos-cloud/byod-management.md rename to docs/conductor/reference/dbos-cloud/byod-management.md diff --git a/docs/production/dbos-cloud/cicd.md b/docs/conductor/reference/dbos-cloud/cicd.md similarity index 79% rename from docs/production/dbos-cloud/cicd.md rename to docs/conductor/reference/dbos-cloud/cicd.md index 9d403fdf7..b383d8244 100644 --- a/docs/production/dbos-cloud/cicd.md +++ b/docs/conductor/reference/dbos-cloud/cicd.md @@ -27,8 +27,8 @@ If you manually specify the application database name by setting `app_db_name` i ::: ## Authentication -You should use [refresh tokens](account-management#authenticating-programatically) to programmatically authenticate your CI/CD user with DBOS Cloud. +You should use [refresh tokens](./account-management#authenticating-programatically) to programmatically authenticate your CI/CD user with DBOS Cloud. :::info -Upgrading to a DBOS Cloud paid plan will unlock [multi-user organizations](account-management#organization-management) which you can use to setup dedicated users for CI/CD. +Upgrading to a DBOS Cloud paid plan will unlock [multi-user organizations](./account-management#organization-management) which you can use to setup dedicated users for CI/CD. ::: diff --git a/docs/production/dbos-cloud/cloud-cli.md b/docs/conductor/reference/dbos-cloud/cloud-cli.md similarity index 100% rename from docs/production/dbos-cloud/cloud-cli.md rename to docs/conductor/reference/dbos-cloud/cloud-cli.md diff --git a/docs/production/dbos-cloud/database-management.md b/docs/conductor/reference/dbos-cloud/database-management.md similarity index 100% rename from docs/production/dbos-cloud/database-management.md rename to docs/conductor/reference/dbos-cloud/database-management.md diff --git a/docs/production/dbos-cloud/deploying-to-cloud.md b/docs/conductor/reference/dbos-cloud/deploying-to-cloud.md similarity index 95% rename from docs/production/dbos-cloud/deploying-to-cloud.md rename to docs/conductor/reference/dbos-cloud/deploying-to-cloud.md index 069f2899e..352e7836f 100644 --- a/docs/production/dbos-cloud/deploying-to-cloud.md +++ b/docs/conductor/reference/dbos-cloud/deploying-to-cloud.md @@ -5,6 +5,9 @@ hide_table_of_contents: true --- import InstallNode from '@site/docs/partials/_install_node.mdx'; +:::info +To use DBOS Cloud, please [contact sales](https://dbos.dev/contact). +::: Any application built with DBOS can be deployed to DBOS Cloud. DBOS Cloud is a serverless platform for durably executed applications. @@ -67,7 +70,7 @@ pip freeze > requirements.txt #### 3. Define a Start Command
-Set the `start` command in the `runtimeConfig` section of your [`dbos-config.yaml`](../../python/reference/configuration.md) to your application's launch command. +Set the `start` command in the `runtimeConfig` section of your [`dbos-config.yaml`](../../../python/reference/configuration.md) to your application's launch command. If your application includes an HTTP server, configure it to listen on port 8000. @@ -132,7 +135,7 @@ npm i @dbos-inc/otel@latest
-Set the `start` command in the `runtimeConfig` section of your [`dbos-config.yaml`](../../typescript/reference/configuration.md) to your application's launch command. +Set the `start` command in the `runtimeConfig` section of your [`dbos-config.yaml`](../../../typescript/reference/configuration.md) to your application's launch command. If your application includes an HTTP server, configure it to listen on port 3000. @@ -203,7 +206,7 @@ npm i -g @dbos-inc/dbos-cloud@latest
-Your DBOSContext [Config](../../golang/reference/dbos-context.md) must be set with: +Your DBOSContext [Config](../../../golang/reference/dbos-context.md) must be set with: - `DatabaseURL` (or your custom `pgxpool`) must point to an environment variable named `DBOS_SYSTEM_DATABASE_URL`
diff --git a/docs/production/dbos-cloud/monitoring-dashboard.md b/docs/conductor/reference/dbos-cloud/monitoring-dashboard.md similarity index 100% rename from docs/production/dbos-cloud/monitoring-dashboard.md rename to docs/conductor/reference/dbos-cloud/monitoring-dashboard.md diff --git a/docs/production/dbos-cloud/otel-integration.md b/docs/conductor/reference/dbos-cloud/otel-integration.md similarity index 100% rename from docs/production/dbos-cloud/otel-integration.md rename to docs/conductor/reference/dbos-cloud/otel-integration.md diff --git a/docs/production/dbos-cloud/retention.md b/docs/conductor/reference/dbos-cloud/retention.md similarity index 94% rename from docs/production/dbos-cloud/retention.md rename to docs/conductor/reference/dbos-cloud/retention.md index e96140f62..47a2d8f86 100644 --- a/docs/production/dbos-cloud/retention.md +++ b/docs/conductor/reference/dbos-cloud/retention.md @@ -4,7 +4,7 @@ title: Workflow Retention Policies --- You can configure workflow history retention policies for your application from the Retention Policy page of the DBOS Console. -These settings let you configure how long workflow history is retained in your application's [system database](../../explanations/system-tables.md). +These settings let you configure how long workflow history is retained in your application's [system database](../../../explanations/system-tables.md). This is useful for managing the database disk usage of workflow history. Retention policies only delete the history of completed workflows (workflows with status `SUCCESS`, `ERROR`, `CANCELLED`, or `MAX_RECOVERY_ATTEMPTS_EXCEEDED`); workflows that are still running, enqueued, or delayed are never deleted. diff --git a/docs/production/dbos-cloud/secrets.md b/docs/conductor/reference/dbos-cloud/secrets.md similarity index 100% rename from docs/production/dbos-cloud/secrets.md rename to docs/conductor/reference/dbos-cloud/secrets.md diff --git a/docs/production/dbos-cloud/workflow-management.md b/docs/conductor/reference/dbos-cloud/workflow-management.md similarity index 97% rename from docs/production/dbos-cloud/workflow-management.md rename to docs/conductor/reference/dbos-cloud/workflow-management.md index 93cf278ab..4e2ecfa1e 100644 --- a/docs/production/dbos-cloud/workflow-management.md +++ b/docs/conductor/reference/dbos-cloud/workflow-management.md @@ -23,7 +23,7 @@ For example, here is the trace of a workflow that processes multiple tasks concu ## Workflow Management -You can manage individual workflows directly from the DBOS console. +You can manage individual workflows directly from the DBOS Console. #### Cancelling Workflows diff --git a/docs/production/dbosctl.md b/docs/conductor/reference/dbosctl.md similarity index 94% rename from docs/production/dbosctl.md rename to docs/conductor/reference/dbosctl.md index 7edd3decd..1ebd764c9 100644 --- a/docs/production/dbosctl.md +++ b/docs/conductor/reference/dbosctl.md @@ -3,9 +3,9 @@ sidebar_position: 34 title: dbosctl CLI Reference --- -`dbosctl` is a command-line client for the [Conductor API](./conductor-api.md). It manages workflows, queues, schedules, applications, and API keys against DBOS-managed Conductor or a [self-hosted Conductor](./hosting-conductor.md), with the target selected by a named **profile**. +`dbosctl` is a command-line client for the [Conductor API](./conductor-api.md). It manages workflows, queues, schedules, applications, and API keys against DBOS-hosted Conductor or a [self-hosted Conductor](../self-hosting/hosting-conductor.md), with the target selected by a named **profile**. -The [`dbosctl sysdb`](#system-database-commands) commands are the exception: they manage the Postgres [system database](../explanations/system-tables.md) directly, so they take a database URL rather than a profile. +The [`dbosctl sysdb`](#system-database-commands) commands are the exception: they manage the Postgres [system database](../../explanations/system-tables.md) directly, so they take a database URL rather than a profile. ## Installation @@ -36,7 +36,7 @@ However you install it, `dbosctl version` reports what you have — a downloaded ## Quick Start -Against DBOS-managed Conductor: +Against DBOS-hosted Conductor: ```shell dbosctl config set managed --managed # create a profile pointing at cloud.dbos.dev @@ -67,11 +67,11 @@ dbosctl app list --profile local A profile is a named bundle of connection settings: which Conductor to talk to, how to authenticate, and the default organization and application. Profiles are stored in `config.yaml` under your OS configuration directory — `~/.config/dbos/config.yaml` on Linux, `~/Library/Application Support/dbos/config.yaml` on macOS. -A profile must target either DBOS-managed Conductor (`--managed`) or a self-hosted one (`--url`); the two are mutually exclusive. There are three common shapes: +A profile must target either DBOS-hosted Conductor (`--managed`) or a self-hosted one (`--url`); the two are mutually exclusive. There are three common shapes: | Shape | How to create it | Authentication | Identity | | --- | --- | --- | --- | -| DBOS-managed | `dbosctl config set --managed` | User JWT or `dbos_` API key | Your real user, or none for an API key | +| DBOS-hosted | `dbosctl config set --managed` | User JWT or `dbos_` API key | Your real user, or none for an API key | | Self-hosted with OIDC | `dbosctl config set --url --issuer --client-id ` | User JWT or `dbos_` API key | Your real user, or none for an API key | | Self-hosted, no auth | `dbosctl config set --url ` | None | Always `local` | @@ -223,7 +223,7 @@ Creates or updates a profile. Only the flags you pass are changed; fields you do **Arguments:** - ``: The profile to create or update. -- `--managed`: Make this a DBOS-managed Conductor profile (production domain `cloud.dbos.dev`). Mutually exclusive with `--url`. +- `--managed`: Make this a DBOS-hosted Conductor profile (production domain `cloud.dbos.dev`). Mutually exclusive with `--url`. - `--url `: Base URL of a self-hosted Conductor. Mutually exclusive with `--managed`. - `--issuer `: OIDC issuer URL. Implies bearer authentication. - `--client-id `: OIDC client ID. Implies bearer authentication. @@ -256,7 +256,7 @@ Shows one application's details. ### `dbosctl app register` **Description:** -Registers an application with Conductor. The name must match the application name in your DBOS configuration — see [Connecting To Conductor](./conductor.md#connecting-to-conductor). +Registers an application with Conductor. The name must match the application name in your DBOS configuration — see [Connecting To Conductor](../overview.md#connecting-to-conductor). **Arguments:** - ``: The application's name. @@ -273,7 +273,7 @@ Updates an application's tuning settings. Only the flags you pass are changed. - ``: The application's name. - `--executor-timeout-secs `: Seconds before an idle executor is considered gone. - `--global-timeout-ms `: Global workflow timeout, in milliseconds. -- `--gc-rows-threshold `: Number of most recently completed workflows whose history is kept; history of older completed workflows is garbage-collected. See [Workflow Retention Policies](./retention.md). +- `--gc-rows-threshold `: Number of most recently completed workflows whose history is kept; history of older completed workflows is garbage-collected. See [Workflow Retention Policies](../retention.md). - `--gc-time-threshold-ms `: Time, in milliseconds, after a workflow completes before its history is garbage-collected. - `--private-mode`: Whether the application is in private mode, in which it does not send workflow payload data — inputs, outputs, and events — to Conductor. Pass `--private-mode=false` to turn it back off. @@ -532,7 +532,7 @@ Lists the organization's API keys. Secrets are not shown. ### `dbosctl api-key create` **Description:** -Creates an API key and prints its secret. **The secret is shown once and cannot be retrieved afterwards.** By default the key is unscoped; narrow it with `--app` and `--permission`. See [Permissions and API Keys](./permissions.md). +Creates an API key and prints its secret. **The secret is shown once and cannot be retrieved afterwards.** By default the key is unscoped; narrow it with `--app` and `--permission`. See [Permissions and API Keys](../permissions.md). **Arguments:** - ``: A name for the key. @@ -572,7 +572,7 @@ Lists the permissions that can be granted to an API key or a role. ## System Database Commands `dbosctl sysdb` groups the commands that open a database instead of calling Conductor. -They connect to a Postgres (or CockroachDB) [system database](../explanations/system-tables.md) directly, so they take a database URL rather than a profile, and accept none of the [common flags](#common-flags) that resolve one. +They connect to a Postgres (or CockroachDB) [system database](../../explanations/system-tables.md) directly, so they take a database URL rather than a profile, and accept none of the [common flags](#common-flags) that resolve one. The system schema is shared by every DBOS SDK and the migrations are built into the `dbosctl` binary. @@ -593,7 +593,7 @@ DBOS_SYSTEM_DATABASE_URL=postgres://user:password@host:5432/dbos_sys dbosctl sys ### `dbosctl sysdb migrate` **Description:** -Creates or upgrades the DBOS [system database](../explanations/system-tables.md), applying every migration the schema is missing and creating the database and schema if they do not exist yet. +Creates or upgrades the DBOS [system database](../../explanations/system-tables.md), applying every migration the schema is missing and creating the database and schema if they do not exist yet. By default, a DBOS application automatically creates these on startup. However, in production environments, a DBOS application may not run with sufficient privilege to create databases or tables. In that case, the `migrate` command can be run with a privileged user to create all DBOS database tables. @@ -688,7 +688,7 @@ The schema itself is left migrated and immediately usable, so the database does Prompts for confirmation when run interactively. **Arguments:** -- `-a, --app `: Empty only the specified application's rows, for a [shared system database](../explanations/sharing-a-system-database.md). Unlike the `--app` argument used by Conductor commands, this argument is only read from the command line — never from `$DBOS_APP` or a profile. +- `-a, --app `: Empty only the specified application's rows, for a [shared system database](../../explanations/sharing-a-system-database.md). Unlike the `--app` argument used by Conductor commands, this argument is only read from the command line — never from `$DBOS_APP` or a profile. - `--drop-database`: Drop the whole database instead of emptying the DBOS tables. Cannot be combined with `--app` or `--schema`. - `--force`: Skip the confirmation prompt. Required when running non-interactively. - `-o, --output `: Output format for the row counts — `table` (default) or `json`. diff --git a/docs/production/retention.md b/docs/conductor/retention.md similarity index 91% rename from docs/production/retention.md rename to docs/conductor/retention.md index 1168df21f..13e6c51bb 100644 --- a/docs/production/retention.md +++ b/docs/conductor/retention.md @@ -1,9 +1,9 @@ --- -sidebar_position: 27 +sidebar_position: 20 title: Workflow Retention Policies --- -If you are using [Conductor](./conductor.md), you can configure workflow history retention policies for your application from the Retention Policy page of the DBOS Console. +If you are using [Conductor](./overview.md), you can configure workflow history retention policies for your application from the Retention Policy page of the DBOS Console. These settings let you configure how long workflow history is retained in your application's [system database](../explanations/system-tables.md). This is useful for managing the database disk usage of workflow history. diff --git a/docs/conductor/self-hosting/_category_.json b/docs/conductor/self-hosting/_category_.json new file mode 100644 index 000000000..560c31811 --- /dev/null +++ b/docs/conductor/self-hosting/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Self-Hosting Conductor", + "position": 90 +} diff --git a/docs/production/hosting-conductor-with-kubernetes.md b/docs/conductor/self-hosting/hosting-conductor-with-kubernetes.md similarity index 87% rename from docs/production/hosting-conductor-with-kubernetes.md rename to docs/conductor/self-hosting/hosting-conductor-with-kubernetes.md index be4123d3b..846cdc299 100644 --- a/docs/production/hosting-conductor-with-kubernetes.md +++ b/docs/conductor/self-hosting/hosting-conductor-with-kubernetes.md @@ -1,6 +1,6 @@ --- -sidebar_position: 70 -title: Self-Hosting Conductor With Kubernetes +sidebar_position: 2 +title: Deploying Conductor on Kubernetes --- :::info @@ -10,7 +10,8 @@ Self-hosting Conductor for commercial or production use requires a [license key] ## Overview -This guide covers deploying DBOS Conductor on Kubernetes so your applications get durable workflow execution, automatic workflow recovery, workflow management and observability — all running on infrastructure you control. +This guide covers deploying DBOS Conductor and the DBOS Console on Kubernetes. +It maps the components and production requirements from the [Self-Hosting Guide](./hosting-conductor.md) onto Kubernetes resources, then walks through a full deployment on AWS EKS. The Kubernetes manifests are portable to any conformant cluster. @@ -18,20 +19,14 @@ The Kubernetes manifests are portable to any conformant cluster. ## Deployments -**Database** — Conductor needs a PostgreSQL database, which we recommend configuring with a dedicated database role. +**Database** — Conductor's [Postgres database](./hosting-conductor.md#components) runs outside the cluster (this guide uses RDS). -**Conductor** — A stateless, single-container Deployment listening on port 8090. -All state lives in PostgreSQL: use a [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and not a [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/). -Required environment variables: -- `DBOS__CONDUCTOR_DB_URL` (connection string to the `dbos_conductor` database) -- `DBOS_CONDUCTOR_LICENSE_KEY` ([obtain a license key](./hosting-conductor.md#licensing)) +**Conductor** — A single-container [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), not a [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/), because all state lives in Postgres. +It listens on port 8090 and reads its [required environment variables](./hosting-conductor.md#conductor) from Secrets. +To run multiple replicas for [high availability](./hosting-conductor.md#high-availability), each pod must advertise its own address; `conductor.yaml` below sets `DBOS__ADVERTISE_ADDRESS` from the pod IP. -Conductor is out of the critical path and a single Conductor instance can serve tens of thousands of application servers. -You can still run multiple replicas for [high availability](./hosting-conductor.md#high-availability); each pod must then advertise its own address, which `conductor.yaml` below does from the pod IP. - -**Console** — A stateless, single-container Deployment listening on port 8080 (the Service in front of it publishes port 80.) -It connects to Conductor using the environment variable `DBOS_CONDUCTOR_URL`, set to a bare `host:port` (for example `conductor.dbos.svc.cluster.local:8090`). -Note that your *applications* also use a variable named `DBOS_CONDUCTOR_URL`, but it takes a full WebSocket URL (see *Register applications* below.) +**Console** — A single-container Deployment listening on port 8080, behind a Service that publishes port 80. +Its `DBOS_CONDUCTOR_URL` is the in-cluster Conductor Service address, `conductor.dbos.svc.cluster.local:8090`. :::info Updating Conductor @@ -41,7 +36,7 @@ Applications seamlessly reconnect to the new Conductor version with no impact on ::: :::info Register applications -After deploying Conductor and Console, [register your application, and generate an API key](./conductor.md#connecting-to-conductor). +After deploying Conductor and Console, [register your application, and generate an API key](../overview.md#connecting-to-conductor). The application connects to Conductor via WebSocket using this API key and the Conductor URL. With the [Ingress](#ingress) below, that URL is your Ingress hostname plus the `/conductor-api` prefix: @@ -56,14 +51,11 @@ Because this is a `wss://` connection, your application verifies the Ingress TLS ## Authentication -Conductor supports OAuth 2.0 with any OIDC-compliant provider. See the [authentication setup guide](./hosting-conductor.md#security). +Conductor supports OAuth 2.0 with any OIDC-compliant provider. See [Security](./hosting-conductor.md#security) for the provider setup and environment variables. :::warning -Conductor performs **no authentication** unless OAuth is enabled. Without it, all -API requests run as a built-in `local` organization admin, and Conductor does not -verify API keys on incoming WebSocket connections. Anyone who can reach the Ingress -can register applications, cancel, resume, fork, or delete workflows, and create API -tokens. Configure OAuth before exposing this deployment to any untrusted network. +Conductor performs **no authentication** unless OAuth is enabled, so anyone who can reach the Ingress has full admin access. +Configure OAuth before exposing this deployment to any untrusted network. ::: When configuring your OAuth provider, the callback URL and allowed web origin are your Ingress hostname (`https:///oauth/callback` and `https://`). @@ -71,21 +63,19 @@ The OAuth settings are not secrets, so they can be set directly in the Deploymen ## Ingress -In this guide, all external traffic enters through a reverse proxy that performs **TLS termination**, supports **WebSockets**, and routes by path: `/conductor-api/...` to Conductor, everything else to the Console. - -This guide uses [ingress-nginx](https://kubernetes.github.io/ingress-nginx/), but any reverse proxy meeting those requirements will work. The `ingress.yaml` below defines the routing it must implement. +All external traffic enters through an Ingress that meets the [reverse proxy requirements](./hosting-conductor.md#reverse-proxy-and-tls) and routes by path: `/conductor-api/...` to Conductor, everything else to the Console. -The DBOS SDK maintains a long-lived WebSocket connection to Conductor, so both the reverse proxy and any cloud load balancer in front of it (e.g., AWS ELB) should have idle timeouts high enough (this guide uses 3600s) to tolerate network hiccups. The DBOS SDK sends periodic pings to keep the connection alive, but a network hiccup that delays pings past the timeout will cause a disconnect. In case of disconnection, the DBOS SDK will reconnect automatically. +This guide uses [ingress-nginx](https://kubernetes.github.io/ingress-nginx/), but any ingress controller meeting those requirements will work. The `ingress.yaml` below defines the routing it must implement. +Set idle timeouts to 3600 seconds on both the ingress controller and the cloud load balancer in front of it (for example, AWS ELB). ## Security Best Practices -**Secret management** — Conductor deployments need credentials for PostgreSQL, a license key, and an API key. -Store these as Kubernetes Secrets and inject them via `secretKeyRef`. +**Secret management** — Store the [Conductor secrets](./hosting-conductor.md#secrets) as Kubernetes Secrets and inject them via `secretKeyRef`. For Git-safe storage, encrypt with [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets), [SOPS](https://github.com/getsops/sops), or a cloud-native secrets manager (AWS Secrets Manager, [Vault](https://developer.hashicorp.com/vault/docs/platform/k8s/vso), etc.). **Network policies** — Apply a default-deny ingress policy to the namespace, then add explicit allow rules for each pod. If Conductor and Console are co-located, allow traffic from the Console to Conductor on port 8090. -Conductor validates its license key against `https://cloud.dbos.dev` at startup and exits if it cannot reach it, so keep outbound HTTPS open from the Conductor pod (this also means its nodes need a route to the internet, such as a NAT gateway for private subnets). +Keep [outbound HTTPS](./hosting-conductor.md#network-access) open from the Conductor pod for license validation, which means its nodes need a route to the internet, such as a NAT gateway for private subnets. **RBAC** — Restrict which ServiceAccounts can read Secrets in the namespace. Conductor credentials (database URLs, license key, API key) should only be accessible to the pods that need them. @@ -756,7 +746,7 @@ console-xxxxxxxxx-xxxxx 1/1 Running 0 30s **Access the Console and Generate an API Key** -At this point, your self-hosted Conductor deployment is fully operational! Open `https:///` in your browser (accept the self-signed cert warning), then follow the [Conductor setup instructions](./conductor.md#connecting-to-conductor) to: +At this point, your self-hosted Conductor deployment is fully operational! Open `https:///` in your browser (accept the self-signed cert warning), then follow the [Conductor setup instructions](../overview.md#connecting-to-conductor) to: 1. Register your application 2. Generate an API key diff --git a/docs/production/hosting-conductor.md b/docs/conductor/self-hosting/hosting-conductor.md similarity index 69% rename from docs/production/hosting-conductor.md rename to docs/conductor/self-hosting/hosting-conductor.md index 8531deccf..6edeeaaf6 100644 --- a/docs/production/hosting-conductor.md +++ b/docs/conductor/self-hosting/hosting-conductor.md @@ -1,15 +1,31 @@ --- -sidebar_position: 15 -title: Self-Hosting Conductor +sidebar_position: 1 +title: Self-Hosting Guide --- :::info Self-hosted Conductor is released under a [proprietary license](https://www.dbos.dev/conductor-license) and requires a [license key](#licensing). ::: -There are many ways to self-host Conductor and the DBOS Console on your own infrastructure. +You can self-host Conductor and the DBOS Console on any infrastructure that runs containers. +This guide covers what a self-hosted deployment consists of and what it needs in production, independent of where you run it. +See our [Kubernetes guide](./hosting-conductor-with-kubernetes.md) for a specific walkthrough. -## Getting Started with Docker Compose +## Components + +A self-hosted deployment has three parts: + +| Component | Image | Port | Role | +|---|---|---|---| +| **Conductor** | [`dbosdev/conductor`](https://hub.docker.com/r/dbosdev/conductor) | 8090 | The control plane your applications connect to over WebSocket. | +| **DBOS Console** | [`dbosdev/console`](https://hub.docker.com/r/dbosdev/console) | 8080 | Conductor's web UI. | +| **Postgres** | Any Postgres | 5432 | Conductor's own database, holding its registry of applications, users, and settings. | + +Conductor's database is separate from the system databases your DBOS applications use. +Conductor never connects to your applications' databases; it exchanges workflow metadata and commands with your applications over their WebSocket connections. +In addition to the Console, you can use [Conductor's API](../reference/conductor-api.md) and the [dbosctl CLI](../reference/dbosctl.md) to manage your applications and their workflows. + +## Trying It Locally with Docker Compose For development and trial purposes, you can self-host Conductor and the DBOS Console on your development machine using Docker Compose. To do this, you need a development license key, which can be obtained from the DBOS Console [here](https://console.dbos.dev/settings/license-key). @@ -134,16 +150,17 @@ volumes: Start Conductor and the DBOS Console with `docker compose up`. After all containers have launched, navigate to http://localhost to view the self-hosted console. -## Connecting to Self-Hosted Conductor +## Connecting Applications -To connect your application to self-hosted Conductor, first [follow these steps](./conductor.md#connecting-to-conductor) in your self-hosted DBOS Console to register an application, generate an API key, and set it in your application. +To connect your application to self-hosted Conductor, first [follow these steps](../overview.md#connecting-to-conductor) in your self-hosted DBOS Console to register an application, generate an API key, and set it in your application. :::tip When self-hosting Conductor, make sure you register your application and generate your key in your self-hosted console, not at https://console.dbos.dev. ::: Then, provide your application with a websockets URL to your self-hosted Conductor server. -For example, for the Docker compose setup above, this URL is `ws://localhost:8090/`. +For example, for the Docker Compose setup above, this URL is `ws://localhost:8090/`. +In production, use a `wss://` URL that goes through your [reverse proxy](#reverse-proxy-and-tls). @@ -208,32 +225,55 @@ For development, testing, or evaluation purposes, you can obtain a trial Conduct You can provide your key to Conductor using the `DBOS_CONDUCTOR_LICENSE_KEY` environment variable. -## Hosting Conductor in Production +## Deploying to Production -You can self-host DBOS Conductor in production by deploying two services: the [Conductor](https://hub.docker.com/r/dbosdev/conductor) service and the [DBOS Console](https://hub.docker.com/r/dbosdev/console). +The Docker Compose setup above is for development only. +A production deployment runs the same containers with a managed Postgres database, a reverse proxy, secret storage, and [authentication](#security). -For a complete Kubernetes walkthrough covering infrastructure, secrets, ingress, and deployment, see [Self-Hosting Conductor with Kubernetes](./hosting-conductor-with-kubernetes.md). +:::tip +For a complete production deployment, including infrastructure, secrets, ingress, and authentication, follow the [Kubernetes guide](./hosting-conductor-with-kubernetes.md). +The requirements below apply to any platform. +::: ### Conductor -To deploy the Conductor service to production, it must connect to a Postgres database. -This database is purely for Conductor internal data (e.g., its registry of applications), it **is not** the database your DBOS applications connect to (Conductor does not need direct access to that database). -You can configure this database by setting the `DBOS__CONDUCTOR_DB_URL` environment variable in the Conductor container. +Run Conductor as a stateless container service with an orchestrator like Kubernetes, ECS, Cloud Run, Nomad, or plain VMs. +Because all state lives in Postgres, instances are interchangeable, and you can run several for [high availability](#high-availability). +Conductor requires these environment variables: -When deploying to production, we recommend placing the Conductor service behind a reverse proxy (e.g., Nginx) for web traffic ingress and TLS termination. -All traffic should be forwarded to the Conductor service container on port 8090. -You should also configure [authentication](#security). +| Environment variable | Description | +|---|---| +| `DBOS__CONDUCTOR_DB_URL` | Connection string for Conductor's Postgres database. We recommend a dedicated database role. | +| `DBOS_CONDUCTOR_LICENSE_KEY` | Your [license key](#licensing). | ### DBOS Console -To deploy the DBOS Console to production, it must connect to your Conductor service. -You can provide the URL of this service by setting the `DBOS_CONDUCTOR_URL` environment variable in your Console container. +Run the Console as a stateless container service listening on port 8080. +Set `DBOS_CONDUCTOR_URL` in the Console container to the bare `host:port` of your Conductor service (for example, `conductor.internal:8090`). +This differs from the `DBOS_CONDUCTOR_URL` your applications use, which is a full WebSocket URL. + +Without [OAuth authentication](#security), the Console has no user or organization management. + +### Reverse Proxy and TLS -When deploying to production, we recommend placing the Console container behind a reverse proxy (e.g., Nginx) for web traffic ingress and TLS termination. -All traffic should be forwarded to the Console service container on port 8080. -You should also configure [authentication](#security). -Without OAuth authentication, there is no user or organization management. -In order to enable these features, you must set up Conductor with an OAuth-compatible single-sign on solution. +Place Conductor and the Console behind a reverse proxy or load balancer (such as Nginx, an ingress controller, or a cloud load balancer) that **supports WebSockets** and does **TLS termination** (Conductor and the Console serve plain HTTP). Route traffic to Conductor on port 8090 and to the Console on port 8080. + +We recommend setting long idle timeouts on the proxy and on any load balancer in front of it to handle network hiccups (for example, 3600 seconds). The DBOS SDK sends periodic pings and reconnects automatically after a disconnect. + +### Network Access + +- **Outbound HTTPS from Conductor.** Conductor validates its license key against `https://cloud.dbos.dev` at startup and exits if it cannot reach it. Hosts in private networks need a route to the internet, such as a NAT gateway. For air-gapped deployments, [contact sales](https://www.dbos.dev/contact). +- **Console to Conductor.** The Console must reach Conductor on port 8090. +- **Conductor to Conductor.** In a [highly available](#high-availability) deployment, Conductor instances must reach each other directly. + +Conductor never needs access to your applications' databases, and your applications need only outbound access to the reverse proxy. + +### Secrets + +Conductor's database URL and license key are secrets. +Store them in your platform's secret store (such as Kubernetes Secrets, AWS Secrets Manager, or Vault) and inject them as environment variables. +The Conductor API keys your applications use to connect are also secrets, and belong in each application's secret store. +The OAuth settings below are not secrets and can be set directly in your deployment configuration. ## High Availability @@ -266,7 +306,13 @@ Set the `DBOS__ADVERTISE_ADDRESS` environment variable to a routable address (a ## Security To securely self-host Conductor in production, you should set up authentication and authorization for all API calls made to it. -Without these, your Conductor service could be accessed by unwanted entities. + +:::warning +Conductor performs **no authentication** unless OAuth is enabled. +Without it, all API requests run as a built-in `local` organization admin, and Conductor does not verify API keys on incoming WebSocket connections. +Anyone who can reach Conductor can register applications, cancel, resume, fork, or delete workflows, and create API keys. +Configure OAuth before exposing Conductor to any untrusted network. +::: You can integrate Conductor with any OAuth-compatible single-sign on (SSO) experience. To do this, first register the DBOS Console as an application and Conductor as an API (audience) with your OAuth provider. diff --git a/docs/production/workflow-management.md b/docs/conductor/workflow-management.md similarity index 95% rename from docs/production/workflow-management.md rename to docs/conductor/workflow-management.md index 7510fbe21..982a73659 100644 --- a/docs/production/workflow-management.md +++ b/docs/conductor/workflow-management.md @@ -1,10 +1,10 @@ --- -sidebar_position: 20 +sidebar_position: 10 title: Workflow Management --- :::info -Workflow observability and management features are only available for applications connected to [Conductor](./conductor.md). +Workflow observability and management features are only available for applications connected to [Conductor](./overview.md). ::: ## Viewing Workflows @@ -27,7 +27,7 @@ For example, here is the trace of a workflow that processes multiple tasks concu ## Workflow Management -You can manage individual workflows directly from the DBOS console. +You can manage individual workflows directly from the DBOS Console. #### Cancelling Workflows diff --git a/docs/explanations/concurrent-executions.md b/docs/explanations/concurrent-executions.md index 6f7ddad67..c894e29e2 100644 --- a/docs/explanations/concurrent-executions.md +++ b/docs/explanations/concurrent-executions.md @@ -6,7 +6,7 @@ description: How DBOS detects concurrent executions of the same workflow and con DBOS guarantees that every workflow runs to completion: if an executor crashes or becomes unreachable, another executor recovers its `PENDING` workflows and re-executes them from their last completed step. -The component responsible for recovery, e.g., [DBOS Conductor](../production/conductor.md), detects unhealthy executors and triggers recovery of its workflows. Sometimes, for example during the rollout of a new application image, that observation can be wrong, and a "zombie" executor could still be running your workflow. +The component responsible for recovery, e.g., [DBOS Conductor](../conductor/overview.md), detects unhealthy executors and triggers recovery of its workflows. Sometimes, for example during the rollout of a new application image, that observation can be wrong, and a "zombie" executor could still be running your workflow. This means the same workflow instance could be running on two executors. (DBOS detects and prevents concurrent executions of the same workflow on the same executor.) diff --git a/docs/explanations/sharing-a-system-database.md b/docs/explanations/sharing-a-system-database.md index 3a88b25ad..8c6335d58 100644 --- a/docs/explanations/sharing-a-system-database.md +++ b/docs/explanations/sharing-a-system-database.md @@ -18,7 +18,7 @@ Ownership determines which application runs what: - A schedule is fired only by the application that created it, and its workflows are owned by that application. - Application versions are tracked per application, so one application's deployments do not affect which version its peers consider latest. -[Retention policies](../production/retention.md) are an exception: their time and rows thresholds apply to the entire system database, including workflows owned by other applications. The global timeout remains scoped to the application that configures it. +[Retention policies](../conductor/retention.md) are an exception: their time and rows thresholds apply to the entire system database, including workflows owned by other applications. The global timeout remains scoped to the application that configures it. Queue, schedule, and version names remain globally unique across all applications sharing a system database; registering a name that a different application already owns raises an error. Workflow IDs are also unique across the entire system database, so ID-addressed operations (retrieving a workflow's handle, status, or result by ID, and sending messages or reading events and streams) work across applications regardless of ownership. @@ -114,7 +114,7 @@ dbosctl sysdb rename-application --to my-app --adopt-unclaimed-rows ## Renaming an Application Because ownership is recorded under the application's name, renaming an application requires transferring ownership of its rows. -To rename an application, first stop it, then run [`dbosctl sysdb rename-application`](../production/dbosctl.md#dbosctl-sysdb-rename-application), then restart it under its new name: +To rename an application, first stop it, then run [`dbosctl sysdb rename-application`](../conductor/reference/dbosctl.md#dbosctl-sysdb-rename-application), then restart it under its new name: ```shell dbosctl sysdb rename-application --from old-name --to new-name --db-url diff --git a/docs/explanations/system-tables.md b/docs/explanations/system-tables.md index f979036fd..afd6fa42b 100644 --- a/docs/explanations/system-tables.md +++ b/docs/explanations/system-tables.md @@ -136,7 +136,7 @@ Each row represents a different workflow execution and is written when the workf **Columns:** - **workflow_uuid**: The unique identifier of the workflow execution. - **inputs**: The serialized inputs of the workflow execution. -- **retention_timestamp**: The epoch timestamp (in milliseconds) when this row was written. Used when applying [retention policies](../production/retention.md). +- **retention_timestamp**: The epoch timestamp (in milliseconds) when this row was written. Used when applying [retention policies](../conductor/retention.md). ### dbos.workflow_output This table stores workflow outputs. @@ -146,7 +146,7 @@ Each row represents a different workflow execution and is written when the workf - **workflow_uuid**: The unique identifier of the workflow execution. - **output**: The serialized workflow output, if any. - **error**: The serialized error thrown by the workflow, if any. -- **retention_timestamp**: The epoch timestamp (in milliseconds) when this row was written. Used when applying [retention policies](../production/retention.md). +- **retention_timestamp**: The epoch timestamp (in milliseconds) when this row was written. Used when applying [retention policies](../conductor/retention.md). ### dbos.operation_outputs This table stores the outputs of workflow steps. @@ -164,7 +164,7 @@ Executions of DBOS methods like `DBOS.sleep` and `DBOS.send` are also recorded h - **completed_at_epoch_ms**: The epoch timestamp of when this step completed. - **serialization**: The name of the serialization format used for this step's output and error. Null if the workflow's default serializer was used. - **application_name**: The application that ran this step. -- **retention_timestamp**: The epoch timestamp (in milliseconds) when this row was written. Used when applying [retention policies](../production/retention.md). +- **retention_timestamp**: The epoch timestamp (in milliseconds) when this row was written. Used when applying [retention policies](../conductor/retention.md). ### dbos.notifications This table stores workflow messages/notifications. diff --git a/docs/faq.md b/docs/faq.md index 1cb13e2f6..7efb6ae4c 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -23,7 +23,7 @@ When sizing your database for DBOS, we recommend using that number (scaled to yo ### Why is my queue stuck? -If a DBOS queue is stuck (workflows are not moving from `ENQUEUED` to `PENDING`), it is likely that either the number of `PENDING` workflows exceeds the queue's global "concurrency" limit or the number of queued workflows in a `PENDING` state on each worker exceeds the queue's "worker concurrency" limit. In either case, new tasks cannot be dequeued until some currently executing tasks complete or are cancelled. You can view all tasks executing on a queue from the "Queues" tab of the [DBOS Console](./production/workflow-management.md) +If a DBOS queue is stuck (workflows are not moving from `ENQUEUED` to `PENDING`), it is likely that either the number of `PENDING` workflows exceeds the queue's global "concurrency" limit or the number of queued workflows in a `PENDING` state on each worker exceeds the queue's "worker concurrency" limit. In either case, new tasks cannot be dequeued until some currently executing tasks complete or are cancelled. You can view all tasks executing on a queue from the "Queues" tab of the [DBOS Console](./conductor/workflow-management.md) If you need to, you can cancel tasks to remove them from the queue. ### Why is my workflow not finishing? @@ -38,7 +38,7 @@ If you are using versioning, check that your app version matches the version of ### How can I cancel or fork a large number of workflows in a batch? -On the [DBOS Console](./production/workflow-management.md), filter for all workflows that meet your criteria, then select them all and apply a batch operation. +On the [DBOS Console](./conductor/workflow-management.md), filter for all workflows that meet your criteria, then select them all and apply a batch operation. Alternatively, write a script using the DBOS Client ([Python](./python/reference/client.md), [TypeScript](./typescript/reference/client.md), [Go](./golang/reference/dbos-context.md#newclient), [Java](./java/reference/client.md)) to list all the workflows that fit your criteria, then process them. ### Why am I seeing errors that objects cannot be deserialized? @@ -67,7 +67,7 @@ To make a workflow deterministic, make sure all non-deterministic operations (su Yes, you can call (or start, or enqueue) a workflow from inside another workflow. That workflow becomes a **child** of its caller and is by default assigned a workflow ID derived from its parent's. -If you view a workflow's trace from the [DBOS console](./production/workflow-management.md), it will include the workflow's children. +If you view a workflow's trace from the [DBOS Console](./conductor/workflow-management.md), it will include the workflow's children. ### Can I call a step from a step? @@ -85,7 +85,7 @@ If you enqueue a workflow with the ID of a workflow that already exists, it's a ### How can I reset all my DBOS state during development? -You can reset your DBOS system database and all internal DBOS state with the [`dbosctl sysdb reset`](./production/dbosctl.md#dbosctl-sysdb-reset) command. +You can reset your DBOS system database and all internal DBOS state with the [`dbosctl sysdb reset`](./conductor/reference/dbosctl.md#dbosctl-sysdb-reset) command. It empties the DBOS tables and leaves the schema migrated, so nothing has to provision the database again between runs, and it works the same way whatever language your application is written in. Pass `--app` to reset just one application's state in a [shared system database](./explanations/sharing-a-system-database.md), or `--drop-database` to drop the database outright. @@ -105,7 +105,7 @@ You can connect a DBOS application to its system database through a connection p DBOS creates tables for its internal state in its [system database](./explanations/system-tables.md). By default, a DBOS application automatically creates these on startup. However, in production environments, a DBOS application may not run with sufficient privilege to create databases or tables. -In that case, the [`dbosctl sysdb migrate`](./production/dbosctl.md#dbosctl-sysdb-migrate) command can be run with a privileged user to create all DBOS system tables. +In that case, the [`dbosctl sysdb migrate`](./conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) command can be run with a privileged user to create all DBOS system tables. Then, a DBOS application can run without privilege (requiring only access to the system database). ### What database privileges does DBOS need, and how do I grant them manually? @@ -131,7 +131,7 @@ ALTER DEFAULT PRIVILEGES IN SCHEMA "dbos" GRANT ALL ON SEQUENCES TO "your_app_ro ALTER DEFAULT PRIVILEGES IN SCHEMA "dbos" GRANT EXECUTE ON FUNCTIONS TO "your_app_role"; ``` -The [`dbosctl sysdb migrate`](./production/dbosctl.md#dbosctl-sysdb-migrate) command does this automatically if you supply an application role with `-r`/`--app-role`. +The [`dbosctl sysdb migrate`](./conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) command does this automatically if you supply an application role with `-r`/`--app-role`. ### How does DBOS scale? @@ -144,7 +144,7 @@ Conductor is required for correct workflow recovery in applications that use mor ### Why is my application not connecting to Conductor? The most common reason an application fails to connect to Conductor is that the name the application is registered with in its DBOS configuration does not match the name it was registered with in Conductor. -Additionally, if you are [self-hosting Conductor](./production/hosting-conductor.md) with a free license, you may connect at most one executor per application to Conductor, so additional executors may see their connections rejected. +Additionally, if you are [self-hosting Conductor](./conductor/self-hosting/hosting-conductor.md) with a free license, you may connect at most one executor per application to Conductor, so additional executors may see their connections rejected. To connect multiple executors, upgrade to a paid license. ### Why is my Conductor dashboard flickering? @@ -152,7 +152,7 @@ To connect multiple executors, upgrade to a paid license. The most common cause of flickering is that you have connected multiple executors using different system databases to the same Conductor application (for example, both an executor from your dev environment and one from your prod environment), causing Conductor to receive inconsistent data. For isolation, you should set up a separate Conductor app for each environment in which you run your DBOS application. For example, you may want to have separate dev, staging, and prod Conductor apps. -See [the docs](./production/conductor.md#managing-conductor-applications) for more information. +See [the docs](./conductor/overview.md#managing-conductor-applications) for more information. ### How are "checkpoints" calculated in Conductor pricing? diff --git a/docs/golang/programming-guide.md b/docs/golang/programming-guide.md index 76431b8c0..9ae57341f 100644 --- a/docs/golang/programming-guide.md +++ b/docs/golang/programming-guide.md @@ -360,12 +360,12 @@ Learn more about DBOS queues [here](./tutorials/queue-tutorial.md). ## 4. Connecting to DBOS Conductor -[Conductor](../production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS console](https://console.dbos.dev). +[Conductor](../conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS Console](https://console.dbos.dev). -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect). -Then, install [`dbosctl`](../production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](../conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead. ```shell @@ -409,8 +409,8 @@ go run main.go ``` Your app is now connected to Conductor! -Launch a workflow by visiting http://localhost:8080, then watch it execute in real time from the [DBOS console](https://console.dbos.dev). -Learn more about Conductor [here](../production/conductor.md). +Launch a workflow by visiting http://localhost:8080, then watch it execute in real time from the [DBOS Console](https://console.dbos.dev). +Learn more about Conductor [here](../conductor/overview.md). Congratulations! You've finished the DBOS Go guide. Next, you should: diff --git a/docs/golang/reference/configuration.md b/docs/golang/reference/configuration.md index 8e7122f94..2e3896be7 100644 --- a/docs/golang/reference/configuration.md +++ b/docs/golang/reference/configuration.md @@ -33,14 +33,14 @@ type Config struct { :::warning Deprecated `AdminServer` and `AdminServerPort` are deprecated and will be removed in v1.5.0. -Use [DBOS Conductor](../../production/conductor.md) for remote workflow management instead. +Use [DBOS Conductor](../../conductor/overview.md) for remote workflow management instead. ::: `ApplicationVersion` and `ExecutorID` are overridden by the `DBOS__APPVERSION` and `DBOS__VMID` environment variables, respectively, when set. `AppName` identifies your application. It must be between 3 and 256 characters long and contain only lowercase letters, numbers, dashes, and underscores. -An application connecting to [Conductor](../../production/conductor.md) (with `ConductorAPIKey` set) fails to start with a name outside that rule, because Conductor refuses to register it; a self-hosted application logs a warning and starts. +An application connecting to [Conductor](../../conductor/overview.md) (with `ConductorAPIKey` set) fails to start with a name outside that rule, because Conductor refuses to register it; a self-hosted application logs a warning and starts. Multiple applications (potentially in different languages) may [share a system database](../../explanations/sharing-a-system-database.md), in which case each must have a distinct name: the name identifies which application owns each workflow, queue, schedule, and application version, and applications only run their own workflows. If you rename an application, transfer ownership of its data with [`RenameApplication`](./methods.md#renameapplication) or the `dbos rename-application` CLI command. diff --git a/docs/golang/reference/methods.md b/docs/golang/reference/methods.md index fa942c2c4..78b828af8 100644 --- a/docs/golang/reference/methods.md +++ b/docs/golang/reference/methods.md @@ -1839,7 +1839,7 @@ func SetAlertHandler(ctx Context, handler AlertHandler) type AlertHandler func(name string, message string, metadata map[string]string) ``` -Register a handler to receive [alerts](../../production/alerting.md) from Conductor. +Register a handler to receive [alerts](../../conductor/alerting.md) from Conductor. The handler function is called with three arguments: - **name**: The type of alert rule. One of `WorkflowFailure`, `SlowQueue`, or `UnresponsiveApplication`. diff --git a/docs/golang/tutorials/workflow-management.md b/docs/golang/tutorials/workflow-management.md index 3dc9ba635..1aac1e766 100644 --- a/docs/golang/tutorials/workflow-management.md +++ b/docs/golang/tutorials/workflow-management.md @@ -3,19 +3,19 @@ sidebar_position: 50 title: Workflow Management --- -You can view and manage your durable workflow executions via the [DBOS Console](../../production/workflow-management.md) or programmatically. +You can view and manage your durable workflow executions via the [DBOS Console](../../conductor/workflow-management.md) or programmatically. ## Listing Workflows You can list your application's workflows programmatically via [`ListWorkflows`](../reference/methods#listworkflows). -You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../production/workflow-management.md). +You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../conductor/workflow-management.md). Workflow List ## Visualizing Workflow Execution -You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../production/workflow-management.md). +You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../conductor/workflow-management.md). For example, here is the trace of a workflow that processes multiple tasks concurrently by enqueueing child workflows: Workflow List diff --git a/docs/index.md b/docs/index.md index 624808fdb..5ccb7f736 100644 --- a/docs/index.md +++ b/docs/index.md @@ -38,8 +38,8 @@ import { FaHackerNews } from "react-icons/fa6"; icon={} /> } @@ -70,8 +70,8 @@ import { FaHackerNews } from "react-icons/fa6"; icon={} /> } @@ -102,8 +102,8 @@ import { FaHackerNews } from "react-icons/fa6"; icon={} /> } @@ -134,8 +134,8 @@ import { FaHackerNews } from "react-icons/fa6"; icon={} /> } @@ -232,7 +232,7 @@ import { PiQueueBold } from "react-icons/pi"; } /> @@ -245,7 +245,7 @@ import { PiQueueBold } from "react-icons/pi"; /> } diff --git a/docs/integrations/logfire.md b/docs/integrations/logfire.md index 57e5a28f2..146a96145 100644 --- a/docs/integrations/logfire.md +++ b/docs/integrations/logfire.md @@ -32,7 +32,7 @@ export OTEL_EXPORTER_OTLP_HEADERS='Authorization=your-write-token' :::tip -If you're deploying your app on DBOS Cloud, make sure to set `OTEL_EXPORTER_OTLP_HEADERS` in your application's [environment variables](../production/dbos-cloud/secrets). +If you're deploying your app on DBOS Cloud, make sure to set `OTEL_EXPORTER_OTLP_HEADERS` in your application's [environment variables](../conductor/reference/dbos-cloud/secrets). ::: @@ -95,7 +95,7 @@ export LOGFIRE_TOKEN='your-write-token' :::tip -If you're deploying your app on DBOS Cloud, make sure to set `LOGFIRE_TOKEN` in your application's [environment variables](../production/dbos-cloud/secrets). +If you're deploying your app on DBOS Cloud, make sure to set `LOGFIRE_TOKEN` in your application's [environment variables](../conductor/reference/dbos-cloud/secrets). ::: diff --git a/docs/integrations/mcp.md b/docs/integrations/mcp.md index 75ae4d4f2..222e0219f 100644 --- a/docs/integrations/mcp.md +++ b/docs/integrations/mcp.md @@ -6,7 +6,7 @@ toc_max_heading_level: 3 You can use the [DBOS Model Context Protocol (MCP) server](https://github.com/dbos-inc/dbos-mcp) to augment your LLM or agent with tools that can analyze and manage your DBOS workflows. This enables your LLM or agent to retrieve information on your applications' workflows and steps, for example to help you debug issues in development or production. -To use the server, your application should be connected to [Conductor](../production/conductor.md). +To use the server, your application should be connected to [Conductor](../conductor/overview.md). You may want to use the MCP server alongside a DBOS prompt or skills ([Python](../python/prompting.md), [TypeScript](../typescript/prompting.md), [Go](../golang/prompting.md), [Java](../java/prompting.md)) so your model has the most up-to-date information on DBOS. diff --git a/docs/integrations/parseable.md b/docs/integrations/parseable.md index a6e657d86..fbaa6deb5 100644 --- a/docs/integrations/parseable.md +++ b/docs/integrations/parseable.md @@ -6,7 +6,7 @@ hide_table_of_contents: false # Use DBOS With Parseable -[Parseable](https://www.parseable.com/) can ingest OpenTelemetry logs and traces from DBOS application processes as well as [Conductor Metrics](../production/metrics.md). +[Parseable](https://www.parseable.com/) can ingest OpenTelemetry logs and traces from DBOS application processes as well as [Conductor Metrics](../conductor/metrics.md). ![DBOS Logs and Traces on Parseable](./assets/parseable-screenshot.png) diff --git a/docs/integrations/supabase-edge-functions.md b/docs/integrations/supabase-edge-functions.md index 840827ceb..5fa3290d2 100644 --- a/docs/integrations/supabase-edge-functions.md +++ b/docs/integrations/supabase-edge-functions.md @@ -13,7 +13,7 @@ We recommend the following architecture: 3. Configure a [pg_cron](https://supabase.com/docs/guides/cron) job that starts your worker whenever there is new work for it to execute. Both functions connect to your project's Postgres database, so DBOS checkpoints your workflows next to the rest of your data. -Because an Edge Function is terminated when it exhausts its CPU or wall-clock budget, connect your worker to [Conductor](../production/conductor.md), which detects the disconnect and recovers interrupted workflows onto the next worker that starts. +Because an Edge Function is terminated when it exhausts its CPU or wall-clock budget, connect your worker to [Conductor](../conductor/overview.md), which detects the disconnect and recovers interrupted workflows onto the next worker that starts. :::info diff --git a/docs/java/programming-guide.md b/docs/java/programming-guide.md index a6b494d78..c2039ad7c 100644 --- a/docs/java/programming-guide.md +++ b/docs/java/programming-guide.md @@ -358,12 +358,12 @@ Learn more about DBOS queues [here](./tutorials/queue-tutorial.md). ## 4. Connecting to DBOS Conductor -[Conductor](../production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS console](https://console.dbos.dev). +[Conductor](../conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS Console](https://console.dbos.dev). -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect). -Then, install [`dbosctl`](../production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](../conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead. ```shell @@ -404,8 +404,8 @@ export DBOS_CONDUCTOR_KEY= ``` Your app is now connected to Conductor! -Launch a workflow by visiting http://localhost:8080, then watch it execute in real time from the [DBOS console](https://console.dbos.dev). -Learn more about Conductor [here](../production/conductor.md). +Launch a workflow by visiting http://localhost:8080, then watch it execute in real time from the [DBOS Console](https://console.dbos.dev). +Learn more about Conductor [here](../conductor/overview.md). Congratulations! You've finished the DBOS Java guide. Next, you should: diff --git a/docs/java/reference/client.md b/docs/java/reference/client.md index 6a1b3a0a2..37bdabffa 100644 --- a/docs/java/reference/client.md +++ b/docs/java/reference/client.md @@ -35,7 +35,7 @@ DBOSClient requires a PostgreSQL database. Providing a non-PostgreSQL `DataSourc The client never creates or migrates the system database. On construction, it checks that the system database schema has been migrated to a version compatible with this DBOS release, and throws `IllegalStateException` if the schema is missing or too old. -Launch a DBOS application (or run [`dbosctl sysdb migrate`](../../production/dbosctl.md#dbosctl-sysdb-migrate)) against the system database first. +Launch a DBOS application (or run [`dbosctl sysdb migrate`](../../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate)) against the system database first. **Parameters:** - **url**: The JDBC URL for your system database. @@ -535,7 +535,7 @@ public record ApplicationRowCounts( ``` Every workflow, step, queue, schedule, and application version is owned by the application that created it. -After renaming an application, use this method (or the [`dbosctl sysdb rename-application`](../../production/dbosctl.md#dbosctl-sysdb-rename-application) command) to transfer everything owned by the old name to the new name. +After renaming an application, use this method (or the [`dbosctl sysdb rename-application`](../../conductor/reference/dbosctl.md#dbosctl-sysdb-rename-application) command) to transfer everything owned by the old name to the new name. Returns the number of rows transferred, by table. The operation is idempotent: if interrupted, running it again resumes where it left off. diff --git a/docs/java/reference/lifecycle.md b/docs/java/reference/lifecycle.md index 40a1e91b4..3cf228d50 100644 --- a/docs/java/reference/lifecycle.md +++ b/docs/java/reference/lifecycle.md @@ -39,9 +39,9 @@ This configuration can be adjusted by using `with` methods that produce new conf - **`withAppName(String appName)`**: Your application's name. Required. It must be between 3 and 256 characters long and contain only lowercase letters, numbers, dashes, and underscores. -An application connecting to [Conductor](../../production/conductor.md) (with a Conductor key set) or running on DBOS Cloud fails to launch with a name outside that rule, because Conductor refuses to register it: `dbos.launch()` throws `IllegalArgumentException`. A self-hosted application logs a warning and launches. +An application connecting to [Conductor](../../conductor/overview.md) (with a Conductor key set) or running on DBOS Cloud fails to launch with a name outside that rule, because Conductor refuses to register it: `dbos.launch()` throws `IllegalArgumentException`. A self-hosted application logs a warning and launches. Multiple applications (potentially in different languages) may [share a system database](../../explanations/sharing-a-system-database.md), in which case each must have a distinct name: the name identifies which application owns each workflow, queue, schedule, and application version, and applications only run their own workflows. -If you rename an application, transfer ownership of its data with [`DBOSClient.renameApplication`](./client.md#renameapplication) or [`dbosctl sysdb rename-application`](../../production/dbosctl.md#dbosctl-sysdb-rename-application). +If you rename an application, transfer ownership of its data with [`DBOSClient.renameApplication`](./client.md#renameapplication) or [`dbosctl sysdb rename-application`](../../conductor/reference/dbosctl.md#dbosctl-sysdb-rename-application). - **`withAppVersion(String appVersion)`**: The code version for this application and its workflows. We recommend always setting it; if it is not set, DBOS computes a version from a hash of your workflow methods, which is only a fallback. Workflow versioning is documented [here](../tutorials/upgrading-workflows.md#versioning). @@ -61,13 +61,13 @@ Using a data source that doesn't support connection pooling like `PGSimpleDataSo - **`withMigrate(boolean enable)`**: If true, attempt to apply migrations to the system database. Defaults to true. -- **`withConductorKey(String key)`**: An API key for [DBOS Conductor](../../production/conductor.md). If provided, application is connected to Conductor. API keys can be created from the [DBOS console](https://console.dbos.dev). +- **`withConductorKey(String key)`**: An API key for [DBOS Conductor](../../conductor/overview.md). If provided, application is connected to Conductor. API keys can be created from the [DBOS Console](https://console.dbos.dev). - **`withConductorDomain(String domain)`**: The domain of the DBOS Conductor instance to connect to. Only needed when using a self-hosted Conductor. - **`withConductorExecutorMetadata(Map metadata)`**: Arbitrary key-value metadata attached to this executor and reported to Conductor. -- **`withAdminServer(boolean enable)`** *(deprecated since 0.9)*: Whether to run the built-in HTTP admin server. Use [DBOS Conductor](../../production/conductor.md) for remote administration instead. +- **`withAdminServer(boolean enable)`** *(deprecated since 0.9)*: Whether to run the built-in HTTP admin server. Use [DBOS Conductor](../../conductor/overview.md) for remote administration instead. - **`enableAdminServer()`** / **`disableAdminServer()`** *(deprecated since 0.9)*: Convenience methods equivalent to `withAdminServer(true)` and `withAdminServer(false)`. @@ -104,9 +104,9 @@ When deploying to DBOS Cloud, several environment variables are automatically se |----------|-------------| | `DBOS__CLOUD` | Set to `true` by DBOS Cloud. Enables cloud mode: `DBOS_APP_NAME` becomes required and the admin server is forced to port 3001. | | `DBOS_APP_NAME` | Overrides `DBOSConfig.appName()`. Required when `DBOS__CLOUD=true`; `launch()` throws if absent. | -| `DBOS__CONDUCTOR_URL` | URL of the DBOS Cloud Conductor. Overrides `withConductorDomain(...)`. | +| `DBOS__CONDUCTOR_URL` | URL of DBOS Conductor. Overrides `withConductorDomain(...)`. | | `DBOS__CONDUCTOR_APP_NAME` | Application name used to identify this executor with Conductor. | -| `DBOS__CONDUCTOR_KEY` | API key for DBOS Cloud Conductor. Overrides `withConductorKey(...)`. Set by the cloud platform; avoids putting credentials in `DBOSConfig`. | +| `DBOS__CONDUCTOR_KEY` | API key for DBOS Conductor. Overrides `withConductorKey(...)`. Set by the cloud platform; avoids putting credentials in `DBOSConfig`. | | `DBOS__VMID` | The executor ID of this process. Overrides `withExecutorId(...)` when `DBOS__CLOUD=true`. | These variables take precedence over any values set in `DBOSConfig`. In local development you do not need to set them. diff --git a/docs/java/reference/methods.md b/docs/java/reference/methods.md index 23dbbbcc6..f2818c470 100644 --- a/docs/java/reference/methods.md +++ b/docs/java/reference/methods.md @@ -1139,7 +1139,7 @@ public class DBOSApplicationNameConflictException extends RuntimeException { Thrown when registering a queue, creating or applying a schedule, or promoting an application version whose name is already owned by a different application sharing the system database. Queue, schedule, and version names are unique across all applications sharing a system database. -Either choose a different name or, if the owning application was renamed, transfer its rows first with [`dbosctl sysdb rename-application`](../../production/dbosctl.md#dbosctl-sysdb-rename-application) or [`DBOSClient.renameApplication`](./client.md#renameapplication). +Either choose a different name or, if the owning application was renamed, transfer its rows first with [`dbosctl sysdb rename-application`](../../conductor/reference/dbosctl.md#dbosctl-sysdb-rename-application) or [`DBOSClient.renameApplication`](./client.md#renameapplication). See [Sharing a System Database](../../explanations/sharing-a-system-database.md). ### DBOSSystemDatabaseException diff --git a/docs/java/reference/spring-boot-starter.md b/docs/java/reference/spring-boot-starter.md index 685787072..d0a17548a 100644 --- a/docs/java/reference/spring-boot-starter.md +++ b/docs/java/reference/spring-boot-starter.md @@ -53,7 +53,7 @@ All properties are in the `dbos.*` namespace. ### Admin Server :::warning -`dbos.admin-server.*` properties are deprecated since 0.9 and will be removed before 1.0. Use [DBOS Conductor](../../production/conductor.md) instead. +`dbos.admin-server.*` properties are deprecated since 0.9 and will be removed before 1.0. Use [DBOS Conductor](../../conductor/overview.md) instead. ::: | Property | Type | Default | Description | diff --git a/docs/java/tutorials/workflow-management.md b/docs/java/tutorials/workflow-management.md index bb3cffec6..239a8856a 100644 --- a/docs/java/tutorials/workflow-management.md +++ b/docs/java/tutorials/workflow-management.md @@ -3,13 +3,13 @@ sidebar_position: 50 title: Workflow Management --- -You can view and manage your durable workflow executions via the [DBOS Console](../../production/workflow-management.md) or programmatically. +You can view and manage your durable workflow executions via the [DBOS Console](../../conductor/workflow-management.md) or programmatically. ## Listing Workflows You can list your application's workflows programmatically via [`dbos.listWorkflows`](../reference/methods.md#listworkflows) or using the [`DBOSClient`](../reference/client.md#listworkflows). -You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../production/workflow-management.md). +You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../conductor/workflow-management.md). Workflow List @@ -17,7 +17,7 @@ You can also view a searchable and expandable list of your application's workflo You can list the steps of a workflow programmatically via [`dbos.listWorkflowSteps`](../reference/methods.md#listworkflowsteps) or using the [`DBOSClient`](../reference/client.md#listworkflowsteps). -You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../production/workflow-management.md). +You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../conductor/workflow-management.md). For example, here is the trace of a workflow that processes multiple tasks concurrently by enqueuing child workflows: Workflow List diff --git a/docs/java/upgrading.md b/docs/java/upgrading.md index d41dd6768..c76dbdfdf 100644 --- a/docs/java/upgrading.md +++ b/docs/java/upgrading.md @@ -7,7 +7,7 @@ title: Upgrading For every DBOS upgrade, migrate the system database before anything that needs the new schema runs. With `withMigrate(true)` (the default), `dbos.launch()` migrates the system database. -If you run with `withMigrate(false)`, run [`dbosctl sysdb migrate`](../production/dbosctl.md#dbosctl-sysdb-migrate) before deploying the upgrade. +If you run with `withMigrate(false)`, run [`dbosctl sysdb migrate`](../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) before deploying the upgrade. [`DBOSClient`](./reference/client.md) never migrates, so upgrade clients only after an upgraded application has launched or you have run `dbosctl sysdb migrate`. An application or client that needs a newer schema than the system database has throws `IllegalStateException` at launch or construction. @@ -31,16 +31,16 @@ If your application servers run 1.0, upgrade all of them to 1.1 before any serve #### Java CLI Removed The Java `dbos` CLI (the `transact-cli` module and its native binaries) has been removed. -Use [`dbosctl`](../production/dbosctl.md#system-database-commands) instead: +Use [`dbosctl`](../conductor/reference/dbosctl.md#system-database-commands) instead: | Before (1.0) | After (1.1) | |---|---| -| `dbos migrate` | [`dbosctl sysdb migrate`](../production/dbosctl.md#dbosctl-sysdb-migrate) | -| `dbos reset` | [`dbosctl sysdb reset`](../production/dbosctl.md#dbosctl-sysdb-reset) | +| `dbos migrate` | [`dbosctl sysdb migrate`](../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) | +| `dbos reset` | [`dbosctl sysdb reset`](../conductor/reference/dbosctl.md#dbosctl-sysdb-reset) | #### Application Names -If you use [Conductor](../production/conductor.md) or DBOS Cloud, `dbos.launch()` now throws `IllegalArgumentException` if your application name doesn't follow the [naming rule](./reference/lifecycle.md#dbosconfig): 3–256 lowercase letters, numbers, dashes, and underscores. +If you use [Conductor](../conductor/overview.md) or DBOS Cloud, `dbos.launch()` now throws `IllegalArgumentException` if your application name doesn't follow the [naming rule](./reference/lifecycle.md#dbosconfig): 3–256 lowercase letters, numbers, dashes, and underscores. Self-hosted applications only log a warning. To fix a name, change the name passed to `DBOSConfig.defaults(...)` or `withAppName`. In Spring Boot, set `dbos.application.name`; without it, DBOS uses `spring.application.name`, which often contains uppercase letters or dots. Rows created before 1.1 aren't owned by any application, so changing the name as part of this upgrade doesn't require transferring ownership. @@ -230,12 +230,12 @@ Note: `Timeout` itself is still used by `StartWorkflowOptions` and `WorkflowOpti The `dbos postgres` and `dbos workflow` subcommand groups have been removed from the CLI. The CLI now only supports `dbos migrate` and `dbos reset`. -Use the [`DBOSClient`](./reference/client.md) API or the [DBOS Console](../production/workflow-management.md) to manage workflows programmatically. +Use the [`DBOSClient`](./reference/client.md) API or the [DBOS Console](../conductor/workflow-management.md) to manage workflows programmatically. Additionally, the CLI now ships as a pre-compiled native binary (via GraalVM AOT compilation) for Linux, macOS, and Windows. Download the appropriate binary from the GitHub Releases page — no JVM required. :::note -The Java CLI was removed in v1.1 in favor of [`dbosctl`](../production/dbosctl.md). See [Java CLI Removed](#java-cli-removed). +The Java CLI was removed in v1.1 in favor of [`dbosctl`](../conductor/reference/dbosctl.md). See [Java CLI Removed](#java-cli-removed). ::: #### Jackson upgraded to 3.1.x diff --git a/docs/production/checklist.md b/docs/production/checklist.md index d28a598b3..01205d101 100644 --- a/docs/production/checklist.md +++ b/docs/production/checklist.md @@ -15,13 +15,13 @@ Here are some recommendations for configuring a Postgres database to best work w **If using a connection pooler, use it in session mode** - Connect your DBOS applications to your Postgres database either directly or using a connection pooler in session mode. Do not use a connection pooler in transaction mode as some Postgres features that DBOS uses (e.g., LISTEN/NOTIFY) are not compatible with it. [This page](https://www.pgbouncer.org/features.html) documents the differences. -**Configure a retention policy** - You should configure a [retention policy](./retention.md) for the workflows in your DBOS application to limit the total amount of storage DBOS uses. +**Configure a retention policy** - You should limit how much workflow history DBOS keeps in your system database. If you use Conductor, you can configure a [retention policy](../conductor/retention.md) for your application from the DBOS Console. **Manage the DBOS schema** - DBOS creates tables for its internal state in its [system database](../explanations/system-tables.md). By default, a DBOS application automatically creates these on startup. However, in production environments, a DBOS application may not run with sufficient privilege to create databases or tables. -In that case, the [`dbosctl sysdb migrate`](./dbosctl.md#dbosctl-sysdb-migrate) command can be run with a privileged user to create all DBOS system tables or migrate them to the latest version. +In that case, the [`dbosctl sysdb migrate`](../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) command can be run with a privileged user to create all DBOS system tables or migrate them to the latest version. Then, a DBOS application can run with lower privilege (requiring only access to the DBOS tables in the system database). If your database is managed by a DBA, `dbosctl sysdb migrate` can also print the SQL for them to apply instead of running it itself. @@ -54,8 +54,7 @@ Note that there is nothing DBOS-specific about this—we recommend following If your Postgres database does become unavailable, all DBOS applications connected to it will pause workflow execution until they reconnect. When your database becomes available again, they will seamlessly resume. -It is worth noting that DBOS Conductor is entirely out-of-band and off your application's workflow execution path. -Thus, its availability does not affect the availability of your applications. +If you use [DBOS Conductor](../conductor/overview.md), note that it is out-of-band and off your application's workflow execution path, so its availability does not affect the availability of your applications. If your connection to Conductor is interrupted, your applications will continue operating normally. All Conductor features (recovery, observability, workflow management) will automatically resume once connectivity is restored. diff --git a/docs/production/hosting-with-cloud-run.md b/docs/production/hosting-with-cloud-run.md index 016cf464d..e3e6a5144 100644 --- a/docs/production/hosting-with-cloud-run.md +++ b/docs/production/hosting-with-cloud-run.md @@ -5,7 +5,7 @@ title: Deploying With Google Cloud Run # Deploying a DBOS App on Google Cloud Run -This guide covers deploying a DBOS application to [Google Cloud Run](https://cloud.google.com/run) with a [Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres) database. It includes best practices for security, availability, and scalability. This guide assumes [DBOS Conductor](./conductor.md) is hosted separately. +This guide covers deploying a DBOS application to [Google Cloud Run](https://cloud.google.com/run) with a [Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres) database. It includes best practices for security, availability, and scalability. This guide assumes [DBOS Conductor](../conductor/overview.md) is hosted separately. ## Choosing a Cloud Run Execution Mode @@ -49,7 +49,7 @@ Deploying a DBOS application to Cloud Run is no different from deploying any oth The one DBOS-specific detail is the **database connection string**: it must be provided in `key=value` format (e.g., `user=postgres password=secret database=myappdb host=/cloudsql/...`). On Cloud Run, use the `--add-cloudsql-instances` flag to mount the [Cloud SQL Auth Proxy](https://cloud.google.com/sql/docs/postgres/connect-run) Unix socket, then pass the socket path as the `host` parameter. This gives your app a private, encrypted path to the database with no public IP. :::tip Schema migration -By default, DBOS creates its [system tables](../explanations/system-tables.md) on startup. If your Cloud Run service account doesn't have DDL privileges, run [`dbosctl sysdb migrate`](./dbosctl.md#dbosctl-sysdb-migrate) with a privileged user before deploying. +By default, DBOS creates its [system tables](../explanations/system-tables.md) on startup. If your Cloud Run service account doesn't have DDL privileges, run [`dbosctl sysdb migrate`](../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) with a privileged user before deploying. :::
@@ -143,7 +143,7 @@ echo -n "[YOUR_STRONG_PASSWORD]" | gcloud secrets create db-password \ --replication-policy="automatic" ``` -Store the [DBOS Conductor](./conductor.md) API key: +Store the [DBOS Conductor](../conductor/overview.md) API key: ```bash echo -n "[YOUR_CONDUCTOR_API_KEY]" | gcloud secrets create conductor-api-key \ @@ -470,7 +470,7 @@ To migrate them, [fork](../golang/tutorials/workflow-management.md#forking-workf #### Patching -With a fixed application version and patching enabled, the new worker pool instances automatically recover workflows from the previous deployment. [Conductor](./conductor.md) detects that the old instances went down and that new instances with the same version are available, triggering recovery without any manual intervention. +With a fixed application version and patching enabled, the new worker pool instances automatically recover workflows from the previous deployment. [Conductor](../conductor/overview.md) detects that the old instances went down and that new instances with the same version are available, triggering recovery without any manual intervention. ### Advanced scenarios diff --git a/docs/production/hosting-with-kubernetes.md b/docs/production/hosting-with-kubernetes.md index d3a632e80..80893dbf6 100644 --- a/docs/production/hosting-with-kubernetes.md +++ b/docs/production/hosting-with-kubernetes.md @@ -17,13 +17,13 @@ Pods are stateless and interchangeable and should use a standard [Deployment](ht ## Configuration -DBOS configuration contains sensitive values: the [system database URL](../explanations/system-tables.md) and, if using [Conductor](./conductor.md), an API key. +DBOS configuration contains sensitive values: the [system database URL](../explanations/system-tables.md) and, if using [Conductor](../conductor/overview.md), an API key. Store these as [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) and inject them via [`secretKeyRef`](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables). For Git-safe storage, encrypt with [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets), [SOPS](https://github.com/getsops/sops), or a cloud-native secrets manager. :::info Connecting to DBOS Conductor -If you use [DBOS managed Conductor](https://console.dbos.dev/), no `DBOS_CONDUCTOR_URL` is needed. The SDK connects automatically. -If you [self-host Conductor](./hosting-conductor.md), set `DBOS_CONDUCTOR_URL` in your application's environment. +If you use [DBOS-hosted Conductor](https://console.dbos.dev/), no `DBOS_CONDUCTOR_URL` is needed. The SDK connects automatically. +If you [self-host Conductor](../conductor/self-hosting/hosting-conductor.md), set `DBOS_CONDUCTOR_URL` in your application's environment. When Conductor is in a different cluster, use `wss://` so the WebSocket connection is encrypted. In the same cluster, use `ws://`, as Conductor requires TLS termination at the ingress layer. ::: @@ -33,7 +33,7 @@ When Conductor is in a different cluster, use `wss://` so the WebSocket connecti DBOS applications store workflow state in [system tables](../explanations/system-tables.md). These tables must be created before the application can start. -Run [`dbosctl sysdb migrate`](./dbosctl.md#dbosctl-sysdb-migrate) with an **admin** role that can create schema and grant permissions, and run the application with a **restricted** role that can only read/write data. Use the `--app-role` flag to grant the necessary schema permissions to the restricted role. +Run [`dbosctl sysdb migrate`](../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) with an **admin** role that can create schema and grant permissions, and run the application with a **restricted** role that can only read/write data. Use the `--app-role` flag to grant the necessary schema permissions to the restricted role. `dbosctl sysdb migrate` works well as a Kubernetes [Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) that you compose into your CI/CD pipeline. It is a single static binary carrying the migrations, so the Job needs no SDK toolchain and no copy of your application. @@ -45,7 +45,7 @@ In addition to [general tips](./checklist.md) for running a DBOS-enabled app in **Resource limits** — DBOS doesn't add significant CPU or memory overhead, but all DBOS SDKs run background tasks; setting more than 1000m CPU can significantly improve the performance of a busy application. -**Replicas** — configure more than one replica. Each replica starts an independent DBOS worker that can process scheduled workflows and handle tasks from DBOS queues. Each replica should have a unique executor ID (which is automatically assigned when using [DBOS Conductor](./conductor.md)) +**Replicas** — configure more than one replica. Each replica starts an independent DBOS worker that can process scheduled workflows and handle tasks from DBOS queues. Each replica should have a unique executor ID (which is automatically assigned when using [DBOS Conductor](../conductor/overview.md)) ## Upgrading Workflow Code @@ -59,7 +59,7 @@ Two patterns support this: ## Scaling with KEDA [KEDA](https://keda.sh/) scales application pods based on external metrics. -A simple pattern for scaling based on DBOS queue depth. When using [DBOS Conductor](./conductor.md), you can install an [autoscaling policy](./autoscaling.md#attaching-a-policy-with-the-api) for your application and configure a KEDA [ScaledObject](https://keda.sh/docs/latest/concepts/scaling-deployments/) to size your application based on the [policy recommendation](./autoscaling.md#reading-the-desired-executor-count). +A simple pattern for scaling based on DBOS queue depth. When using [DBOS Conductor](../conductor/overview.md), you can install an [autoscaling policy](../conductor/autoscaling.md#attaching-a-policy-with-the-api) for your application and configure a KEDA [ScaledObject](https://keda.sh/docs/latest/concepts/scaling-deployments/) to size your application based on the [policy recommendation](../conductor/autoscaling.md#reading-the-desired-executor-count). --- @@ -93,7 +93,7 @@ APP_ROLE_PASSWORD='choose-another-secure-password' CONDUCTOR_API_KEY='your-api-key' # Conductor URL -# DBOS Cloud: wss://cloud.dbos.dev/conductor/v1alpha1 +# DBOS-hosted Conductor: wss://cloud.dbos.dev/conductor/v1alpha1 # Self-hosted (same cluster): ws://conductor.dbos.svc.cluster.local:8090 # Self-hosted (external): wss://your-conductor-hostname/conductor/ CONDUCTOR_URL='wss://cloud.dbos.dev/conductor/v1alpha1' @@ -127,9 +127,9 @@ aws sts get-caller-identity **DBOS Conductor** -This walkthrough connects the application to [DBOS Conductor](./conductor.md) for workflow recovery and observability. -You can use either [DBOS Cloud](https://console.dbos.dev/) or a [self-hosted Conductor](./hosting-conductor-with-kubernetes.md). -You'll need the **Conductor URL** and an **API key** — both are available from the Console after [registering your application](./conductor.md#connecting-to-conductor). +This walkthrough connects the application to [DBOS Conductor](../conductor/overview.md) for workflow recovery and observability. +You can use either [DBOS-hosted Conductor](https://console.dbos.dev/) or a [self-hosted Conductor](../conductor/self-hosting/hosting-conductor-with-kubernetes.md). +You'll need the **Conductor URL** and an **API key** — both are available from the Console after [registering your application](../conductor/overview.md#connecting-to-conductor). **Create an EKS Cluster** @@ -431,7 +431,7 @@ postgres-admin Opaque 2 10s DBOS applications store workflow state in [system tables](../explanations/system-tables.md). These tables must be created before the application can start. -We use a separate Kubernetes Job that runs [`dbosctl sysdb migrate`](./dbosctl.md#dbosctl-sysdb-migrate) with **admin** credentials, then the application itself runs with a **restricted** role that can only read/write data — not modify schema. +We use a separate Kubernetes Job that runs [`dbosctl sysdb migrate`](../conductor/reference/dbosctl.md#dbosctl-sysdb-migrate) with **admin** credentials, then the application itself runs with a **restricted** role that can only read/write data — not modify schema. This separation follows the principle of least privilege: the application never holds the keys to alter its own schema. @@ -649,7 +649,7 @@ spec: ``` Replace `${CONDUCTOR_URL}` with the value you set earlier: -- **DBOS Cloud**: `wss://cloud.dbos.dev/conductor/v1alpha1` +- **DBOS-hosted Conductor**: `wss://cloud.dbos.dev/conductor/v1alpha1` - **Self-hosted (same cluster)**: `ws://conductor.dbos.svc.cluster.local:8090` - **Self-hosted (external)**: `wss:///conductor/` @@ -700,7 +700,7 @@ Store the certificate as a Kubernetes Secret, mount it into an init container th
:::tip -When using DBOS managed Conductor, you don't need to set `DBOS_CONDUCTOR_URL` in the manifest. +When using DBOS-hosted Conductor, you don't need to set `DBOS_CONDUCTOR_URL` in the manifest. ::: **Deploy the Application** diff --git a/docs/production/workflow-recovery.md b/docs/production/workflow-recovery.md index 59fe5d219..d368d99ea 100644 --- a/docs/production/workflow-recovery.md +++ b/docs/production/workflow-recovery.md @@ -22,14 +22,5 @@ When an application with an executor ID restarts, it only recovers pending workf ### Recovery With Conductor -If your application is connected to [DBOS Conductor](./conductor.md), workflow recovery is automatic. -When Conductor detects that an executor is unhealthy, it automatically signals another executor to recover its workflows. - -When an executor disconnects from Conductor, its status is changed to `DISCONNECTED` while Conductor waits for it to reconnect. -If it has not reconnected after a certain period of time, its status is changed to `DEAD` and Conductor signals another executor to recover its workflows. -After recovery is confirmed, Conductor deletes its record of the executor. - -By default, the executor timeout is 60 seconds, so Conductor waits 60 seconds after an executor disconnects before recovering its workflows. -You can configure the executor timeout per application from the DBOS Console. - -Workflow Timeout Configuration +If your application is connected to [DBOS Conductor](../conductor/overview.md), workflow recovery is automatic: when Conductor detects that an executor is unhealthy, it signals another executor to recover its workflows. +See [Distributed Recovery](../conductor/distributed-recovery.md) for details. diff --git a/docs/python/examples/customer-service.md b/docs/python/examples/customer-service.md index af401ff91..07e27e343 100644 --- a/docs/python/examples/customer-service.md +++ b/docs/python/examples/customer-service.md @@ -3,7 +3,7 @@ sidebar_position: 50 title: Reliable Customer Service Agent --- -In this example, you'll learn how to build a reliable AI-powered customer service agent with DBOS and [LangGraph](https://langchain-ai.github.io/langgraph/) and serverlessly deploy it to DBOS Cloud. This example demonstrates how **DBOS makes it easy to connect your AI agent to your existing production systems**, especially when integrating **human decision-making** into automated processes. +In this example, you'll learn how to build a reliable AI-powered customer service agent with DBOS and [LangGraph](https://langchain-ai.github.io/langgraph/). This example demonstrates how **DBOS makes it easy to connect your AI agent to your existing production systems**, especially when integrating **human decision-making** into automated processes. You can chat with this LLM-powered AI agent to check the status of your purchase order, or request a refund for your order. Even if the agent is interrupted during refund processing, upon restart it automatically recovers, finishes processing the refund, then proceeds to the next step in its workflow. diff --git a/docs/python/programming-guide.md b/docs/python/programming-guide.md index 0c7177e03..961394bff 100644 --- a/docs/python/programming-guide.md +++ b/docs/python/programming-guide.md @@ -245,12 +245,12 @@ Learn more about DBOS queues [here](./tutorials/queue-tutorial.md). ## 4. Connecting to DBOS Conductor -[Conductor](../production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS console](https://console.dbos.dev). +[Conductor](../conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS Console](https://console.dbos.dev). -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect). -Then, install [`dbosctl`](../production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](../conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead. ```shell @@ -294,8 +294,8 @@ python3 main.py ``` Your app is now connected to Conductor! -Launch a workflow by visiting http://localhost:8000, then watch it execute in real time from the [DBOS console](https://console.dbos.dev). -Learn more about Conductor [here](../production/conductor.md). +Launch a workflow by visiting http://localhost:8000, then watch it execute in real time from the [DBOS Console](https://console.dbos.dev). +Learn more about Conductor [here](../conductor/overview.md). Congratulations! You've finished the DBOS Python guide. Next, you should: diff --git a/docs/python/prompting.md b/docs/python/prompting.md index b24eece07..076867ac0 100644 --- a/docs/python/prompting.md +++ b/docs/python/prompting.md @@ -1845,7 +1845,7 @@ sqlite:///[application_name].sqlite - **system_database_engine**: A custom SQLAlchemy engine to use to connect to your system database. If provided, DBOS will not create an engine but use this instead. - **use_listen_notify**: Whether to use PostgreSQL LISTEN/NOTIFY (`True`) or polling (`False`) to await notifications and events. Defaults to `True`. Ignored in SQLite, which always uses polling. - **observability_query_timeout_sec**: The statement timeout, in seconds, applied to observability queries (such as listing workflows, queued workflows, and workflow steps) on a Postgres system database, so a slow query on a large database does not hold resources indefinitely. A query that exceeds the timeout raises `DBOSQueryTimeoutError`. Defaults to 30 seconds. Set to zero or a negative value to disable the timeout. -- **conductor_key**: An API key for DBOS Conductor. If provided, application is connected to Conductor. API keys can be created from the DBOS console. +- **conductor_key**: An API key for DBOS Conductor. If provided, application is connected to Conductor. API keys can be created from the DBOS Console. - **conductor_url**: The URL of the Conductor service to connect to. Only set if you are self-hosting Conductor. - **enable_otlp**: Enable DBOS OpenTelemetry tracing and export. Defaults to False. - **otlp_traces_endpoints**: DBOS operations automatically generate OpenTelemetry Traces. Use this field to declare a list of OTLP-compatible trace receivers. Requires `enable_otlp` to be True. diff --git a/docs/python/reference/configuration.md b/docs/python/reference/configuration.md index 77db37402..3782c6ee5 100644 --- a/docs/python/reference/configuration.md +++ b/docs/python/reference/configuration.md @@ -119,10 +119,10 @@ A system database ahead of the required version is accepted, so a process with m ### Conductor Settings -- **conductor_key**: An API key for [DBOS Conductor](../../production/conductor.md). If provided, application connects to Conductor. API keys can be created from the [DBOS console](https://console.dbos.dev). +- **conductor_key**: An API key for [DBOS Conductor](../../conductor/overview.md). If provided, application connects to Conductor. API keys can be created from the [DBOS Console](https://console.dbos.dev). - **conductor_url**: The URL of the Conductor service to connect to. Only set if you are self-hosting Conductor. - **conductor_executor_metadata**: A JSON-serializable dictionary of metadata to associate with this executor. This metadata is sent to Conductor and displayed on the dashboard, making it easier to identify executors (e.g., by region, instance type, or deployment environment). -- **conductor_metadata_only_mode**: If `True`, this process sends only workflow metadata to Conductor, never workflow data (inputs, outputs, errors, step outputs, events, messages, streams, or schedule context), regardless of the [metadata-only mode](../../production/conductor.md#metadata-only-mode) setting in the Conductor console. Defaults to `False`. +- **conductor_metadata_only_mode**: If `True`, this process sends only workflow metadata to Conductor, never workflow data (inputs, outputs, errors, step outputs, events, messages, streams, or schedule context), regardless of the [metadata-only mode](../../conductor/overview.md#metadata-only-mode) setting in the Conductor console. Defaults to `False`. ### Logging and Tracing Settings @@ -154,7 +154,7 @@ A system database ahead of the required version is accepted, so a process with m ## DBOS Configuration File -Some tools in the DBOS ecosystem, including [DBOS Cloud](../../production/dbos-cloud/deploying-to-cloud.md) and the [DBOS CLI](./cli.md), are configured by a `dbos-config.yaml` file. +Some tools in the DBOS ecosystem, including [DBOS Cloud](../../conductor/reference/dbos-cloud/deploying-to-cloud.md) and the [DBOS CLI](./cli.md), are configured by a `dbos-config.yaml` file. You can create a `dbos-config.yaml` with default parameters with: @@ -177,7 +177,7 @@ This connection string is used by the DBOS [CLI](cli.md). It has the same format as the `system_database_url` you pass to the DBOS constructor. - **runtimeConfig**: - **start**: (required only in DBOS Cloud) The command(s) with which to start your app. Called from [`dbos start`](../reference/cli.md#dbos-start), which is used to start your app in DBOS Cloud. - - **setup**: Setup commands to run before your application is built in DBOS Cloud. Used only in DBOS Cloud. Documentation [here](../../production/dbos-cloud/application-management.md#customizing-microvm-setup). + - **setup**: Setup commands to run before your application is built in DBOS Cloud. Used only in DBOS Cloud. Documentation [here](../../conductor/reference/dbos-cloud/application-management.md#customizing-microvm-setup). ### Configuration Schema File diff --git a/docs/python/reference/contexts.md b/docs/python/reference/contexts.md index 28e077365..14b494402 100644 --- a/docs/python/reference/contexts.md +++ b/docs/python/reference/contexts.md @@ -2138,7 +2138,7 @@ def my_handler(rule_type: str, message: str, metadata: Dict[str, str]) -> None: ... ``` -Register a function to handle [alerts](../../production/alerting.md) received from Conductor. +Register a function to handle [alerts](../../conductor/alerting.md) received from Conductor. The handler function is called with three arguments: - **rule_type**: The type of alert rule. One of `WorkflowFailure`, `SlowQueue`, or `UnresponsiveApplication`. diff --git a/docs/python/tutorials/logging-and-tracing.md b/docs/python/tutorials/logging-and-tracing.md index 051975984..d2a277815 100644 --- a/docs/python/tutorials/logging-and-tracing.md +++ b/docs/python/tutorials/logging-and-tracing.md @@ -239,4 +239,4 @@ For example, try using [Jaeger](https://www.jaegertracing.io/docs/latest/getting ### Metrics -Using [Conductor](../../production/conductor.md), you can also scrape metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint. See [Metrics](../../production/metrics.md) for details. +Using [Conductor](../../conductor/overview.md), you can also scrape metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint. See [Metrics](../../conductor/metrics.md) for details. diff --git a/docs/python/tutorials/workflow-management.md b/docs/python/tutorials/workflow-management.md index 6042996c3..37b1e0639 100644 --- a/docs/python/tutorials/workflow-management.md +++ b/docs/python/tutorials/workflow-management.md @@ -3,13 +3,13 @@ sidebar_position: 60 title: Workflow Management --- -You can view and manage your durable workflow executions via the [DBOS Console](../../production/workflow-management.md), programmatically, or via command line. +You can view and manage your durable workflow executions via the [DBOS Console](../../conductor/workflow-management.md), programmatically, or via command line. ## Listing Workflows You can list your application's workflows programmatically via [`DBOS.list_workflows`](../reference/contexts.md#list_workflows) or from the command line with [`dbos workflow list`](../reference/cli.md#dbos-workflow-list). -You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../production/workflow-management.md). +You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../conductor/workflow-management.md). Workflow List @@ -17,7 +17,7 @@ You can also view a searchable and expandable list of your application's workflo You can list the steps of a workflow programmatically via [`DBOS.list_workflow_steps`](../reference/contexts.md#list_workflow_steps) or from the command line with [`dbos workflow steps`](../reference/cli.md#dbos-workflow-steps). -You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../production/workflow-management.md). +You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../conductor/workflow-management.md). For example, here is the trace of a workflow that processes multiple tasks concurrently by enqueueing child workflows: Workflow List diff --git a/docs/quickstart.md b/docs/quickstart.md index 284debc4e..a5b9d066c 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -118,16 +118,16 @@ Congratulations, you've run your first durable workflow with DBOS!
-[Conductor](./production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +[Conductor](./conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect).
-Then, install [`dbosctl`](./production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](./conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead.
@@ -175,7 +175,7 @@ python3 main.py
Your app is now connected to Conductor! -You can view and manage its workflows from the [DBOS console](https://console.dbos.dev). +You can view and manage its workflows from the [DBOS Console](https://console.dbos.dev).
@@ -289,16 +289,16 @@ Congratulations, you've run your first durable workflow with DBOS!
-[Conductor](./production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +[Conductor](./conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect).
-Then, install [`dbosctl`](./production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](./conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead.
@@ -346,7 +346,7 @@ npm run start
Your app is now connected to Conductor! -You can view and manage its workflows from the [DBOS console](https://console.dbos.dev). +You can view and manage its workflows from the [DBOS Console](https://console.dbos.dev).
@@ -446,16 +446,16 @@ Congratulations, you've run your first durable workflow with DBOS!
-[Conductor](./production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +[Conductor](./conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect).
-Then, install [`dbosctl`](./production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](./conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead.
@@ -503,7 +503,7 @@ go run main.go
Your app is now connected to Conductor! -You can view and manage its workflows from the [DBOS console](https://console.dbos.dev). +You can view and manage its workflows from the [DBOS Console](https://console.dbos.dev).
@@ -608,16 +608,16 @@ Congratulations, you've run your first durable workflow with DBOS!
-[Conductor](./production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +[Conductor](./conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect).
-Then, install [`dbosctl`](./production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](./conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead.
@@ -665,7 +665,7 @@ export DBOS_CONDUCTOR_KEY=
Your app is now connected to Conductor! -You can view and manage its workflows from the [DBOS console](https://console.dbos.dev). +You can view and manage its workflows from the [DBOS Console](https://console.dbos.dev).
diff --git a/docs/typescript/programming-guide.md b/docs/typescript/programming-guide.md index f70994f7e..21b7209bd 100644 --- a/docs/typescript/programming-guide.md +++ b/docs/typescript/programming-guide.md @@ -257,12 +257,12 @@ Learn more about DBOS queues [here](./tutorials/queue-tutorial.md). ## 4. Connecting to DBOS Conductor -[Conductor](../production/conductor.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. -Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS console](https://console.dbos.dev). +[Conductor](../conductor/overview.md) is the control plane for your durable workflows, providing distributed workflow recovery, observability, and management. +Once you connect your app to Conductor, you can view and manage all its workflows and queued tasks from the [DBOS Console](https://console.dbos.dev). -To connect your app to Conductor, first sign up for an account on the [DBOS console](https://console.dbos.dev/login-redirect). +To connect your app to Conductor, first sign up for an account on the [DBOS Console](https://console.dbos.dev/login-redirect). -Then, install [`dbosctl`](../production/dbosctl.md), the Conductor command-line client. +Then, install [`dbosctl`](../conductor/reference/dbosctl.md), the Conductor command-line client. On Windows, [download a release binary](https://github.com/dbos-inc/dbos-ctl/releases) instead. ```shell @@ -302,8 +302,8 @@ npm run start ``` Your app is now connected to Conductor! -Launch a workflow by visiting http://localhost:3000, then watch it execute in real time from the [DBOS console](https://console.dbos.dev). -Learn more about Conductor [here](../production/conductor.md). +Launch a workflow by visiting http://localhost:3000, then watch it execute in real time from the [DBOS Console](https://console.dbos.dev). +Learn more about Conductor [here](../conductor/overview.md). Congratulations! You've finished the DBOS TypeScript guide. Next, you should: diff --git a/docs/typescript/reference/configuration.md b/docs/typescript/reference/configuration.md index ca892c79c..91a38bbc8 100644 --- a/docs/typescript/reference/configuration.md +++ b/docs/typescript/reference/configuration.md @@ -55,7 +55,7 @@ export interface DBOSConfig { } ``` -In [DBOS Cloud](../../production/dbos-cloud/deploying-to-cloud.md), DBOS takes your application's name, system database URL, and OTLP endpoints from environment variables supplied by DBOS Cloud (`DBOS_APP_NAME`, `DBOS_SYSTEM_DATABASE_URL`, `DBOS__OTLP_TRACES_ENDPOINT`, and `DBOS__OTLP_LOGS_ENDPOINT`), overriding `name` and `systemDatabaseUrl` and adding to `otlpTracesEndpoints` and `otlpLogsEndpoints`. +In [DBOS Cloud](../../conductor/reference/dbos-cloud/deploying-to-cloud.md), DBOS takes your application's name, system database URL, and OTLP endpoints from environment variables supplied by DBOS Cloud (`DBOS_APP_NAME`, `DBOS_SYSTEM_DATABASE_URL`, `DBOS__OTLP_TRACES_ENDPOINT`, and `DBOS__OTLP_LOGS_ENDPOINT`), overriding `name` and `systemDatabaseUrl` and adding to `otlpTracesEndpoints` and `otlpLogsEndpoints`. The application version and executor ID also come from DBOS Cloud (`DBOS__APPVERSION` and `DBOS__VMID`), so `applicationVersion` and `executorID` are ignored there, as is `enablePatching`'s effect on the application version (it still enables [`DBOS.patch`](./workflows-steps.md#patch)). ### Application Settings @@ -161,7 +161,7 @@ await DBOS.launch(); ## DBOS Configuration File -Some tools in the DBOS ecosystem, including [DBOS Cloud](../../production/dbos-cloud/deploying-to-cloud.md) and the [DBOS CLI](./cli.md), are configured by a `dbos-config.yaml` file. +Some tools in the DBOS ecosystem, including [DBOS Cloud](../../conductor/reference/dbos-cloud/deploying-to-cloud.md) and the [DBOS CLI](./cli.md), are configured by a `dbos-config.yaml` file. Your application itself does not read this file; configure it with [`DBOS.setConfig`](#configuring-dbos). Here is an example configuration file with default parameters: @@ -190,7 +190,7 @@ This connection string is used by the DBOS [CLI](cli.md). It has the same format as the `systemDatabaseUrl` you pass to `DBOS.setConfig()`. - **runtimeConfig**: - **start**: (required only in DBOS Cloud) The command(s) with which to start your app. Called from [`npx dbos start`](./cli.md#npx-dbos-start), which is used to start your app in DBOS Cloud. - - **setup**: (optional) Setup commands to run before your application is built in DBOS Cloud. Used only in DBOS Cloud. Documentation [here](../../production/dbos-cloud/application-management.md#customizing-microvm-setup). + - **setup**: (optional) Setup commands to run before your application is built in DBOS Cloud. Used only in DBOS Cloud. Documentation [here](../../conductor/reference/dbos-cloud/application-management.md#customizing-microvm-setup). ### Configuration Schema File diff --git a/docs/typescript/reference/dbos-class.md b/docs/typescript/reference/dbos-class.md index 4ddc5208e..432d37a3d 100644 --- a/docs/typescript/reference/dbos-class.md +++ b/docs/typescript/reference/dbos-class.md @@ -62,10 +62,10 @@ main().catch(console.log); ``` **Parameters:** -- **conductorKey**: An API key for [DBOS Conductor](../../production/conductor.md). If provided, application connects to Conductor. API keys can be created from the [DBOS console](https://console.dbos.dev). +- **conductorKey**: An API key for [DBOS Conductor](../../conductor/overview.md). If provided, application connects to Conductor. API keys can be created from the [DBOS Console](https://console.dbos.dev). - **conductorURL**: The URL of the Conductor service to connect to. Only set if you are self-hosting Conductor. - **conductorExecutorMetadata**: A JSON-serializable dictionary of metadata to associate with this executor. This metadata is sent to Conductor and displayed on the dashboard, making it easier to identify executors (e.g., by region, instance type, or deployment environment). -- **conductorMetadataOnlyMode**: If `true`, this process sends only workflow metadata to Conductor, never workflow data (inputs, outputs, errors, step outputs, events, messages, streams, or schedule context), regardless of the [metadata-only mode](../../production/conductor.md#metadata-only-mode) setting in the Conductor console. Defaults to `false`. +- **conductorMetadataOnlyMode**: If `true`, this process sends only workflow metadata to Conductor, never workflow data (inputs, outputs, errors, step outputs, events, messages, streams, or schedule context), regardless of the [metadata-only mode](../../conductor/overview.md#metadata-only-mode) setting in the Conductor console. Defaults to `false`. ### DBOS.shutdown diff --git a/docs/typescript/reference/methods.md b/docs/typescript/reference/methods.md index 899d97baa..ea2c36914 100644 --- a/docs/typescript/reference/methods.md +++ b/docs/typescript/reference/methods.md @@ -1297,7 +1297,7 @@ DBOS.setAlertHandler( ): void ``` -Register a handler to receive [alerts](../../production/alerting.md) from Conductor. +Register a handler to receive [alerts](../../conductor/alerting.md) from Conductor. The handler function is called with three arguments: - **ruleType**: The type of alert rule. One of `WorkflowFailure`, `SlowQueue`, or `UnresponsiveApplication`. diff --git a/docs/typescript/tutorials/logging.md b/docs/typescript/tutorials/logging.md index d0d4383e0..f26e9027d 100644 --- a/docs/typescript/tutorials/logging.md +++ b/docs/typescript/tutorials/logging.md @@ -241,4 +241,4 @@ For example, try using [Jaeger](https://www.jaegertracing.io/docs/latest/getting ### Metrics -Using [Conductor](../../production/conductor.md), you can also scrape metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint. See [Metrics](../../production/metrics.md) for details. +Using [Conductor](../../conductor/overview.md), you can also scrape metrics about your applications' workflows, steps, and executors from a Prometheus-compatible endpoint. See [Metrics](../../conductor/metrics.md) for details. diff --git a/docs/typescript/tutorials/workflow-management.md b/docs/typescript/tutorials/workflow-management.md index fa08e69c6..88b40a901 100644 --- a/docs/typescript/tutorials/workflow-management.md +++ b/docs/typescript/tutorials/workflow-management.md @@ -3,13 +3,13 @@ sidebar_position: 50 title: Workflow Management --- -You can view and manage your durable workflow executions via the [DBOS Console](../../production/workflow-management.md), programmatically, or via command line. +You can view and manage your durable workflow executions via the [DBOS Console](../../conductor/workflow-management.md), programmatically, or via command line. ## Listing Workflows You can list your application's workflows programmatically via [`DBOS.listWorkflows`](../reference/methods.md#dboslistworkflows) or from the command line with [`npx dbos workflow list`](../reference/cli.md#npx-dbos-workflow-list). -You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../production/workflow-management.md). +You can also view a searchable and expandable list of your application's workflows from its page on the [DBOS Console](../../conductor/workflow-management.md). Workflow List @@ -17,7 +17,7 @@ You can also view a searchable and expandable list of your application's workflo You can list the steps of a workflow programmatically via [`DBOS.listWorkflowSteps`](../reference/methods.md#dboslistworkflowsteps) or from the command line with [`npx dbos workflow steps`](../reference/cli.md#npx-dbos-workflow-steps). -You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../production/workflow-management.md). +You can also visualize a workflow's execution as a trace timeline (showing the workflow, its steps, and its child workflows and their steps) from its page on the [DBOS Console](../../conductor/workflow-management.md). For example, here is the trace of a workflow that processes multiple tasks concurrently by enqueueing child workflows: Workflow List diff --git a/docs/typescript/upgrading.md b/docs/typescript/upgrading.md index 69d3f92c5..a767ceac6 100644 --- a/docs/typescript/upgrading.md +++ b/docs/typescript/upgrading.md @@ -201,7 +201,7 @@ DBOS.setConfig({ await DBOS.launch(); ``` -The [DBOS CLI](./reference/cli.md) and [DBOS Cloud](../production/dbos-cloud/deploying-to-cloud.md) still use [`dbos-config.yaml`](./reference/configuration.md#dbos-configuration-file). +The [DBOS CLI](./reference/cli.md) and [DBOS Cloud](../conductor/reference/dbos-cloud/deploying-to-cloud.md) still use [`dbos-config.yaml`](./reference/configuration.md#dbos-configuration-file). ### HTTP Serving and Role-Based Authorization diff --git a/docusaurus.config.js b/docusaurus.config.js index dce21d955..a46afcf04 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -199,10 +199,30 @@ const config = { to: '/ai/ai-quickstart', }, ], - // Blanket redirect from /cloud-tutorials to /production/dbos-cloud + // Conductor docs moved from /production to /conductor, and DBOS Cloud docs + // moved from /cloud-tutorials and /production/dbos-cloud to /conductor/reference/dbos-cloud. createRedirects(existingPath) { - if (existingPath.startsWith('/production/dbos-cloud')) { - return [existingPath.replace('/production/dbos-cloud', '/cloud-tutorials')]; + if (existingPath.startsWith('/conductor/reference/dbos-cloud')) { + return ['/cloud-tutorials', '/production/dbos-cloud'].map((oldPrefix) => + existingPath.replace('/conductor/reference/dbos-cloud', oldPrefix), + ); + } + const movedFromProduction = { + '/conductor/overview': '/production/conductor', + '/conductor/workflow-management': '/production/workflow-management', + '/conductor/retention': '/production/retention', + '/conductor/metrics': '/production/metrics', + '/conductor/alerting': '/production/alerting', + '/conductor/autoscaling': '/production/autoscaling', + '/conductor/permissions': '/production/permissions', + '/conductor/audit-logs': '/production/audit-logs', + '/conductor/self-hosting/hosting-conductor': '/production/hosting-conductor', + '/conductor/self-hosting/hosting-conductor-with-kubernetes': '/production/hosting-conductor-with-kubernetes', + '/conductor/reference/conductor-api': '/production/conductor-api', + '/conductor/reference/dbosctl': '/production/dbosctl', + }; + if (movedFromProduction[existingPath]) { + return [movedFromProduction[existingPath]]; } return undefined; }, diff --git a/sidebars.js b/sidebars.js index c0124ca69..12465ff5c 100644 --- a/sidebars.js +++ b/sidebars.js @@ -76,7 +76,7 @@ const sidebars = { }, { type: 'category', - label: 'Deploy To Production', + label: 'Deploy to Production', items: [ { type: 'autogenerated', @@ -84,6 +84,16 @@ const sidebars = { } ], }, + { + type: 'category', + label: 'DBOS Conductor', + items: [ + { + type: 'autogenerated', + dirName: 'conductor', + } + ], + }, { type: 'category', label: 'Build Durable AI Agents',