Assistant Config Versioning, Rollback & Publishing

Overview

For teams running assistants across multiple environments, Sulus recommends a Git-based, environment-promotion approach to managing configuration: every assistant, tool, and squad definition lives in Git as the single source of truth, changes are validated and tested in a dev environment, and promotion to production happens through a gated CI/CD pipeline rather than direct dashboard edits. This page walks through that workflow end to end.

The recommended approach is built on five principles:

• Isolation — separate organizations per environment (dev, uat, prod)
• Config as code — all configs stored as JSON/YAML in Git
• Immutability + promotion — create in dev, validate in uat, promote to prod via automation
• Least privilege — RBAC, secrets isolation, and data boundaries per environment
• Reproducibility — idempotent apply, drift detection, and rollbacks from Git history

Assistant and tool configurations are stored as declarative YAML files:

kind: Assistant
apiVersion: v1
metadata:
  name: order-agent
  description: Handles order inquiries
spec:
  systemPromptRef: prompts/order-agent.md
  model: gpt-4.1
  tools:
    - ref: jira
    - ref: zendesk
  knowledge:
    - ref: product-faqs
  safetyPolicyRef: policies/safety.yaml

A typical repository layout separates assistants, squads, tools, and knowledge sources, plus an environments.yaml mapping each environment to its org ID, model defaults, and endpoints:

/platform
  /assistants
    order-agent.yaml
    support-agent.yaml
  /squads
    support-level1.yaml
  /tools
    jira.yaml
    zendesk.yaml
  /knowledge
    product-faqs.yaml
  /policies
    safety.yaml
  environments.yaml       # maps env → org IDs, model defaults, endpoints
  schemas/                # JSONSchema for validation

Secrets are never committed to Git — store them in a secret manager and reference them via placeholders resolved at apply time.

Labeling for traceability: every resource is tagged with env, app, owner, and sha labels, so any live resource can be traced back to the exact environment, application, owner, and Git commit that produced it.

All changes start in the DEV environment:

1. Create or modify configs in Git
2. Validate with local schema/lint checks
3. Plan/diff the change against the dev environment
4. Apply to dev and run unit/integration tests

Three layers of validation are recommended at this stage:

• Static — JSONSchema validation, lint refs, and schema compatibility checks
• Dynamic — dry-run/plan renders and diffs
• Behavioral — golden-path chat transcripts, tool execution smoke tests, and canary testing

Applies use idempotency keys so re-running the pipeline safely converges to the desired state without side effects:

# Apply (create or update) — safe to re-run
curl -sS -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -X PUT https://api.sulus.ai/v1/assistants/order-agent \
  --data-binary @rendered/order-agent.dev.json

Use one idempotency key per resource per pipeline run, and detect drift by fetching current state, computing a diff, and failing the pipeline on unmanaged drift.

Publishing is treated as promotion, not a direct edit. All changes to production flow exclusively through CI/CD — no direct writes are permitted.

1. Open a pull request against main. CI automatically runs a plan against uat and posts the diff for reviewers.
2. UAT signoff — QA and subject-matter experts review the behavioral test results (golden-path transcripts, tool smoke tests) in the uat environment, and signoff is recorded before the PR can be merged.
3. Promote to PROD — once merged to main, CI runs a plan against prod, requires owner approvals, applies the config to prod via a dedicated CI service principal, and records the change set and artifacts for audit.

Example CI/CD pipeline (GitHub Actions):

name: Platform Deploy

on:
  pull_request:
  push:
    branches: [ main ]

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Validate
        run: npm run validate:all
      - name: Plan UAT
        env:
          ORG_ID: ${{ secrets.UAT_ORG_ID }}
          API_TOKEN: ${{ secrets.UAT_TOKEN }}
        run: npm run plan -- --env uat --out plan-uat.txt

  deploy-prod:
    if: github.ref == 'refs/heads/main'
    needs: [ plan ]
    runs-on: ubuntu-latest
    environment:
      name: prod
    steps:
      - name: Apply PROD
        env:
          ORG_ID: ${{ secrets.PROD_ORG_ID }}
          API_TOKEN: ${{ secrets.PROD_TOKEN }}
        run: npm run apply -- --env prod --approve

Developers have no direct write access to prod — only CI service principals can apply changes there.

Rollback is a simple, auditable operation: revert the Git commit that introduced the problem, and re-run the CI apply pipeline.

# Revert the last commit and push to trigger CI
git revert HEAD
git push origin main

Because applies are idempotent, re-running the pipeline with the reverted config safely restores the previous state. Confirm the rollback succeeded by checking that resource labels have reverted to the previous Git SHA.

Always back up before applying. Before any apply, export the current state:

# Export (backup) current state
curl -sS -H "Authorization: Bearer $TOKEN" \
  https://api.sulus.ai/v1/assistants?label=order-agent > backups/order-agent-prod.json

Keep backup exports from every apply job for audit purposes.

Before merging and promoting a change to production, verify all of the following:

In summary, treat every assistant configuration as code: draft and validate in dev, gain signoff in uat, promote to prod only through an automated, approved CI/CD pipeline, and keep rollback as simple as a Git revert plus a re-run of the apply job.