BUILD-AND-DEPLOY.md — pulumi/registry

Table of Contents#

1. Overview
2. Architecture Overview
3. Local Development
4. Build System
   • 4.1 Makefile Targets
   • 4.2 Hugo Build
   • 4.3 resourcedocsgen Tool
   • 4.4 CI Build Script
   • 4.5 Versioned Documentation
5. GitHub Actions Workflows
   • 5.1 pull-request.yml — PR Validation + Preview Deploy
   • 5.2 push.yml — Production Build + Deploy
   • 5.3 Other Workflows
6. Deployment Infrastructure
   • 6.1 Pulumi IaC
   • 6.2 AWS Resources
   • 6.3 S3 Preview Bucket Lifecycle
   • 6.4 The Pinned Preview Comment
   • 6.5 Custom Redirects
7. Registry Publication
8. Testing Strategy
   • 8.1 Go Unit Tests
   • 8.2 Go Linting
   • 8.3 Markdown Linting
   • 8.4 Script Linting
   • 8.5 Dark Logo Variants
   • 8.6 Provider API Docs Tests
   • 8.7 Browser Tests (Cypress)
   • 8.8 Link Checking
9. Environment & Secret Management
   • 9.1 Pulumi ESC
   • 9.2 Environment Variables Reference
   • 9.3 Tool Version Matrix
10. Scheduled & Maintenance Tasks
11. Troubleshooting
12. Pulumi Cloud Service Integration

• 12.1 Two Parallel Systems
• 12.2 How the Publishers Bridge Them
• 12.3 Pulumi Cloud Roles Summary

Overview#

Quick Reference: Common Commands#

Key Concepts#

• Hugo: Static site generator that produces the final HTML from templates and content files.
• Pulumi ESC (Environments, Secrets, and Configuration): Used for OIDC-based secret exchange; no long-lived secrets are stored directly in GitHub.
• resourcedocsgen: A Go tool (in tools/resourcedocsgen/) that generates provider API documentation from Pulumi provider schemas.
• S3 origin bucket model: Each build produces its own uniquely-named S3 bucket. A Pulumi IaC program reads a metadata file to determine which bucket to point CloudFront at. This means each PR commit gets its own preview URL, and production deploys atomically swap the CloudFront origin.

Prerequisites#

The canonical tool versions are tracked in mise.toml:

Install and manage all tools via mise:

mise trust && mise install

Architecture Overview#

                   ┌─────────────────────────────────────┐
                   │        pulumi/registry repo          │
                   │                                      │
                   │  themes/default/data/registry/       │
                   │    packages/*.yaml  ─────────────┐   │
                   │                                  ▼   │
                   │  tools/resourcedocsgen/ ──► generates API docs
                   │                                  │   │
                   │  themes/default/content/         │   │
                   │    registry/packages/ ◄──────────┤   │
                   │                                  │   │
                   │  llm-docs-out/registry/packages/ │   │
                   │    (LLM-friendly JSON) ◄─────────┘   │
                   │                                      │
                   │  Hugo build ◄── (reads content/)     │
                   │       │                              │
                   │       ▼                              │
                   │  public/  (built site)               │
                   └─────────────────────────────────────┘
                              │
                              │  scripts/ci/sync.sh
                              ▼
              ┌─────────────────────────────┐
              │  AWS S3 Origin Bucket        │
              │  registry-{env}-origin-      │
              │  {build-id}                  │
              └──────────────┬──────────────┘
                             │
                             │  Pulumi IaC (infrastructure/)
                             │  reads origin-bucket-metadata.json
                             ▼
              ┌─────────────────────────────┐
              │  AWS CloudFront Distribution │
              │  (per environment)           │
              └──────────────┬──────────────┘
                             │
                             ▼
                   pulumi.com/registry

Two Environments#

Component Roles#

Multi-Repo Touchpoints#

• pulumi/pulumi-* provider repos: Trigger publish-provider-update.yml via repository_dispatch when a new provider version is released.

Local Development#

Tool Installation#

# Trust mise config and install all declared tools
mise trust && mise install

First-Time Setup#

# 1. Install Node.js dependencies (yarn packages, etc.)
make ensure
 
# 2. Build theme CSS and JavaScript assets
make build-assets
 
# 3. Start local dev server
make serve

make serve runs:

hugo serve --buildDrafts --buildFuture --ignoreVendorPaths="github.com/pulumi/registry/**/*"

REF_NOT_FOUND Hugo warnings are suppressed automatically.

Hot-Reload Development#

To watch both Hugo content changes and theme asset changes simultaneously:

make serve-all

This uses concurrently to run the Hugo server and the Yarn asset watcher (yarn --cwd ./themes/default/theme run start) in parallel. If either process exits, both are killed.

To watch only theme assets (without serving Hugo):

make serve-assets

Generating API Docs for One Package#

make api-docs/<package>

For example, to generate docs for the aws provider:

make api-docs/aws

This runs bin/resourcedocsgen docs registry for the named package, reading its YAML from themes/default/data/registry/packages/aws.yaml and writing generated content to content/registry/packages/aws/ (at the repository root — note this differs from the path used by scripts/ci/build.sh in CI, which writes to themes/default/content/registry/packages/).

The Makefile uses .SECONDEXPANSION and a .make/ sentinel directory so that make api-docs/aws is a no-op if neither the YAML file nor the existing content has changed since the last run.

Building a Single Provider Locally vs. Full Build#

For a single provider:

make api-docs/<package>

For a full local build (all providers, all content):

make build

make build runs make build-assets, then scripts/apply-fixes.js, then Hugo.

Memory note: A full build requires significant RAM. The CI build sets NODE_OPTIONS=--max_old_space_size=8192 to give Node 8 GB of heap. For full local builds, consider setting this in your shell:

export NODE_OPTIONS="--max_old_space_size=8192"

32 GB+ of RAM is recommended for a complete build.

Build System#

4.1 Makefile Targets#

All targets are defined in the repository root Makefile.

4.2 Hugo Build#

Hugo reads from the following key locations:

Build order for a full CI build:

1. make build-assets — compiles theme JS and CSS via yarn --cwd ./themes/default/theme run build
2. resourcedocsgen docs registry — generates provider API docs into themes/default/content/registry/packages/
3. node scripts/apply-fixes.js — post-processes Hugo content (see script for details)
4. hugo --minify --buildFuture --templateMetrics -e <environment> — generates public/
5. yarn run minify-css (i.e., node scripts/minify-css.js) — purges and minifies CSS in public/

The Hugo base URL differs between build modes:

• preview: http://<origin-bucket-prefix>-<build-id>.s3-website.us-west-2.amazonaws.com
• update (production): uses production base URL; Hugo env is set to production

4.3 resourcedocsgen Tool#

Location: tools/resourcedocsgen/

Building:

# Via Makefile (recommended — uses helpmakego for dependency tracking)
make bin/resourcedocsgen
 
# Manual
go build -C tools/resourcedocsgen -o ../../bin ./...

The compiled binary is placed at bin/resourcedocsgen.

Subcommands:

Key flags for docs registry:

bin/resourcedocsgen docs registry \
    --baseDocsOutDir themes/default/content/registry/packages \
    --basePackageTreeJSONOutDir themes/default/static/registry/packages/navs \
    --baseSchemasOutDir themes/default/static/registry/packages \
    --baseLLMDocsOutDir ./llm-docs-out/registry/packages \
    [<package-name>]

Note: The paths above reflect scripts/ci/build.sh (CI runs). The Makefile's make api-docs/<pkg> target uses ./content/registry/packages (repo root, without the themes/default/ prefix) for --baseDocsOutDir, and similarly for the other --base* flags.

When <package-name> is omitted, all packages listed in themes/default/data/registry/packages/ are processed.

Input: YAML files at themes/default/data/registry/packages/*.yaml — each file describes one provider (name, version, repo URL, schema file path, publisher, etc.)

Output:

• Generated docs at themes/default/content/registry/packages/<pkg>/api-docs/
• Package navigation JSON at themes/default/static/registry/packages/navs/<pkg>.json
• Schema JSON at themes/default/static/registry/packages/<pkg>.json
• LLM docs JSON at llm-docs-out/registry/packages/<pkg>/api-docs/llm-docs.json (only when --baseLLMDocsOutDir is set)

The format of llm-docs.json is specified in docs/llm-markdown-spec.md.

4.4 CI Build Script (scripts/ci/build.sh)#

This is the master build script for CI runs. It accepts one argument: preview or update.

Steps executed by build.sh:

1. Calls make build-assets (theme CSS/JS compilation).
2. Computes a build_identifier (for preview: pr-<number>-<sha8>; for push: push-<sha8>).
3. Sets asset bundle paths:
   • CSS_BUNDLE=static/css/styles.<id>.css
   • JS_BUNDLE=static/js/bundle.min.<id>.js
4. Restores cached API docs from .cache/api-docs/ into the Hugo content/static trees (see section 4.7 for details).
5. Runs make api-docs, which compiles resourcedocsgen and generates all provider API docs. The tool skips unchanged packages using sentinel files.
6. Saves API docs output (including sentinel files) back to .cache/api-docs/ for the next run. Versioned packages (@-suffixed) are excluded — they have their own cache. LLM docs from llm-docs-out/ are also cached to .cache/api-docs/llm-docs/ and restored on the next run.
7. Runs node ./scripts/apply-fixes.js.
8. Runs Hugo with --minify --buildFuture --templateMetrics:
   • preview mode: sets HUGO_BASEURL to the S3 website URL and uses -e preview
   • update mode: uses -e production
9. Runs yarn run minify-css to purge and minify CSS.

4.5 Versioned documentation#

Location: scripts/generate-versioned-docs.sh

Blessed packages (first-party Pulumi providers listed in ci-mgmt) get versioned API docs for the last 3 major versions. URLs use the pkg@X.x format:

• /registry/packages/aws/ - latest version (e.g., v7.x)
• /registry/packages/aws@6.x/ - previous major version
• /registry/packages/aws@5.x/ - older major version

How it works:

1. make api-docs/<pkg> runs resourcedocsgen for the latest version, then calls generate-versioned-docs.sh
2. The script fetches the blessed packages list from ci-mgmt
3. For blessed packages, it discovers the last 3 major versions using registry-mirror-discover
4. For each older major version:
   • Downloads and caches the schema (schemas are 50MB+)
   • Fetches version-specific _index.md from GitHub or falls back to latest
   • Generates API docs with resourcedocsgen
   • Creates versioned nav JSON (e.g., aws@6.x.json)
   • Generates metadata YAML in package_versions/ directory

Generated output:

Dependencies:

• registry-mirror-discover: built via go install from github.com/pulumi/registry-mirror-tools at a pinned commit. In CI, the binary is cached by GitHub Actions to avoid rebuilding on every run (see section 4.7).

Hugo template handling:

Templates parse versioned package names to extract the base name and version slug:

{{ $rawPackageName := index $directories 2 }}
{{ $isVersioned := strings.Contains $rawPackageName "@" }}
{{ $basePackageName := cond $isVersioned (index (split $rawPackageName "@") 0) $rawPackageName }}

The version selector dropdown appears on package pages when multiple versions exist, reading from $.Site.Data.registry.package_versions.

4.7 Build Caching#

CI builds use multiple cache layers to avoid redundant work. All caches are stored in GitHub Actions cache and restored at the start of each build.

GitHub Actions–level caches#

The docs cache uses restore-keys: docs-cache- so it falls back to the most recent previous run's cache when an exact match isn't found (the key includes run_id, so it's always unique).

The resourcedocsgen Go job keys on the go.sum of the module it compiles, via cache-dependency-path. setup-go restores on an exact primary-key match and exposes no restore-keys. Jobs that compile no Go module set cache: false rather than falling back to the repo-root go.mod, which describes the Hugo theme module and never changes.

Incremental API docs generation#

The resourcedocsgen tool skips unchanged packages using sentinel files. Each generated package directory contains a .generated file recording a cache key composed of:

• SHA-256 of the package YAML metadata — changes when the package version or config is updated.
• Go toolchain version — changes on Go upgrades.
• Source hash — a SHA-256 of all .go, .tmpl, and go.sum files in tools/resourcedocsgen/, injected at build time via -ldflags. Changes when the doc generation logic or templates change.

On each run, resourcedocsgen compares the computed cache key against the sentinel. If they match and the expected output files (api-docs, nav JSON, schema JSON) all exist, the package is skipped. Otherwise it regenerates.

The scripts/ci/build.sh script manages the cache lifecycle:

1. Restore: copies cached content from .cache/api-docs/ into the Hugo content/static trees before running resourcedocsgen.
2. Generate: make api-docs runs resourcedocsgen, which skips fresh packages and regenerates stale ones.
3. Save: copies the generated output (including updated sentinel files) back to .cache/api-docs/ for the next run.

LLM docs follow the same lifecycle: on restore, .cache/api-docs/llm-docs/<pkg>/api-docs/ is copied to llm-docs-out/registry/packages/<pkg>/api-docs/; on save, the reverse copy is performed. Only schema.json (not the entire directory tree) is cached per package in the schema layer, to avoid persisting stale LLM doc files from older builds. LLM docs are stored uncompressed in the cache; sync.sh gzip-compresses them in place immediately before uploading to S3 (with Content-Encoding: gzip).

Versioned docs cache#

Versioned docs (older major versions of blessed packages) are cached separately in .cache/versioned-docs/, which has three subdirectories: content/, navs/, and metadata/. The generate-versioned-docs.sh script restores from this cache before processing and saves back to it afterward.

Each versioned package directory has a .generated sentinel file, but unlike the API docs cache, it stores only the schema URL (not a full composite key). If the schema URL in the sentinel matches the current version's schema URL, generation is skipped. This means versioned docs only regenerate when the schema URL changes (i.e., when a new version is published for that major version line).

Provider schemas themselves are cached separately in .cache/schemas/ (keyed by <package>-v<version>.json) since individual schemas can be 50MB+.

Cache invalidation triggers#

GitHub Actions Workflows#

All workflow files live in .github/workflows/.

Workflow Summary#

5.1 pull-request.yml — PR Validation + Preview Deploy#

Trigger: Pull request to master

ESC environment: github-secrets/pulumi-registry (OIDC, no long-lived secrets exported automatically)

Flow:

PR opened / committed
        │
        ├── resourcedocsgen (calls check-go.yml)
        │       ├── lint (golangci-lint)
        │       └── test (go test ./...)
        │
        ├── lint-markdown
        │       └── yarn install → make lint-markdown
        │
        ├── lint-scripts
        │       └── yarn install → yarn run lint
        │
        ├── lint-dark-logos
        │       └── make lint-dark-logos
        │
        ├── test-live-publish
        │       └── uv run publish_to_registry.py --validate-all
        │
        ├── test-provider-api-docs
        │       └── make ensure build-assets → make test_provider_api_docs
        │
        ├── preview  (skipped for fork PRs; skipped for automation/tfgen-provider-docs label)
        │       ├── Fetch ESC secrets
        │       ├── Install Node 22, Go 1.26, Hugo 0.157
        │       ├── Validate community-packages/package-list.json
        │       ├── Configure AWS credentials → assume testing account role
        │       ├── Install s5cmd v2.3.0
        │       └── make ci-pull-request
        │               ├── scripts/ci/validate-packages.sh
        │               ├── scripts/ci/build.sh preview
        │               └── scripts/ci/sync.sh preview
        │                       ├── Create / reuse S3 bucket
        │                       ├── s5cmd sync public/ → bucket
        │                       ├── gzip -9 llm-docs-out/ (parallel pre-compress)
        │                       ├── s5cmd sync llm-docs-out/ → bucket (Content-Encoding: gzip)
        │                       ├── Run browser tests (Cypress smoke test)
        │                       ├── Write origin-bucket-metadata.json
        │                       └── Update the pinned PR comment (preview URL + changed pages)
        │
        └── sentinel (depends on all jobs above)
                └── Writes "Sentinel" GitHub status check = success

Skipping preview for fork PRs: Fork PRs are excluded at the workflow level. The preview job has an if: condition that only allows it to run when github.event.pull_request.head.repo.full_name == github.repository, so the job is never scheduled for PRs from forks. pull-request.sh also contains a defensive credential check (for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and PULUMI_ACCESS_TOKEN) as a fallback, but this code path is not expected to be reached in practice.

The sentinel job: The sentinel job creates a GitHub status check called "Sentinel" only after all required jobs pass. This single required check simplifies branch protection rules. The sentinel job runs for non-fork PRs and for repository_dispatch events (condition: github.event_name == 'repository_dispatch' || github.event.pull_request.head.repo.full_name == github.repository).

Key environment variables in preview job:

Runner: pulumi-service-ubuntu-24.04-16core (large runner required for the full build)

5.2 push.yml — Production Build + Deploy#

Trigger: Push to master

ESC environment: github-secrets/pulumi-registry (OIDC via id-token: write)

Flow:

Push to master
        │
        └── build job
                ├── Fetch ESC secrets (OIDC)
                ├── Install Node 22, Go 1.26, Hugo 0.157
                ├── Checkout (using PULUMI_BOT_TOKEN for private module access)
                ├── Configure AWS credentials
                │       └── Assume arn:aws:iam::388588623842:role/ContinuousDelivery
                ├── Install s5cmd v2.3.0
                ├── Install Pulumi CLI
                ├── make ci_push
                │       ├── scripts/ci/login.sh
                │       │       └── pulumi login && pulumi -C infrastructure stack select $PULUMI_STACK_NAME
                │       ├── scripts/ci/build.sh update
                │       ├── scripts/ci/sync.sh update
                │       │       ├── Create / reuse S3 bucket
                │       │       ├── s5cmd sync public/ → bucket
                │       │       ├── gzip -9 llm-docs-out/ (parallel pre-compress)
                │       │       ├── s5cmd sync llm-docs-out/ → bucket (Content-Encoding: gzip)
                │       │       ├── Run browser tests (Cypress smoke test)
                │       │       └── Write origin-bucket-metadata.json
                │       ├── scripts/generate-search-index.sh
                │       │       └── Upload search-index.json to S3
                │       ├── node scripts/await-in-progress.js
                │       ├── scripts/ci/run-pulumi.sh update
                │       │       └── pulumi -C infrastructure update --yes
                │       │           (reads origin-bucket-metadata.json, updates CloudFront)
                │       └── scripts/ci/make-s3-redirects.sh
                │               └── Apply 301 redirects from scripts/redirects/
                ├── Archive origin-bucket-metadata.json as artifact
                ├── uv run publish_to_registry.py
                │       └── Publish new provider versions to registry service
                └── uv run push-registry.py   (only if the above failed)
                        └── Legacy fallback publisher

Runner: pulumi-service-ubuntu-24.04-16core

Key environment variables:

5.3 Other Workflows#

testing-deploy.yml — Manual Test Environment Deploy#

Identical to push.yml but triggered manually via workflow_dispatch and deploys to the testing environment (account 571684982431, role arn:aws:iam::571684982431:role/ContinuousDelivery). Uses Go 1.21.x (note: older than production).

pull-request-closed.yml — PR Preview Cleanup#

Trigger: PR closed (any PR from this repo, not forks)

Runs make ci-pull-request-closed which calls scripts/ci/pull-request-closed.sh:

1. Fetches all commits associated with the closed PR from GitHub API.
2. For each commit, looks up the associated S3 bucket in SSM Parameter Store.
3. If a bucket is found and accessible, deletes it (aws s3 rb ... --force).
4. Posts a PR comment noting that previews have been removed.

Requires: ESC secrets + AWS credentials (testing account role via AWS_CI_ROLE_ARN from ESC).

check-go.yml — Reusable Go Lint + Test#

A reusable workflow_call workflow. Called by pull-request.yml for tools/resourcedocsgen/.

Jobs:

• lint: Sparse checkout → golangci-lint v2.1.6 with --config .golangci.yml
• test: Sparse checkout → go test ./... -v

check-links.yml — Link Validation#

Trigger: Every Monday at 3:00 PM UTC (cron: 0 15 * * MON); also workflow_dispatch

Runs make check_links which calls yarn run check-links, which runs node scripts/link-checker/check-links.js "https://www.pulumi.com/registry" 2 (2 retries on failure). Broken links are reported to the #registry-ops Slack channel.

Node version: 22.x; Hugo 0.157.0 installed but not explicitly used.

run-browser-tests.yml — Scheduled Browser Tests#

Trigger: Daily at 2:00 PM UTC; also workflow_dispatch

Runs make run-browser-tests on a pulumi-ubuntu-8core runner. Assumes the production AWS role (388588623842:role/ContinuousDelivery) to be able to reach the live site.

Node version: 22.x; Hugo 0.157.0 installed.

generate-package-metadata.yml — Nightly Community Package Check#

Trigger: Daily at 5:30 AM UTC and 5:30 PM UTC; also workflow_dispatch

Flow:

1. generate-packages-list job: Runs python generate_package_list.py in community-packages/ to build a matrix of community provider repos to check.
2. check-for-package-update job (matrix, max-parallel: 1): For each provider, runs resourcedocsgen pkgversion to check if a new version is available. If so, runs resourcedocsgen metadata from-github to generate updated metadata and opens a PR via .github/actions/new-provider-version-pr.
3. PRs are skipped if an open PR already exists for that provider (deduplication check via list_pull_requests in scripts/common.sh).

community-package-*.yml — Community Package Verified Check#

The check pipeline gives a contributor who adds one entry to community-packages/package-list.json an automated, security-reviewed fact-sheet before a maintainer approves. It runs in two planes that never share a job: a secret-free plane that touches contributor input, and a privileged plane that never runs contributor code. community-package-policy.yml fails CI if any workflow mixes the two (SecretCodeSeparationTests).

• community-package-check.yml (workflow_dispatch, two jobs): its check job is the secret-free plane. It fetches the PR's file list with a read-only token, refuses the PR outright if it touches anything outside the allowlist, then reads the package's schema and docs at its latest GitHub release and probes without executing the package's code — installs the plugin (blocking), resolves the npm/PyPI/Go SDKs and lints the docs (advisory). It writes a fact-sheet artifact and a one-word verdict. The plugin install is the only blocking check, alongside successful docs generation and a present docs/_index.md.
• Its report job (write token, no secrets, no contributor code) downloads that artifact and does two things: it edits the pinned fact-sheet comment in place, and it posts the verdict as a new comment. Editing notifies nobody, so without that second comment a contributor has to keep refreshing the page to learn the result.
• community-package-check-command.yml (issue_comment): dispatches a fresh check run when the author or a maintainer comments /check on its own line, authorized and rate-limited. It dispatches rather than re-runs, so the check also reaches a pull request whose own run GitHub parked.
• community-package-sweep.yml (schedule, every 5 minutes): the check has no pull_request trigger, because a fork PR from a first-time contributor parks such a run in action_required until a maintainer approves it, leaving the contributor with no fact-sheet and nothing for /check to re-run. The sweep is the automatic trigger instead: it dispatches a check for every open package-list PR, once per head commit. A dispatched run starts in the base repo, so GitHub does not gate it. The sweep only dispatches: it never runs a contributor's code. If the sweep itself fails, it opens a p1 issue that the daily priority digest surfaces, and leaves the existing one alone if there already is one.
• community-package-preview-command.yml (issue_comment): builds an on-demand site preview when a maintainer comments /preview. A fork's own pull_request build gets no secrets, so this maintainer-triggered run stands in for it: it materializes the fork's entry as data and reuses the build-and-deploy-preview action, never running the fork's code.
• community-package-policy.yml: runs the toolchain's unit tests and mypy --strict, including the plane-separation test, on any PR touching the pipeline sources. It is not among Sentinel Tower's needs, so it does not gate a merge today.

After merge, generate-package-metadata.yml (above) generates and publishes the package's docs metadata.

publish-provider-update.yml — Provider Doc Update via Repository Dispatch#

Trigger: repository_dispatch with event types resource-provider or push-provider-update

Used by first-party Pulumi provider repos to trigger documentation regeneration when a new provider version is released.

For resource-provider: Calls resourcedocsgen metadata from-github → creates a PR. For push-provider-update: Downloads schema from schema-url, extracts version from schema, calls resourcedocsgen metadata from-urls → creates a PR.

bucket-cleanup.yml — Remove Stale S3 Preview Buckets#

Trigger: Daily at 3:00 PM UTC; also workflow_dispatch

Runs make ci_bucket_cleanup which calls scripts/ci/bucket-cleanup.sh, which in turn calls scripts/ci/remove-buckets.sh push and scripts/ci/remove-buckets.sh pr.

For each deletable bucket (associated with a closed PR):

1. Applies a lifecycle policy: all objects expire after 1 day.
2. Adds a CleanupStarted tag with a timestamp.
3. If the bucket has been in cleanup state for 48+ hours, attempts aws s3 rb.
4. Gives up (with an error) if cleanup has been stalled for 7+ days.

Runs in the production environment (388588623842:role/ContinuousDelivery). Node 18.x / Go 1.20.x (older versions pinned in this workflow).

priority-digest.yml — Post Open P0 and P1 Issues to Slack#

Trigger: Daily at 3:00 PM UTC; also workflow_dispatch with a dry-run input

Runs scripts/ci/priority_digest.py, which searches GitHub for open issues labelled p0 or p1 across pulumi/registry and pulumi/terraform-to-pulumi-registry-pipeline, then posts them to Slack, oldest first, with each issue's age and assignee.

Replaces a Metabase subscription that posted the same query as a screenshot. Uses PULUMI_BOT_TOKEN and SLACK_ACCESS_TOKEN from ESC and posts via chat.postMessage to the channel ID in the SLACK_TEAM_CHANNEL repository variable; the bot must be a member of that channel. --dry-run prints the message to the job log instead of posting.

export-repo-secrets.yml — Sync GitHub Secrets → ESC#

Trigger: workflow_dispatch only

Exports all GitHub repository secrets (except EXPORT_SECRETS_PRIVATE_KEY) to the github-secrets/pulumi-registry ESC environment using the pulumi/esc-export-secrets-action.

add-triage-label.yml — Auto-label New Issues#

Trigger: Issue opened or reopened

Adds the needs-triage label to all new issues automatically.

add-to-project.yml — Add Issues to GitHub Project#

Trigger: Issue opened or reopened

Adds new issues to the Pulumi Docs GitHub project (project #79) using PULUMI_BOT_GHA_MARKETING token from ESC.

Deployment Infrastructure#

6.1 Pulumi IaC#

Location: infrastructure/

Runtime: Node.js (runtime: nodejs, per infrastructure/Pulumi.yaml)

Purpose: Manages AWS resources. After a successful build and sync, the CI pipeline runs pulumi -C infrastructure update --yes, which reads origin-bucket-metadata.json to determine the newly-built S3 bucket and updates the CloudFront origin accordingly.

Stacks:

Key config values (both stacks):

6.2 AWS Resources#

IAM roles: Both accounts have an arn:aws:iam::<account>:role/ContinuousDelivery role that CI assumes via OIDC or static key → role assumption.

S3 origin buckets: Public static website buckets named by the origin_bucket_prefix() + build_identifier() functions in scripts/ci/common.sh:

• Prefix: registry-<deployment-env>-origin (e.g., registry-testing-origin)
• Identifier for PRs: pr-<number>-<sha8> (e.g., registry-testing-origin-pr-42-a1b2c3d4)
• Identifier for pushes: push-<sha8>

Each bucket is created as an S3 static website with index.html / 404.html and public-read ACL.

AWS SSM Parameter Store: Each commit's bucket name is stored at:

/registry/commits/<full-sha>/bucket

This mapping is used by pull-request-closed.sh to find and delete preview buckets when a PR closes.

CloudFront distributions: Managed by the Pulumi IaC program. After each production push, Pulumi reads origin-bucket-metadata.json and updates the CloudFront origin to point to the new S3 bucket.

6.3 S3 Preview Bucket Lifecycle#

PR commit pushed
       │
       ▼
scripts/ci/sync.sh preview
  1. aws s3 mb registry-testing-origin-pr-<N>-<sha8>
  2. Enable static website hosting
  3. s5cmd sync public/ → bucket (--delete)
  3a. gzip -9 llm-docs.json files in llm-docs-out/ (parallel)
  3b. s5cmd sync llm-docs-out/ → bucket (Content-Encoding: gzip)
  4. Run Cypress smoke tests
  5. Write origin-bucket-metadata.json
  6. aws ssm put-parameter /registry/commits/<sha>/bucket = <bucket-name>
  7. Create or update the pinned PR comment with the preview URL and changed pages

PR merged or closed
       │
       ▼
scripts/ci/pull-request-closed.sh
  1. List all commits for the PR (GitHub API)
  2. For each commit: aws ssm get-parameter /registry/commits/<sha>/bucket
  3. If bucket exists: aws s3 rb s3://<bucket> --force
  4. aws ssm delete-parameter /registry/commits/<sha>/bucket
  5. Post PR comment: "Site previews have been removed."

Daily bucket-cleanup.yml (3:00 PM UTC)
  - Catches any buckets not cleaned up by pull-request-closed.sh
  - Applies 1-day lifecycle expiration policy
  - Deletes bucket after 48+ hours in cleanup state

6.4 The Pinned Preview Comment#

Each preview build maintains a single comment on the PR rather than adding one per commit. The comment is written by post_preview_comment in scripts/ci/sync.sh, and carries:

1. The preview URL for the current commit (<bucket-website>/registry/).
2. A Changed pages list — direct links to the pages the PR changed, so a reviewer lands on them instead of navigating the preview by hand.

How it stays pinned: the body opens with the HTML marker <!-- registry-preview-link -->. upsert_github_pr_comment (scripts/ci/common.sh) pages through the PR's comments looking for that marker on a comment authored by github-actions[bot] or pulumi-bot, then PATCHes that comment; it only POSTs a new one when no match exists. Matching on the author as well as the marker means a contributor can't redirect the pinned comment by quoting the marker. The comment list is paginated deliberately — GitHub returns 30 comments per page by default, and an unpaginated search would miss the marker on a long PR and post a duplicate on every build.

How changed pages are resolved: changed_pages_section (scripts/ci/common.sh) reads the changed-file list from the GitHub API (/pulls/<n>/files), not a local git diff, so it works identically for the pull_request build and the maintainer-triggered /preview command. It collects every API page before mapping — changed_paths_to_urls de-duplicates only within a single invocation, so mapping page by page would double-list a package whose YAML and landing page straddle the 100-file page boundary. Each path is mapped under two rules:

The YAML rule is the one that matters most here: the generated api-docs/ content is gitignored and never appears in a PR diff, so without it the list would be empty on most registry PRs. Results are de-duplicated, then filtered to URLs that actually rendered (public/<url>index.html exists), which drops removed files and url:/alias overrides rather than linking them as dead URLs. The list is capped at 50 entries with an "…and N more" line.

The whole block is reporting, not deployment: it is invoked as post_preview_comment || log … so a GitHub API hiccup can never fail an otherwise-good build. Conversely, because sync.sh runs under set -o errexit after the Cypress smoke test, a failed build posts nothing.

make test-preview-comment (scripts/ci/test-preview-comment.sh) covers the mapping, the de-duplication, and the existence gate offline; it runs in the Lint Scripts PR job.

6.5 Custom Redirects#

Source files: scripts/redirects/ — pipe-delimited text files with key | location entries.

Applied by: scripts/ci/make-s3-redirects.sh as part of every production push (make ci_push).

Mechanism: For each redirect entry, creates an S3 object key with a WebsiteRedirectLocation header, causing S3 to return a proper 301 HTTP redirect rather than an HTML meta-refresh. This improves SEO and supports URL anchors.

Registry Publication#

Publication is separate from the site deploy above. The deploy syncs the rendered Hugo pages to S3; publication writes each package's schema and docs into the Pulumi Cloud registry service at https://api.pulumi.com/api/registry/packages/{source}/{publisher}/{name}/versions/{version}, which is what pulumi package add resolves against and what the Pulumi Cloud console's registry UI reads. The two stores are updated by different steps and can disagree; a package can render on the site without having a registry entry, and vice versa.

There are two publishers, a primary and a fallback.

Primary: publish_to_registry.py#

Location: scripts/ci/publish_to_registry.py

Runtime: Python 3 (via uv run --with pyyaml)

Invoked:

• On every push to master (in push.yml, Publish to registry (primary), after the build+deploy completes). continue-on-error: true, so its failure hands off to the fallback rather than failing the job.
• In --validate-all mode on every PR (in pull-request.yml, test-live-publish job). That mode only turns every package YAML into a publish spec and reports the ones it cannot; it publishes nothing.

What it does:

1. Reads the package YAMLs changed in the last commit (git diff --name-only HEAD~1 against themes/default/data/registry/packages/*.yaml).
2. Turns each into a {source}/{publisher}/{name}@{version} spec, skipping DEPRECATED publishers and the azure-native-v* / aws-v<N> legacy aliases.
3. Pipes those specs through registry-mirror-discover | registry-mirror-publish, both installed with go install from github.com/pulumi/registry-mirror-tools at the commit pinned in REGISTRY_MIRROR_TOOLS_COMMIT. Retries up to 3 times with exponential backoff (10s to 30s).

Fallback: push-registry.py#

Location: scripts/ci/push-registry.py

Runtime: Python 3 (via uv run --with requests,pyyaml)

Invoked: only when the primary publisher fails — push.yml's Push to the Live Registry (legacy fallback) step is gated on steps.mirror-publish.outcome == 'failure'. It is also continue-on-error: true; if both publishers fail, a Slack notification goes to #registry-ops. Nothing runs it on a PR.

What it does:

1. Reads all YAML files from themes/default/data/registry/packages/*.yaml.
2. For each package:
   • Skips packages where publisher == "DEPRECATED".
   • Skips packages whose name matches azure-native-v* (except azure-native itself) — these are aliases.
   • Skips packages whose name matches aws-v<N> — these are legacy versioned packages.
   • Calls the registry API above to check if this version already exists.
   • If it does not exist (404): downloads the schema from the provider repo or schema_file_url, corrects the version field if needed, and calls pulumi package publish.
   • If the package has an installation-configuration.md page (it is optional — only _index.md is required), passes it to pulumi package publish as --installation-configuration.
3. In --dry-run mode: prints the pulumi package publish command instead of running it.

pulumi package publish --help describes itself as publishing to the "Private Registry"; the hidden --source flag this script passes is what targets the public namespace instead of an organization's. The flag does not appear in --help, but it is accepted.

Required environment variable: PULUMI_ACCESS_TOKEN

Optional: PULUMI_BACKEND_URL (defaults to https://api.pulumi.com/api)

Package source: Determined by schema_url:

• Contains registry.opentofu.org → source = "opentofu"
• Otherwise → source = "pulumi"

Publisher lookup: tools/resourcedocsgen/pkg/publishers/publisher-names.json maps display names (as in YAML) to canonical publisher IDs.

Testing Strategy#

8.1 Go Unit Tests#

make test
# Runs: cd tools/resourcedocsgen && go test ./...

Also run in CI via check-go.yml for tools/resourcedocsgen/.

8.2 Go Linting#

make lint-go
# Runs golangci-lint in:
#   tools/resourcedocsgen/

Config: .golangci.yml (repo root).

golangci-lint version: v2.1.6 (specified in both mise.toml and check-go.yml).

8.3 Markdown Linting#

make lint-markdown
# Runs: scripts/lint/lint-markdown.js

Node version used in CI: 23.x (in pull-request.yml lint-markdown job).

8.4 Script Linting#

yarn run lint
# Runs: eslint scripts && prettier scripts --check

Lints all files under scripts/ with ESLint and Prettier. Config defined in the root ESLint and Prettier config files.

Node version used in CI: 23.x.

To auto-fix formatting:

yarn run format
# Runs: prettier scripts --write

8.5 Dark Logo Variants#

make lint-dark-logos
# Runs: python3 scripts/generate-dark-logos.py --check

The dark-mode package marks under themes/default/assets/fingerprinted/logos/pkg/ (<name>-on-dark.svg) are generated from their light siblings, so adding or replacing a local logo leaves them stale. The check is deterministic, offline and stdlib-only, and runs in PR CI as the lint-dark-logos job. Regenerate with:

python3 scripts/generate-dark-logos.py

Its sibling, scripts/classify-external-logos.py, decides which packages with a third-party logo_url need a light chip in dark mode and writes themes/default/data/registry/external_logo_treatment.yaml. It downloads every external logo (and shells out to macOS sips for non-PNG rasters), so it is not wired into CI — run it by hand after adding a package with a logo_url, or when a vendor changes their logo. Its --check mode exits 2, rather than claiming the file is stale, if any logo could not be measured.

8.6 Provider API Docs Tests#

make test_provider_api_docs
# Runs: scripts/ci/run-provider-tests.sh

Requires: ensure, build-assets, and bin/resourcedocsgen to have been built. Runs in PR CI as the test-provider-api-docs job.

Node version in CI: 23.x. Go version: stable (latest).

8.7 Browser Tests (Cypress)#

make run-browser-tests
# Runs: scripts/run-api-docs-tests.sh

Config: cypress.config.js — default base URL is http://localhost:1313.

Test location: cypress/ directory.

Reporters: cypress-multi-reporters (config in reporter-config.json).

Browser tests are run:

1. As a smoke test inside scripts/ci/sync.sh after each S3 deploy (both preview and production), using the deployed S3 website URL.
2. Daily at 2:00 PM UTC via run-browser-tests.yml against the live production site.

8.8 Link Checking#

make check_links
# Runs: yarn run check-links
# Which runs: node scripts/link-checker/check-links.js "https://www.pulumi.com/registry" 2

• Fetches the sitemap from https://www.pulumi.com/registry/sitemap.xml.
• Excludes API docs pages, SDK reference pages, and install/versions pages.
• Uses broken-link-checker with filterLevel: 1 and GET requests.
• Up to 2 retries if broken links are found.
• Reports broken links to the #registry-ops Slack channel via SLACK_ACCESS_TOKEN.
• Many known-flaky domains are excluded (LinkedIn, YouTube, Twitter, etc.).

Runs every Monday at 3:00 PM UTC in CI.

Environment & Secret Management#

9.1 Pulumi ESC#

Organization: pulumi Environment: github-secrets/pulumi-registry

All workflows use OIDC token exchange to authenticate with Pulumi ESC — no long-lived secrets are stored in GitHub Actions secrets directly (except for AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY used in the testing environment for PR preview deploys).

The ESC action is configured via workflow-level environment variables:

ESC_ACTION_OIDC_AUTH: true
ESC_ACTION_OIDC_ORGANIZATION: pulumi
ESC_ACTION_OIDC_REQUESTED_TOKEN_TYPE: urn:pulumi:token-type:access_token:organization
ESC_ACTION_ENVIRONMENT: github-secrets/pulumi-registry
ESC_ACTION_EXPORT_ENVIRONMENT_VARIABLES: false  # (or a specific mapping)

The export-repo-secrets.yml workflow provides a manual escape hatch to sync GitHub repository secrets into ESC.

9.2 Environment Variables Reference#

9.3 Tool Version Matrix#

Note: mise.toml specifies Node 20 for local development, while CI workflows use Node 22. The lint-markdown and lint-scripts jobs in pull-request.yml use Node 23.x. The bucket-cleanup.yml workflow uses Node 18.x.

Scheduled & Maintenance Tasks#

Troubleshooting#

Hugo version mismatch#

Symptom: Build fails with template errors or unexpected output.

Fix: Ensure you are using Hugo 0.157 extended. Check with hugo version. If using mise: mise install will install the correct version. Ensure you have the extended variant (required for SCSS processing).

Node out-of-memory error#

Symptom: Node process killed during build with FATAL ERROR: Reached heap limit.

Fix: Set the Node memory limit before running the build:

export NODE_OPTIONS="--max_old_space_size=8192"
make build

This is set automatically in CI. If local builds still fail, consider increasing the value or ensuring adequate system RAM (32 GB+ recommended for full builds).

PR preview not posting#

Symptom: PR has no preview comment; the preview job shows "Missing secret tokens, possibly due to a forked PR."

Cause: PRs from forks do not have access to repository secrets. scripts/ci/pull-request.sh detects the absence of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or PULUMI_ACCESS_TOKEN and skips the S3 sync. The build still runs; only the deployment is skipped.

Fix: This is expected behavior for fork PRs. If you need a full preview, merge the fork into a branch on the upstream repo.

resourcedocsgen schema fetch failure#

Symptom: resourcedocsgen docs registry exits with a non-zero status; error mentions a specific provider.

Cause: The provider's schema URL (in its YAML file under themes/default/data/registry/packages/) may be unreachable, or the version field may reference a tag that does not exist in the provider's GitHub repo.

Fix: Check the YAML file for the failing provider. Verify that the version tag exists on the provider's GitHub repo and that the schema file path is correct.

S3 bucket cleanup failures#

Symptom: bucket-cleanup.yml fails or leaves stale buckets.

Cause: The lifecycle policy approach means buckets are not immediately deleted. The 2-step process (apply lifecycle → wait 48h → delete) is intentional. Buckets that have been in cleanup state for 7+ days will cause the cleanup script to exit non-zero.

Fix: Manually inspect and delete the affected bucket via AWS console or CLI. Remove the associated SSM parameter.

Pulumi stack selection failure#

Symptom: scripts/ci/login.sh fails with "stack not found" or similar.

Cause: PULUMI_STACK_NAME environment variable is not set or does not match an available stack.

Fix: Ensure PULUMI_STACK_NAME is set correctly in the GitHub Actions environment variables for the relevant environment (testing or production). The stack must exist in the Pulumi Cloud org. Run pulumi -C infrastructure stack ls locally (with correct credentials) to confirm available stacks.

push-registry.py publisher not found#

Symptom: push-registry.py raises Exception: Missing publisher entry for "<publisher-name>".

Cause: A package YAML file references a publisher display name that is not listed in tools/resourcedocsgen/pkg/publishers/publisher-names.json.

Fix: Add the publisher to publisher-names.json with the correct canonical identifier, or update the package YAML to use an already-registered publisher name.

Pulumi Cloud Service Integration#

12.1 Two Parallel Systems#

The Pulumi Registry is actually two separate but tightly coupled systems that must remain in sync:

These two systems are not the same thing and do not share a data store. The static site is rebuilt from YAML files on every push to master; it does not query the Pulumi Cloud API at runtime. The Pulumi Cloud API is a live service that stores package metadata independently.

The bridge between them is the publish step that runs after every production build: scripts/ci/publish_to_registry.py primarily, with scripts/ci/push-registry.py as a fallback if that fails. Either one publishes new package versions to the Pulumi Cloud API.

YAML files in repo                            Pulumi Cloud Registry API
(source of truth for                          (source of truth for CLI
 the static Hugo site)                         package resolution)
        │                                               │
        │  publish_to_registry.py, or push-registry.py  │
        │  if that fails (both on production push)      │
        └──────────────────────────────────────────────►│
                  registry-mirror-publish, or           │
                  pulumi package publish                │

Consequence: If publication fails on a particular package, the Hugo site will show the package correctly but the Pulumi CLI will not be able to resolve it. The two systems can drift. Both publish steps are continue-on-error: true, so a failure of either leaves the workflow green — only the Slack notification to #registry-ops, which fires when both fail, surfaces it.

Consequence: The static site does not support version browsing (no "select a version" dropdown) because Hugo generates a fixed set of pages from a fixed set of YAML files. Versioned snapshots (e.g., aws-v6) are implemented as entirely separate YAML files, separate Hugo pages, and separate API publication entries — not as a first-class versioned concept.

12.2 How the Publishers Bridge Them#

On every push to master, after the Hugo site is built and deployed, scripts/ci/publish_to_registry.py reads the package YAMLs changed in the last commit, turns each into a {source}/{publisher}/{name}@{version} spec, and pipes those through registry-mirror-discover | registry-mirror-publish.

If that step fails, scripts/ci/push-registry.py runs as the legacy fallback. It sweeps every package rather than just the changed ones:

1. Reads every YAML file from themes/default/data/registry/packages/.
2. For each package, queries GET /api/registry/packages/{source}/{publisher}/{name}/versions/{version} to check whether this exact version already exists in the API.
3. If it does not exist (404):
   • Downloads the provider schema JSON from the URL in the YAML file.
   • Corrects the version field in the schema if it is absent or inconsistent with the YAML.
   • Runs pulumi package publish <schema.json> --readme <_index.md> --source <source> --publisher <publisher> to register the version.
4. If it does exist (200): no-op.
5. Skips deprecated packages, azure-native-v* aliases, and aws-v* legacy versioned packages.

On PRs: the test-live-publish CI job runs publish_to_registry.py --validate-all, which turns every package YAML into a publish spec and reports the ones it cannot — catching an unparseable YAML, a missing version or publisher, or a publisher absent from publisher-names.json. It publishes nothing and never contacts the production API. push-registry.py does not run on PRs at all.

Required credential: PULUMI_ACCESS_TOKEN must be set. This is sourced from Pulumi ESC in CI.

12.3 Pulumi Cloud Roles Summary#

Pulumi Cloud plays four distinct roles in this system: