Skip to main content

Syncing organization documentation

Each day, the sync copies pages from projects in tuna-os into this site. Fix copied content in its source repository. Direct edits to generated pages can be lost on the next sync.

The implementation has three main parts:

How sync works​

The sync-org-docs workflow runs at 06:00 UTC each day (0 6 * * *). Maintainers can also run it through workflow_dispatch. Its job runs only in tuna-os/docs, so a fork does not run the sync job.

  1. The workflow checks out the site, sets up Node.js 24, and runs npm ci.
  2. The script uses gh api to list repositories in the organization, across all pages. It excludes archived repositories and names in SKIP.
  3. It maps each remaining repository to a lowercase slug. PROJECTS can override its label, slug, and status. A slug in HAND_AUTHORED is left alone.
  4. It makes a shallow clone of each source repository's default branch. The branch can be dev or master; it does not assume main.
  5. It writes selected documents into docs/<slug>/, then removes eligible files that the source no longer supplies. It removes each temporary clone after it processes that repository.
  6. It reports coverage, fails if any repository failed, and checks all paths under docs/ for names that differ only in case.
  7. It checks each page written by this run for MDX compatibility. A page that cannot compile gets mdx: {format: md} in its front matter. Docusaurus then reads it as CommonMark; raw HTML can appear as text.
  8. The workflow runs npm run build. Only after that succeeds does it create or update a PR from automation/sync-org-docs, titled chore: sync org-wide project docs. It opens no PR when nothing changed.

A maintainer must review and merge the sync PR. A successful run alone does not publish content. A push to main starts the production build and deployment.

Source revision and freshness​

Synced pages show a compact source indicator after their first refresh with provenance enabled. Its link names the source file at the commit that the sync inspected. The timestamp is the last successful source fetch, not the time of a site build or a change to the source file. It does not claim that the source has no newer commits.

The sync generates src/data/synced-docs.json from the files it copies, including renamed pages. This is a checked snapshot, not a project registry. Protected pages have no generated source entry. Existing pages gain entries on the next successful sync; the first snapshot is for rust-atomicwrites.

If a refresh fails, the sync retains the prior revision, paths, content, and fetch time. It records a stale/error status. The indicator then reports the failed refresh. The command still exits nonzero, and the workflow still stops before it opens a publication PR. A failed run does not automatically publish its stale status. A successful retry replaces it with the new snapshot.

npm run build reads checked pages and metadata. It does not run the sync, contact source repositories, or rewrite provenance. A build of an old or stale snapshot cannot give it a new fetch time. Run the sync explicitly to refresh sources, and review its content and metadata together.

Which files become pages​

SourceDestination or rule
Root README.mddocs/<slug>/index.md, with generated front matter and an optional status banner
Selected root documentsSame filename in docs/<slug>/; default names are ARCHITECTURE, CONTRIBUTING, ROADMAP, TODO, SECURITY, and SPEC, with .md or .rst extensions
Files directly inside docs/.md and .rst files, except SUMMARY.md, copied to docs/<slug>/

The script strips leading Markdown titles and adds page metadata. For the root README, it changes local links to that title to point to the page top. If a file in docs/ would share a filename or route with another page, the script adds a docs- prefix. For example, docs/README.md becomes docs-README.md because README.md and index.md share a Docusaurus route. It repeats the prefix if that name is also in use.

An entry in ROOT_DOC_FILTER replaces the default selection of root files. DOCS_SUBDIR can select another source directory. Despite the script's recursive-folder comment, the current loop copies only files directly inside that directory. It does not copy nested folders or image assets.

Contributors maintain navigation in sidebars.ts by hand. A new project can have generated pages without a sidebar entry. Use npm run check-sidebar-coverage to find generated project trees that readers cannot reach from the sidebar.

Ownership and removals​

Contributors write pages at the top level of docs/, including this guide, in this repository. HAND_AUTHORED protects each listed directory from writes and removals. SKIP_DOCS_DIR excludes a source's documentation folder but still copies selected documents from the root. For example, its tunaOS entry protects pages such as docs/tunaos/introduction.md. The source README still supplies docs/tunaos/index.md.

For each repository it syncs, the script compares existing direct files with files written in this run. It can remove old subpages without a content check. It removes an old index.md only if the page matches the generated-index template. The script excludes subdirectories from removal. For SKIP_DOCS_DIR repositories, removal is limited to eligible root documents.

Check these rules before you add a hand-written file beside generated files: the sync can overwrite or remove it. The script skips archived and excluded repositories, so this step leaves their existing pages in place. Review those pages separately when you retire a project.

Pipeline resilience​

Discovery requests per_page=100 and uses --paginate. It removes duplicates from the active names and keeps a separate count of archived repositories. The listing check compares the total with the organization's reported public_repos, which includes archived public repositories.

  • An empty listing, or a total below public_repos, is a fatal discovery result.
  • A fatal result triggers one retry after five seconds. A second fatal result stops the sync before it changes project pages.
  • Exactly 30 repositories, or a whole multiple of 100, produces a warning about a possible page boundary. A warning alone does not stop the run.
  • If the script cannot read the organization count, it skips the independent count check. The empty-list check and page-boundary warnings still apply.

This handles an incomplete listing that clears on retry, such as a result cut short by a secondary rate limit. Other failures have no automatic retry: a failed command to list repositories can stop discovery immediately. A clone failure does not get a retry. Each clone has a 60-second timeout.

The script collects failures and continues with other repositories. It then exits with a failure if any repository failed, so a partial run cannot produce a successful sync PR.

For the history behind these guards, see #103, #242, and #260.

fixRelativeLinks resolves relative paths from the source document's directory, collapses . and .., and uses the branch found in the clone. It sends file links to GitHub and images to raw image bytes. It does not copy the linked files into the site.

For a source document docs/user-guide.md on branch dev:

Source targetRewritten destination
../CONTRIBUTING.md#setuphttps://github.com/tuna-os/REPO/blob/dev/CONTRIBUTING.md#setup
screenshots/start.png in a Markdown imagehttps://raw.githubusercontent.com/tuna-os/REPO/dev/docs/screenshots/start.png
../LICENSEhttps://github.com/tuna-os/REPO/blob/dev/LICENSE
../releaseshttps://github.com/tuna-os/REPO/releases

The image pass handles .png, .svg, .jpg, .jpeg, .gif, and .webp. It also rewrites relative src values in HTML img tags with double quotes. Absolute URLs, anchors such as #setup, and paths that start with / stay as written. A path that starts with / must therefore already make sense on the site. HTML cleanup also converts common patterns from GitHub Markdown into safe MDX. Examples include autolinks, void elements, and less-than signs in prose.

To investigate a broken link, compare the source path and branch with the generated URL. Open the rewritten file link and load the image itself; a GitHub blob page returns HTML and cannot serve as an image source. just build checks how the pages render, but does not verify URLs on GitHub. The internal-link check also does not validate external destinations.

Troubleshooting​

Start with the workflow run and its failed step. These commands only read GitHub data; use your configured GH_TOKEN:

gh run list --repo tuna-os/docs --workflow sync-org-docs.yml --limit 5
gh run view RUN_ID --repo tuna-os/docs --log-failed
gh pr list --repo tuna-os/docs --head automation/sync-org-docs
gh api rate_limit

Replace RUN_ID with the run ID from the list. To inspect a successful run's coverage summary, use gh run view RUN_ID --repo tuna-os/docs --log.

API limits or incomplete discovery​

Look for repo listing looked truncated, listing is truncated, or an API error. Compare the logged listed count, archived count, and public count. Confirm that --paginate is still present in listOrgRepos. For a rate-limit error, inspect the response and wait for its reset or retry time before a maintainer reruns the workflow. Repeated immediate reruns can extend the problem. Keep credentials out of logs and issue reports.

A source change is missing from the site​

Trace the change through these steps:

  1. Confirm that the source repository contains the change on its default branch. The sync excludes open source PRs and changes on other branches.
  2. Check the filters below and confirm the filename and source directory are eligible. Look for hand-authored or a repository failure in the sync log.
  3. Inspect the latest sync PR for the generated file. If the sync or build fails, the workflow skips the PR step. A no-change run creates no new PR.
  4. Confirm that the sync PR merged and the production deployment succeeded. If the page exists by URL but has no navigation entry, update sidebars.ts.

For an archived repository, a successful sync does not refresh or delete its pages. Review a removal or an archive notice in this repository instead.

Find the named page and its source document. Fix incorrect links or prose in the source repository. If the transform breaks a valid document from GitHub, fix the helpers here. Add a regression case to scripts/__tests__/sync-org-docs.test.mjs. Check both prose and code samples: HTML cleanup should preserve fenced and inline code during its prose rewrites.

Look for not valid MDX warnings and the CommonMark front matter when a page builds but displays raw markup. For a case-collision error, remove or rename one of the named paths and check whether the next sync would recreate it. A rename of only the generated copy leaves the source unchanged.

Maintenance​

The sync discovers new active repositories automatically. A root README.md supplies their index without an entry in PROJECTS. Check the generated output, add a sidebar entry, and review the project's status and label.

The settings in scripts/sync-org-docs.mjs and scripts/sync-org-docs.config.json have distinct purposes:

SettingWhen to change it
PROJECTSSet a display label, slug, or status for a known repository
SKIPExclude a moved, retired, or internal repository entirely
protected_repos in the JSON configurationProtect an entire site-owned project directory; loaded as HAND_AUTHORED
SKIP_DOCS_DIRKeep source developer docs out while still copying root documents
sync_filters in the JSON configurationUse an exact root-filename allowlist for a repository; loaded as ROOT_DOC_FILTER
DOCS_SUBDIRRead a different source documentation directory

Treat sync_filters values as exact filenames. The script uses includes(file): it is case-sensitive, and an empty array allows no additional root documents. The README has a separate step.

Each protected_repos entry records a slug, reason, and protection date. The script stops if it cannot load the configuration or an existing protected slug is absent. To remove protection, also update HAND_AUTHORED_EXPECTED in the script after you review the ownership change.

Before you change a filter, inspect existing pages for ownership and possible removals. A slug change can leave an old directory behind. The script does not clean up old directories when a repository changes its name or joins the skip list.

To reproduce a sync locally, follow the repository's setup and validation instructions. Use a clean, disposable worktree with dependencies installed, GitHub CLI, Git, Node.js 24, and a configured GH_TOKEN. The command contacts GitHub, clones repositories, and writes and removes files under docs/:

node scripts/sync-org-docs.mjs
git diff --stat
git diff -- docs/
node scripts/__tests__/sync-org-docs.test.mjs
just preflight
just build

There is no dry-run option. The local script does not create a PR or deploy the site; those steps belong to the workflow and merge process. Review the diff before you propose changes, and keep lint budgets at or below their current values.