Pipelines
Guide to the UCD.js pipeline system, from core concepts to debugging
The @ucdjs/pipeline-* packages provide the internal extraction and data-production path for UCD.js. A pipeline definition describes:
- which versions to process
- which sources provide files
- which routes should match those files
- which transforms and dependencies should shape execution order
- which outputs should be emitted or published
These docs focus on the live authoring model used in the repo today. That means route dependencies are modeled with depends and ctx.getRouteData(...), execution is handled by @ucdjs/pipeline-executor, and published results are configured through route outputs.
Mental model
At a high level, pipeline authoring works like this:
- Define a pipeline in a
*.ucd-pipeline.tsfile. - Add one or more sources that expose versioned files.
- Add routes with filters, parsers, optional transforms, and resolvers.
- Express route dependencies with
depends. - Run the definition through the CLI or
@ucdjs/pipeline-executor. - Inspect results in the Pipeline Server when you need execution-level debugging.
Package roles
The pipeline system is split across focused packages:
@ucdjs/pipeline-core: definitions, filters, route/source/output types, transforms, and helpers@ucdjs/pipeline-executor: runtime execution, logging hooks, traces, caching, and pipeline ordering@ucdjs/pipeline-loader: discovery and loading of*.ucd-pipeline.tsfiles from local or remote sources@ucdjs/pipeline-presets: shared parsers, resolvers, routes, and pipeline factories@ucdjs/pipeline-server: local UI and server for inspection, execution history, logs, traces, and graphs
Start where you need to start
Definitions
Learn the fields on a pipeline definition and how DAG validation works.
Sources
Understand backends, source helpers, includes/excludes, and pipeline output sources.
Routes
See how filters, parsers, transforms, dependencies, resolvers, cache, and outputs fit together.
Outputs
Learn how emitted and published route results are declared and consumed.
Write Your First Pipeline
Start from the smallest runnable pipeline in the playground.
Debug Pipelines
Use the CLI and Pipeline Server to inspect discovery, execution, logs, and graphs.
These packages are powerful and heavily used inside the monorepo, but they should still be treated as advanced infrastructure. Prefer the documented helpers and playground patterns over ad-hoc experimentation.