Pipeline Core
Architecture notes for @ucdjs/pipeline-core
@ucdjs/pipeline-core defines the pipeline DSL used to describe Unicode data processing.
Role
- Defines pipelines, sources, routes, filters, transforms, outputs, and tracing shapes.
- Validates route dependency graphs at definition time.
- Provides the pure data model that executors and visualizers consume.
Related Docs
Pipeline Executor
The executor runs the definitions produced by pipeline-core.
Pipeline Graph
Graph tools visualize and analyze the dependency model defined here.
Pipeline Presets
Presets package prebuilds common sources, parsers, and routes on top of the DSL.
Pipeline Loader
The loader discovers files that export pipeline definitions from this package.
Mental Model
pipeline-core is definition-time code, not runtime orchestration.
The key concepts are:
- pipeline: a complete processing plan across versions
- source: where files come from
- route: how matching files are parsed and resolved
- fallback: what happens to unmatched files
- DAG: route dependency order
The package returns immutable data structures that other packages execute.
Method Flows
definePipeline()
Route execution contract
Source and dependency model
Design Notes
- Definition-time validation is deliberate so invalid pipelines fail before execution starts.
- The package does not know about filesystem layout, caching, or UI.
PipelineDefinitionis intentionally pure data plus precomputed DAG metadata.- The DSL is expressive enough for both tiny sample pipelines and large preset-driven pipelines.
Testing Use
- invalid DAG detection
- filter composition and matching behavior
- route dependency typing and ordering
- output normalization contracts
- transform chaining and fallback behavior