Skip to main content
bijection-airflow maps finite Bijection pipelines to typed Airflow operators. Airflow schedules tasks and backfills; Bijection captures exact inputs and dispatches to your installed external compute resource. Workers return complete accepted output references, while triggerers observe work asynchronously. Install the application Python SDK and packages/airflow in DAG processor, worker and triggerer environments. Supported versions are Python 3.12+ and Airflow >=3.3.2,<3.4.
For a daily pipeline with native UTC daily partitioning:
Use an explicit data-interval timetable. Actual intervals must span one aligned UTC day, including manual runs. For unpartitioned pipelines omit partition and use constants or declared DAG/task parameters through ba.param. Create a bijection Airflow connection with credential_file and journal_uri extras. The journal must be durable and shared by workers: a mounted filesystem supporting atomic hard links, or S3 with the SDK’s orchestration-s3 extra. Reference storage retains admission and acceptance; dataset payloads stay with their existing owners. No credential belongs in a result reference or trigger. Retries and clearing within one run recover the original occurrence before resolving upstream XComs. A new DAG run creates a new occurrence. Explicit recovery_param names a declared parameter containing the original run ID for same-task recovery. Changed admitted meaning or unavailable prior submission history is refused. Timeouts detach observation and do not cancel compute. Use ba.upstream_result(refresh.outputs.revenue) for exact upstream versions. Selectors add dependencies and consume a complete result-reference envelope from the same DAG run, with no latest or prior-date fallback. Input contracts and partitions must match. Exact and selected inputs cannot be mixed. Completion may be accepted, selected, or served. Selection is an explicit application decision; serving requires native proof of the exact group. Configure pools to include deferred tasks when bounding outstanding remote work. Native admission retains its limit of sixteen open evaluations per pipeline.
The runtime artifact file maps dag_processor, worker, and triggerer to immutable sha256:… image identities. The release records exact bindings, application identities, generated code, DAG bundle hashes and installed versions. Keep the release outside the captured bundle and mount it separately. Select the binding and release paths through bijection.json’s orchestration configuration. Connection extras release_file, runtime_artifacts, generated_package, and bundle_directory enable local artifact checks before admission. Triggerers validate their role and versions without loading the DAG package. Run bijection-airflow probe-runtime --connection bijection_finance --role worker in each environment with its corresponding role, then preflight the deployed Bijection target using bijection orchestration check --target … --credential …. Artifact identities are deployment claims; checks do not attest remote execution. Optional lineage_namespace and lineage_required connection extras report Airflow/native correlations through the installed OpenLineage reporting source. Required reporting may fail projection after acceptance; retries recover the same result. Reports confer no native acceptance or selection authority. Supported bindings are finite, unpartitioned or UTC daily, with ordinary tasks or static DAG generation. Dynamic mapping, other timetables and continuous jobs are refused. Dagster integration uses the same SDK.