Prefect Flows and Deployments
Prefect flows and deployments are Prefect’s way of turning Python workflow code into managed, observable pipeline runs. A flow defines the workflow and calls smaller tasks; a deployment defines how that flow is run later, with parameters, scheduling, execution infrastructure, logging, retries, and run history.
The need appears when a useful script becomes operationally important. Manual runs do not tell you reliably what happened, which inputs were used, where output landed, or whether a failure was temporary or caused by a real code defect. Data pipelines also need repeatable runs for different dates, files, tables, or environments. Prefect adds orchestration around the Python code so the pipeline has state, logs, retry behaviour, and a record of each run.
A flow is a Python function marked as the top-level workflow. Tasks are smaller decorated functions inside it, such as extracting, validating, transforming, or loading data. When the flow runs, Prefect tracks task boundaries, captures state transitions, stores logs, applies retry rules, and accepts parameters passed at runtime. A deployment then registers that flow with default inputs, scheduling rules, and the work pool or infrastructure that should execute it outside a local session.
The trade-off is that orchestration is not a cure for bad pipeline design. Retries help with transient faults such as temporary network, API, or database failures, but they can make deterministic bugs noisier and slower to diagnose if used indiscriminately. Parameters improve reuse, but poorly chosen defaults can make runs ambiguous. Logging adds observability only when it records meaningful facts: input values, row counts, validation outcomes, output paths, and failure context.
Engineers usually meet flows while converting Python data jobs into production workflows, and deployments when those workflows need to run on demand, from an API or UI, or on a schedule. A common misunderstanding is that a deployment is just a schedule. It is broader than that: it is the operational contract that says which flow to invoke, with what defaults, on which execution environment, and under what timing or trigger conditions.
Common questions
- What is the difference between a Prefect flow and a task?
- A flow is the main workflow boundary: it represents a complete pipeline run and coordinates the work. A task is a smaller unit inside the flow, such as reading a source, checking data, or writing output. Prefect tracks both, but tasks are where retries and granular state are often most useful.
- What is a Prefect deployment for?
- A deployment makes a flow runnable outside the developer’s current Python process. It records how the flow should be invoked, including default parameters, schedule, and execution environment. Without a deployment, a flow can still run locally, but it is not yet packaged as an operational workflow for repeatable automated execution.
- Should every function in a pipeline be a Prefect task?
- No. Use tasks where orchestration adds value: retry boundaries, observability, separate failure reporting, or clear units of pipeline work. Very small helper functions often work better as plain Python. The honest answer depends on how independently the step can fail, how much visibility you need, and whether retrying it alone is safe.