07.04 · Walkthrough
Prefect Flows and Deployments
Build and deploy a Prefect flow with task retries, parameters, logging, and a scheduled run.
Prefect turns Python data pipeline code into managed workflows with retries, parameters, logging, run history and scheduled execution. A flow defines the workflow, tasks define units of work, and a deployment connects that code to an execution environment so it can run repeatably outside a local session.
What this lesson answers
- how do Prefect flows and tasks work
- how do I schedule a Prefect deployment
- when should Prefect task retries be used
Notes
Prefect is an orchestration tool that helps engineers turn ordinary Python code into observable, repeatable data workflows. In Prefect, a flow represents the main workflow, while tasks represent smaller units of work inside it, such as extracting data, validating records, transforming a dataframe, or loading results into storage. The key idea is that orchestration adds operational behavior around your code: retries when a task fails, structured logging, parameters for different runtime inputs, visibility into run state, and a history of what happened.
Common questions
- What is a Prefect flow?
- A Prefect flow is the top-level workflow for a piece of Python pipeline code. It coordinates smaller tasks such as extracting data, validating inputs, transforming records or writing outputs. The value is not replacing Python, but adding orchestration behaviour around it: state tracking, observability, parameters, retries and repeatable execution.
- What belongs in a Prefect task?
- A task should represent a meaningful unit of work that may need retries, logging or independent visibility. Good candidates include API calls, database reads, validation checks and load steps. Avoid wrapping every tiny line of code as a task, and avoid using retries to mask deterministic failures such as invalid schemas or broken SQL.
- What does a Prefect deployment do?
- A deployment makes a flow runnable outside the developer’s local process. It records how the flow should be invoked, default parameter values, where it should execute and when it should run. In practice, it is the operational contract that turns a Python workflow into something schedulable, observable and repeatable.
Short definition: what is Prefect Flows and Deployments?