Skip to main content

dbt and recipes

Warehouse tables are built by recipes, not by raw DDL. Which authoring path you use depends on the installation's engine:

project.engine()      # "dbt" or "native"
project.uses_dbt()    # True on a dbt installation
  • dbt — author dbt model files through project.dbt and build them with dbt.
  • native — an installation without dbt. Author SQL recipes through Recipe and build them with project.build_native().

engine() is cached after the first call (refresh=True re-reads it). See the native engine page for how the platform chooses.

dbt: the DbtIDE handle​

project.dbt is a property that returns a DbtIDE handle over the platform's dbt API, the same one the platform's dbt editor uses. Most of these calls need org-admin permission.

Files and models​

dbt = project.dbt

dbt.write_file("models/staging/stg_sales.sql",
               "select * from {{ source('erp', 'sales') }}")
dbt.read_file("models/staging/stg_sales.sql")     # {path, content, size_bytes, modified}
dbt.list_files("models")                          # the file tree under a dbt directory
dbt.delete_file("models/staging/old_model.sql")

dbt.create_model("stg_sales", schema_name="staging",
                 sql="select * from {{ source('erp', 'sales') }}",
                 description="Sales lines, typed")
dbt.list_models()
dbt.compile("models/staging/stg_sales.sql")       # resolved SQL in "compiled_sql"

write_file creates or overwrites a file; the platform registers new models in the catalog. create_model scaffolds a new model with its dataset registration and schema entry and returns {name, path, schema, sql}; schema_name is the layer (staging, intermediate or marts).

Run, test, build​

res = dbt.run(select="stg_sales+")
print(res["success"], res["return_code"], res["duration_seconds"])
print(res["output"])

dbt.test(select="stg_sales")
dbt.build(select="+fact_orders")
dbt.execute("seed")          # run | test | compile | build | seed | snapshot | deps | ls
dbt.runs(limit=20)           # recent dbt executions

These calls run the dbt command and return its result ({success, return_code, output, duration_seconds, command}) when it finishes, so set a generous client timeout for long builds.

Version control​

dbt.git_status()
dbt.git_commit("add stg_sales", files=["models/staging/stg_sales.sql"], push=True)
dbt.git_push()

Building through the job pipeline​

project.run_dbt(select) and Recipe.build() start the build as a pipeline job instead of a direct dbt call. They return as soon as the job is started; with wait=True they block until the run finishes and return the run row (0.1.8+: in 0.1.7, wait=True returned immediately).

run = project.run_dbt("stg_sales+", wait=True, timeout=3600)
print(run["status"])         # "success" / "failed" / …

project.get_recipe("fact_orders").build(downstream=True, wait=True)

build(downstream=True) selects model+, upstream=True selects +model. See Scenarios and jobs for the underlying trigger_and_wait.

Recipes​

project.get_recipe(model_name) returns a Recipe handle for a flow recipe, addressed by its output model name.

r = project.get_recipe("int_orders_enriched")
r.get_metadata()        # {id, recipe_type, inputs, steps, version, …}
r.recipe_id
r.runs(limit=5)         # recent executions
r.engine_status()       # selected engine, pinned or default, and which engines are selectable

Author a prepare recipe​

r = project.get_recipe("stg_orders_clean")
r.save_prepare_steps(
    source="orders_raw",
    steps=[...],                 # prepare steps in the Flow Builder's step format
    materialized="table",
)

save_prepare_steps stores the steps, generates the SQL and (on a dbt installation) the model file. Pass expected_version (from get_metadata()["version"]) for optimistic locking: a stale version is refused with 409.

Author any recipe type​

r = project.get_recipe("orders_by_store")
r.save(recipe_type="sql_script",
       steps=[],
       inputs=["fact_orders"],
       output_datasets=["orders_by_store"],
       inline_code=(
           "DROP TABLE IF EXISTS marts.orders_by_store;\n"
           "CREATE TABLE marts.orders_by_store AS "
           "SELECT store_key, COUNT(*) AS orders FROM marts.fact_orders GROUP BY store_key;"
       ))

save writes a recipe of any type (prepare, python, sql_query, sql_script, split, …) with its configuration in the Flow Builder's shape for that type. The platform validates prepare steps; for sql_script, the script goes in inline_code.

Native engine: land SQL and build (0.1.8+)​

On a native-engine installation, save only compiles inline_code for prepare recipes. For the other SQL-shaped types, land the SQL with set_sql, then build:

r = project.get_recipe("orders_by_store")
r.set_sql("""
    SELECT store_key, COUNT(*) AS orders
    FROM {{ ref('fact_orders') }}
    GROUP BY store_key
""")

project.build_native(select="orders_by_store+", dry_run=True)   # resolved order only
result = project.build_native(select="orders_by_store+")
print(result["ok"], result["built"], result["total"])

set_sql resolves {{ ref(...) }} and {{ source(...) }} against the project's datasets and refuses the write with 400 if any template expression is left. A dbt installation answers 409: author the model through project.dbt instead.

build_native(select=None, *, dry_run=False, background=False, wait=False) builds the project's SQL recipes in dependency order and stops at the first failure. select uses dbt syntax (model, model+, +model, comma-separated). background=True returns {run_id, status: "running", …} at once; add wait=True to block on that run.

Change recipe engines​

project.change_recipe_engines(["orders_by_store", "fact_orders"], "sql", dry_run=True)

target is sql (pin to the in-database engine), dbt, or auto (clear pins). A recipe that cannot compile to the requested engine is reported and left unchanged. Needs org admin.

Plugin recipes​

Plugins your organization has enabled can contribute custom recipe types. Discover them, then create and run one:

for t in client.plugin_recipe_types():
    print(t["type"], t["plugin"], t["display_name"])

rec = project.create_plugin_recipe(
    "row_tools.rowcount", "count_orders",      # a type from plugin_recipe_types()
    config={"input_table": "marts.fact_orders"},
    run=True,
)
print(rec.last_result)

rec.update_config({"input_table": "marts.fact_orders_archive"})   # merges by default
rec.run(timeout=600)
rec.runs(limit=5)
rec.delete()

create_plugin_recipe(recipe_type, name, *, config=None, inputs=None, output_datasets=None, run=False, wait=False) returns a PluginRecipe. The platform refuses creation with 403 when the owning plugin is not enabled for your organization. run() is synchronous: it returns when the plugin's handler finishes, and timeout= bounds that HTTP call. The result is also kept on rec.last_result.

To write a plugin of your own, see Plugins.

Verified against honeyframeapi 0.1.8.