Summary

• Databricks FileStore allows you to store files that are accessible via download links
• You can programmatically generate download URLs for files in FileStore
• While FileStore works, using external cloud storage like Azure Blob Storage or AWS S3 is generally recommended for file sharing

Detail

Databricks FileStore provides a convenient way to store files that you want to make available for download. By copying files into the special /FileStore directory in DBFS, you can easily generate URLs that allow users to download those files directly from rendered notebooks.

Here’s how you can create a download link for a file using FileStore in a Python notebook:

# File paths
temp_file_path = "/tmp/output_data.xlsx"
filestore_path = "/FileStore/output_data.xlsx"

# Code to generate the Excel file and save to temp_file_path 
# goes here...

# Copy the file from temp directory to FileStore
dbutils.fs.cp(f"file:{temp_file_path}", filestore_path)

# Get the URL parameters 
workspace_url = spark.conf.get("spark.databricks.workspaceUrl")

# Generate the download URL
download_url = f"https://{workspace_url}/files/output_data.xlsx"

# Render the download link using displayHTML
displayHTML(f"<a href='{download_url}'>Click here to download the Excel file</a>")

The key steps are:

1. Copy the file into FileStore using dbutils.fs.cp
2. Retrieve the workspace URL from the Spark context
3. Construct the download URL
4. Render the download link using displayHTML

When a user clicks the rendered download link, it will prompt their browser to download the file directly from FileStore.

Reference

For more details on the FileStore and other Databricks file system concepts, check out:

• Databricks File System Overview

• The Databricks FileStore

• Download File Store StackOverflow

Unlocking the Power of Databricks Magic Commands

Databricks notebooks offer a powerful and flexible environment for data exploration, analysis, and collaboration. One of the key features that makes Databricks notebooks so productivity-enhancing is the availability of built-in IPython magic commands. These magic commands allow you to interact with the notebook environment, control code execution, and extend functionality.

%lsmagic

%time df = spark.sql("SELECT * FROM large_table")

CPU times: user 456 ms, sys: 123 ms, total: 579 ms
Wall time: 5.67 s

%%timeit
df = spark.sql("SELECT * FROM large_table")
df.count()  

2.31 s ± 135 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%logstart
df = spark.sql("SELECT * FROM large_table")
df.count()

%logstop

%logstate

Logging is currently ON
Log file at: <log_path>.log

# IPython log file
get_ipython().run_line_magic('logstart', '')
df = spark.sql("SELECT * FROM default.my_test_delta_table")
df.count()
get_ipython().run_line_magic('logstate', '')
get_ipython().run_cell_magic('logstop', '', '')

%sh 
echo "Hello from bash!"

%perl
print "Hello from Perl!\n";

Summary

• Use spark.catalog.list methods to programmatically explore catalogs, databases, tables and columns
• Recursively traverse the metadata hierarchy to build a catalog structure dictionary
• Compare catalog structures to identify differences between environments

Overview

As your lakehouse platform grows to encompass more use cases, teams, and workloads, keeping track of all the metadata - catalogs, databases, tables, and columns - can become increasingly challenging. Fortunately, the Spark Catalog APIs provide programmatic methods to explore and analyze this metadata.

In this post, we’ll walk through an example of how to use these APIs in PySpark to:

1. Recursively list all databases and tables within a catalog
2. Build a dictionary representing the hierarchical catalog structure
3. Compare catalog structures across environments to identify metadata differences

Exploring Metadata Programmatically

The key to programmatic metadata exploration in Databricks is the family of spark.catalog.list methods:

• spark.catalog.listCatalogs() - List all available catalogs
• spark.catalog.listDatabases() - List databases in the current catalog
• spark.catalog.listTables() - List tables in a specified database
• spark.catalog.listColumns() - List columns in a specified table

Let’s see a quick example of using these methods to explore all databases and tables in each catalog:

for catalog in spark.catalog.listCatalogs():
    print("Catalog is ",catalog.name)
    spark.catalog.setCurrentCatalog(catalog.name)
    for database in spark.catalog.listDatabases():
        print("Database is:", database.name)
        for table in spark.catalog.listTables(database.name):
            print("Table is:", table.name)

This will output something like:

Catalog is hive_metastore  
Database is: default
Table is: customers
Table is: orders
Database is: example
Table is: flights
Catalog is unity_dev_environment
Database is: dlt_pipeline_metrics
Table is: pipeline_status

Building a Catalog Structure

While the above snippet is useful for exploration, let’s see how we can build a more programmatic representation of our catalog structure that will enable richer analysis and comparison.

def get_catalog_structure():
    catalog_structure = {}
    all_paths = []
   
    for catalog in spark.catalog.listCatalogs():
        catalog_structure[catalog.name] = {}
        spark.catalog.setCurrentCatalog(catalog.name)
       
        for database in spark.catalog.listDatabases():
            catalog_structure[catalog.name][database.name] = []
           
            tables = spark.catalog.listTables(database.name)
            for table in tables:
                catalog_structure[catalog.name][database.name].append(table.name)
                all_paths.append(f"{catalog.name}.{database.name}.{table.name}")

    return catalog_structure, all_paths

This recursively traverses catalogs and databases, building a dictionary catalog_structure where: - Keys are catalog names - Values are dictionaries where: - Keys are database names - Values are lists of tables in that database

It also collects a list all_paths containing the fully-qualified paths of each table in “catalog.database.table” notation.

Comparing Catalog Structures

Now that we can capture a catalog structure, let’s put it to use by comparing structures across environments to identify metadata diffs.

def compute_catalog_diffs(catalog_structure):
    # Get list of all catalogs
    catalogs = list(catalog_structure.keys())
    diffs = {}
   
    # Compare each catalog with every other catalog
    for i in range(len(catalogs)):
        for j in range(i + 1, len(catalogs)):
            cat1, cat2 = catalogs[i], catalogs[j]
           
            # Get all paths for catalog 1
            cat1_paths = set()
            for db in catalog_structure[cat1]:
                for table in catalog_structure[cat1][db]:
                    cat1_paths.add(f"{db}.{table}")
           
            # Get all paths for catalog 2
            cat2_paths = set()
            for db in catalog_structure[cat2]:
                for table in catalog_structure[cat2][db]:
                    cat2_paths.add(f"{db}.{table}")
           
            # Compute differences
            diffs[f"{cat1} vs {cat2}"] = {
                'onlyin' + cat1: cat1_paths - cat2_paths,
                'onlyin' + cat2: cat2_paths - cat1_paths,
                'common': cat1_paths & cat2_paths
            }
   
    return diffs

This compares every unique pair of catalogs, building path sets for each, and computing 3 diffs: 1. only_in_X - paths only in catalog X 2. only_in_Y - paths only in catalog Y 3. common - paths in both

structure, all_paths = get_catalog_structure()
differences = compute_catalog_diffs(structure) 

Resources

• PySpark listCatalogs()
• PySpark listDatabases()
  
• PySpark listTables()

```

Summary

• Learn how to create custom IPython magics % to streamline common Databricks notebook operations
• Implement practical examples for cluster information, SQL execution plans, and configuration management

Introduction

If you’re working with Databricks notebooks regularly, you’ve probably used magic commands like %sql or %md. But did you know you can create your own custom magics to automate repetitive tasks and enhance your notebook workflow? In this tip, we’ll explore how to create custom magics specifically designed for Databricks environments.

Understanding IPython Magics

IPython magics come in two flavors: - Line magics (prefixed with %): Execute on a single line of input - Cell magics (prefixed with %%): Execute on multiple lines of cell content

from IPython.core.magic import (register_line_magic,register_cell_magic,register_line_cell_magic)

    @register_line_magic
    def my_line_magic(self, line):
        """Simple line magic that processes input after the command."""
        return f"You passed: {line}"

    %my_line_magic hello world
    # Output: "You passed: hello world"

    @register_cell_magic
    def my_cell_magic(self, line, cell):
        """Cell magic that processes both command line and cell content."""
        return f"Command line: {line}\nCell contents:\n{cell}"

%%my_cell_magic optional command line args
This is the
cell content
over multiple lines
# Output: 
# Command line: optional command line args
# Cell contents:
# This is the
# cell content
# over multiple lines

@register_line_cell_magic
def my_flexible_magic(line, cell=None):
        """Magic that works in both line and cell modes."""
        if cell is None:
            # Line mode
            return f"Line mode: {line}"
        else:
            # Cell mode
            return f"Line: {line}, Cell:\n{cell}"

%my_flexible_magic some_args
# Output: "Line mode: some_args"

%%my_flexible_magic command line
cell
content
# Output:
# Line: command line, Cell:
# cell
# content

Custom Magic Examples for Databricks

Let’s explore some practical custom magics that can enhance your Databricks workflow:

from IPython.core.magic import register_line_magic

@register_line_magic
def spark_info(line):
    """Print Spark version and environment info."""
    print(f"Spark Version: {spark.version}")
    
    # If on Databricks, often `dbutils` is available for additional info
    if 'dbutils' in globals():
        cluster_name = dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().get("clusterId").get()
        print(f"Running on cluster: {cluster_name}")
    else:
        print("dbutils not found. Cannot retrieve cluster info.")

Spark Version: 3.5.0
Running on cluster: 097-221359-d1i6gxa1

@register_line_magic
def show_tables(line):
    """List tables in the given database: %show_tables dbname"""
    dbname = line.strip() or "default"
    tables = spark.catalog.listTables(dbname)
    for t in tables:
        print(f"{t.name}\t{t.tableType}")

@register_line_magic
def explain_sql(line):
    """
    Explain the Spark execution plan for a given SQL query.
    Usage: %explain_sql SELECT * FROM my_table
    """
@register_cell_magic
def explain_sql(line, cell):
    """
    Explain the Spark execution plan for a given SQL query.
    Usage: %explain_sql SELECT * FROM my_table
    """
    query = cell.strip()
    df = spark.sql(query)
    print(df._jdf.queryExecution().simpleString())

4. Notebook Context Magic

Here’s a more advanced example that demonstrates how to access and utilize the Databricks notebook context:

@register_line_magic
def notebook_context(line):
    """Display current notebook context information"""
    context = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
    
    info = {
        "notebook_path": context.notebookPath().get(),
        "cluster_id": context.clusterId().get(),
        "workspace_id": context.workspaceId().get()
    }
    
    return info

from IPython.core import events as ip_events
from IPython.core.magic import (register_line_magic,register_cell_magic,register_line_cell_magic)

def logging_pre(info):

    try:

        print("Debug Logging to Notebook" )

        print(info.__dict__)

    except Exception:

        info.raw_cell = ''

def logging_post(info):

    try:

        print("Debug Logging to Notebook cell succesfully executed:", info.raw_cell)

        print(info.__dict__)

    except Exception:

        info.raw_cell = ''

@register_line_cell_magic

def enable_debug_logging(line):
    #Line could include logging level etc
    ipython = get_ipython()

    ipython.events.register('pre_run_cell', logging_pre)

    ipython.events.register('post_run_cell', logging_post)

Best Practices for Custom Magics

1. Documentation: Always include detailed docstrings explaining usage and parameters
2. Error Handling: Implement robust error checking for different environments
3. Performance: Keep magics focused and lightweight to maintain notebook responsiveness
4. Reusability: Place commonly used magics in a shared module that can be imported

Loading Custom Magics

To use your custom magics, create a Python module (e.g., custom_magics.py) and load it in your notebook:

%load_ext custom_magics

You can also create classes of magics, see the documentation.

You can also register them directly in code

ip = get_ipython()
#e.g. ip.register_magics(<magic_function>)
ip.register_magics(my_flexible_magic)

Conclusion

Custom IPython magics can streamline your Databricks workflow by automating common tasks and providing quick access to important information. Start with these examples and build your own magics tailored to your specific needs.

Further Reading

• Official IPython Magic Documentation
• Databricks Notebook Content

Introduction

Databricks notebooks are powerful, but sometimes you need more control over execution flow and logging. Enter IPython events - a mechanism that allows you to hook into the notebook execution lifecycle.

Understanding IPython Events

IPython provides several event hooks (code that is called when a specfic activity happens e.g. someone runs a notebook cell in databricks) that allow you to execute code before and after cell execution. The main events we’ll focus on are:

• pre_run_cell: Fires before a cell execution
• post_run_cell: Fires after a cell execution
• pre_execute: Fires before any code execution (including non-interactive)
• post_execute: Fires after any code execution

Basic Implementation

Here’s the basic code to call some in your Databricks notebook everytime before a cell runs and after its ran:

ipython = get_ipython()
ipython.events.register('pre_run_cell', <Some Code or Function You Want to run each time this event is called >)
ipython.events.register('post_run_cell', <Some Code or Function You Want to run each time this event is called >)

This prints hello from your favourite tips site everytime a cell is queued to be run.

Result:

ipython = get_ipython()
ipython.events.register('pre_run_cell', lambda: print()"Hello From DailyDatabricks"))

This means you can start to build up generic code that runs every time a specfic event is triggered. Often this can lead to useful AOP patterns

Benefits and Applications

• Automated Quality Checks: Validate DataFrame transformations automatically
• Enhanced Debugging: Track execution flow and catch issues early
• Audit Trail: Maintain comprehensive logs of notebook execution
• Performance Optimization: Monitor execution time and resource usage
• Cross-Cutting Concerns: Implement security checks, logging, and monitoring without cluttering business logic

Logging Implementation

Here’s how to implement execution logging:

def logging_pre(info):
    try:
        print("Debug Logging to Console or Your Preferened Logging tool:")
        # You can access cell content via info.raw_cell
    except Exception:
        info.raw_cell = ''

def logging_post(info):
    try:
        print(f"Debug Logging to Console or Your Preferened Logging tool. Cell successfully executed: {info.raw_cell}")
    except Exception:
        info.raw_cell = ''

Advanced Use Case: DataFrame Validation

def inspect_spark_dataframes(code_namespace):
    """
    Inspects a given namespace for Spark DataFrames and prints their counts.
    """
    for name, obj in code_namespace.items():
        if not name.startswith('_') and isinstance(obj, DataFrame):
            try:
                count = obj.count()
                print(f"The dataframe called '{name}' has the following count: {count}")
            except Exception as e:
                print(f"Error counting DataFrame '{name}': {str(e)}")

def pre_counts(info):
    print("Validating pre transformation Dataframe counts")
    inspect_spark_dataframes(globals())

def post_counts(info):
    print("Validating post transformation Dataframe counts")
    inspect_spark_dataframes(globals())

ipython = get_ipython()
ipython.events.register('pre_run_cell', pre_counts)
ipython.events.register('post_run_cell', post_counts)

Additional AOP Patterns Suggestions

Here are some other useful patterns you can implement:

1. Performance Monitoring:

def performance_pre(info):
    info.start_time = time.time()

def performance_post(info):
    duration = time.time() - info.start_time
    print(f"Cell execution took {duration:.2f} seconds")

2. Memory Usage Tracking:

def memory_tracking(info):
    from pyspark.sql import SparkSession
    spark = SparkSession.builder.getOrCreate()
    print(spark.sparkContext.statusTracker().getExecutorMetrics())

Further Reading

• IPython Events Documentation
• PySpark DataFrame API

Remember to initialize these patterns at the start of your notebook.

Detail

Window functions are a powerful tool for performing calculations across a set of rows related to the current row. However, when you need to apply the same window definition to multiple functions, the query can quickly become verbose and hard to read.

Named window specifications solve this by allowing you to declare the window spec once and reference it by name. Here’s how it works:

WINDOW { window_name AS window_spec } [, ...]

SELECT product, 
       SUM(sales) OVER monthly_window AS monthly_sales,
       AVG(sales) OVER monthly_window AS avg_monthly_sales
FROM sales_data
WINDOW monthly_window AS (PARTITION BY product ORDER BY date)

from pyspark.sql import Window
from pyspark.sql.functions import *

window = Window.partitionBy("product").orderBy("date")

sales_data = sales_data.select(
    "product",
    sum("sales").over(window).alias("monthly_sales"),
    avg("sales").over(window).alias("avg_monthly_sales")
)

Benefits

Further Reading

• Databricks SQL - SELECT Named Window
• PySpark Window Functions

Summary

• Understanding and utilizing Spark’s NOOP (No Operation) write format
• Benefits of using NOOP in development and testing
• Practical code examples in Python

Introduction

Apache Spark is a powerful tool for big data processing. However, developing and testing Spark applications can be challenging, especially when dealing with large datasets. The Spark write NOOP operation provides a solution for testing data processing without the overhead of actual data output.

Code Examples

dataframe.write.format("noop").mode("overwrite").save()

# Example DataFrame transformation
transformed_df = dataframe.withColumn("new_column", f.expr("existing_column * 2"))

# Writing with NOOP for testing
transformed_df.write.format("noop").mode("overwrite").save()

dataframe.write.format("noop").mode("overwrite").save()

Detail

The write NOOP operation in Spark is a unique feature that allows for the execution of data processing tasks without performing any actual write operation. This feature is particularly useful in scenarios where the primary goal is to test the processing logic of Spark jobs, without the need to persist the output.

References & Further Reading

Introduction

When working with Delta Lake tables in Databricks, one often overlooked but powerful feature is the ability to log custom metadata. This capability allows you to embed valuable context directly into your Delta table’s commit history, making it easier to track data lineage, audit changes, and maintain documentation. In this post, we’ll explore how to implement custom metadata logging and discuss some practical applications.

Setting Up Custom Metadata

# Using SparkConf
spark.conf.set(
    "spark.databricks.delta.commitInfo.userMetadata",
    "Global session metadata - Data processed by ETL pipeline v2.1"
)

df.write.format("delta") \
    .option("userMetadata", "Custom metadata for this specific write operation") \
    .mode("append") \
    .saveAsTable("my_table")

SET spark.databricks.delta.commitInfo.userMetadata = 'Global session metadata - Data processed by ETL pipeline v2.1';

library(SparkR)
sparkR.session()
sparkR.session(sparkConfig = list("spark.databricks.delta.commitInfo.userMetadata" =
    "Global session metadata - Data processed by ETL pipeline v2.3"))

Practical Use Cases

from datetime import datetime
import json

pipeline_metadata = {
    "pipeline_id": "ETL_123",
    "source_system": "CRM",
    "processing_timestamp": datetime.now().isoformat(),
    "validation_rules_version": "1.2.0"
}

df.write.format("delta") \
    .option("userMetadata", json.dumps(pipeline_metadata)) \
    .mode("append") \
    .saveAsTable("customer_data")

quality_metadata = {
    "null_percentage": df.filter(col("important_field").isNull()).count() / df.count() * 100,
    "distinct_values": df.select("category_field").distinct().count(),
    "validation_status": "PASSED",
    "quality_score": 0.98
}

df.write.format("delta") \
    .option("userMetadata", json.dumps(quality_metadata)) \
    .mode("overwrite") \
    .saveAsTable("validated_transactions")

audit_metadata = {
    "approved_by": "data.governance@company.com",
    "approval_ticket": "TICK-123",
    "compliance_check_version": "2.0",
    "retention_policy": "7_years"
}

Best Practices and Tips

1. Structured Metadata: Use JSON format for complex metadata to maintain consistency and queryability
2. Size Considerations: Keep metadata concise - it’s stored with every commit
3. Automation: Implement automated metadata logging in your ETL frameworks
4. Documentation: Include metadata schemas in your data documentation

Querying Metadata History

There are several ways to access the commit history and metadata of your Delta tables. Let’s explore the different approaches:

-- View complete history with metadata
DESCRIBE HISTORY my_delta_table;

from delta.tables import DeltaTable

# Get the DeltaTable instance
deltaTable = DeltaTable.forName(spark, "my_delta_table")

# Get complete history
history_df = deltaTable.history()

# Select specific columns and filter
metadata_history = (history_df
    .select("version", "timestamp", "operation", "userMetadata")
    .where("userMetadata IS NOT NULL")
    .orderBy("version")
)

# Display the results
metadata_history.show(truncate=False)

import io.delta.tables._

// Get the DeltaTable instance
val deltaTable = DeltaTable.forName("my_delta_table")

// Get complete history
val historyDF = deltaTable.history()

// Select specific columns and filter
val metadataHistory = historyDF
  .select("version", "timestamp", "operation", "userMetadata")
  .where("userMetadata IS NOT NULL")
  .orderBy("version")

// Display the results
metadataHistory.show(false)

Conclusion

Custom metadata logging in Delta tables provides a powerful way to enhance your data lake’s observability and governance. By implementing these practices, you can build more maintainable and traceable data pipelines.

Further Reading

• Enrich Delta Lake tables with custom metadata
• Delta Lake Documentation on Transaction Log

Introduction

Every Databricks workspace ships with SQL Warehouse configuration parameters set to very permissive defaults. A query can run for 2 full days before it times out. ANSI mode may be off for older accounts, hiding silent behaviour. These are the kind of settings that, left untouched, lead to bill shock and hard-to-diagnose data quality issues.

The good news is that workspace admins can override these defaults in a single place and have them apply to every SQL Warehouse in the workspace. This post walks through the key parameters and recommends sensible starting points.

Where to Set These Parameters

Navigate to Admin Settings > Compute > SQL Warehouses > SQL Configuration Parameters.

Enter one key-value pair per line, separating the parameter name from its value with a space:

STATEMENT_TIMEOUT 3600
ANSI_MODE true
TIMEZONE UTC

You can also set these via the SQL Warehouses API for infrastructure-as-code workflows.

Recommended Defaults

-- Set at workspace level via Admin Settings, or per-session:
SET STATEMENT_TIMEOUT = 3600;

SET ANSI_MODE = true;

SET TIMEZONE = 'UTC';

Putting It All Together

Here is a recommended workspace-level configuration block you can paste directly into the SQL Configuration Parameters textbox:

STATEMENT_TIMEOUT 3600
ANSI_MODE true
TIMEZONE UTC
USE_CACHED_RESULT true
LEGACY_TIME_PARSER_POLICY EXCEPTION

Managing as Infrastructure as Code

Clicking through the Admin UI is fine for a single workspace, but if you manage multiple environments (dev, staging, prod) you want these settings version-controlled and reproducible.

resource "databricks_sql_global_config" "this" {
  sql_config_params = {
    "STATEMENT_TIMEOUT"         = "3600"
    "ANSI_MODE"                 = "true"
    "TIMEZONE"                  = "UTC"
    "USE_CACHED_RESULT"         = "true"
    "LEGACY_TIME_PARSER_POLICY" = "EXCEPTION"
  }
}

Quick Reference Table

*FALSE for accounts created before October 2022

References & Further Reading

• SQL Warehouse Admin Settings
• Configuration Parameters Reference
• STATEMENT_TIMEOUT Parameter
• ANSI_MODE Parameter
• TIMEZONE Parameter
• databricks_sql_global_config Terraform Resource
• Databricks Asset Bundles Resources

The Problem With MODIFY

Until now, Unity Catalog offered one write privilege: MODIFY. Granting it to an ingestion job so the job can append rows also lets that job drop a column, truncate the table, or rewrite the schema. Most write workloads need one verb, so MODIFY grants far more than they use.

As of August 2026, you can grant that single verb. INSERT, UPDATE, and DELETE are child privileges of MODIFY, and each covers a subset of its write access.

Working Example

The example below builds an append-only ingestion path. The order-loaders group can add orders and read them back. It cannot delete rows, and it cannot change the table’s shape.

CREATE SCHEMA IF NOT EXISTS main.dml_demo;

CREATE TABLE IF NOT EXISTS main.dml_demo.orders (
  order_id BIGINT,
  customer STRING,
  amount   DECIMAL(10, 2)
);

-- Traversal. Required to reach the table at all.
GRANT USE CATALOG ON CATALOG main TO `order-loaders`;
GRANT USE SCHEMA ON SCHEMA main.dml_demo TO `order-loaders`;

-- Read the data, and append to it. Nothing else.
GRANT SELECT ON TABLE main.dml_demo.orders TO `order-loaders`;
GRANT INSERT ON TABLE main.dml_demo.orders TO `order-loaders`;

SHOW GRANTS `order-loaders` ON TABLE main.dml_demo.orders;

principal      actionType  objectType  objectKey
-------------  ----------  ----------  ----------------------
order-loaders  SELECT      TABLE       main.dml_demo.orders
order-loaders  INSERT      TABLE       main.dml_demo.orders

-- Succeeds. Covered by INSERT.
INSERT INTO main.dml_demo.orders VALUES (1, 'Acme Corp', 250.00);

-- Succeeds. Covered by SELECT.
SELECT * FROM main.dml_demo.orders;

-- Fails. Requires DELETE.
DELETE FROM main.dml_demo.orders WHERE order_id = 1;

-- Fails. Requires INSERT and DELETE, because it replaces existing rows.
INSERT OVERWRITE main.dml_demo.orders VALUES (2, 'Globex', 99.00);

-- Fails. Schema changes still require MODIFY.
ALTER TABLE main.dml_demo.orders ADD COLUMN region STRING;

Which Privilege Covers Which Operation

MERGE INTO is the useful case. A merge that only matches and updates needs UPDATE; add a WHEN NOT MATCHED THEN INSERT clause and it also needs INSERT. Grant the clauses your statement actually uses:

GRANT SELECT, UPDATE, INSERT ON TABLE main.dml_demo.orders TO `order-upserters`;

Two Behaviours to Watch

REVOKE MODIFY ON TABLE main.dml_demo.orders FROM `order-loaders`;
-- An explicitly granted INSERT survives this. Revoke it too.
REVOKE INSERT ON TABLE main.dml_demo.orders FROM `order-loaders`;

-- Every current and future table in the schema becomes appendable.
GRANT INSERT ON SCHEMA main.dml_demo TO `order-loaders`;

When to Use Each

References & Further Reading

• Fine-grained DML privileges
• Unity Catalog privileges reference
• Privilege inheritance
• August 2026 platform release notes

The problem

A pipeline that debits one table, credits another, and appends to an audit log runs three statements. Each one commits on its own. If the second statement fails, the first is already durable and the third never runs. The tables now disagree, and you repair them by hand.

As of July 2026, transactions on Unity Catalog managed Delta tables are generally available. Group the statements and the lakehouse guarantees all or nothing.

Before you begin

You need the following:

• A SQL warehouse, serverless compute, or a cluster running Databricks Runtime 18.0 or above.
• Unity Catalog managed tables with the catalogManaged table feature on every write target.
• Permission to create a schema and tables in a catalog. This guide uses main.

Create the tables

Run the following statements outside a transaction. Transactions don’t support DDL.

Set delta.feature.catalogManaged at creation time. Catalog commits move commit coordination from the file system to Unity Catalog, which is what lets one commit span two tables.

CREATE SCHEMA IF NOT EXISTS main.txn_demo;

CREATE TABLE main.txn_demo.accounts (
  account_id BIGINT,
  owner      STRING,
  balance    DECIMAL(12, 2)
) TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported');

CREATE TABLE main.txn_demo.transfer_log (
  from_account BIGINT,
  to_account   BIGINT,
  amount       DECIMAL(12, 2),
  logged_at    TIMESTAMP
) TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported');

Add a constraint so the example has a rule to break, then seed two accounts:

ALTER TABLE main.txn_demo.accounts
  ADD CONSTRAINT positive_balance CHECK (balance >= 0);

INSERT INTO main.txn_demo.accounts VALUES
  (1, 'Ada',   500.00),
  (2, 'Grace', 125.00);

To confirm that catalog commits are on, run DESCRIBE DETAIL main.txn_demo.accounts and look for catalogManaged in the tableFeatures column.

Move money in one commit

The following transaction debits one account, credits another, and writes the audit row. All three statements commit together.

BEGIN ATOMIC
  UPDATE main.txn_demo.accounts SET balance = balance - 100.00 WHERE account_id = 1;
  UPDATE main.txn_demo.accounts SET balance = balance + 100.00 WHERE account_id = 2;
  INSERT INTO main.txn_demo.transfer_log
    VALUES (1, 2, 100.00, current_timestamp());
END;

Check the result:

SELECT account_id, owner, balance FROM main.txn_demo.accounts ORDER BY account_id;

account_id  owner   balance
----------  ------  -------
1           Ada     400.00
2           Grace   225.00

Watch it roll back

Now run a transfer that Grace can’t cover. The audit row and the credit come first and both succeed. The debit is last, and it violates positive_balance.

BEGIN ATOMIC
  INSERT INTO main.txn_demo.transfer_log
    VALUES (2, 1, 5000.00, current_timestamp());
  UPDATE main.txn_demo.accounts SET balance = balance + 5000.00 WHERE account_id = 1;
  UPDATE main.txn_demo.accounts SET balance = balance - 5000.00 WHERE account_id = 2;
END;

The statement fails with DELTA_VIOLATE_CONSTRAINT_WITH_VALUES. Verify that the two successful statements were undone as well:

SELECT
  (SELECT balance FROM main.txn_demo.accounts WHERE account_id = 1) AS ada,
  (SELECT count(*) FROM main.txn_demo.transfer_log)                 AS log_rows;

ada     log_rows
------  --------
400.00  1

Ada’s balance is untouched and the log still holds one row. Without the transaction, you would be holding a phantom 5000.00 credit and an audit entry for a transfer that never happened.

Fail fast instead

The rollback above does the right thing, but it does the work first and discards it. Validate up front with SIGNAL, which raises an error and triggers the same automatic rollback:

BEGIN ATOMIC
  IF (SELECT balance FROM main.txn_demo.accounts WHERE account_id = 2) < 5000.00 THEN
    SIGNAL SQLSTATE '75001'
      SET MESSAGE_TEXT = 'Insufficient funds in account 2.';
  END IF;

  UPDATE main.txn_demo.accounts SET balance = balance - 5000.00 WHERE account_id = 2;
  UPDATE main.txn_demo.accounts SET balance = balance + 5000.00 WHERE account_id = 1;
END;

Choose a mode

Prefer BEGIN ATOMIC. It detects conflicts at the row level, so two transactions can write different rows of the same file without colliding, and it can’t leave a session holding an open transaction.

Reach for interactive transactions when a client outside SQL decides whether to commit. Start those sessions with a ROLLBACK to clear any leftover state, and note that they roll back after 10 minutes of inactivity.

Behaviour to plan for

Reads are repeatable. The first time a transaction touches a table, it pins a snapshot. Every later read of that table in the same transaction sees that snapshot, even if someone else commits to it meanwhile.

Commits are optimistic. Nothing locks. Conflicts surface at commit time, and the loser fails. Retry failed transactions against fresh data rather than assuming they’ll succeed.

One transaction is one Delta log entry. However many statements ran, the table history shows a single commit, with the individual operations as JSON metadata. Auditing and rollback stay simple.

Limits worth knowing

• No DDL. Run CREATE, ALTER, and DROP outside the transaction.
• No time travel, no SHOW TABLES, and no queries against system tables inside a transaction.
• No path-based access. Selecting straight from a storage path fails with PATH_BASED_ACCESS. To read a non-transactional source, register it as a table and add the WITH (allow_nontransactional_read = true) hint.
• Up to 100 tables written or read, and up to 100 views read, per transaction.
• Every transaction rolls back after 48 hours.

Clean up

DROP SCHEMA main.txn_demo CASCADE;

References & Further Reading

• Transactions
• Transaction modes
• Catalog commits
• ATOMIC compound statement
• July 2026 platform release notes

Understanding the Error

[RequestId=<GUID> ErrorClass=INVALID_STATE.RESTORATION_PERIOD_EXPIRED] Cannot undrop table because the table was deleted before the maximum supported restorable period of 7 days.

When working with data in Databricks, particularly with the Unity Catalog managed tables, users may encounter the INVALID_STATE.RESTORATION_PERIOD_EXPIRED error. This error suggests an attempt to recover a table that was deleted more than 7 days ago, exceeding the maximum restorable period supported by Databricks. However, may not always be the case.

The INVALID_STATE.RESTORATION_PERIOD_EXPIRED error occurs under the following conditions:

• Time Limit Exceeded: The primary reason is attempting to undrop a table after the 7-day restoration window has closed.

• Recreated Table: If a table with the same name has been recreated after being dropped, it can complicate the restoration process.

• Table Name Issues: Mistakes in the table name during the restoration command might lead to failure in identifying the correct dropped table.

Solution

SHOW TABLES DROPPED IN my_schema;

ALTER TABLE existing_table RENAME TO new_table_name;  -- Rename the existing table
UNDROP TABLE WITH ID 'your_table_id_here';            -- Restore the dropped table

UNDROP TABLE WITH ID 'your_table_id_here';

References & Further Reading

Summary

• Wrapping individual .withColumn() calls in functions and chaining with .transform() hides column-level intent — you have to read every function definition to know what columns are being added
• .withColumns() (Spark 3.3+ / DBR 12.0+) keeps every column name and expression visible at the call site, in a single plan node
• .transform() shines at pipeline-level chaining: deduplication, SCD logic, audit metadata — not individual column expressions
• Both approaches are testable.

from faker import Faker
from pyspark.sql import functions as F
from pyspark.sql.types import *
import random

fake = Faker()
Faker.seed(42)
random.seed(42)

rows = []
for _ in range(500):
    rows.append((
        fake.first_name(),
        fake.last_name(),
        random.randint(18, 75),
        f"  {fake.email()}  ",  # whitespace to clean
        random.choice(["active", "inactive", "pending", "suspended"]),
    ))

schema = StructType([
    StructField("first",  StringType()),
    StructField("last",   StringType()),
    StructField("age",    IntegerType()),
    StructField("email",  StringType()),
    StructField("status", StringType()),
])

df = spark.createDataFrame(rows, schema)
df.show(5, truncate=False)

+--------+--------+---+-----------------------------+--------+
|first   |last    |age|email                        |status  |
+--------+--------+---+-----------------------------+--------+
|Danielle|Johnson |58 |  john21@example.net         |active  |
|Joy     |Gardner |19 |  fjohnson@example.org       |pending |
|Jesse   |Guzman  |33 |  jennifermiles@example.com  |inactive|
|Jeffrey |Lawrence|26 |  blakeerik@example.com      |active  |
|Matthew |Moore   |61 |  curtis61@example.com       |active  |
+--------+--------+---+-----------------------------+--------+
only showing top 5 rows

Approach 1: .transform() with Per-Column Functions

This is the pattern from the original post — each column gets its own function:

from pyspark.sql import functions as F

def add_full_name(df):
    return df.withColumn(
        "full_name", F.concat_ws(" ", F.col("first"), F.col("last"))
    )

def categorize_age(df):
    return df.withColumn(
        "age_group",
        F.when(F.col("age") < 30, "young")
         .when(F.col("age") < 50, "mid")
         .otherwise("senior"),
    )

def normalize_email(df):
    return df.withColumn("email_clean", F.lower(F.trim(F.col("email"))))

def clean_status(df):
    return df.withColumn(
        "status_clean",
        F.when(F.col("status") == "active", "ACTIVE").otherwise("INACTIVE"),
    )

df_transform = (
    df
    .transform(add_full_name)
    .transform(categorize_age)
    .transform(normalize_email)
    .transform(clean_status)
)

df_transform.show(5, truncate=False)

+--------+--------+---+-----------------------------+--------+----------------+---------+-------------------------+------------+
|first   |last    |age|email                        |status  |full_name       |age_group|email_clean              |status_clean|
+--------+--------+---+-----------------------------+--------+----------------+---------+-------------------------+------------+
|Danielle|Johnson |58 |  john21@example.net         |active  |Danielle Johnson|senior   |john21@example.net       |ACTIVE      |
|Joy     |Gardner |19 |  fjohnson@example.org       |pending |Joy Gardner     |young    |fjohnson@example.org     |INACTIVE    |
|Jesse   |Guzman  |33 |  jennifermiles@example.com  |inactive|Jesse Guzman    |mid      |jennifermiles@example.com|INACTIVE    |
|Jeffrey |Lawrence|26 |  blakeerik@example.com      |active  |Jeffrey Lawrence|young    |blakeerik@example.com    |ACTIVE      |
|Matthew |Moore   |61 |  curtis61@example.com       |active  |Matthew Moore   |senior   |curtis61@example.com     |ACTIVE      |
+--------+--------+---+-----------------------------+--------+----------------+---------+-------------------------+------------+
only showing top 5 rows

Approach 2: .withColumns() — Explicit and Compact

from pyspark.sql import functions as F

df_with_columns = df.withColumns({
    "full_name":    F.concat_ws(" ", F.col("first"), F.col("last")),
    "age_group":    F.when(F.col("age") < 30, "young")
                     .when(F.col("age") < 50, "mid")
                     .otherwise("senior"),
    "email_clean":  F.lower(F.trim(F.col("email"))),
})

df_with_columns.show(5, truncate=False)

+--------+--------+---+-----------------------------+--------+----------------+---------+-------------------------+
|first   |last    |age|email                        |status  |full_name       |age_group|email_clean              |
+--------+--------+---+-----------------------------+--------+----------------+---------+-------------------------+
|Danielle|Johnson |58 |  john21@example.net         |active  |Danielle Johnson|senior   |john21@example.net       |
|Joy     |Gardner |19 |  fjohnson@example.org       |pending |Joy Gardner     |young    |fjohnson@example.org     |
|Jesse   |Guzman  |33 |  jennifermiles@example.com  |inactive|Jesse Guzman    |mid      |jennifermiles@example.com|
|Jeffrey |Lawrence|26 |  blakeerik@example.com      |active  |Jeffrey Lawrence|young    |blakeerik@example.com    |
|Matthew |Moore   |61 |  curtis61@example.com       |active  |Matthew Moore   |senior   |curtis61@example.com     |
+--------+--------+---+-----------------------------+--------+----------------+---------+-------------------------+
only showing top 5 rows

When .transform() IS the Right Tool

.transform() is excellent at the pipeline grain — composing meaningful DataFrame stages:

from pyspark.sql import functions as F
from pyspark.sql.window import Window

def select_latest_per_email(df):
    """Keep the most recent record per email."""
    w = Window.partitionBy("email").orderBy(F.col("age").desc())
    return (df
        .withColumn("_rn", F.row_number().over(w))
        .filter("_rn = 1")
        .drop("_rn"))

def enrich_columns(df):
    """Add derived columns in one pass."""
    return df.withColumns({
        "full_name":    F.concat_ws(" ", F.col("first"), F.col("last")),
        "age_group":    F.when(F.col("age") < 30, "young")
                         .when(F.col("age") < 50, "mid")
                         .otherwise("senior"),
        "email_clean":  F.lower(F.trim(F.col("email"))),
        "status_clean": F.when(F.col("status") == "active", "ACTIVE")
                         .otherwise("INACTIVE"),
    })

def add_audit_metadata(df):
    """Standard audit columns."""
    return df.withColumns({
        "ingested_at": F.current_timestamp(),
        "source":      F.lit("CRM"),
    })

# Pipeline-level chaining — THIS is the right grain for .transform()
df_final = (
    df
    .transform(select_latest_per_email)
    .transform(enrich_columns)
    .transform(add_audit_metadata)
)

df_final.show(5, truncate=False)

+-------+---------+---+-----------------------------+---------+---------------+---------+-------------------------+------------+--------------------------+------+
|first  |last     |age|email                        |status   |full_name      |age_group|email_clean              |status_clean|ingested_at               |source|
+-------+---------+---+-----------------------------+---------+---------------+---------+-------------------------+------------+--------------------------+------+
|Clayton|Gray     |47 |  abrown@example.net         |suspended|Clayton Gray   |mid      |abrown@example.net       |INACTIVE    |2026-02-27 19:08:57.959623|CRM   |
|James  |Rodriguez|59 |  adam65@example.net         |suspended|James Rodriguez|senior   |adam65@example.net       |INACTIVE    |2026-02-27 19:08:57.959623|CRM   |
|Tanya  |Riley    |60 |  adkinsbrian@example.net    |pending  |Tanya Riley    |senior   |adkinsbrian@example.net  |INACTIVE    |2026-02-27 19:08:57.959623|CRM   |
|Larry  |Gibson   |35 |  adrianabailey@example.org  |pending  |Larry Gibson   |mid      |adrianabailey@example.org|INACTIVE    |2026-02-27 19:08:57.959623|CRM   |
|Kelly  |Raymond  |73 |  agarcia@example.net        |pending  |Kelly Raymond  |senior   |agarcia@example.net      |INACTIVE    |2026-02-27 19:08:57.959623|CRM   |
+-------+---------+---+-----------------------------+---------+---------------+---------+-------------------------+------------+--------------------------+------+
only showing top 5 rows

What About Testing?

A fair point for .transform(): you can test each function in isolation with a tiny DataFrame. But .withColumns() is equally testable:

from pyspark.sql import functions as F

# Define expressions as reusable variables
_age_group = (
    F.when(F.col("age") < 30, "young")
     .when(F.col("age") < 50, "mid")
     .otherwise("senior")
)

# Test against a 3-row DataFrame — same as you would with a function
test_df = spark.createDataFrame([(25,), (45,), (65,)], ["age"])
test_df.select(_age_group.alias("age_group")).show()

# Or test the full enrichment dict
ENRICHMENT_COLS = {
    "age_group":    _age_group,
    "status_clean": F.when(F.col("status") == "active", "ACTIVE")
                     .otherwise("INACTIVE"),
}

test_df2 = spark.createDataFrame(
    [(25, "active"), (45, "inactive")],
    ["age", "status"],
)
test_df2.withColumns(ENRICHMENT_COLS).show()

+---------+
|age_group|
+---------+
|    young|
|      mid|
|   senior|
+---------+

+---+--------+---------+------------+
|age|  status|age_group|status_clean|
+---+--------+---------+------------+
| 25|  active|    young|      ACTIVE|
| 45|inactive|      mid|    INACTIVE|
+---+--------+---------+------------+

References & Further Reading

• PySpark DataFrame.withColumns API
• PySpark DataFrame.transform API
• From 20 withColumn() Calls to 1 withColumns()
• Rename Multiple Columns with withColumnsRenamed()

Summary

• Calling .withColumn() in a loop creates a new DataFrame per iteration, generating deeply nested logical plans that slow down the Catalyst optimizer
• .withColumns() (Spark 3.3+ / DBR 12.0+) accepts a dictionary of {column_name: expression} and applies all transformations in a single plan step
• Real-world use cases: metadata enrichment, column standardisation, acronym expansion, cleaning messy column names

Introduction

One of the most common PySpark anti-patterns is wrapping .withColumn() in a for loop. It reads well, feels Pythonic, and works… until it doesn’t.

Every call to .withColumn() returns a brand-new DataFrame object. Spark stacks each transformation as a separate Project node in the logical plan. With 50 columns in your loop, that’s 50 nested Project nodes the Catalyst optimizer has to traverse, analyze, and optimize through. With hundreds of columns (common in wide fact tables or IoT data) you can even hit a StackOverflowError because plan tree traversal is recursive.

.withColumns() was introduced in PySpark 3.3 to solve exactly this. One dictionary in, one plan node out.

df.withColumns(colsMap: dict[str, Column]) -> DataFrame

from faker import Faker
from pyspark.sql import functions as F
from pyspark.sql.types import *
import random

fake = Faker()
Faker.seed(42)
random.seed(42)

# Generate 1000 rows of realistic messy data
rows = []
for _ in range(1000):
    rows.append((
        fake.random_int(min=10000, max=99999),          # CustID
        f"  {fake.name()}  ",                            # custName (with whitespace)
        fake.date_between("-2y", "today").isoformat(),   # trnx_date
        round(random.uniform(5.0, 9999.99), 4),         # trnxAmt (extra decimals)
        round(random.uniform(-500.0, 50000.0), 2),      # ACCT-bal
        random.choice(["pending", "complete", "failed", "REVERSED"]),  # pmt$Status
        fake.street_address(),                           # addr.Line1
    ))

schema = StructType([
    StructField("CustID",     IntegerType()),
    StructField("custName",   StringType()),
    StructField("trnx_date",  StringType()),
    StructField("trnxAmt",    DoubleType()),
    StructField("ACCT-bal",   DoubleType()),
    StructField("pmt$Status", StringType()),
    StructField("addr.Line1", StringType()),
])

df = spark.createDataFrame(rows, schema)
df.show(5, truncate=False)

+------+---------------------+----------+---------+--------+----------+------------------------------+
|CustID|custName             |trnx_date |trnxAmt  |ACCT-bal|pmt$Status|addr.Line1                    |
+------+---------------------+----------+---------+--------+----------+------------------------------+
|93810 |  Patrick Sanchez    |2024-05-30|6396.0645|763.04  |failed    |819 Johnson Course            |
|38657 |  Lance Hoffman      |2025-07-25|2452.6916|6546.67 |pending   |79402 Peterson Drives Apt. 511|
|59797 |  Ryan Munoz         |2025-09-30|6768.6046|44555.07|pending   |161 Calderon River Suite 931  |
|16006 |  Patricia Galloway  |2025-11-04|5906.9668|1105.03 |pending   |4752 Kelly Skyway             |
|44993 |  Melinda Jones      |2025-05-27|2190.2844|25020.44|pending   |76483 Cameron Trail           |
+------+---------------------+----------+---------+--------+----------+------------------------------+
only showing top 5 rows

Code Examples

from pyspark.sql import functions as F

# Each iteration creates a new DataFrame with a deeper logical plan
columns_to_add = {
    "trnx_date_upper":   F.upper(F.col("trnx_date")),
    "cust_name_trimmed":  F.trim(F.col("custName")),
    "acct_balance_abs":   F.abs(F.col("`ACCT-bal`")),
    "txn_amt_rounded":    F.round(F.col("trnxAmt"), 2),
    "load_timestamp":     F.current_timestamp(),
}

df_result = df
for col_name, expression in columns_to_add.items():
    df_result = df_result.withColumn(col_name, expression)
# 5 iterations = 5 nested Project nodes in the plan

df_result.show(5, truncate=False)

+------+---------------------+----------+---------+--------+----------+------------------------------+---------------+-----------------+----------------+---------------+--------------------------+
|CustID|custName             |trnx_date |trnxAmt  |ACCT-bal|pmt$Status|addr.Line1                    |trnx_date_upper|cust_name_trimmed|acct_balance_abs|txn_amt_rounded|load_timestamp            |
+------+---------------------+----------+---------+--------+----------+------------------------------+---------------+-----------------+----------------+---------------+--------------------------+
|93810 |  Patrick Sanchez    |2024-05-30|6396.0645|763.04  |failed    |819 Johnson Course            |2024-05-30     |Patrick Sanchez  |763.04          |6396.06        |2026-02-18 12:13:29.416136|
|38657 |  Lance Hoffman      |2025-07-25|2452.6916|6546.67 |pending   |79402 Peterson Drives Apt. 511|2025-07-25     |Lance Hoffman    |6546.67         |2452.69        |2026-02-18 12:13:29.416136|
|59797 |  Ryan Munoz         |2025-09-30|6768.6046|44555.07|pending   |161 Calderon River Suite 931  |2025-09-30     |Ryan Munoz       |44555.07        |6768.6         |2026-02-18 12:13:29.416136|
|16006 |  Patricia Galloway  |2025-11-04|5906.9668|1105.03 |pending   |4752 Kelly Skyway             |2025-11-04     |Patricia Galloway|1105.03         |5906.97        |2026-02-18 12:13:29.416136|
|44993 |  Melinda Jones      |2025-05-27|2190.2844|25020.44|pending   |76483 Cameron Trail           |2025-05-27     |Melinda Jones    |25020.44        |2190.28        |2026-02-18 12:13:29.416136|
+------+---------------------+----------+---------+--------+----------+------------------------------+---------------+-----------------+----------------+---------------+--------------------------+
only showing top 5 rows

from pyspark.sql import functions as F

# Single plan node, single pass, same result
df_result = df.withColumns({
    "trnx_date_upper":   F.upper(F.col("trnx_date")),
    "cust_name_trimmed":  F.trim(F.col("custName")),
    "acct_balance_abs":   F.abs(F.col("`ACCT-bal`")),
    "txn_amt_rounded":    F.round(F.col("trnxAmt"), 2),
    "load_timestamp":     F.current_timestamp(),
})

df_result.show(5, truncate=False)

+------+---------------------+----------+---------+--------+----------+------------------------------+---------------+-----------------+----------------+---------------+--------------------------+
|CustID|custName             |trnx_date |trnxAmt  |ACCT-bal|pmt$Status|addr.Line1                    |trnx_date_upper|cust_name_trimmed|acct_balance_abs|txn_amt_rounded|load_timestamp            |
+------+---------------------+----------+---------+--------+----------+------------------------------+---------------+-----------------+----------------+---------------+--------------------------+
|93810 |  Patrick Sanchez    |2024-05-30|6396.0645|763.04  |failed    |819 Johnson Course            |2024-05-30     |Patrick Sanchez  |763.04          |6396.06        |2026-02-18 12:13:31.643766|
|38657 |  Lance Hoffman      |2025-07-25|2452.6916|6546.67 |pending   |79402 Peterson Drives Apt. 511|2025-07-25     |Lance Hoffman    |6546.67         |2452.69        |2026-02-18 12:13:31.643766|
|59797 |  Ryan Munoz         |2025-09-30|6768.6046|44555.07|pending   |161 Calderon River Suite 931  |2025-09-30     |Ryan Munoz       |44555.07        |6768.6         |2026-02-18 12:13:31.643766|
|16006 |  Patricia Galloway  |2025-11-04|5906.9668|1105.03 |pending   |4752 Kelly Skyway             |2025-11-04     |Patricia Galloway|1105.03         |5906.97        |2026-02-18 12:13:31.643766|
|44993 |  Melinda Jones      |2025-05-27|2190.2844|25020.44|pending   |76483 Cameron Trail           |2025-05-27     |Melinda Jones    |25020.44        |2190.28        |2026-02-18 12:13:31.643766|
+------+---------------------+----------+---------+--------+----------+------------------------------+---------------+-----------------+----------------+---------------+--------------------------+
only showing top 5 rows

from pyspark.sql import functions as F
import re

def to_snake_case(name: str) -> str:
    """Convert any naming convention to snake_case."""
    name = re.sub(r'[.$\-@#!]', '_', name)            # special chars to _
    name = re.sub(r'([a-z])([A-Z])', r'\1_\2', name)  # camelCase split
    name = re.sub(r'_{2,}', '_', name)                 # collapse doubles
    return name.lower().strip('_')

ABBREVIATIONS = {
    "trnx": "transaction", "cust": "customer",
    "acct": "account",     "pmt":  "payment",
    "addr": "address",     "amt":  "amount",
    "bal":  "balance",     "qty":  "quantity",
    "desc": "description", "dt":   "date",
    "nbr":  "number",      "cd":   "code",
    "nm":   "name",
}

def expand_abbreviations(name: str) -> str:
    """Expand common data abbreviations in a column name."""
    parts = name.split("_")
    return "_".join(ABBREVIATIONS.get(p, p) for p in parts)

# Build the transformation dict dynamically
transformations = {}
for col_name in df.columns:
    clean_name = expand_abbreviations(to_snake_case(col_name))
    transformations[clean_name] = F.col(f"`{col_name}`")

df_clean = df.withColumns(transformations)
print(df_clean.columns)

['CustID', 'custName', 'trnx_date', 'trnxAmt', 'ACCT-bal', 'pmt$Status', 'addr.Line1', 'customer_id', 'customer_name', 'transaction_date', 'transaction_amount', 'account_balance', 'payment_status', 'address_line1']

from pyspark.sql import functions as F

pipeline_run_id = "run-2026-02-16-001"

df_enriched = df.withColumns({
    "ingestion_timestamp": F.current_timestamp(),
    "source_system":       F.lit("CRM_PROD"),
    "pipeline_run_id":     F.lit(pipeline_run_id),
    "row_hash":            F.sha2(F.concat_ws("|", *[F.col(f"`{c}`").cast("string") for c in df.columns]), 256),
    "is_valid": F.when(
        F.col("trnxAmt").isNotNull() & (F.col("trnxAmt") > 0), True
    ).otherwise(False),
})

df_enriched.select("CustID", "ingestion_timestamp", "source_system", "row_hash", "is_valid").show(5, truncate=40)

+------+--------------------------+-------------+----------------------------------------+--------+
|CustID|       ingestion_timestamp|source_system|                                row_hash|is_valid|
+------+--------------------------+-------------+----------------------------------------+--------+
| 93810|2026-02-18 12:13:32.732663|     CRM_PROD|0a16938d05f57ecdac3a93290a6b091113435...|    true|
| 38657|2026-02-18 12:13:32.732663|     CRM_PROD|2852cc60ce03fa94327046eaa23647536574e...|    true|
| 59797|2026-02-18 12:13:32.732663|     CRM_PROD|8fdc5170c3c067b621370b9b015f84dd4e66c...|    true|
| 16006|2026-02-18 12:13:32.732663|     CRM_PROD|15e0ed67555e8ed6a965c5fa76a6071d3b953...|    true|
| 44993|2026-02-18 12:13:32.732663|     CRM_PROD|7dcd0fb0db5ffe761939d1de92d8b1798783e...|    true|
+------+--------------------------+-------------+----------------------------------------+--------+
only showing top 5 rows

import org.apache.spark.sql.functions._

// Scala uses a Map of column name to expression
val dfResult = df.withColumns(Map(
  "trnx_date_upper"   -> upper(col("trnx_date")),
  "cust_name_trimmed"  -> trim(col("custName")),
  "acct_balance_abs"   -> abs(col("`ACCT-bal`")),
  "txn_amt_rounded"    -> round(col("trnxAmt"), 2),
  "load_timestamp"     -> current_timestamp()
))

Detail

References & Further Reading

• PySpark DataFrame.withColumns API Documentation
• Companion Tip: Rename Multiple Columns in PySpark with withColumnsRenamed()

Summary

• Calling .withColumnRenamed() in a loop has the same problem as .withColumn() loops: each call adds a new Project node to the logical plan
• .withColumnsRenamed() (Spark 3.4+ / DBR 13.0+) takes a dictionary of {old_name: new_name} and applies all renames in a single plan step
• Perfect for converting to snake_case, expanding abbreviations, removing special characters, and applying generic naming conventions across entire DataFrames

Introduction

This is the companion anti-pattern to calling .withColumn() in a loop. If you’ve ever ingested data from vendor feeds, legacy systems, APIs, or CSVs, you know the pain. Column names come in every flavour: CustID, trnxAmt, ACCT-bal, pmt$Status, addr.Line1. The instinct is to write a loop renaming them one by one.

The problem is identical to .withColumn() loops. Each .withColumnRenamed() call returns a new DataFrame with a new Project node stacked on the logical plan. With dozens of renames, the Catalyst optimizer bogs down traversing and optimizing through all those nested nodes.

.withColumnsRenamed() was introduced in PySpark 3.4 to solve this. One dictionary mapping old names to new names, one plan node.

df.withColumnsRenamed(colsMap: dict[str, str]) -> DataFrame

from faker import Faker
from pyspark.sql.types import *
import random

fake = Faker()
Faker.seed(42)
random.seed(42)

# Generate 1000 rows with intentionally messy column names
rows = []
for _ in range(1000):
    rows.append((
        fake.random_int(min=10000, max=99999),
        fake.name(),
        fake.date_between("-2y", "today").isoformat(),
        round(random.uniform(5.0, 9999.99), 4),
        round(random.uniform(-500.0, 50000.0), 2),
        random.choice(["pending", "complete", "failed", "REVERSED"]),
        fake.street_address(),
    ))

schema = StructType([
    StructField("CustID",     IntegerType()),
    StructField("custName",   StringType()),
    StructField("trnx_date",  StringType()),
    StructField("trnxAmt",    DoubleType()),
    StructField("ACCT-bal",   DoubleType()),
    StructField("pmt$Status", StringType()),
    StructField("addr.Line1", StringType()),
])

df = spark.createDataFrame(rows, schema)
print(df.columns)
df.show(5, truncate=False)

['CustID', 'custName', 'trnx_date', 'trnxAmt', 'ACCT-bal', 'pmt$Status', 'addr.Line1']
+------+-----------------+----------+---------+--------+----------+------------------------------+
|CustID|custName         |trnx_date |trnxAmt  |ACCT-bal|pmt$Status|addr.Line1                    |
+------+-----------------+----------+---------+--------+----------+------------------------------+
|93810 |Patrick Sanchez  |2024-06-01|6396.0645|763.04  |failed    |819 Johnson Course            |
|38657 |Lance Hoffman    |2025-07-27|2452.6916|6546.67 |pending   |79402 Peterson Drives Apt. 511|
|59797 |Ryan Munoz       |2025-10-02|6768.6046|44555.07|pending   |161 Calderon River Suite 931  |
|16006 |Patricia Galloway|2025-11-06|5906.9668|1105.03 |pending   |4752 Kelly Skyway             |
|44993 |Melinda Jones    |2025-05-29|2190.2844|25020.44|pending   |76483 Cameron Trail           |
+------+-----------------+----------+---------+--------+----------+------------------------------+
only showing top 5 rows

Code Examples

# Each iteration creates a new DataFrame with a deeper logical plan
rename_map = {
    "CustID":     "customer_id",
    "trnxAmt":    "transaction_amount",
    "ACCT-bal":   "account_balance",
    "pmt$Status": "payment_status",
    "addr.Line1": "address_line_1",
}

df_result = df
for old_name, new_name in rename_map.items():
    df_result = df_result.withColumnRenamed(old_name, new_name)
# 5 iterations = 5 nested Project nodes

print(df_result.columns)

['customer_id', 'custName', 'trnx_date', 'transaction_amount', 'account_balance', 'payment_status', 'address_line_1']

# Single plan node, single pass, same result
df_result = df.withColumnsRenamed({
    "CustID":     "customer_id",
    "trnxAmt":    "transaction_amount",
    "ACCT-bal":   "account_balance",
    "pmt$Status": "payment_status",
    "addr.Line1": "address_line_1",
})

print(df_result.columns)

['customer_id', 'custName', 'trnx_date', 'transaction_amount', 'account_balance', 'payment_status', 'address_line_1']

import re

def to_snake_case(name: str) -> str:
    """Convert any naming convention to clean snake_case."""
    name = re.sub(r'[.$\-@#!&*()+=/\\]', '_', name)      # special chars to _
    name = re.sub(r'([a-z0-9])([A-Z])', r'\1_\2', name)   # camelCase split
    name = re.sub(r'([A-Z]+)([A-Z][a-z])', r'\1_\2', name) # ACRONYMWord split
    name = re.sub(r'_{2,}', '_', name)                      # collapse doubles
    return name.lower().strip('_')

# Build rename dict from current column names
rename_map = {col: to_snake_case(col) for col in df.columns}
print(rename_map)

# Apply all renames in one pass
df_clean = df.withColumnsRenamed(rename_map)
print(df_clean.columns)

{'CustID': 'cust_id', 'custName': 'cust_name', 'trnx_date': 'trnx_date', 'trnxAmt': 'trnx_amt', 'ACCT-bal': 'acct_bal', 'pmt$Status': 'pmt_status', 'addr.Line1': 'addr_line1'}
['cust_id', 'cust_name', 'trnx_date', 'trnx_amt', 'acct_bal', 'pmt_status', 'addr_line1']

import re

ABBREVIATIONS = {
    "trnx": "transaction", "cust": "customer",
    "acct": "account",     "pmt":  "payment",
    "addr": "address",     "amt":  "amount",
    "bal":  "balance",     "qty":  "quantity",
    "desc": "description", "dt":   "date",
    "ts":   "timestamp",   "nbr":  "number",
    "num":  "number",      "cd":   "code",
    "nm":   "name",
}

def expand_column_name(name: str) -> str:
    """snake_case + expand common abbreviations."""
    # First convert to snake_case
    snake = re.sub(r'[.$\-@#!]', '_', name)
    snake = re.sub(r'([a-z0-9])([A-Z])', r'\1_\2', snake)
    snake = re.sub(r'_{2,}', '_', snake).lower().strip('_')
    # Then expand abbreviations
    parts = snake.split("_")
    return "_".join(ABBREVIATIONS.get(p, p) for p in parts)

rename_map = {col: expand_column_name(col) for col in df.columns}
print(rename_map)

df_clean = df.withColumnsRenamed(rename_map)
print(df_clean.columns)

{'CustID': 'customer_id', 'custName': 'customer_name', 'trnx_date': 'transaction_date', 'trnxAmt': 'transaction_amount', 'ACCT-bal': 'account_balance', 'pmt$Status': 'payment_status', 'addr.Line1': 'address_line1'}
['customer_id', 'customer_name', 'transaction_date', 'transaction_amount', 'account_balance', 'payment_status', 'address_line1']

# First simulate vendor-prefixed data from our setup df
df_vendor = df.withColumnsRenamed({col: f"SRC_{col}" for col in df.columns})
print("Before:", df_vendor.columns)

# Now strip the prefix in one pass
prefix = "SRC_"
rename_map = {
    col: col[len(prefix):] if col.startswith(prefix) else col
    for col in df_vendor.columns
}
df_clean = df_vendor.withColumnsRenamed(rename_map)
print("After: ", df_clean.columns)

Before: ['SRC_CustID', 'SRC_custName', 'SRC_trnx_date', 'SRC_trnxAmt', 'SRC_ACCT-bal', 'SRC_pmt$Status', 'SRC_addr.Line1']
After:  ['CustID', 'custName', 'trnx_date', 'trnxAmt', 'ACCT-bal', 'pmt$Status', 'addr.Line1']

import org.apache.spark.sql.functions._

// Scala uses a Map of old name to new name
val dfResult = df.withColumnsRenamed(Map(
  "CustID"     -> "customer_id",
  "trnxAmt"    -> "transaction_amount",
  "ACCT-bal"   -> "account_balance",
  "pmt$Status" -> "payment_status",
  "addr.Line1" -> "address_line_1"
))

Detail

References & Further Reading

• PySpark DataFrame.withColumnsRenamed API Documentation
• Companion Tip: From 20 withColumn() Calls to 1 withColumns()

Summary

• Get Metadata & Context on your current notebook environment
• User Credentials (email, IPs, azuread object ids)
• Browser details & language
• Cluster Ids

Code

dbutils.notebook.entry_point.getDbutils().notebook().getContext()

dbutils.notebook.entry_point.getDbutils().notebook().getContext() \
  .notebookPath() \
  .toString() 

import json
json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())

dbutils.notebook.getContext()

dbutils.notebook.getContext().notebookPath \
  .toString() 

val context = dbutils.notebook.getContext() // Returns MAP[String,String] no need to map it

# List only the objects that begin with "spark"
ls(pattern = "^spark")

 spark.databricks.api.url  

[1] "spark.databricks.api.url"                                              
[2] "spark.databricks.driverNfs.clusterWideVirtualEnv"                      
[3] "spark.databricks.env"                                                  
[4] "spark.databricks.inherited.credentials.keys.spark.databricks.api.token"
[5] "spark.databricks.notebook.id"                                          
[6] "spark.databricks.notebook.path"                                        
[7] "spark.databricks.replId"                                               
[8] "spark.databricks.token"                                                
[9] "spark.scheduler.pool"     
[10] "orgId"

# Get the names of objects that begin with "spark"
spark_names <- ls(pattern = "^spark")

# Get the values of these objects
spark_values <- mget(spark_names)

# Convert the named vector to a list
spark_list <- as.list(spark_values)

{'rootRunId': None,
 'currentRunId': None,
 'jobGroup': '3390617077668218965_7460294295175766547_89e1d0119f914af39bafdb017336882e',
 'tags': {'opId': 'ServerBackend-54787e706ebf268a',
  'shardName': 'az-eastus-c3',
  'opTarget': 'com.databricks.backend.common.rpc.InternalDriverBackendMessages$StartRepl',
  'clusterMemory': '8192',
  'serverBackendName': 'com.databricks.backend.daemon.driver.DriverCorral',
  'notebookId': '2048538015852092',
  'projectName': 'driver',
  'tier': 'tier-multitenant',
  'eventWindowTime': '2254196.899999991',
  'httpTarget': '/notebook/2048538015852092/command/3362446343388373',
  'commandRunId': 'dddd47ee-68fe-4fd2-9042-fbde8fd28a8e',
  'buildHash': '',
  'workspaceRoutingTarget': 'null',
  'browserHash': '#notebook/2048538015852092/command/3362446343388373',
  'host': '10.139.64.4',
  'browserPathName': '/',
  'notebookLanguage': 'sql',
  'workspaceRoutingBucket': 'null',
  'sparkVersion': '12.1.x-scala2.12',
  'hostName': 'cons-webapp-0',
  'httpMethod': 'POST',
  'browserIdleTime': '411',
  'jettyRpcJettyVersion': '9',
  'browserLanguage': 'en-GB',
  'browserTabId': '99fc6c79-18df-4ff9-b0e7-35df3da8bc08',
  'sourceIpAddress': '<AN_IP>',
  'loadedUiVersions': 'Map(monolith -> f03dcdb1e979dfefcf2f878284c318bd21970a5d)',
  'browserUserAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'orgId': '2675080094955785',
  'userAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'clusterId': '0120-154009-7bof0jyo',
  'serverEventId': 'CgwIusXzngYQ9+mEqAIQgAQYiQE6EKeKn3/v7UOlvUQlALPm2wI=',
  'rootOpId': 'ServiceMain-8ef19d61f07f0002',
  'requestIdWasMissing': 'true',
  'sessionId': 'd9be357a4121697a1de52d1c3b447874315900e2b5a5054b8aa92576f1e865be',
  'clusterCreator': '<EMAIL_ADDRESS>',
  'originatedFromEnvoy': 'true',
  'clientBranchName': 'webapp_2023-01-27_23.45.48Z_master_b5b81920_1040710074',
  'clientTimestamp': '1675421567053',
  'clusterType': 'spot',
  'requestId': 'CgwIusXzngYQ4/r7pwIQgAQYiQE6EKeKn3/v7UOlvUQlALPm2wI=',
  'browserHasFocus': 'true',
  'userId': '8925728005363108',
  'browserIsHidden': 'false',
  'principalIdpId': '<GUID>',
  'clientLocale': 'en',
  'branchName': 'webapp_2023-01-27_23.45.48Z_master_b5b81920_1040710074',
  'opType': 'ServerBackend',
  'sourcePortNumber': '0',
  'user': '<EMAIL_ADDRESS>',
  'principalIdpObjectId': '<GUID>',
  'browserHostName': '<WORKSPACE>.azuredatabricks.net',
  'parentOpId': 'RPCClient-8ef19d61f07f11ae',
  'jettyRpcType': 'InternalDriverBackendMessages$DriverBackendRequest'},
 'extraContext': {'allowStdin': 'true',
  'non_uc_api_token': '',
  'commandResultJsonMaxBytes': '20971520',
  'notebook_path': '/sqlsdamples',
  'thresholdForStoringInDbfs': '10000',
  'enableStoringResultsInDbfs': 'true',
  'api_url': 'https://eastus-c3.azuredatabricks.net',
  'aclPathOfAclRoot': '/workspace/<ID>',
  'api_token': '[REDACTED]'},
 'credentialKeys': ['adls_aad_token',
  'adls_gen2_aad_token',
  'synapse_aad_token']}
  

Detail

The code snippet dbutils.notebook.entry_point.getDbutils().notebook().getContext() may appear intimidating at first glance, but fret not, let’s break it down:

1. dbutils: This is a utility library in Databricks, providing a collection of utility functions and classes that help in interacting with Databricks functionalities.

2. notebook: This refers to the notebook module within dbutils, which contains methods for interacting with the notebook environment.

3. entry_point: The entry_point attribute within the notebook module refers to the starting point of the notebook execution.

4. getDbutils: This is a method call which returns an instance of dbutils.

5. notebook(): This method call, following getDbutils, returns an instance of the notebook module.

6. getContext(): Finally, this method retrieves the current Notebook Context, encapsulating information like notebook path, user, and other contextual details.

Benefits of Accessing Notebook Context: Accessing the Notebook Context can serve various purposes:

• Identifying User Information: By accessing the Notebook Context, you can retrieve information about the user executing the notebook. This could be useful for auditing or personalizing the notebook behavior based on the user.

• Fetching Notebook Path: If your logic depends on the particular notebook or its location within the Databricks workspace, accessing the Notebook Context is invaluable.

• Managing Execution Flow: Utilizing the Notebook Context can aid in managing the flow of notebook execution, especially in complex, multi-notebook workflows.

References & Further Reading

Summary

• Dynamic Test Case Discovery: Automatically loads test cases from the current notebook module.
• Reflection Magic: Uses reflection to introspect classes, searching for unittest.TestCase subclasses.
• Test Suite Management: Organizes tests into a suite for easier management.

Introduction

Have you ever had to define a unit test suite and got fed up with having to create a list of the unit test classes to include into the suite for running? Well this tip is for you, it looks up the objects defined in the current namespace that are of type UnitTest case.

Code

import unittest

def load_test_classes_from_notebook():
    try:
        ns = get_ipython().user_ns #UserNameSpace
    except NameError:
        ns = globals()

    suite = unittest.TestSuite()

    # Limit to classes that look like *your* tests
    nb_name = ns.get('__name__', '__main__')
    for name, obj in list(ns.items()):
        if isinstance(obj, type) and issubclass(obj, unittest.TestCase) and obj is not unittest.TestCase:
            if getattr(obj, '__module__', None) in (nb_name, '__main__'): # Maybe Overkill todo this filtering 
                suite.addTests(unittest.TestLoader().loadTestsFromTestCase(obj))
    return suite

    
suite = load_test_classes_from_notebook()
runner = unittest.TextTestRunner(verbosity=2)
runner.run(suite)

def add(a, b):
    """Perform addition of two numeric values."""
    return a + b

def multiply(a, b):  
    """Perform multiplication of two numeric values."""
    return a * b

import unittest

class TestGenericMathOperations(unittest.TestCase):
    def test_addition(self):
        # Test with numeric values
        self.assertEqual(add(2, 3), 5, "Addition of 2 and 3 should equal 5")
        self.assertEqual(add(-1, 5), 4, "Addition of -1 and 5 should equal 4") 
        self.assertAlmostEqual(add(0.1, 0.2), 0.3, places=1, msg="Addition of 0.1 and 0.2 should approximate 0.3")
        
        # Test with an invalid case
        with self.assertRaises(TypeError, msg="Adding a number and a string should raise a TypeError"):
            add(5, "hello")
            
    def test_multiplication(self):
        # Test with numeric values  
        self.assertEqual(multiply(2, 3), 6, "Multiplication of 2 and 3 should equal 6")
        self.assertEqual(multiply(-2, 4), -8, "Multiplication of -2 and 4 should equal -8")
        self.assertAlmostEqual(multiply(0.5, 0.4), 0.2, places=1, msg="Multiplying 0.5 and 0.4 should approximate 0.2")
        
        # Test with an invalid case
        with self.assertRaises(TypeError, msg="Multiplying a number and a string should raise a TypeError"):
            multiply(3, "world")

Details

In the spirit of enabling flexible test discovery, this snippet takes advantage of reflection to dynamically load all test cases in the users ipython notebook namespace. Let’s dive into the finer details:

1. TestSuite Setup: We initialize an empty TestSuite to collect individual test cases.

2. Current Namespace Context: Accessing get_ipython().user_ns grants a reference to the current namespace, a key tool for reflection.

3. Reflection and Filtering:

   • The dir function retrieves all objects in the module.
     
   • getattr allows accessing the object itself by name.
   • We then ensure the object is a subclass of unittest.TestCase but not the base class itself.

4. Loading Tests: unittest.TestLoader().loadTestsFromTestCase loads tests from each discovered class, and the suite aggregates them.

Benefits

• Automation and Flexibility: Automatically loading tests simplifies running new tests without manual suite updates.
• Dynamic Testing: Great for scenarios where test classes change frequently or are dynamically generated.

Cautions

• Over-Discovery: This method could inadvertently load experimental or partial test classes, leading to unintended test runs.
• Opaque Testing: The reliance on reflection makes the test suite implicit and less readable. Reviewers may struggle to understand the source of loaded tests.
• Performance: The reflective lookup over the entire module can be inefficient, especially in larger projects.
• pytest: Is more likely to align with your requirements and use concepts like fixtures to support you.

Summary

• Overview: An introduction to Databricks widgets, their usage, and how to retrieve their values.
• Dynamic Bindings: Explanation of dynamically fetching notebook bindings and when this is appropriate.
• Run Parameters: How to get job parameters when a notebook is executed as a job.

dbutils.widgets.text("input", "default")
input_value = dbutils.widgets.get("input")
print(f"The value of the input widget is: {input_value}")

run_parameters = dbutils.notebook.entry_point.getCurrentBindings()
print(run_parameters)

References & Further Reading

Summary

• Get Metadata & Context on your current notebook environment
• User Credentials (email, IPs, azuread object ids)
• Browser details & language
• Cluster Ids

Code

dbutils.notebook.entry_point.getDbutils().notebook().getContext()

dbutils.notebook.entry_point.getDbutils().notebook().getContext() \
  .notebookPath() \
  .toString() 

import json
json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())

dbutils.notebook.getContext()

dbutils.notebook.getContext().notebookPath \
  .toString() 

val context = dbutils.notebook.getContext() // Returns MAP[String,String] no need to map it

# List only the objects that begin with "spark"
ls(pattern = "^spark")

 spark.databricks.api.url  

[1] "spark.databricks.api.url"                                              
[2] "spark.databricks.driverNfs.clusterWideVirtualEnv"                      
[3] "spark.databricks.env"                                                  
[4] "spark.databricks.inherited.credentials.keys.spark.databricks.api.token"
[5] "spark.databricks.notebook.id"                                          
[6] "spark.databricks.notebook.path"                                        
[7] "spark.databricks.replId"                                               
[8] "spark.databricks.token"                                                
[9] "spark.scheduler.pool"     
[10] "orgId"

# Get the names of objects that begin with "spark"
spark_names <- ls(pattern = "^spark")

# Get the values of these objects
spark_values <- mget(spark_names)

# Convert the named vector to a list
spark_list <- as.list(spark_values)

{'rootRunId': None,
 'currentRunId': None,
 'jobGroup': '3390617077668218965_7460294295175766547_89e1d0119f914af39bafdb017336882e',
 'tags': {'opId': 'ServerBackend-54787e706ebf268a',
  'shardName': 'az-eastus-c3',
  'opTarget': 'com.databricks.backend.common.rpc.InternalDriverBackendMessages$StartRepl',
  'clusterMemory': '8192',
  'serverBackendName': 'com.databricks.backend.daemon.driver.DriverCorral',
  'notebookId': '2048538015852092',
  'projectName': 'driver',
  'tier': 'tier-multitenant',
  'eventWindowTime': '2254196.899999991',
  'httpTarget': '/notebook/2048538015852092/command/3362446343388373',
  'commandRunId': 'dddd47ee-68fe-4fd2-9042-fbde8fd28a8e',
  'buildHash': '',
  'workspaceRoutingTarget': 'null',
  'browserHash': '#notebook/2048538015852092/command/3362446343388373',
  'host': '10.139.64.4',
  'browserPathName': '/',
  'notebookLanguage': 'sql',
  'workspaceRoutingBucket': 'null',
  'sparkVersion': '12.1.x-scala2.12',
  'hostName': 'cons-webapp-0',
  'httpMethod': 'POST',
  'browserIdleTime': '411',
  'jettyRpcJettyVersion': '9',
  'browserLanguage': 'en-GB',
  'browserTabId': '99fc6c79-18df-4ff9-b0e7-35df3da8bc08',
  'sourceIpAddress': '<AN_IP>',
  'loadedUiVersions': 'Map(monolith -> f03dcdb1e979dfefcf2f878284c318bd21970a5d)',
  'browserUserAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'orgId': '2675080094955785',
  'userAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'clusterId': '0120-154009-7bof0jyo',
  'serverEventId': 'CgwIusXzngYQ9+mEqAIQgAQYiQE6EKeKn3/v7UOlvUQlALPm2wI=',
  'rootOpId': 'ServiceMain-8ef19d61f07f0002',
  'requestIdWasMissing': 'true',
  'sessionId': 'd9be357a4121697a1de52d1c3b447874315900e2b5a5054b8aa92576f1e865be',
  'clusterCreator': '<EMAIL_ADDRESS>',
  'originatedFromEnvoy': 'true',
  'clientBranchName': 'webapp_2023-01-27_23.45.48Z_master_b5b81920_1040710074',
  'clientTimestamp': '1675421567053',
  'clusterType': 'spot',
  'requestId': 'CgwIusXzngYQ4/r7pwIQgAQYiQE6EKeKn3/v7UOlvUQlALPm2wI=',
  'browserHasFocus': 'true',
  'userId': '8925728005363108',
  'browserIsHidden': 'false',
  'principalIdpId': '<GUID>',
  'clientLocale': 'en',
  'branchName': 'webapp_2023-01-27_23.45.48Z_master_b5b81920_1040710074',
  'opType': 'ServerBackend',
  'sourcePortNumber': '0',
  'user': '<EMAIL_ADDRESS>',
  'principalIdpObjectId': '<GUID>',
  'browserHostName': '<WORKSPACE>.azuredatabricks.net',
  'parentOpId': 'RPCClient-8ef19d61f07f11ae',
  'jettyRpcType': 'InternalDriverBackendMessages$DriverBackendRequest'},
 'extraContext': {'allowStdin': 'true',
  'non_uc_api_token': '',
  'commandResultJsonMaxBytes': '20971520',
  'notebook_path': '/sqlsdamples',
  'thresholdForStoringInDbfs': '10000',
  'enableStoringResultsInDbfs': 'true',
  'api_url': 'https://eastus-c3.azuredatabricks.net',
  'aclPathOfAclRoot': '/workspace/<ID>',
  'api_token': '[REDACTED]'},
 'credentialKeys': ['adls_aad_token',
  'adls_gen2_aad_token',
  'synapse_aad_token']}
  

Detail

The code snippet dbutils.notebook.entry_point.getDbutils().notebook().getContext() may appear intimidating at first glance, but fret not, let’s break it down:

1. dbutils: This is a utility library in Databricks, providing a collection of utility functions and classes that help in interacting with Databricks functionalities.

2. notebook: This refers to the notebook module within dbutils, which contains methods for interacting with the notebook environment.

3. entry_point: The entry_point attribute within the notebook module refers to the starting point of the notebook execution.

4. getDbutils: This is a method call which returns an instance of dbutils.

5. notebook(): This method call, following getDbutils, returns an instance of the notebook module.

6. getContext(): Finally, this method retrieves the current Notebook Context, encapsulating information like notebook path, user, and other contextual details.

Benefits of Accessing Notebook Context: Accessing the Notebook Context can serve various purposes:

• Identifying User Information: By accessing the Notebook Context, you can retrieve information about the user executing the notebook. This could be useful for auditing or personalizing the notebook behavior based on the user.

• Fetching Notebook Path: If your logic depends on the particular notebook or its location within the Databricks workspace, accessing the Notebook Context is invaluable.

• Managing Execution Flow: Utilizing the Notebook Context can aid in managing the flow of notebook execution, especially in complex, multi-notebook workflows.

References & Further Reading

Introduction

Understanding what your Databricks jobs are doing — and what they are costing you — has always required stitching together information from the Jobs UI, cluster logs, and billing exports. With the GA release of the Lakeflow system tables in January 2026, Databricks now provides a unified, SQL-queryable record of every job and job run across your entire account.

These tables live in the system.lakeflow schema (previously called system.workflow — the content is identical) and cover all workspaces deployed in the same cloud region. They retain 365 days of data at no additional cost, and they support streaming reads so you can build real-time monitoring pipelines on top of them.

What Tables Are Available?

The system.lakeflow schema contains four GA tables and two in Public Preview:

Practical Examples

SELECT
    j.name AS job_name,
    r.run_id,
    r.result_state,
    r.termination_code,
    r.period_start_time,
    r.period_end_time
FROM system.lakeflow.job_run_timeline r
JOIN system.lakeflow.jobs j
    ON r.workspace_id = j.workspace_id
    AND r.job_id = j.job_id
WHERE r.result_state = 'FAILED'
    AND r.period_start_time > current_timestamp() - INTERVAL 24 HOURS
ORDER BY r.period_start_time DESC;

SELECT
    j.name AS job_name,
    r.run_id,
    r.result_state,
    SUM(b.usage_quantity * lp.pricing.default) AS estimated_cost
FROM system.lakeflow.job_run_timeline r
JOIN system.lakeflow.jobs j
    ON r.workspace_id = j.workspace_id
    AND r.job_id = j.job_id
JOIN system.billing.usage b
    ON r.job_id = b.usage_metadata.job_id
    AND r.run_id = b.usage_metadata.job_run_id
JOIN system.billing.list_prices lp
    ON b.sku_name = lp.sku_name
    AND b.usage_date = lp.price_start_time
GROUP BY j.name, r.run_id, r.result_state
ORDER BY estimated_cost DESC;

SELECT
    j.name AS job_name,
    DATE(r.period_start_time) AS run_date,
    AVG(r.run_duration_ms) / 1000 AS avg_duration_seconds,
    COUNT(*) AS run_count
FROM system.lakeflow.job_run_timeline r
JOIN system.lakeflow.jobs j
    ON r.workspace_id = j.workspace_id
    AND r.job_id = j.job_id
WHERE r.period_start_time > current_timestamp() - INTERVAL 30 DAYS
    AND r.result_state = 'SUCCESS'
GROUP BY j.name, DATE(r.period_start_time)
ORDER BY j.name, run_date;

Access Requirements

To query these tables, you need one of:

• Metastore admin and account admin roles, or
• Explicit USE and SELECT grants on the system.lakeflow schema

Further Reading

• Jobs System Table Reference
• Monitoring and Observability for Lakeflow Jobs
• Lakeflow Jobs Overview
• January 2026 Release Notes