Validate Polars DataFrames: A Guide to Dataframely for Robust Data Pipelines

Are you building data pipelines with Polars and struggling to ensure data quality? Dataframely is your answer! This Python package makes your data pipelines more robust and readable. Learn how to validate your Polars DataFrames with ease!

What is Dataframely and Why Should You Use It?

Dataframely is designed to validate schemas and content within Polars DataFrames. It ensures your data meets your expectations, adds helpful schema information to DataFrame type hints, and ultimately makes your data pipelines more reliable. If you're tired of unexpected data errors causing headaches, Dataframely is for you.

• Robust Pipelines: Prevent errors by validating data against predefined schemas.
• Improved Readability: Add schema information to DataFrame type hints for better understanding.
• Simplified Debugging: Quickly identify data quality issues.

Installation: Get Started in Seconds

Installing Dataframely is straightforward. Use your preferred package manager to get started:

Using pip

pip install dataframely

Using pixi

pixi add dataframely

Usage: Define and Validate Your DataFrames

Here's how to use Dataframely to define a schema and validate your data.

1. Defining a DataFrame Schema

First, define a schema by creating a class that inherits from dy.Schema. Inside this class, define each column with its corresponding type and constraints. This gives you fine-grained control over what is considered valid.

import dataframely as dy
import polars as pl

class HouseSchema(dy.Schema):
    zip_code = dy.String(nullable=False, min_length=3)
    num_bedrooms = dy.UInt8(nullable=False)
    num_bathrooms = dy.UInt8(nullable=False)
    price = dy.Float64(nullable=False)

2. Adding Custom Rules

You can also define custom rules to enforce more complex validation logic. These rules can be applied to the entire DataFrame or grouped by specific columns.

    @dy.rule()
    def reasonable_bathroom_to_bedrooom_ratio() -> pl.Expr:
        ratio = pl.col("num_bathrooms") / pl.col("num_bedrooms")
        return (ratio >= 1 / 3) & (ratio <= 3)

    @dy.rule(group_by=["zip_code"])
    def minimum_zip_code_count() -> pl.Expr:
        return pl.len() >= 2

3. Validating Data Against the Schema

Now, let's validate a DataFrame against the defined schema. Dataframely will check if your data conforms to your expectations.

import polars as pl

df = pl.DataFrame({
    "zip_code": ["01234", "01234", "1", "213", "123", "213"],
    "num_bedrooms": [2, 2, 1, None, None, 2],
    "num_bathrooms": [1, 2, 1, 1, 0, 8],
    "price": [100_000, 110_000, 50_000, 80_000, 60_000, 160_000]
})

validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)

The validate method checks the DataFrame and can also cast columns to the expected types using cast=True. This helps automatically correct minor data type discrepancies.

Take Your Data Pipelines to the Next Level

Dataframely offers a powerful and flexible way to validate Polars DataFrames. By defining schemas and custom rules, you can ensure data quality, improve pipeline robustness, and simplify debugging. Explore the Dataframely documentation for advanced usage examples and unlock the full potential of this valuable tool.