Validate Polars DataFrames Easily: A Guide to Dataframely for Python

Ensuring data quality in your Python data pipelines is crucial. Dataframely, a Python package, offers a robust and declarative way to validate the schema and content of Polars data frames. This article covers installation, usage, and key features for a more reliable data flow.

Why Use Dataframely for Polars Data Frame Validation?

Dataframely enhances data pipeline robustness. It also improves code readability adding schema information to data frame type hints. Benefits include:

• Schema Validation: Enforce data types, nullability, and other constraints.
• Content Validation: Define custom rules to check for data integrity.
• Improved Readability: Schema definitions act as documentation.
• Early Error Detection: Catch data quality issues early in the pipeline.

Simple Installation: Get Started with Dataframely

Install Dataframely using your preferred package manager. It integrates seamlessly with your existing Python environment.

pixi add dataframely

Or using pip:

pip install dataframely

Defining Your Data Frame Schema with Dataframely

Dataframely uses a declarative approach. Define your schema using Python classes, making it easy to understand and maintain.

import dataframely as dy
import polars as pl

class HouseSchema(dy.Schema):
    zip_code = dy.String(nullable=False, min_length=3)
    num_bedrooms = dy.UInt8(nullable=False)
    num_bathrooms = dy.UInt8(nullable=False)
    price = dy.Float64(nullable=False)

    @dy.rule()
    def reasonable_bathroom_to_bedrooom_ratio() -> pl.Expr:
        ratio = pl.col("num_bathrooms") / pl.col("num_bedrooms")
        return (ratio >= 1 / 3) & (ratio <= 3)

    @dy.rule(group_by=["zip_code"])
    def minimum_zip_code_count() -> pl.Expr:
        return pl.len() >= 2

In this example, we define a schema HouseSchema with fields like zip_code, num_bedrooms, num_bathrooms, and price, each with specified data types and constraints. We also added data validation rules using the @dy.rule() decorator.

Validating Your Data: Ensuring Quality with Dataframely

Validating your data against the defined schema is straightforward. Dataframely detects and reports any inconsistencies.

import polars as pl

df = pl.DataFrame({
    "zip_code": ["01234", "01234", "1", "213", "123", "213"],
    "num_bedrooms": [2, 2, 1, None, None, 2],
    "num_bathrooms": [1, 2, 1, 1, 0, 8],
    "price": [100_000, 110_000, 50_000, 80_000, 60_000, 160_000]
})

# Validate the data and cast columns to expected types
validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)

The validate method checks the DataFrame against the schema. The cast=True parameter automatically casts column types to match the schema.

Advanced Usage: Custom Rules and Data Transformations

Dataframely's power lies in its custom rule support. Define complex validation logic tailored to your data. Group by operations enable validation across related data points.

Dataframely: A Tool for Robust Data Pipelines

Dataframely simplifies Polars DataFrame validation. It improves data quality, code readability, and pipeline reliability. Integrate it into your workflow for confident data processing.