Validate Polars DataFrames with Dataframely: Ensure Data Quality and Robust Pipelines

Dataframely is a Python package expertly designed to validate the schema and content of your Polars DataFrames. Are you tired of unexpected data types or missing values crashing your data pipelines? Dataframely ensures your data meets your expectations, leading to more reliable and readable code. Keep reading to learn how you can start using Dataframely for Polars today!

Why Use Dataframely for Polars Data Validation?

Dataframely offers crucial benefits to data scientists and engineers. It improves the robustness and maintainability of your data workflows:

• Schema Validation: Define schemas upfront, ensuring data adheres to specific types and constraints.
• Improved Readability: Add schema information to DataFrame type hints for enhanced code clarity.
• Robust Data Pipelines: Catch data inconsistencies early, preventing errors downstream.
• Custom Validation Rules: Implement custom rules tailored to your specific data requirements.

Easy Installation: Get Started in Minutes

Installing Dataframely is a breeze using your favorite Python package manager. Choose your preferred method:

pixi add dataframely

or

pip install dataframely

With just one line of code, you're ready to start using Dataframely with Polars to validate your data.

Defining a DataFrame Schema: Set Your Data's Expectations

Dataframely uses a declarative approach, allowing you to define your schema in a clear and concise way. Here's an example defining a schema for housing data:

import dataframely as dy
import polars as pl

class HouseSchema(dy.Schema):
    zip_code = dy.String(nullable=False, min_length=3)
    num_bedrooms = dy.UInt8(nullable=False)
    num_bathrooms = dy.UInt8(nullable=False)
    price = dy.Float64(nullable=False)

    @dy.rule()
    def reasonable_bathroom_to_bedrooom_ratio() -> pl.Expr:
        ratio = pl.col("num_bathrooms") / pl.col("num_bedrooms")
        return (ratio >= 1 / 3) & (ratio <= 3)

    @dy.rule(group_by=["zip_code"])
    def minimum_zip_code_count() -> pl.Expr:
        return pl.len() >= 2

This example defines the data types, nullability, and even custom validation rules such as a reasonable bathroom-to-bedroom ratio and a minimum count per zip code.

Validating Your Data: Ensuring Quality and Consistency

Once your schema is defined, validating your data is simple. Dataframely will automatically check if the dataframe matches the schema you've defined.

import polars as pl

df = pl.DataFrame({
    "zip_code": ["01234", "01234", "1", "213", "123", "213"],
    "num_bedrooms": [2, 2, 1, None, None, 2],
    "num_bathrooms": [1, 2, 1, 1, 0, 8],
    "price": [100_000, 110_000, 50_000, 80_000, 60_000, 160_000]
})

# Validate the data and cast columns to expected types
validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)

The validate method checks your DataFrame against the defined schema. The cast=True argument automatically casts columns to the expected data types. By using Dataframely with Polars you can automatically flag any issues from incorrect data types or missing values.

Benefits of Using Dataframely in Data Pipelines

Implementing Dataframely for Polars data pipelines offers significant advantages:

• Early Error Detection: Find and fix data quality issues early in the process.
• Reduced Debugging Time: Clear schema definitions make it easier to understand and debug data-related issues.
• Improved Data Governance: Enforce data quality standards across your organization.
• More Reliable Insights: Ensure your analysis is based on clean, validated data.

Explore Advanced Usage and Documentation

Ready to take Dataframely to the next level? Consult the official Dataframely documentation for in-depth examples and advanced features. Start building more robust and reliable data pipelines today.