Discover Structured Cloud Compute Data with SpareCores sc-data: A Python Package Tutorial

Are you looking for a way to easily access and standardize cloud compute resource data? The SpareCores sc-data package provides structured data collected by sc-crawler, offering a streamlined way to pull and utilize information about cloud resources. This article will guide you through the features, installation, and usage of this powerful tool, perfect for researchers, developers, and anyone interested in cloud computing economics.

What is SpareCores sc-data?

SpareCores sc-data is a Python package designed to provide structured data on cloud compute resources. It uses sparecore-crawler to gather and standardize data, with the crawler running regularly to update spot prices and cloud resource information. The data is stored in an internal SCD table and a public SQLite snapshot, ensuring you have access to the latest information.

Key Features and Benefits:

• Structured Data: Access well-organized and standardized data on cloud compute resources.
• Regular Updates: Benefit from data updated every 5 minutes for spot prices and hourly for all cloud resources.
• Easy Access: Obtain data via a convenient Python package and SQLite database.
• Open Source: Utilize a tool under the CC-BY-SA-4.0 license, encouraging collaboration and innovation.
• NGI Search Open Call Beneficiary: Supported by organizations committed to open search and data accessibility.

Quick Installation Guide

Getting started with SpareCores sc-data is simple. Here's how to install it:

Install the Stable Version from PyPI:

pip install sparecores-data

This command fetches and installs the stable release directly from the Python Package Index.

Install the Latest Version from GitHub:

pip install "sparecores-data @ git+https://[email protected]/SpareCores/sc-data.git"

Use this command to get the most up-to-date version directly from the GitHub repository.

Simple Usage Example

Accessing the SQLite database file is straightforward. The db object in the sc_data package runs an updater thread to keep the SQLite file current.

from sc_data import db
print(db.path)

This code snippet imports the db object and prints the path to the SQLite database, allowing you to start querying the data immediately. The database updates every 600 seconds by default, but you can change this refresh interval.

Customizing the Database Refresh Interval

You can customize how often the SQLite database updates using either a built-in attribute or an environment variable.

• Built-in Attribute: sc_data_db_refresh_seconds
• Environment Variable: SC_DATA_DB_REFRESH_SECONDS

Setting these allows you to control the frequency of updates to suit your specific needs.

Useful Resources and References

To maximize your use of SpareCores sc-data, here are some helpful links:

• sparecores-crawler documentation: Learn more about the crawler that collects the data.
• Database schemas: Understand the structure of the database.
• Latest SQLite database release: Download the most recent SQLite database snapshot.
• sparecores.com: Explore the SpareCores website for more information.

Why Choose SpareCores sc-data?

Choosing SpareCores sc-data provides you with:

• Reliable Data: Regularly updated information ensures you're working with current data.
• Ease of Use: Simple installation and access make it easy to integrate into your projects.
• Community Support: Benefit from an open-source project with available documentation and resources.
• Comprehensive Coverage: Access a wide range of cloud compute resources and spot pricing data.

By using SpareCores sc-data, you gain efficient access to structured cloud compute data, empowering you to make informed decisions and drive innovation in your projects. Start exploring today and unlock the potential of cloud resource data!