Unlock the Power of LLMs: Effortlessly Create Fine-Tuning Datasets with Easy Dataset

Tired of wrestling with complex data pipelines to fine-tune your Large Language Models (LLMs)? Easy Dataset is your all-in-one solution for streamlined dataset creation, designed to make LLM fine-tuning accessible and efficient. Get ready to transform your domain expertise into high-quality training data in minutes.

Why Choose Easy Dataset for LLM Fine-Tuning?

• Boost Model Accuracy: Fine-tune your LLMs with domain-specific data, resulting in significantly improved accuracy and relevance.
• Save Time and Resources: Say goodbye to manual data wrangling. Easy Dataset automates key steps, freeing up your time for model optimization.
• User-Friendly Interface: Whether you're a seasoned developer or a domain expert, Easy Dataset's intuitive UI makes dataset creation a breeze.

Transform Your Documents into LLM Training Gold

Easy Dataset simplifies the process of creating effective fine-tuning datasets. Here's how:

• Intelligent Document Processing: Upload your Markdown files and watch as Easy Dataset expertly splits them into meaningful segments, ready for question generation.
• Smart Question Generation: Automatically extract relevant questions from each text segment, accelerating the creation of question-answer pairs.
• Answer Generation: Leverage LLM APIs to generate comprehensive answers for each question, ensuring high-quality training data.

Key Features That Will Revolutionize Your LLM Fine-Tuning

Easy Dataset is packed with features that empower you to create the perfect dataset:

• Intelligent Document Processing: Upload Markdown files and automatically split them into meaningful segments. No more tedious manual splitting!
• Smart Question Generation: Extracts relevant questions from each text segment, saving you hours of brainstorming.
• Answer Generation: Generates comprehensive answers for each question using LLM APIs, ensuring high-quality training data.
• Multiple Export Formats: Export your datasets in various formats (Alpaca, ShareGPT) and file types (JSON, JSONL) for seamless integration with your LLM pipeline. Easy Dataset provides the flexibility you need.
• Wide Model Support: Compatible with all LLM APIs that follow the OpenAI format, giving you the freedom to choose the best model for your needs.
• Customizable System Prompts: Fine-tune the behavior of your LLM by adding custom system prompts.

Getting Started with Easy Dataset

Ready to unlock the potential of Easy Dataset? Here's how to get started:

1. Clone the Repository:

   git clone https://github.com/ConardLi/easy-dataset.git
   cd easy-dataset
   

   Copy
2. Install Dependencies:

   npm install
   

   Copy
3. Start the Development Server:

   npm run build
   npm run start
   

   Copy
4. Open your browser and navigate to http://localhost:1717

Alternatively, you can build and run the application using Docker:

1. Clone the Repository:

   git clone https://github.com/ConardLi/easy-dataset.git
   cd easy-dataset
   

   Copy
2. Build the Docker image:

   docker build -t easy-dataset .
   

   Copy
3. Run the container:

   docker run -d -p 1717:1717 -v {YOUR_LOCAL_DB_PATH}:/app/local-db --name easy-dataset easy-dataset
   

   Copy

Note: Replace {YOUR_LOCAL_DB_PATH} with the actual path where you want to store the local database.

Creating Your First Dataset with Easy Dataset

Follow these simple steps to create your first fine-tuning dataset:

1. Create a Project: Click the "Create Project" button, enter a name & description, and configure your LLM API settings.
2. Process Documents: Upload your Markdown files in the "Text Split" section and review/adjust the automatically split segments.
3. Generate Questions: Navigate to the "Questions" section, select text segments, and review/edit the generated questions, organizing them with tags.
4. Create Datasets: Go to the "Datasets" section, select questions, generate answers using your configured LLM, and review/edit the results.
5. Export Datasets: Click the "Export" button, choose your preferred format (Alpaca or ShareGPT) and file type (JSON or JSONL), and add custom system prompts if needed.

Unleash the Potential of Your LLMs Today

Easy Dataset empowers you to create high-quality datasets for LLM fine-tuning with ease. Whether you're working with niche topics or broad concepts, leveraging a specialized tool for dataset generation will allow for efficiency gains from end-to-end. Start building better models and unlock the true potential of your LLMs today!