"Outfit Matchmaker": Combining GPT-4o Mini and RAG for Personalized Clothing Recommendations

Want to create a personalized styling experience? Discover how to expertly combine the power of GPT-4o mini with Retrieval-Augmented Generation (RAG) to build an intelligent clothing recommendation system, and find outfits that complement your style, leveraging the strengths of both generative and retrieval-based AI techniques.

What is the Clothing Matchmaker App?

The Clothing Matchmaker App analyzes images of clothing using GPT-4o mini, extracting crucial features like color, style, and type. Then, leveraging a custom algorithm and the RAG technique, it searches a knowledge base for complementary items, taking into account color compatibility and style coherence.

Key Benefits of Using GPT-4o Mini + RAG for Clothing Recommendations

Why combine these technologies? Here's how they enhance the styling experience:

• Contextual Understanding: GPT-4o mini analyzes images to understand context, deciphering objects, scenes, and activities.
• Rich Knowledge Base: RAG accesses a vast information corpus, providing suggestions based on various knowledge in different fields.
• Customization: The approach customizes suggestions to specific needs or preferences, tailoring recommendations.

Setting Up Your Environment

To get started, ensure you have the necessary libraries installed:

pip install openai -- quiet
pip install tenacity -- quiet
pip install tqdm -- quiet
pip install numpy -- quiet
pip install typing -- quiet
pip install tiktoken -- quiet
pip install concurrent -- quiet

Then, import the libraries that will be used to build the application to find the desired similar clothing items:

import pandas as pd
import numpy as np
import json
import ast
import tiktoken
import concurrent
from openai import OpenAI
from tqdm import tqdm
from tenacity import retry, wait_random_exponential, stop_after_attempt
from IPython.display import Image, display, HTML
from typing import List

client = OpenAI()

GPT_MODEL = "gpt-4o-mini"
EMBEDDING_MODEL = "text-embedding-3-large"
EMBEDDING_COST_PER_1K_TOKENS = 0.00013

Creating Embeddings for Your Clothing Data

To build a robust knowledge base, generate embeddings for your clothing dataset. Here's how to prepare it:

1. Choose a Database: Either use a CSV file or opt for an ready-to-use vector database.

2. Load Your Data: Load a .csv file of clothing items and their attributes into a Pandas DataFrame. Each row should represent a clothing item, with columns like productDisplayName, articleType, gender, etc.

   styles_filepath = "data/sample_clothes/sample_styles.csv"
   styles_df = pd.read_csv(styles_filepath, on_bad_lines = 'skip')
   print (styles_df.head())
   print ( "Opened dataset successfully. Dataset has {} items of clothing.".format( len (styles_df)))
   

   Copy

3. Generate Embeddings: Use OpenAI's embedding models to convert text descriptions of clothing items into numerical vectors, capturing semantic information.

   • Batch Embedding Logic: Turn text input into batches and then into a list of embeddings.
   • Parallel Processing: Improve the speed for creating embeddings for larger datasets.

   # Simple function to take in a list of text objects and return them as a list of embeddings
   @retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(10))
   def get_embeddings(input: List):
       response = client.embeddings.create(
           input=input,
           model=EMBEDDING_MODEL
       ).data
       return [data.embedding for data in response]
   
   # Splits an iterable into batches of size n.
   def batchify(iterable, n=1):
       l = len(iterable)
       for ndx in range(0, l, n):
           yield iterable[ndx:min(ndx + n, l)]
   
   # Function for batching and parallel processing the embeddings
   def embed_corpus(
       corpus: List[str],
       batch_size=64,
       num_workers=8,
       max_context_len=8191,
   ):
       # Encode the corpus, truncating to max_context_len
       encoding = tiktoken.get_encoding("cl100k_base")
       encoded_corpus = [
           encoded_article[:max_context_len] for encoded_article in encoding.encode_batch(corpus)
       ]
   
       # Calculate corpus statistics: the number of inputs, the total number of tokens, and the estimated cost to embed
       num_tokens = sum(len(article) for article in encoded_corpus)
       cost_to_embed_tokens = num_tokens / 1000 * EMBEDDING_COST_PER_1K_TOKENS
       print(
           f"num_articles={len(encoded_corpus)}, num_tokens={num_tokens}, est_embedding_cost={cost_to_embed_tokens:.2f} USD"
       )
   
       # Embed the corpus
       with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
   
           futures = [
               executor.submit(get_embeddings, text_batch)
               for text_batch in batchify(encoded_corpus, batch_size)
           ]
   
           with tqdm(total=len(encoded_corpus)) as pbar:
               for _ in concurrent.futures.as_completed(futures):
                   pbar.update(batch_size)
   
           embeddings = []
           for future in futures:
               data = future.result()
               embeddings.extend(data)
   
       return embeddings
   
   # Function to generate embeddings for a given column in a DataFrame
   def generate_embeddings(df, column_name):
       # Initialize an empty list to store embeddings
       descriptions = df[column_name].astype(str).tolist()
       embeddings = embed_corpus(descriptions)
   
       # Add the embeddings as a new column to the DataFrame
       df['embeddings'] = embeddings
       print("Embeddings created successfully.")
   
   generate_embeddings(styles_df, 'productDisplayName')
   print ( "Writing embeddings to file ...")
   styles_df.to_csv( 'data/sample_clothes/sample_styles_with_embeddings.csv', index = False)
   print ( "Embeddings successfully stored in sample_styles_with_embeddings.csv")
   

   Copy

Crafting a Matching Algorithm for Clothing Items

Create a custom cosine similarity retrieval algorithm. This will find similar items within your dataset.

1. Cosine Similarity Function: Create a function to calculate the cosine similarity between two vectors.

   def cosine_similarity_manual(vec1, vec2):
       """Calculate the cosine similarity between two vectors."""
       vec1 = np.array(vec1, dtype=float)
       vec2 = np.array(vec2, dtype=float)
   
       dot_product = np.dot(vec1, vec2)
       norm_vec1 = np.linalg.norm(vec1)
       norm_vec2 = np.linalg.norm(vec2)
       return dot_product / (norm_vec1 * norm_vec2)
   

   Copy

2. Find Similar Items: Develop a function to find the most similar items based on cosine similarity.

   def find_similar_items(input_embedding, embeddings, threshold=0.5, top_k=2):
       """Find the most similar items based on cosine similarity."""
   
       # Calculate cosine similarity between the input embedding and all other embeddings
       similarities = [(index, cosine_similarity_manual(input_embedding, vec)) for index, vec in enumerate(embeddings)]
   
       # Filter out any similarities below the threshold
       filtered_similarities = [(index, sim) for index, sim in similarities if sim >= threshold]
   
       # Sort the filtered similarities by similarity score
       sorted_indices = sorted(filtered_similarities, key=lambda x: x[1], reverse=True)[:top_k]
   
       # Return the top-k most similar items
       return sorted_indices
   
   def find_matching_items_with_rag(df_items, item_descs):
       """Take the input item descriptions and find the most similar items based on cosine similarity for each description."""
   
       # Select the embeddings from the DataFrame.
       embeddings = df_items['embeddings'].tolist()
   
       similar_items = []
       for desc in item_descs:
   
           # Generate the embedding for the input item
           input_embedding = get_embeddings([desc])
   
           # Find the most similar items based on cosine similarity
           similar_indices = find_similar_items(input_embedding, embeddings, threshold=0.6)
           similar_items += [df_items.iloc[i] for i in similar_indices]
   
       return similar_items
   

   Copy

Image Analysis Module Leveraging GPT-4o Mini

Use GPT-4o mini to analyze input images and extract important features

1. API Call: Analyze images using an API call. Provide the image URL for analysis and request identification of relevant features.

2. Specific techniques: Set output format, clear and concise instructions, and a one-shot example.

3. Expected Model Output:

   • items (str[]) - A list of concise clothing titles, including the item's style, color, and gender.
   • category (str) - The most representative article type for the analyzed item.
   • gender (str) - The gender intended for the item. Options: [Men, Women, Boys, Girls, Unisex].

   def analyze_image(image_base64, subcategories):
       response = client.chat.completions.create(
           model=GPT_MODEL,
           messages=[
               {
                   "role": "user",
                   "content": [
                       {
                           "type": "text",
                           "text": """Given an image of an item of clothing, analyze the item and generate a JSON output with the following fields: "items", "category", and "gender".
                           Use your understanding of fashion trends, styles, and gender preferences to provide accurate and relevant suggestions for how to complete the outfit.
                           The items field should be a list of items that would go well with the item in the picture. Each item should represent a title of an item of clothing that contains the style, color, and gender of the item.
                           The category needs to be chosen between the types in this list: {subcategories}.
                           You have to choose between the genders in this list: [Men, Women, Boys, Girls, Unisex]
                           Do not include the description of the item in the picture. Do not include the ```json ``` tag in the output.
   
                           Example Input: An image representing a black leather jacket.
   
                           Example Output: {"items": ["Fitted White Women's T-shirt", "White Canvas Sneakers", "Women's Black Skinny Jeans"], "category": "Jackets", "gender": "Women"}
                           """,
                       },
                       {
                           "type": "image_url",
                           "image_url": {
                               "url": f"data:image/jpeg;base64, {image_base64}",
                           },
                       }
                   ],
               }
           ]
       )
       # Extract relevant features from the response
       features = response.choices[0].message.content
       return features
   

   Copy

Testing Your Prompt with Images

Now, test the setup using a few sample images:

1. Encode Images: Encode the test images into base64 format.

   import base64
   
   def encode_image_to_base64(image_path):
       with open(image_path, 'rb') as image_file:
           encoded_image = base64.b64encode(image_file.read())
       return encoded_image.decode('utf-8')
   
   # Set the path to the images and select a test image
   image_path = "data/sample_clothes/sample_images/"
   test_images = ["2133.jpg", "7143.jpg", "4226.jpg"]
   
   # Encode the test image to base64
   reference_image = image_path + test_images[0]
   encoded_image = encode_image_to_base64(reference_image)
   
   # Select the unique subcategories from the DataFrame
   unique_subcategories = styles_df['articleType'].unique()
   
   # Analyze the image and return the results
   analysis = analyze_image(encoded_image, unique_subcategories)
   image_analysis = json.loads(analysis)
   
   # Display the image and the analysis results
   display(Image(filename=reference_image))
   print(image_analysis)
   

   Copy