Build Your Own Browser (BYOB) Tool for Web Browsing: Get Up-to-Date Info with AI

Large Language Models (LLMs) are powerful, but they have a knowledge cutoff date. This means they lack current information, impacting their ability to provide accurate and relevant responses.

Want to equip your LLM with the latest web data for up-to-date answers? This article shows you how to build a Bring Your Own Browser (BYOB) tool in Python. You'll learn how to integrate web search capabilities with an LLM, enabling it to generate responses based on the freshest information available online, like recent OpenAI developments.

Why Build a Bring Your Own Browser (BYOB) Tool?

• Overcome Knowledge Cutoffs: Access real-time information beyond your LLM's training data.
• Provide Up-to-Date Answers: Ensure accurate and relevant responses based on the latest web content.
• Automate Web Browsing: Programmatically perform web searches and data extraction tasks.

What Will You Learn to Build?

This guide walks you through creating a BYOB tool that:

1. Sets Up a Search Engine: Uses Google's Custom Search API to perform web searches.
2. Builds a Search Dictionary: Collects titles, URLs, and summaries of web pages into a structured dictionary.
3. Generates a RAG Response: Implements Retrieval-Augmented Generation (RAG) by feeding the gathered information to the LLM.

Real-World Example: OpenAI Product Launches

Imagine you want to list recent OpenAI product launches in chronological order. Without current data, your LLM might miss key developments like the o1-preview model. See how a BYOB tool can fill this gap.

Here's the initial query to the model:

search_query ="List the latest OpenAI product launches in chronological order from latest to oldest in the past 2 years"

The unmodified LLM struggles with this task due to its knowledge cutoff. Let's improve this with a "bring your own browser" (BYOB) setup.

Setting Up Your BYOB Tool in 3 Steps to Enhance LLM Accuracy

1. Set Up a Search Engine: Fetch web search results using Google's Custom Search API.
2. Build a Search Dictionary: Create a data structure with titles, URLs, and webpage summaries.
3. Generate a RAG Response: Pass the information to the LLM to answer the user's query with current data.

Step 1: Configure Your Search Engine to Find Latest Information

Use publicly available web search APIs. This example uses Google's Custom Search API to get updated and relevant product launch information.

a. Get Your API Key: Navigate to the Programmable Search Engine Link to set up an API key and Custom Search Engine ID (CSE ID).

b. The search Function: The function below sets up the search based on search term, the API and CSE ID keys, as well as number of search results to return. We'll introduce a parameter site_filter to restrict the output to only openai.com

import requests # For making HTTP requests to APIs and websites

defsearch(search_item, api_key, cse_id, search_depth=10, site_filter=None):
 service_url ='https://www.googleapis.com/customsearch/v1'

 params = {
'q': search_item,
'key': api_key,
'cx': cse_id,
'num': search_depth
 }

try:
 response = requests.get(service_url, params=params)
 response.raise_for_status()
 results = response.json()

# Check if 'items' exists in the results
if'items'in results:
if site_filter isnotNone:

# Filter results to include only those with site_filter in the link
 filtered_results = [result for result in results['items'] if site_filter in result['link']]

if filtered_results:
return filtered_results
else:
print(f"No results with {site_filter} found.")
return []
else:
if'items'in results:
return results['items']
else:
print("No search results found.")
return []

except requests.exceptions.RequestException as e:
print(f"An error occurred during the search: {e}")
return []

c. Identify Search Terms: Use query expansion to get better API search results. Expand the original search query with related terms, synonyms, or variations. This technique helps search engines match a range of related terms.

search_term = client.chat.completions.create(
model="gpt-4o",
messages=[
 {"role": "system", "content": "Provide a google search term based on search query provided below in 3-4 words"},
 {"role": "user", "content": search_query}]
).choices[0].message.content

print(search_term)

This code transforms the user query into a succinct phrase like "Latest OpenAI product launches".

d. Invoke the Search Function: Retrieve results from the Google API. The output only contains the web page link and a snippet, more information can be extracted in the next step.

from dotenv import load_dotenv
import os

load_dotenv('.env')

api_key = os.getenv('API_KEY')
cse_id = os.getenv('CSE_ID')

search_items = search(search_item=search_term, api_key=api_key, cse_id=cse_id, search_depth=10, site_filter="https://openai.com")

Step 2: Build Your Search Dictionary for Structured Information

Extract information to pass to the LLM for the eventual output.

a. Scrape Web Page Content: Retrieve the web page for each URL from the search results and extract only textual data.

b. Summarize Content: Employ an LLM to generate concise summaries of the scraped content, focusing on key details related to OpenAI product launches. This model is provided with the initial search text to give better summarizations.

c. Create a Structured Dictionary: Organize the data into a dictionary containing each page's title, link, and summary. This structured format facilitates easy information retrieval and enhances the LLM's ability to generate comprehensive summaries with proper citations.

import requests
from bs4 import BeautifulSoup

TRUNCATE_SCRAPED_TEXT=50000# Adjust based on your model's context window
SEARCH_DEPTH=5

defretrieve_content(url, max_tokens=TRUNCATE_SCRAPED_TEXT):
try:
 headers = {'User-Agent': 'Mozilla/5.0'}
 response = requests.get(url, headers=headers, timeout=10)
 response.raise_for_status()

 soup = BeautifulSoup(response.content, 'html.parser')
for script_or_style in soup(['script', 'style']):
 script_or_style.decompose()

 text = soup.get_text(separator=' ', strip=True)
 characters = max_tokens *4# Approximate conversion
 text = text[:characters]
return text
except requests.exceptions.RequestException as e:
print(f"Failed to retrieve {url}: {e}")
returnNone

defsummarize_content(content, search_term, character_limit=500):
 prompt = (
f"You are an AI assistant tasked with summarizing content relevant to '{search_term}'. "
f"Please provide a concise summary in {character_limit} characters or less."
 )
try:
 response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
 {"role": "system", "content": prompt},
 {"role": "user", "content": content}]
 )
 summary = response.choices[0].message.content
return summary
exceptExceptionas e:
print(f"An error occurred during summarization: {e}")
returnNone

defget_search_results(search_items, character_limit=500):
# Generate a summary of search results for the given search term
 results_list = []
for idx, item inenumerate(search_items, start=1):
 url = item.get('link')

 snippet = item.get('snippet', '')
 web_content = retrieve_content(url, TRUNCATE_SCRAPED_TEXT)

if web_content isNone:
print(f"Error: skipped URL: {url}")
else:
 summary = summarize_content(web_content, search_term, character_limit)
 result_dict = {
'order': idx,
'link': url,
'title': snippet,
'Summary': summary
 }
 results_list.append(result_dict)
return results_list

Give Your LLM the Power of Web Access

By building a Bring Your Own Browser (BYOB) tool, you bridge the gap between your LLM's knowledge cutoff and the ever-evolving world of online information. With the capability to process current data from the web, your models are set to deliver more insightful, accurate, and relevant responses. Start building today and witness the difference!