How We Started Using Python For SEO

It’s coming to the midpoint of 2024, and I finally got round to properly learning and incorporating Python for SEO. For those who are not familiar with Python or its potential for SEO, Python’s flexibility and ease of use is known for streamlining repetitive tasks, and managing large datasets efficiently.

Having recently embraced Python in my SEO toolkit, I’ve quickly recognized its tremendous potential to transform the way we approach search engine optimisation. This powerful programming language has proven invaluable for automating complex SEO workflows and uncovering insights that would typically require cumbersome, manual processes. 

Its early days but the potential unlocked value from Python can not only optimise our SEO operations but also open up new avenues for innovation and effectiveness in our campaigns.

Python Streamlines SEO Processes

In the realm of SEO, where data is king, Python’s ability to streamline complex processes is invaluable. It reduces reliance on manual interventions, speeds up data processing, and facilitates a more agile response to the dynamic nature of search engine algorithms. 

By automating repetitive tasks, Python frees up SEO specialists to focus on strategic decision-making and long-term planning. This shift not only boosts productivity but also enhances the accuracy and effectiveness of SEO campaigns. Python’s capability to integrate with various APIs and online tools further extends its usefulness, allowing for a more cohesive and comprehensive approach to SEO.

Getting Started With Python

Basic Setup and Tools Needed

1. Python Installation: First, you’ll need to install Python. It’s available for free on the official Python website. Download and install Python, ensuring to check the box that says ‘Add Python to PATH’ during installation to simplify future steps.
2. Integrated Development Environment (IDE): While you can use Python directly through your command line, an IDE can make your coding experience much smoother. For beginners, I recommend starting with Google Colab because it doesn’t require any installation and includes helpful features like code suggestions and easy-to-manage notebooks. Alternatively, you might consider installing Visual Studio Code or PyCharm, which are great for more advanced projects.
3. Pip: Python’s package installer, Pip, will likely be installed with Python. This tool is essential for managing additional libraries that you’ll need for SEO tasks, such as requests for web crawling or Pandas for data analysis.

Test With A Good Old “Hello World” Script

To kick off your Python coding journey, let’s write a classic simple “Hello World” script. This is a traditional first step in learning any programming language as it teaches you the basic syntax and gives you a quick win.

Step 1: Open your chosen IDE or a notebook in Google Colab.

Step 2: In a new file or cell, type the following Python code:

print(“Hello, SEO world!”)

Step 3: Run the code. You should see the output Hello, SEO world! displayed. Congratulations, you’ve just run your first Python script!

This simple exercise introduces you to the basic syntax of Python—writing statements, executing code, and seeing immediate results. As we move forward, we’ll dive into more specific scripts that can automate and enhance your SEO efforts, making your workflows not only faster but also more data-driven.

Python Applications for SEO

In this section, we’ll explore a series of practical Python projects tailored specifically for SEO professionals. These projects will help automate routine tasks, provide deep insights through data analysis, and enhance your overall SEO strategy. Each project is designed to be both accessible for beginners and valuable for experienced SEO practitioners, ensuring you can implement them regardless of your skill level.

Application 1: Using Python with Google Indexing API

The Google Indexing API is a powerful tool designed for developers to interact directly with Google’s search index. The primary function of this API is to allow developers to notify Google immediately when web pages are added or updated. This notification helps speed up the indexing process, ensuring that your content becomes searchable more quickly than it would through the normal crawling process. To use the Indexing API, there are several prerequisites:

Google Search Console Verification: You must verify ownership of your website in Google Search Console. This verification is crucial as it confirms that you have the authority to make requests about the website.

API Limitations: Note that this API is intended primarily for websites with job posting or live streaming content. However, it can be used experimentally for other types of content as well.

I have tested this extensively and have incorporated it into my workflow for most SEO campaigns. Thus far, I have found it to get pages indexed much faster as compared to GSC requests or third party indexing tools. The main caveat here being that you would need owner access of the GSC property.

Setting Up

To start using the Google Indexing API with Python, follow these setup steps:

• Visit the Google API Console and create a new project.
• Under the “Library” section, search for the Google Indexing API and enable it for your project.
• In the API Console, go to the “Credentials” section.
• Click on “Create Credentials” and select OAuth client ID.
• Follow the prompts to configure your OAuth consent screen, and select “Web application” as the application type.
• Save the credentials, which will include your client ID and client secret.

Installing Python Libraries

Ensure Python is installed on your system along with pip.

Install the necessary Python libraries by running the following commands in your terminal:

pip install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client

Python Script for Requesting Indexing

Let’s create a Python script to authenticate with the Google API using OAuth 2.0 and request indexing for a URL:

Authentication

from google_auth_oauthlib.flow import InstalledAppFlow

# Define the scope

SCOPES = [‘https://www.googleapis.com/auth/indexing’]

# Create the flow using the client secrets file from the API Console

flow = InstalledAppFlow.from_client_secrets_file(‘client_secrets.json’, SCOPES)

# Run the flow to obtain credentials

credentials = flow.run_local_server(port=0)

Requesting Indexing

from googleapiclient.discovery import build

# Build the service from the credentials

service = build(‘indexing’, ‘v3’, credentials=credentials)

# Define the URL to be indexed

url = ‘https://example.com/new-page’

body = {

    ‘url’: url,

    ‘type’: ‘URL_UPDATED’

}

# Execute the indexing request

response = service.urlNotifications().publish(body=body).execute()

# Print the response to console

print(response)

This script first authenticates with Google, then constructs and sends a request to the Indexing API. It notifies Google of a new or updated URL, which can significantly expedite the indexing process. This script is an excellent example of how Python can be used to interact with Google services, making SEO tasks more efficient and effective.

Application 2: Using Python For Site Speed Analysis

The PageSpeed Insights API is an invaluable tool provided by Google that allows developers to measure the performance of a page for both mobile and desktop devices. It provides insights into how well a page adheres to best practices for speed and offers suggestions for improvements.

Setting Up

Setting up your environment to use the PageSpeed Insights API with Python involves a few key steps:

Visit the Google API Console, create a new project, or select an existing one.

Search for the PageSpeed Insights API, enable it, and then go to the “Credentials” section to create an API key.

Ensure you have Python and pip installed.

Install the necessary libraries for making HTTP requests:

pip install requests

Python Script for Analyzing Site Speed

Create a Python script to fetch and report site speed data for a list of URLs using the PageSpeed Insights API:

Import Libraries:

import requests

import json

Define Function to Fetch Site Speed:

def fetch_site_speed(url, api_key):

    endpoint = f”https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url={url}&key={api_key}”

    response = requests.get(endpoint)

    speed_data = response.json()

    return speed_data

Analyze Multiple URLs:

Create a list of URLs you want to analyse.

Iterate over the list, calling the function for each URL, and print out relevant data such as First Contentful Paint (FCP) and Time to Interactive (TTI).

urls = [‘https://example.com’, ‘https://another-example.com’]

api_key = ‘YOUR_API_KEY_HERE’

for url in urls:

    result = fetch_site_speed(url, api_key)

    print(f”Results for {url}:”)

    print(f”First Contentful Paint: {result[‘lighthouseResult’][‘audits’][‘first-contentful-paint’][‘displayValue’]}”)

    print(f”Time to Interactive: {result[‘lighthouseResult’][‘audits’][‘interactive’][‘displayValue’]}”)

Interpret Results:

After running the script, you’ll receive detailed performance metrics for each URL.

Use these metrics to determine areas of improvement and optimize the pages accordingly.

Application 3: Using Python For Site Speed Analysis

Keyword research is a fundamental SEO task that involves identifying popular words and phrases people enter into search engines. Understanding these keywords helps marketers create content that is likely to attract more traffic from search engines. Automation in keyword research not only speeds up the process but also allows for deeper data analysis.

Setting Up

Decide on which platforms or tools you want to fetch keyword data from. Common choices include Google Keyword Planner, SEMrush, Ahrefs, or even scraping Google Search results.

Note: API access may require authentication and possibly a subscription.

Installing Python Libraries

Make sure Python and pip are installed on your system.

Install libraries that may be needed for API requests, data handling, and web scraping:

pip install requests pandas beautifulsoup4

Python Script for Keyword Research

Here’s how to create a simple Python script to automate keyword research tasks:

Import Libraries:

import requests

import pandas as pd

from bs4 import BeautifulSoup

Define Function to Fetch Keywords from Google Autocomplete:

This example uses Google Autocomplete to fetch keyword suggestions.

def fetch_keywords(base_keyword, user_agent):

    endpoint = f”http://suggestqueries.google.com/complete/search?output=toolbar&hl=en&q={base_keyword}”

    headers = {‘User-Agent’: user_agent}

    response = requests.get(endpoint, headers=headers)

    soup = BeautifulSoup(response.content, ‘xml’)

    suggestions = [suggestion.text for suggestion in soup.find_all(‘suggestion’)]

    return suggestions

Analyze Keywords and Compile Data

Compile keywords into a DataFrame for analysis:

base_keyword = ‘digital marketing’

user_agent = ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3’

keywords = fetch_keywords(base_keyword, user_agent)

# Create a DataFrame

df_keywords = pd.DataFrame(keywords, columns=[‘Keyword’])

print(df_keywords)

Expand Functionality:

To enhance this script, consider integrating API calls to SEO tools like SEMrush or Ahrefs for additional keyword data like search volume, competition, and CPC.

Analyse keyword trends over time or across regions to further refine your keyword strategy.

This project demonstrates a basic approach to automating keyword research using Python. By leveraging APIs and scraping techniques, SEO professionals can gather extensive keyword data efficiently. This allows for more time to analyse trends, refine strategies, and ultimately, produce content that aligns with user search behaviours.

Application 4: Creating Redirect Maps

A redirect map is an essential component of website migrations, where URLs from an old site are mapped to their new locations on a redesigned or restructured site. Properly implemented redirects prevent 404 errors, preserve SEO rankings, and maintain user experience by ensuring that bookmarks and external links continue to function.

Setting Up

Before you begin creating a Python script for redirect maps, ensure you have the necessary setup:

Python Installation:

Verify that Python is installed on your system. If not, download and install it from the official Python website.

Required Python Libraries:

Install Python libraries that will help in reading, writing, and manipulating data:

pip install pandas openpyxl

Python Script for Creating Redirect Maps

Here’s a guide on how to write a Python script that automates the creation of redirect maps:

Import Libraries:

import pandas as pd

Load Old and New URLs:

Assume you have a spreadsheet with two columns: ‘Old URL’ and ‘New URL’, listing the old URLs and their corresponding new URLs.

def load_urls(file_path):

    return pd.read_excel(file_path)

Generate Redirect Map:

This function reads the URLs and prepares a list of redirect rules.

def create_redirect_map(df):

    redirect_map = []

    for index, row in df.iterrows():

        rule = f”Redirect 301 {row[‘Old URL’]} {row[‘New URL’]}”

        redirect_map.append(rule)

    return redirect_map

Save Redirect Map to File:

Write the redirect rules to a text file that can be used in .htaccess or server config.

def save_redirect_map(redirect_map, output_path):

    with open(output_path, ‘w’) as f:

        for rule in redirect_map:

            f.write(rule + ‘\n’)

Putting It All Together:

Load the data, create the redirect map, and save it to a file.

if __name__ == ‘__main__’:

    url_file_path = ‘path_to_your_spreadsheet.xlsx’

    output_file_path = ‘redirect_map.txt’

    urls_df = load_urls(url_file_path)

    redirect_map = create_redirect_map(urls_df)

    save_redirect_map(redirect_map, output_file_path)

    print(“Redirect map created successfully.”)

This script simplifies the process of creating redirect maps by automating the reading of old and new URLs from a spreadsheet, generating appropriate redirect rules, and saving them in a format that can be directly used by web servers. This automation helps ensure a smooth transition during site migrations, protecting SEO integrity and enhancing user experience.

Application 5: Analysing Backlink Profiles

Backlink analysis is crucial for understanding the health and authority of a website’s link profile. It involves examining the sources of inbound links and assessing their quality, relevance, and impact on SEO. Effective backlink analysis helps identify opportunities for improvement and potential risk factors in a website’s SEO strategy.

Setting Up

API Access:

Obtain API keys from SEMrush, Ahrefs, or any other SEO tool that provides backlink data. This usually requires a paid subscription.

Familiarise yourself with the API documentation to understand the endpoints available for backlink data.

Installing Python Libraries:

Ensure Python is installed along with pip.

Install the requests library to make HTTP requests to the APIs:

pip install requests

Python Script for Fetching and Analysing Backlink Data

Here’s how to write a Python script that fetches and analyses backlink data:

Import Libraries:

import requests

import json

Define Function to Fetch Backlink Data:

This function makes an API call to fetch backlink data for a specified URL.

def fetch_backlink_data(url, api_key):

    api_url = f”https://api.semrush.com/?type=backlinks_overview&key={api_key}&target={url}”

    response = requests.get(api_url)

    return response.json()

Analyze Backlink Data:

Extract and analyze specific metrics such as the number of backlinks, referring domains, and the geographic distribution of backlinks.

def analyze_backlinks(data):

    print(“Total Backlinks:”, data[‘total_backlinks’])

    print(“Referring Domains:”, data[‘referring_domains’])

    print(“Top Countries by Backlinks:”)

    for country, count in data[‘top_countries’].items():

        print(f”{country}: {count}”)

Putting It All Together:

Use the functions to fetch data for a URL and display the analysis.

if __name__ == ‘__main__’:

    website_url = ‘https://example.com’

    api_key = ‘YOUR_SEMRUSH_API_KEY’

    backlink_data = fetch_backlink_data(website_url, api_key)

    analyze_backlinks(backlink_data)

This script allows SEO professionals to automate the retrieval and preliminary analysis of backlink profiles using popular SEO tools’ APIs. By integrating these capabilities into their workflow, they can more efficiently monitor and refine their backlink strategies, ultimately enhancing their website’s SEO performance.

Application 6: Automated Meta Tag Checker

Meta tags play a crucial role in SEO as they provide search engines with metadata about the content of a page. Properly configured meta tags can enhance visibility in search engine results, improve click-through rates, and are essential for effective SEO management.

Setting Up

Python Installation:

Confirm that Python is installed on your system. If not, install it from the official Python website.

Required Python Libraries:

Install Python libraries that are essential for web scraping and parsing HTML:

pip install requests beautifulsoup4

Python Script for Checking Meta Tags

Import Libraries:

import requests

from bs4 import BeautifulSoup

Define Function to Fetch and Parse a Web Page:

This function retrieves the HTML content of a web page and parses it to find meta tags.

def check_meta_tags(url):

    response = requests.get(url)

    if response.status_code == 200:

        soup = BeautifulSoup(response.text, ‘html.parser’)

        title_tag = soup.find(‘title’)

        description_tag = soup.find(‘meta’, attrs={‘name’: ‘description’})

        return {

            ‘title’: title_tag.text if title_tag else “No title tag found”,

            ‘description’: description_tag[‘content’] if description_tag else “No description meta tag found”

        }

    else:

        return {“error”: f”Failed to retrieve page. Status code: {response.status_code}”}

Checking Multiple URLs:

You can extend this function to handle a list of URLs and report the status of meta tags for each.

urls = [‘https://example.com’, ‘https://another-example.com’]

for url in urls:

    result = check_meta_tags(url)

    print(f”Results for {url}:”)

    print(f”Title: {result.get(‘title’)}”)

    print(f”Description: {result.get(‘description’)}”)

This script provides a straightforward and effective way to automate the process of checking meta tags across multiple web pages. By implementing this tool, SEO professionals can easily monitor and ensure that key SEO elements are in place, which is essential for maintaining and improving search engine visibility.

Enhancing SEO Skills through Python Training

For those looking to advance their Python skills, the AI100 course at HeiCoders Academy is an excellent resource. This course focuses on Python programming and data visualisation, taught by industry practitioners from renowned companies such as DBS, IBM, Carousell, Spotify, and Google. Here’s why this beginner python course is beneficial for SEO professionals:

Industry-Relevant Curriculum:

The course covers practical programming skills and advanced data handling techniques, which can be directly applied to SEO tasks such as analysing backlink profiles or optimising site structure based on user data.

Taught by Experts:

Learning from practitioners who have experience at top companies ensures that the teaching is grounded in real-world applications, providing insights that go beyond theoretical knowledge.

Networking Opportunities:

Interacting with professionals from various industries can provide SEOs with networking opportunities and potential collaborations, which can be beneficial for career growth.

Enhancement of Analytical Skills:

The course emphasises data visualisation, a key skill in interpreting SEO data effectively to make strategic decisions. Visualisations help in presenting data in meetings and reports, making complex data accessible to non-technical stakeholders.

Final Thoughts

As the landscape of SEO continues to evolve, staying ahead requires not only keeping up with current trends but also continuously seeking out and integrating new technologies and methodologies. First Page, a leading SEO agency, exemplifies this proactive approach by constantly exploring innovative techniques to enhance the effectiveness of client campaigns.

Our commitment to innovation is mirrored in our dedication to leveraging the power of tools like Python, which can significantly optimise SEO processes from data analysis to automated task execution. This commitment ensures that our clients always benefit from cutting-edge strategies that drive substantial, measurable results in their digital marketing efforts.