Generate SEO-friendly robots.txt with OpenAI and Google Drive

Description

Quick overview This workflow runs manually to check a website’s existing robots.txt, detect the site’s CMS from the homepage HTML using OpenAI, and generate an optimized robots.txt (including the provided sitemap URL) that is saved as a file and uploaded to Google Drive. How it works Starts when you click “Execute workflow” and sets the target website URL and sitemap URL. Requests the site’s existing robots.txt via HTTP and checks whether the response returns status code 200. If robots.txt exists, the workflow stops without making changes. If robots.txt is missing, it fetches the website homepage HTML via HTTP. Sends the HTML to OpenAI to detect the CMS and parses the model output into structured JSON. Uses OpenAI again to generate a production-ready robots.txt tailored to the detected CMS and including the sitemap directive. Converts the generated text into a robots.txt file and uploads it to a chosen folder in Google Drive. Setup Add an OpenAI API credential and ensure the selected chat model is available for both CMS detection and robots.txt generation. Add a Google Drive OAuth2 credential and choose the target Drive and folder for the uploaded robots.txt file. Update the website URL and sitemap URL values to match your site. Ensure the target website is publicly reachable so the HTTP requests can fetch /robots.txt and the homepage HTML. Additional info MY NEW YOUTUBE CHANNEL 👉 Subscribe to my new YouTube channel. Here I'll share videos and Shorts with practical tutorials and FREE templates for n8n. Need help customizing? Contact me for consulting and support or add me on Linkedin. An n8n automation workflow template by Davide Boizza.

Categories

Tags

Capabilities

• 5 nodes
• 2 views