How to Create an XML Sitemap Using Screaming Frog

How to Create an XML Sitemap Using Screaming Frog

Anna Kowalska

There are plenty of reasons to create your own custom XML sitemap, not least to improve SEO. By formatting the full list of your website URLs in an easy to understand layout for the search engine bots, you’ll have the best chance of crawling up the rankings.

A selection of free and paid tools allow you to create your own XML Sitemap, but you needn’t look further than Screaming Frog. For both the large and small business, it provides and effective and affordable way to realise the SEO benefits from a proper formatted XML Sitemap. What’s more, it’s easy to configure, even for the novice webmaster. Follow this step by step approach and create your own Sitemap without the help of a developer.

Step 1

Firstly, you’ll want to crawl the site which needs the sitemap. This can be done by hitting the ‘Start’ button on the top admin panel of the Screaming Fog homepage. Once this process has completed, go to ‘Sitemaps’ followed by ‘Created XML Sitemap’ found in the Screaming Fog menu.

A selection of tabs will now be shown which allows you to modify the settings for the XML sitemap. Although each individual case is different, the following configuration is recommended for most.

Step 2

Under the ‘Pages’ tab, be sure to uncheck everything aside from ‘Include PDFs’. Everything else is a waste of time as there’s little need to include pages which already instruct Google not to index them. Best not to confuse the search engines with mixed signals.

Step 3

Under the tab ‘Last Modified’ you’ll have the option to include a tag on every URL instructing search engines of when a recent update was made. Given the current emphasis on updated content, this feature is particularly useful. Opt for the option to ‘use server response’ as this will avoid you having to input the time on every occasion that a page is updated. Instead, any modifications are auto-updated.

Step 4

The next tab is ‘Priority’ which as the name would suggest, enables search engines to understand which pages have the most importance. Each page is assigned a number from 0.1 to 1 – 1 being the top priority which would usually be the home page.

From here, the major category pages usually receive a 0.9 identification and smaller category pages a 0.8. Fortunately, Screaming Frog takes us all the stress out of identifying pages as it does it automatically during the crawling process. Of course, these can be manually changed at your discretion.

Step 5

The ‘Change Frequency’ configuration isn’t essential for smaller sites and can often be overlooked without any adverse effect. However, for those looking for an added boost, this optional tag can improve Google’s understanding of how frequently content on your site is updated. By using content tree levels, you can indicate the frequency of how often pages are updated.

Step 6

Lastly, the ‘Images’ tab allows you to include the URLs of images on your website within your main XML sitemap. For beginners and those looking for the simplest solution, it’s recommend to keep this setting unchecked. There’s always the option to create an image sitemap further down the line.

Step 7

Now you’ve configured the Screaming Frog settings, your last step is to upload the sitemap to your site. To do this, first reference the file’s location in your robots.txt file before uploading to the Google Search Console under ‘Crawl’ then ‘Sitemaps’.

After you’ve uploaded, be sure to use Webmaster Tools to monitor for any potential errors which may require another update.