Screaming Frog SEO Review: A Modern File Analyzer

Editor’s Note: While the US uses the spelling “Analyzer,” the UK uses “Analyser,” and since Screaming Frog Log Analyser is a UK company, the article spells the word as such.


In recent years, log file analysis has made a comeback in technical SEO, as SEOs increasingly realize the value of digging into a website’s server log files.

Log files are something that many people overlook. Back then, when analyzing a website’s performance, we only had log files. However, when JavaScript-based analytics became popular, many people swiftly abandoned log files in favor of the new, flashy reports tools. We didn’t realize how beneficial log files can be for other things.

This long-lost information is being unearthed now. With numerous log file analyst tools appearing, some old-school SEOs never stopped looking at log files.

Screaming Frog joined the fray a few years ago to develop Screaming Frog Log File Analyser, a brother to the well-known Screaming Frog SEO Spider, a mainstay of every SEO toolset.

SEO Log File Analyser Overview

SEO Log File Analyser is a Search Engine Optimization (SEO) tool created by Screaming Frog, a UK-based website search business. You may submit your log files, identify crawled URLs, check search engine bots, and analyze search bot data and behavior using SEO Log File Analyser for essential SEO information. You could delegate those time-consuming tasks to Screaming Frog.

You can view what the search engines have encountered by analyzing your log files directly from the server. The Screaming Frog Log File Analyser gives you a quick overview of important data in a log file and lets you dig deeper.

What is the value of log file analysis for my site’s SEO?

A log file keeps track of all content requests made to your hosting web server. Data from log files is compelling since it shows what occurred when a bot from a search engine visits your website.

You’ll have an immensely powerful way of researching and comprehending search engine crawl behavior for your website after you have your log file data. The information you obtain is superior to that obtained by any other external crawl tool, and the result is improved SEO channel performance. To get the raw log files, it’s as simple as exporting the data and searching for Googlebot requests through user-agent and IP range.

The problem arises when we try to analyze these log files. Due to the difficulty of the files’ format and enormous size, studying log files has always been a pain. SEOs frequently use GREP to collect raw Googlebot log lines and analyze them in a spreadsheet, or they employ log solutions that were not designed with SEO in mind. Although open-source log monitoring software exists, configuring and applying it might not be easy.

The Screaming Frog Log File Analyser was created to simplify the process and provide SEOs with immediate insight into search bot behavior without requiring complicated configuration.


screaming frog

SEO Log File Analyser stood out during our testing for its quick searches and extensive feature set. Aside from studying log files, you can also find crawling URLs, broken links and problems, and find server and client errors, among other things. Here’s the breakdown:

Ease of Use

The SEO Log File Analyser is designed to be simple to use. To begin, you’ll need the server log files for your website. These are usually saved in the /logs/ or /access logs/ folders on the webserver.

After obtaining the files, drag and drop them into the app’s interface; the Log File Analyser will detect the log file format and combine the data into a local database that stores millions of log events. You can drag in multiple logs at once, they can be zipped/gzipped, and they can come from any server, including Apache, ISS, and NGINX.

To begin, you may want to gather log data from the most recent full month and start analyzing from there. You’ll quickly figure out whether this is enough data to work with or if you’ll need more.

Identify URLs Crawled and Examine Its Frequency

Using the search bot user-agent lets you see which URLs have been crawled by search engines and analyze your most often visited pages or directories. Data can be examined, segmented, filtered, sorted, searched, and exported using the Log File Analyser.

To uncover waste and enhance crawl efficiency, you can also examine your site’s most and least scanned URLs and directories. You can even detect orphan or unknown pages or URLs that Googlebot hasn’t crawled by importing a list of URLs and matching them against the log file data.

Locate Slow and Large URLs

The Log File Analyser displays the average number of bytes downloaded and the time taken for search bots for each URL requested, making it simple to spot huge sites or performance concerns. In the UI, you can easily sort the database with this information.

Locate Server and Client Errors

You can rapidly examine the various responses that search engines have received throughout time. The Log File Analyser divides response codes into groups: (1XX, 2XX, 3XX, 4XX, 5XX). The division will aid you in the investigation of response inconsistencies and provide response code filters. You can opt to see URLs with a 5XX response code from all events throughout the period, or only the most recent response code, for example.

Import a Crawl or Other Data

screaming frog

You can also import crawl data into the Log File Analyser using the ‘Imported URL Data’ tab. Import any data with a ‘URLs’ column and compare it to log file data. You can even input crawls, directives, or external link data for a more comprehensive analysis.

Other Features

Other capabilities of SEO Log File Analyser include audit redirects, which you can use to identify temporary and permanent redirects seen by search bots that aren’t found in a browser or simulated crawl.

You may also find out how long each URL takes to respond on average, see which URLs have inconsistent response codes over time, automatically verify search engine bots, and more.

Screaming Frog updates its software regularly, including minor and major patches and new features if the demand arises.

Pricing and Free Tier

Screaming Frog’s SEO Log File Analyser is available in two versions. One is free, while the other costs £99 per year (about $132.5). If you work for a large company and require several licenses so that everyone on your team can use the program, Screaming Frog will give you a discount if you purchase more than five licenses at once. The more you buy, the lower the price of an individual license.

The free version of Screaming Frog’s Log File Analyser allows you to import up to 1k log events at a time. With the free version, all of the features and functionalities are available to you, except for the unlocked log event limit and project limit, as well as technical support. With the free tier, you can only have one project at a time with no technical assistance.

This limitation should not deter you from using the free edition of SEO Log File Analyser unless you routinely study log events of a large website with hundreds of distinct subdomains and pages or if you work on many projects.

User Interface

At first sight, The UI design of Screaming Frog’s SEO Log File Analyser appears to be outdated for today’s standards. Nonetheless, it maintains a tidy appearance and is simple to navigate.

Users who have never used an SEO analysis tool before may find the interface a little intimidating. Despite that, everything is well-organized, and even beginners should be able to use this software after a few hours of ‘knowing each other.’

The software is also relatively easy to install, and it works on any machine that runs Windows, Ubuntu, or Mac OS. Once you’ve finished installing, you can use the demo log file if you don’t have a log file to experiment at the start. It’s ideal for beginners compared to starting from scratch on a new project without prior experience.

screaming frog

Once you load in the demo file, It’s best to go through all of the tabs, such as “Overview,” “URLs,” “Response Codes,” “User Agents,” and so on, to get a sense of what each one does. If you ever feel unsure about something, you can always select User Guide from the Help menu at the top. Each tab has its own detailed guide. You can also rely on technical help if you have purchased the commercial version.

Should I use SEO Log File Analyser?

Screaming Frog is one of the industry’s “pioneers.” That’s how it became so well-known. It’s also run by a company specializing in SEO consultation as part of its core business. So you may rest assured that they’ll maintain it up to date with all of the industry’s changes.

As for their SEO Log File Analyser, it may not have the most beautiful and user-friendly interface, but it is a great tool for gathering detailed information on your website’s SEO design and performance. Thanks to advanced features like broken links and errors locator and crawled URLs identifier, you’ll likely get your money’s worth with this tidy little tool.

Leave a Reply

Your email address will not be published.

Previous Post

Strategies to Stop Encrypted Hack for Bitcoin

Next Post
seo software tools

8 Best SEO Software or Tools in 2022

Related Posts