Screaming Frog SEO Spider

Screaming Frog SEO Spider is a desktop-based website crawler for technical SEO auditing and analysis. Developed by Screaming Frog Ltd, a search marketing agency founded in 2010 by Daniel Pickard and based in Henley-on-Thames, United Kingdom. The SEO Spider is the industry-standard tool used by SEO professionals at over 35,000 companies including Apple, Amazon, Disney, and Google. Available in two editions: free (crawl up to 500 URLs) and licensed (unlimited crawling with additional features). Key features: website crawling: systematically crawl websites of any size, following internal links to discover all URLs. Configurable crawl depth, speed, and scope. URL analysis: analyzes each URL for status codes, indexability, page titles, meta descriptions, meta keywords, headings (H1-H6), image ALT text, canonical tags, and hreflang attributes. Identifies missing, duplicate, long, or short elements. Redirect chains: identifies permanent (301) and temporary (302) redirects, redirect chains, and loops. Broken links: finds broken internal links (404 errors) and broken external links pointing to other domains. XML Sitemap generation: automatically generates valid XML sitemaps from crawl data. Custom extraction: extract data from web pages using XPath, CSS Path, or regex selectors. Perfect for scraping structured data. Integration with Google Analytics, Search Console, and PageSpeed Insights for combined data. Duplicate content detection: identifies duplicate page titles, meta descriptions, and content. Response time analysis: measures page load times to identify slow pages. Robots.txt and directives: respects and reports on robots.txt directives, meta robots, and X-Robots-Tag headers. Structured data (schema markup) parsing. Visualization: crawl diagrams and directory tree views. Java. Proprietary with free tier.