Deepcrawl’s website intelligence platform introduces ultra-fast crawling and expanded capabilities
- Deepcrawl’s enterprise-scale SaaS platform for technical SEO and website intelligence leverages best-in-class serverless architecture design in its latest crawler to drastically enhance website crawling speeds
- Roll-out of the enhanced crawler is underway for new and existing Deepcrawl users
Deepcrawl, an enterprise-scale website intelligence and technical SEO platform, has launched a new, enhanced version of its market-leading website crawler. The latest crawler developments will introduce ultra-fast crawl speeds to the Deepcrawl platform, breaking earlier records with speeds of up to 450 URLs per second for non-rendered pages and 350 URLs per second for JavaScript-rendered pages.
These crawl speeds were reached in field testing on enterprise-scale websites and offer a major time-saving boon for teams working on large sites that frequently house hundreds of thousands, if not millions, of pages. The enhanced crawler allows for unprecedented scalability on a platform that already serves many of the world’s largest brands, opening the door to new efficiencies in website analysis. Roll-out of the new crawler has begun, with all users expected to have access to the enhanced platform by September 2022.
Crawling software is a foundational aspect of SEO and website intelligence platforms. It traverses a website’s pages to collate the raw data required for sophisticated website analytics and serves as the first step in understanding and optimizing a website’s technical health and search performance. Earlier crawling technology could take several days to finish crawling an enterprise-size website – by comparison, Deepcrawl’s enhanced crawler can crawl a million pages in just one hour.
Increased flexibility in data collection is also built into this latest crawler release. The information that can be captured in website crawls has been expanded beyond traditional technical SEO metrics, paving the way for new built-in metrics and reporting options within the Deepcrawl platform. Presently, users have access to 250+ built-in reports as well as the ability to design custom extractions that collect data on any specified website element. The added flexibility in data collection allows for new use cases to be developed via Deepcrawl’s crawler, with a recently released GraphQL API providing an easy implementation for developers.
Deepcrawl’s latest advancements in crawling technology allow for tailor-made website intelligence at unparalleled speeds. The enhanced crawler makes it easier than ever for businesses to make effective decisions about their online presence”
– Ali Habibzadeh, Chief Technology Officer at Deepcrawl.
