URL Scanner API
URL scanner API allows you to scan any URL for phishing activity, malware infections and other suspicious files. These tools use a variety of different methods to examine the web link against domain risk history information, checking for buried malicious code that could be downloaded onto your device and start infecting it with a range of viruses and worms.
Some of the major tools include VirusTotal, which uses aggregated and comprehensive data to validate links against a database of known phishing and malware threats. Others such as Cyren URL classification cloud process over 1 billion URLs each day to categorize them into 84 categories, including 13 security categories for web threats like phishing and malware.
A tool such as Cyren also offers APIs that allow you to submit new locations for scanning and search existing scans by domain names, IP addresses or Autonomous System Numbers (AS) numbers. These tools work with a variety of other third-party applications to automatically detect and block suspicious URLs, reducing the number of potentially dangerous downloads and helping protect your devices.
The GitHub pages phishing scanner API was exposed earlier this month, allowing users to access sensitive data that can be used by attackers. This is a serious issue because it exposes a large amount of confidential data from GitHub Pages websites that are meant to be private and secure. While GitHub has been working to correct this vulnerability, the damage has already been done. In a blog post published on November 2, Positive Security warns users about this type of URL scanner API integration and how it can reveal an unprotected wealth of sensitive information in the form of metadata such as email, passwords, user IDs and account settings.
You can import a RESTful API web service definition file via the Links/API Definitions tab in Invicti to make the scanner aware of the API routes and parameters that it should consider scanning during the crawling stage. The definition file can be imported from a file or directly from a URL.
The URL Scanner API also provides a way for analysts in security operations centers to quickly understand the footprint of a potential threat by obtaining scan results, screenshots and other referred information on an indicator such as an IP address or a domain. This can help them to triage the information that they are investigating much faster.
When you are submitting new locations for scans via the API, it is a good idea to limit concurrent requests and use exponential backoffs for all types of requests. This will help ensure that you do not exceed your rate-limit, which may result in your request being ignored. The overallState field in the reports shows you how many of your available rate windows remain, or how long it will take for all remaining requests to be processed.
You can see the status of a request in progress by looking at the Request Status field in the reports, and in the API response. The status will tell you whether the endpoint is still processing your request or if the request was skipped because it exceeded the allowed rate. You can also find out why the endpoint has been skipped by examining the event log.