在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:breenmachine/httpscreenshot开源软件地址:https://github.com/breenmachine/httpscreenshot开源编程语言:Python 88.7%开源软件介绍:httpscreenshotInstallation via Docker
Installation on UbuntuVia ScriptRun This script has been tested on Ubuntu 20.04 as root (sudo). Manually
If you run into: 'module' object has no attribute 'PhantomJS' then If installing on Kali Linux, PhantomJS might not be in the repositories, you can download from https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2 and symlink to
README and Use CasesHTTPScreenshot is a tool for grabbing screenshots and HTML of large numbers of websites. The goal is for it to be both thorough and fast which can sometimes oppose each other. Before getting into documentation - this is what I USUALLY use for options if I want to screenshot a bunch of sites:
Notice there are a ton of worker threads (40). This can be problematic, I make up for failures that could have been a result of too many threads with a second run:
YMMV The options are as follows: -h, --help show this help message and exit -l LIST, --list LIST List of input URLs -i INPUT, --input INPUT nmap gnmap output file -p, --headless Run in headless mode (using phantomjs) -w WORKERS, --workers WORKERS number of threads -t TIMEOUT, --timeout TIMEOUT time to wait for pageload before killing the browser -v, --verbose turn on verbose debugging -a, --autodetect Automatically detect if listening services are HTTP or HTTPS. Ignores NMAP service detction and URL schemes. -vH, --vhosts Attempt to scrape hostnames from SSL certificates and add these to the URL queue -dB DNS_BRUTE, --dns_brute DNS_BRUTE Specify a DNS subdomain wordlist for bruteforcing on wildcard SSL certs -r RETRIES, --retries RETRIES Number of retries if a URL fails or timesout -tG, --trygui Try to fetch the page with FireFox when headless fails -sF, --smartfetch Enables smart fetching to reduce network traffic, also increases speed if certain conditions are met. -pX PROXY, --proxy PROXY SOCKS5 Proxy in host:port format Some of the above options have non-obvious use-cases, so the following provides some more detail: -l, --list -> Takes as input a file with a simple list of input URLs in the format "http(s)://<URL>" -i, --input -> Takes a gnmap file as input. This includes masscan gnmap output. -p, --headless -> I find myself using this option more and more. By default the script "drives" Firefox. As the number of threads increases this becomes really ugly - 20,30 Firefox windows open at once. This options uses "phantomjs" which doesn't have a GUI but will still do a decent job parsing javascript. -w, --workers -> The number of threads to use. Increase for more speed. The list of input URL's is automatically shuffled to avoid hammering at IP addresses that are close to each other when possible. If you add too many threads, you might start seeing timeouts in responses - adjust for your network and machine. -t TIMEOUT, --timeout -> How long to wait for a response from the server before calling it quits -v, --verbose -> Will spit out some extra debugging output. -a, --autodetect -> Without this option enabled, HTTPScreenshot will behave as follows:
-vH, --vhosts -> Often when visiting websites by their IP address (e.g: https://192.168.1.30), we will receive a different page than expected or an error. This is because the site is expecting a certain "virtual host" or hostname instead of the IP address, sometimes a single HTTP server will respond with many different pages for different hostnames.
-dB, --dns_brute -> Must use with -vH for it to make sense. This flag specifies a file containing a list of potential subdomains. For any wildcard certificate e.g: "*.google.com", HTTPScreenshot will try to bruteforce valid subdomains and add them to the list of URLs to be screenshotted. -r, --retries -> Sometimes Firefox or ghostscript timeout when fetching a page. This could be due to a number of factors, sometimes you just have too many threads going, a network hiccup, etc. This specifies the number of times to "retry" a given host when it fails. -tG, --trygui -> Upon failure to fetch with the headless browser phantomJS, will pop open FireFox and try again. -sF, --smartfetch -> Enables smart fetching to reduce network traffic, also increases speed if certain conditions are met. |
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论