Tools and techniques helpful in reverse engineering cmopanies engaged in questionable actitivies.
- Automator Virtual Input (Apple) – Extends Apple’s Automator automation program to give it control over the keyboard and mouse cursor. Automator Virtual Input allows Automator to access anything you can by simulating mouse clicks, movement and drags, etc.
- IOpus – web automation, testing & extraction tool.
- mjtnet – Windows automation software
- ScrapeBox – Powerful URL harvesting, analysis, and management tool
- Web Data Parser – an inexpensive and pretty effective web data extraction tool for a lot of purposes
- Ubot Studio – extremely robust automation tool for developing “bots” through a visual scripting language
- URL Detective – Whitehat URL harvester that uses the Google API
- WinAutomation - Windows Automation Software
“Swiss Army Knife” type tools:
- Maltego – A short demo of Maltego (extremely powerful investigation tool with a free version). “Maltego is an open source intelligence and forensics application. It will offer you timous mining and gathering of information as well as the representation of this information in a easy to understand format.”
- Outwit Hub – “An all-in-one application for extracting and organizing data from online sources; a powerful automation engine that explores for you large series of Web sources and brings back the answers instead of telling you where they are.”
- DomainTools.com (especially the paid version) – incredibly powerful tool for exploring almost anything you can think of when it comes to domains.
- Builtwith.com – determines site technology used as well as types of analytics, etc.
- DomainCrawler.com – DomainCrawler 2.0 includes extensive functionality such as thumbnails, backlink checks, pagerank history and the ability to monitor a domain for change as well as more traditional domain/SEO-related information.
- DNSStuff – Extensive collection of DNS analysis, tracking, and alert tools.
- IP-Address.com – Several IP tools, including the most accurate IP locator tool (other than MaxMind.com’s bulk GEO IP locator) I’ve found yet.
- MarkosWeb.com – Uses around 25+ open API sources researching the mo st popular domains and keywords. Helps webmasters check their competitors and understand why their ranks are so high and includes useful summary tables for webmasters.
- Network Tools – Traceroute, Ping, Domain Name Server Lookup, Whois, and more
- Push To Check – offers a service that checks different parameters for domains and subdomains and saves their history. So you can return again later and check for progress of your sites or sites of your competitors.
- QuarkBase.com – See people, traffic data, similar sites, social comments, description, social popularity and much more about website
- Robtex.com – Swiss Army Knife internet tool – network exploration, blacklist checking, and a lot more
- SensePost Tools – Numerous Paid and Free Tools
- Serverniff.net (NOT .com) Your free “Swiss Army Knife” for networking, serverchecks and routing with many many little toys and tools for administrators, webmasters, developers, powerusers und security-aware users.
- Siteopsys – 30 research and analytical SEO tools. One nice tool is the bulk reverse-ip look up tool. Not as accurate as using WebDataParser against DomainTools, but a nice quick and dirty way to find potential groups of sites on IPs to explore further.
Tools developed specifically to determine potential ownership between websites
- AdsSpy.com (closed?)
- eWhois.com (free service)
- OwnerSpy.com (paid service)
- SameOwner.com (paid service)
- W3Who.com (free service)
- WebBoar.com (free service)
- ReverseInternet.com (I have some questions about how they are populating this database)
Tools to pinpoint potential footprints to run through Footprint Finding Tools
Of course almost any of the tools on this page can be used to pinpoint potential footprints. But the following tools are great tools to run against the results of the OTHER tools on this page to more easily pinpoint potential footprints to run through Footprint Finding Tool
- Ghostery – A privacy tool which can be used to determine what types of analytics, advertising, and tracking codes a site is using.
- Maltego – Extremely powerful investigation tool. A great way to find potential footprints you may have never considered
- ServerSiders.com – has a section which sometimes shows if a site is running Google Analytics or Google Adsense and provides the IDs. Not always accurate and is best use to feed the Ids through the tools developed specifically to determine potential ownership between websites.
- OutWit Hub – “An all-in-one application for extracting and organizing data from online sources; a powerful automation engine that explores for you large series of Web sources and brings back the answers instead of telling you where they are.”
Footprint Finding Tools
In a sense most of the tools on this entire page could be used as “Footprint Finding Tools” – but these are Footprint Finding Tools I find myself using time after time.
- CopyScape – Find duplicate versions of content
- ScrapeBox – Powerful URL harvesting, analysis, and management tool
- Tineye – Currently the best reverse-image search engine online
Tools To Determine Uniqueness Of A Footprint
- pHash.org - Open source perceptual hash library
- md5deep and hashdeep – md5deep is a set of programs to compute MD5, SHA-1, SHA-256, Tiger, or Whirlpool message digests on an arbitrary number of file
- Image information viewer (hasn’t been working lately as of 7/26/2010)
- Similar Page Checker – Determine the percentage of similarity between two page
Keyword Research And Tracking Tools (Spy on competitors keywords, ads, campaigns, and more)
- Keyword Spy (Paid Version) – Uncover the competition landscape by knowing your competitors’ PPC and Organic campaign strategies in seconds.
- PPC Web Spy (Paid Version) – PPC Web Spy allows you to secretly spy on & uncover anyones Google-Adwords keywords while you browse Google in real time.
- iSpionage (Paid Version) – Discover competitors’ top keyword, ad copy, and landing page strategies
- SEMRush – SEMRush shows Google keywords for any site, Adwords keywords for any site, list of competitors in organic and paid Google search results, & estimated SE and Adwords traffic for any domain
- SpyFu (Paid Version) SpyFu allows you to get information on the competition, including how they are using keywords in paid advertising and organic search as well as other information on their advertising strategies.
- PPC Bully (Paid Version)
Other Traffic Analysis Tools
- Alexa – “Swiss Army Knife” like collection of traffic analysis/competitive intelligence tools – Alexa has become MUCH more useful than it has been in the past. And it’s free.
- Compete – Another “Swiss Army Knife” collection of traffic analysis/competitive intelligence tools. The best data from Compete is not free.
- Quantcast – Provides detailed audience profiles for the advertising marketplace
- Web Site Traffic Spy – A traffic aggregator using Alexa, Compete, Quantcast, Google Trends, and the IP location information from IP-Address.com
Proxy Detection Tools
- IP2Proxy free online service from FraudLabs. Limit is 20 queries per day. Free DeskTop version is also available.
- FraudLabs.com IP2Proxy Proxy Detection Web service – hosted, programmable XML Web Service that allows instant detection of anonymous proxy by IP address. The IP2Proxy™ Proxy Detection Web Service enables you to check proxy servers for anonymity.
- MaxMind Proxy Detection -
- WhatIsMyIp - IP Lookup at WhatIsMyIp
Website/Web Page Change Detection Tools
- Archive.org - Archival history of websites and their pages
- ChangeAlarm.com – free website change detection tool (free use subject to Terms of Service)
- ChangeDetection.com – free service provides page change monitoring and notification services to internet users worldwide. Anyone can use our service to monitor any website page for changes. Free since 1999 (free use subject to Terms of Service)
- URLy Warning – free, easy-to-use Windows software that lets you monitor Web pages. It works great with any XP, Vista or Windows 2000 PC. Know instantly when a page changes. See exactly what’s changed on that page.
- WatchThatPage.com – a service that enables you to automatically collect new information from your favorite pages on the Internet. You select which pages to monitor, and WatchThatPage will find which pages have changed, and collect all the new content for you. The new information is presented to you in an email and/or a personal web page. You can specify when the changes will be collected, so they are fresh when you want to read them. The service is free!* *Professional and heavy users are requested to pay a fee. See terms & conditions
Website Exploration Tools
- Fox Splitter – splits the content area of the browser window almost any way you would want.
- HTTrack Website Copier – Offline Brower
- OutWitHub.com – “An all-in-one application for extracting and organizing data from online sources; a powerful automation engine that explores for you large series of Web sources and brings back the answers instead of telling you where they are.”
More Specialized Utilities/SitesTools
- Ip2Location.com – IP2Location™ is a geo IP solution to help you to identify visitor’s geographical location. Highly accurate.
- Bulk GEO IP locator at MaxMind.com – Find countries and cities by entering IP addresses/hostnames/domains
- Domain Status Reporter – Obtain and monitor the registry status (active, registrar-lock, registrar-hold, etc), created, updated and expiry dates of global top level domains (com, Net, org, Edu, Biz and Info).
Competitive Intelligence – Media Monitoring Services
- LiveDash – Search What’s Being Mentioned On T.V.
- Publisher’s Information Bureau – check on the competition by seeing what publications they are using repeatedly.
- Experian Simmons – extensive consumer studies report over 60,000 data variables in over 8,000 product categories and 450 plus brands, major media usage behavior, and in-depth consumer behavior demographics to give our clients the ability to make the most of their marketing budget.
- Whos Mailing What - Direct Mail Competitive Intelligence Archive
News & Media Analysis Tools
- Recorded Future – Recorded Future’s linguistics and statistics algorithms extract time-related information and through temporal reasoning helps users understand relationships between entities and events over time, to form the world’s first temporal analytics engine
- SiloBreaker – aggregates content on global issues, science, technology, energy, business, sports and entertainment from tens of thousands of news sources, blogs, multimedia, and other forms of news media from around the world. With the engine’s focus on finding and connecting related data in the information flow, Silobreaker’s user tools and visualizations are ideal for bringing meaning to content from either today’s Web or the evolving Semantic Web, or both.
Forum Buzz & Reputation Tools
- Omgili Graphs – Measure and compare the Buzz of any term. The Buzz is the percentage of the term out of the total number of discussions Omgili covered on a specific date.
- Boardreader – Forum search engine
- BoardTracker (Premium version) – Social media metrics with detailed graphs and advanced analytics based on quality data.
Social Media Buzz, Analysis, & Reputation Tools
- 123People.com – (Note: not as good as Pipl) – free people search
- BrandsEye.com – Inexpensive reputation management tool
- BuzzDing – Monitor your reputation online and connect with the community surrounding you, your company, and your products
- Gabberface – Find out what everyone is posting about on Facebook
- HowSociable? – Type in a brand name, keyword, website, etc.. Find out how visible it is on the social webs.
- KnowEm – Check a company’s or person’s profile presence across 400 social media networks
- PeekYou – asdf
- Pipl – “People search” deep web search engine. Can also search by email, username, phone number, and business name
- ScoutLabs – Track, Manage, and Analyze discussions across social media platforms. Includes sentiment analysis engine.
- socialmention – Real-time social media search and analysis
- SocialWhois – Type a nickname and select a service (Twitter or FriendFeed) and SocialWhois will try to give you some information about the company or person associated with that nickname.
- Spokeo – Spokeo is a search engine specialized in organizing people-related information from phone books, social networks, marketing lists, business sites, and other public sources
- Trackur – Suite of social media monitoring tools
- Trendistic – see trends in Twitter
- Whos Talkin – Social media search tool
- Yasni.com – Find a person is mentioned on the web
Social Network Diagraming Tools
- Gephi – an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs.
- NodeXL – Free Network Overview, Discovery and Exploration Template for Excel
- Pajek - Windows program for analysis and visualization of large networks – freely available, for noncommercial use.
- SocioGram - A simple program to generate sociogram-graphics showing the relationships between individuals
- Software list at INSNA (obviously I haven’t tried all of these)
Image Analysis Tools
- Tineye – Currently the best reverse-image search engine online
- Focus Magic – uses advanced forensic strength deconvolution technology to literally “undo” blur and recover lost detail. It can repair both out-of-focus blur and motion blur (camera shake) in an image. Extremely inexpensive considering how powerful it is.
- DeblurMyImage - Deblurring software for out of focus correction and motion correction
- DenoiseMyImage – allows you to remove digital noise from your photos while preserving edge.
- ABBYY Tools – Especially the FineReader OCR solution for extracting text from images.
Text Manipulation Tools
- Clean Haven – Mac, OS X, Windows, and Linux based tool for cleaning text
- Convert Case – Simple online tool to convert case of characters
- Data Mystic – Data Extraction, Data Conversion, Report Mining and Text Manipulation Software
- SortMyList – Online tool with a large variety of ways to sort lists
- Text Crawler – Enables you to instantly find and replace words and phrases across multiple files and folders. It utilises a powerful Regular Expression engine to enable you to create sophisticated searches, preview replace, perform batch operations, extract text from files and more. It is fast and easy to use, and as powerful as you need it to be.
- Text_Diff – Show differences between 2 works of text
- Text Mechanic – An extensive collection of free, online, browser-based, text manipulation tools. Includes harder to find tools such as text randomizing tools, text obfuscation tools, permutation & combination generators and more. Also offers a basic online html WYSIWYG editor.