🔎 40+ open source intelligence tools for various internet investigations
Rather than go into how you do online competitive intelligence, I’m just going to run through a free list of tools that have earned themselves and honorable mention here and give you some awesome tidbits for finding information online.
I review competitive information on a daily basis; I have to warn you that its a lot of information.
Don’t be afraid of simply sitting down and playing with one or two tools a week and seeing what you come up with. The real magic begins to happen when you start discovering that one tool provides the secret ingredient to turbo-charge the next tool.
Every tool on this list is free 🌟
(some of them may have premium paid upgrades, but they aren’t necessarily required.)
If you would like to find out about the how and why behind using these tools, check out my competitive intelligence posts.
Google Tools
You can’t beat Google when it comes to free information and tons of data. The biggest issue with using Google effectively is understanding how to sift through massive volumes of data to find actionable nuggets of wisdom. To help you do that I’ve include two link to a advanced search parameter cheat sheet and a walk-through of using advanced parameters. If you have a few minutes, you can also watch this official Google training video on advanced search parameters.
Google Cheat Sheet – Advanced Operators http://www.googleguide.com/advanced_operators_reference.html
Using Advanced Search Parameters on Google
http://www.googleguide.com/using_advanced_operators.html
The actual Google Tools…
Google News
http://news.google.com/nwshp
This is probably one of the easiest tools to keep track of your industry, your competitors, and your prospects/clients. When you are signed into Google you have the option of personalizing the news topics and the sources they draw from (In my personal use I have it set to review TechCrunch, Mashable, GeekWire, Venturebeat, etc to cover the start-up/tech scene.)
Custom Google Search Engine
www.google.com/cse/
This is another awesome tool from Google. It allows you to control the sites Google uses for search results. If you have a list of 100 competitive sites you can use it to search them all for investor reports, instructional guides, quarterly results, client lists, etc.
Google Trends
www.google.com/trends/
A great tool for understanding how keyword volume has changed over time. The chart to the right shows a review of Obama vs. Romney for the past twelve months and provides me with the ability to look at noteworthy news stories, drill down to local regions, or explore associated Google results.
trends.google.com/websites
Provides slightly different information when you enter a web domain (.com, .net, etc)
This information details demand generation over time. A quick look at BestBuy.com shows us the buying patterns around holiday spikes for the past five years, the top ten states driving that demand, and the top cities within those states.
Google Insights
http://www.google.com/insights/search/
Google Insights is one of my favorite ‘go to’ tools: it allows me to narrow data to specific categories, like finance, health, and tech. It shows me search volume distributed across regions and cities.
Social Media Tools 🏅
Addictomatic.com
http://addictomatic.com/
Is a highly useful social search site that presents a ‘widget’ for each site it finds related information on. These widgets can be sorted, added/deleted, and bookmarked or shared with coworkers.
SocialMention.com
http://socialmention.com/
Has a variety of advanced selection tools to sort blogs, video, microblogs, comments, and more. Results provide multiple breakdowns such as top users, frequency, and popular keywords/tags
Monitter.com
http://monitter.com/
Provides a ‘tweetdeck’ like interface to search Twitter topics quickly. In my daily use I often plug 5-10 topics into the tool and simply let it run on one of my secondary monitors so that I can contemplate trends through-0ut the day.
FeedCompare
http://www.feedcompare.com/
Allows you to examine RSS subscribers on popular sites, a metric that can be very useful for determine where different audiences are coming from and if there are unusual trends that can be associated to marketing and awareness campaigns.
Site Based Tools ⚛️
Quantcast.com
http://www.quantcast.com/
Provides a wealth of information about sites on the web: demographics, traffic trends, related websites, and topic affinity.If you examine your top ten competitors using this tool, you can often find a few related properties for press announcements and influencer campaigns.
XINU Returns
http://tools.seobook.com/xinu/
An old but still useful web script that reports on 100+ data points that include Pagerank, backlinks, social shares, code validation, and various technical details.
Quarkbase
http://www.quarkbase.com/
Provides a range of statistics regarding a web domain, including key people (according to Lead411), Alexa and Compete Rank, similar sites, and socially popular sites.
Alexa
http://www.alexa.com/
Alexa is a provider of free, global web metrics: discover the most successful sites on the web by keyword,category, or country.
SEO Based Tools
SEObook Competitive Research Tool
http://tools.seobook.com/competitive-research-tools/
While there are dozens of paid SEO tools, SEObook has multiple free tools that include Keyword Suggestion and Spider Tests, along with three different Firefox plugins
- Rank Checker– check a domain for ranking on dozens of keywords (good for your site or competitor sites.)
- SEOtoolbar – get 10+ metrics at your fingertips, along with access to more tools.
- SEO for Firefox – provides metrics directly inside your Google search results (no other steps needed.
Web Tools 🖥️
ChangeDetect
http://www.changedetect.com/
A service that alerts you to any changes on a specific web domain. It allows you to note things like product updates, team changes, or public announcements on auto pilot.
WayBack Machine
http://archive.org/web/web.php
Keeps track of web site changes over time. This is a critical item for understanding how competitors have changed branding and positioning.
Ever want to see what BestBuy.com looked like on Dec 25th of 1996?
Favorite Plugins 🤓
Here are a few plugins I use on a daily basis. Some of them have versions for other browsers if you are an IE, Chrome, or Safari person.
Search Sidebar by Webmynd
https://addons.mozilla.org/En-us/firefox/addon/the-search-sidebar-by-webmynd/
Provides a sidebar on your search results page that includes additional results from Facebook, Twitter, Linkedin, Wikipedia, and many more content sites.
Search Engine Tags
https://addons.mozilla.org/en-US/firefox/addon/SearchEngineTags/
This allows you to search for information using an additional layer of meta data that you normally don’t see. It provides access to site and topic specific results based on using tags in your searches (it requires a little practice to learn, but provides some highly useful results.)
Autopager
https://addons.mozilla.org/En-us/firefox/addon/autopager/
Allows you to automatically load results from the next page… and the next (into infinity.) When using sites like Google you can see 10 to 10,000 results to search for specific items hidden well beyond page one of the results.
Fireshot
https://addons.mozilla.org/En-us/firefox/addon/fireshot/
Allows you to take screenshots and has dozens of very useful features.When used for competitive intelligence it allows you to quickly gather visual records and note them.
Firebug
https://addons.mozilla.org/En-us/firefox/addon/firebug/
A development tool that allows you to see the code structure for any web page. Very useful for finding software vendors who provide different platform capabilities to your competitors.
Scrapbook
https://addons.mozilla.org/En-us/firefox/addon/scrapbook/
Add visual bookmarks into a catalog that can be tagged and categorized. Competitive Intelligence professionals can quickly sort reference sites and create visual archives for team collaboration.
List Open URLs
https://addons.mozilla.org/En-us/firefox/addon/list-open-urls/
When you have multiple tabs open on your browser this saves all the URLs to a text file. Very useful for opening 5 to 25 tabs on a research project and documenting the list of source sites for later reference.
HR Related Tools 📞
LinkedIn
http://linkedin.com
If you don’t use Linkedin for competitive and market intelligence you are really missing something. Linkedin provides all sorts of data about employees, relationships, company data, new hires, people that left, press releases, org charts, open positions, and more…
Glassdoor
http://glassdoor.com
Past employees share a treasure trove about the marketplace you deal with. As a standard competitive intelligence sweep I usually recommend checking the top five to ten competitors, top client/prospects, and your vendors.
DATA Sources 📈
Copernic
http://www.copernic.com/en/products/agent/personal.html
Copernix is a piece of software that accesses multiple search engines using one interface. It aggregates the information, provides multiple filter options, saves search records, highlights keywords, and allows import/export of the results.
Yahoo Pipes
http://pipes.yahoo.com/pipes/
This isn’t really a data source, but a web based tool that allows you to aggregate, mash-up, and syndicate other sources of data. It is incredibly useful as a tool to build ‘minimum viable models’ for web research (but does have a learning curve.)
GNIP
http://gnip.com
Gnip provides data from dozens of top social sites ranging from Facebook and Twitter to Youtube and Vimeo. By using a single API it allows you to access thousands of data fields that can be extracted for all sorts of competitive intelligence on businesses, community audiences, and specific people.
DataSift
http://datasift.com
Is another social media data provider. They have a tremendous amount of information available (some overlap with Gnip, some is unique.) It is one of the few social media data providers that can provide full access to the Twitter firehose.
Spinn3r
http://spinn3r.com
Yet another social media data company, but with focus on real-time indexing of the blogosphere or content that you specifically want. Very useful for processing large amounts of social content found on niche community sites that don’t make it onto Gnip or Datasift’s lists.
Visualization Tools 📊
Tabelau
http://www.tableausoftware.com/
Provides an incredibly useful ‘community edition’ visualization tool on the web that allows you to sort and sift data using a variety of tools.
Pentaho
http://www.pentaho.com/
Business analytics tool for playing with numbers. Open source version is limited, but has some great tools for the price (FREE!)
Splunk Free
http://www.splunk.com/view/free-vs-enterprise/SP-CAAAE8W
Variety of indexing, search, and reporting tools built into a dashboarding tool with apps and developer APIs.
Google Charts
https://developers.google.com/chart/
Google Charts is a go-to resource for free charting capabilities. You can serve up all sorts of data into it, embed them, and even use spreadsheets from Google docs.