This article is translated from the WeChat official account "Intelligencer1".
There is a lot of Open-Source Intelligence (OSINT) information on the Internet, where open-source refers to information or programs that are easily accessible and distributed by the public; intelligence refers to the ability to acquire and apply knowledge. In general, open-source intelligence is the process of obtaining, collecting, and analyzing information from public sources to generate valuable intelligence. These sources include the Internet, social media, academic and professional journals, newspapers, television, and even certain data that has been leaked in violation of regulations.
1. OSINT Framework#
OSINT Framework is a website that contains different tools. Users can use these tools to query open-source intelligence information in different systems and knowledge bases, including email addresses, social media, domain names, search engines, public records, documents, and even phone numbers.
For example, if a user wants to learn about "social media" related information, OSINT Framework will divide it into subcategories such as Facebook, Twitter, Instagram, Reddit, LinkedIn, etc. Users can further select subcategories, which will include searching, analyzing, images, locations, and storing/deleting tweets. When selecting "analyze", it will also present subcategories such as tweet metadata, Birdwatcher, Tinfoleak, etc.
Application link: https://osintframework.com/
2. Wayback Machine#
Wayback Machine is known as the digital archive of the Internet. It is used to manage, capture, and archive snapshots of websites over a period of time, so that even if the original web pages are deleted or modified, users can still view past snapshots of the web pages. Currently, Wayback Machine receives over 1 billion requests per day (including retrieval and archiving requests) and has cached nearly 700 billion different web pages at different points in time.
Users can archive billions of websites through Wayback Machine's "Save Page" service. If a free account is registered, users can also request the "outlinks" of relevant pages to be crawled and receive reports on the crawling status via email.
When historical information needs to be traced, Wayback Machine presents all saved records of the website to users in the form of "year selectors" and "date selectors". If you want to compare the differences between two different web page archives, you can use Wayback Machine's "Changes" feature to achieve this.
Application link: http://www.wayback.com
3. Maltego#
Maltego mainly uses open-source intelligence technology to query sources such as DNS records, whois records, search engines, social networks, various online APIs, and metadata extraction to discover the links between this information. The Maltego program works by automatically searching different public data sources, so users can click a button and perform multiple queries, with each query returning up to 12 entities. Since the program uses public interfaces to perform searches, it is compatible with almost any information source that has a public interface.
Once information is collected, Maltego establishes links to reveal hidden relationships between names, email addresses, aliases, companies, websites, document owners, subsidiaries, and other information that may be useful for investigations or to identify potential future issues. It can be downloaded on Windows, Linux, and Mac, and comes pre-installed on Kali Linux.
The program has a free version with limited functionality called Maltego CE. The Maltego PRO version is primarily designed for professional investigators and small teams, and includes the Maltego One desktop, all Maltego standard transforms, and access to the commercial transform hub (see the complete list of features).
Application link: https://www.maltego.com/
4. Have I Been Pwned#
Have I Been Pwned is a free website resource created by Troy Hunt. The website collects a large number of breached databases for users to check if their personal information has been compromised. Users only need to enter their name or phone number in the search box, and it will search the leaked databases to see if the corresponding credentials have been compromised. If relevant breach events are found, the leaked information can be viewed by scrolling down the page.
Application link: https://haveibeenpwned.com/
5. Shodan#
Shodan is a specialized search engine used to search the Internet and different network servers connected to it. This means that with Shodan, users can browse its database to find devices connected to the Internet, such as routers, Internet of Things (IoT) devices, monitors, security cameras, traffic lights, etc.
As a powerful OSINT tool, Shodan can monitor and search an astonishing range of scope. It is one of the few engines that can check Operational Technology (OT). Without tools like Shodan, there would be a significant gap in collecting open-source intelligence in industries that deploy IT and OT.
Creating an account on Shodan is free, but the information that can be queried for free is very limited. If users want to further query more information, they need to purchase Shodan's membership service. In addition to the personal version, Shodan also offers paid versions for small businesses and enterprise advanced versions. The small business version can scan up to 65,536 IP addresses and return up to 20 million results; the enterprise advanced version provides unlimited results and up to 327,680 IP scans per month, as well as vulnerability search filters and advanced support services.
Application link: https://account.shodan.io/
6. TinEye#
TinEye is a reverse image search and image recognition website developed by Idée. It was officially launched in 2008. It uses computer vision, pattern recognition, neural networks, and machine learning to provide fast and accurate search solutions. To use TinEye, simply upload the image of interest to the website. The site will retrieve information about the image's location, source, usage, and even higher resolutions.
Application link: www.TinEye.com
7. Censys Search#
Censys Search is a web-based search engine that was initially created for academic research. It is also a public welfare project completed in cooperation with the University of Michigan and Rapid7. On the Censys official website, this search engine is defined as follows: "Censys Search is a search engine that allows computer scientists to understand the devices and networks that make up the Internet. Censys is driven by Internet-wide scans, allowing researchers to find specific hosts and create a comprehensive report on the configuration and deployment information of devices, websites, and certificates." Like Shodan, it searches for servers and connected devices on the Internet. In addition, it can also identify connected industrial control systems and platforms.
Application link: https://censys.io
8. BuiltWith#
BuiltWith is a tool used to discover popular website building technologies and programming languages. It is a website analyzer, business intelligence (BI), lead generation, and competitive analysis tool. BuiltWith also generates a complete list of known JavaScript/CSS libraries used by a website, such as jQuery or Bootstrap. In addition, the service provides a list of plugins, frameworks, server information, analytics and tracking information installed on the website. In practical applications, when used in conjunction with website security scanning programs such as WPScan, BuiltWith makes it easier for security personnel to discover common vulnerabilities that affect website security.
Application link: https://builtwith.com/
9. Nmap#
Nmap is a free network and port scanner used to discover services, operating systems, hosts, and open ports running on a network or website. It is available on the web and supported on most common operating systems.
Nmap's features include:
• Host discovery: Identifying hosts on a network;
• Port scanning: Enumerating open ports on target hosts;
• Version detection: Interrogating network services on remote devices to determine application names and versions;
• OS detection: Determining the operating system and hardware characteristics of network devices;
• Scriptable interaction: Using the Nmap Scripting Engine (NSE) and the Lua programming language.
Application link: https://nmap.org/
10. Recon-ng#
Recon-ng is an information gathering tool developed in Python that operates through the command line. It can perform DNS information gathering, domain information gathering, email information gathering, and information gathering combined with search engines. It is a powerful web reconnaissance framework.
Recon-ng adopts a modular design framework and has many built-in features, making it easy for even novice Python developers to create publicly available data searches and return good results. Developers do not need to write Recon-ng scripts to perform searches, they only need to select the functionalities they want it to perform, and they can build an automated module within minutes.
Application link: https://bitbucket.org/LaNMaSteR53/recon-ng
11. The Harvester#
The Harvester is an open-source intelligence tool used to gather information about email addresses, websites, subdomains, open ports, virtual hosts, and more. The Harvester uses popular search engines such as Bing and Google, as well as less common search engines such as dogpile, DNSdumpster, and the Exalead metadata engine as sources. It can even use the Shodan search engine to find open ports on discovered hosts.
The Harvester can access most public resources without any special preparations. However, some of its features require an API key. In addition, Python 3.6 or higher must be present in the user's environment.
Application link: https://github.com/laramies/theHarvester
Original article link: https://mp.weixin.qq.com/s/hIcmJK0ozuNkQH3t10i6pQ