Website OSINT: Discovery and Exploration of Web Resources

The links below are a companion to the talk Micah Hoffman gave in June 2021 at the NCPTF conference. The slides have not been and will not be posted.

Researching the IP/Domain

• General/DNS sites: host.io, dnsdumpster.com
• WHOIS: whoxy.com, domainbigdata.com
• IP Location and Details: ip2location.com
• Infrastructure site: (requires free account signup/login) shodan.io
  • References:
    • https://osintcurio.us/2021/05/13/searching-with-shodan/
    • https://github.com/JavierOlmedo/shodan-filters

---

Examine the Web Server Configuration

• General/Analytics: builtwith.com
  • Reference: https://youtu.be/cA35RGMhHNg
• HTTPS: censys.io
  • Reference: https://www.youtube.com/watch?v=XHltHamQVoA

---

Discovery of Files

• Search engines
  • Create keywords: name of site, company, people names, etc…
  • Google search operators: site:, inurl:
• Archive sites
  • Archive.org
    • Searching with wildcards (example: https://web.archive.org/web/*/osintcurio.us/*)
    • Changes
  • archive.today
    • Searching using wildcards
  • urlscan.io
    • On-demand searching (active)
    • Using their search (passive
  • References:
    • https://osintcurio.us/2021/03/03/using-archive-org-for-osint-investigations/
    • https://osintcurio.us/2019/02/12/osint-on-deleted-content/
• robots.txt file (example: https://www.apple.com/robots.txt)
  • Reference: https://youtu.be/aEGbPiliJpg
• Decoding complex URLs: https://dfir.blog/unfurl/

---

OSINT Curious Resources

• Website: https://osintcurio.us
• YouTube Channel: https://youtube.com/theosintcuriousproject
• Link to Google Doc Index of our Resources: https://bit.ly/osintcuriousresources