The tool cURL is already over twenty years old, but it remains one of the most used tools by people, especially the ones working in the IT Security branch. It is used directly on the command line and can send or receive data from a website or IP address. Whether you need to retrieve information over HTTP, HTTPS, FTP, SMTP or any other kind of popular protocol, or send data to a server while emulating a browser, it can all be done with cURL. It comes preinstalled in some major Linux or UNIX OS’s or can easily be installed, even in Windows if you need it.
How to use cURL
The most basic cURL method is to request information from a web server. For that we open a command shell or prompt and we simply type: curl ipinfo.io. This command will contact the web server running on https://ipinfo.io, which sees it is the cURL tool that requests the information, and sends back a reply with your personal IP address:
sector035@Tuvok:~$ curl ipinfo.io
{
"ip": "82.102.19.147",
"city": "Zaventem",
"region": "Vlaams-Brabant",
"country": "BE",
"loc": "50.8833,4.4667",
"postal": "1930",
"org": "AS9009 M247 Ltd"
}
But the tool is much more powerful. For instance, there is an option to only request the headers that the web server sends to the client, so if we would want to know what a web server is running, we can add the option “-I”:
Sector035@Tuvok:~$ curl -I https://osintcurio.us HTTP/2 200 server: nginx date: Tue, 14 May 2019 14:35:50 GMT content-type: text/html; charset=UTF-8 strict-transport-security: max-age=86400 vary: Accept-Encoding vary: Cookie x-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header. link: <https://wp.me/avIqQ>; rel=shortlink x-ac: 3.ams _dca
To gather a bit more information, one can also change the option to ‘-vI’ where the ‘v’ stands for verbose and gives information on the security of the connection offered, information about the certificate that is assigned, protocols that are being used et cetera.
To post data, for instance to a form, there are a few things we need to specify. First of all, the default ‘HTTP Method’ is GET. Which means that cURL will retrieve (or GET) information from a web server by default unless you specify anything else. To enable cURL to actively send data to a server, we need to specify the desired ‘HTTP Method’ called ‘POST’ (or ‘PUT’, depending on the service you are using). Secondly, we need to specify the ‘payload’ or data that we want to send. For that we have the option ‘-d’. Here is an example made to the awesome website httpbin.org that you can use to play with cURL:
sector035@Tuvok:~$ curl -X POST "http://httpbin.org/anything" -d testpayload="Whatever You Want"
{
"args": {},
"data": "",
"files": {},
"form": {
"testpayload": "Whatever You Want"
},
"headers": {
"Accept": "/",
"Content-Length": "22",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "curl/7.58.0"
},
"json": null,
"method": "POST",
"origin": "82.102.19.147, 82.102.19.147",
"url": "https://httpbin.org/anything"
}
In the first line we see that I had cURL post data to the URL ‘http://httpbin.org/anything’. It responds with information about my request. First it sends back the arguments, form information or whatever data I have sent, followed by information sent about the client I used (cURL), the URL I targeted and my IP address.
A lot of things can be accomplished by visiting a website with a browser, but sometimes a website provides a special service that gives you the ability to retrieve information in a raw and easy to handle format, therefore making it perfect to use cURL and save the result in a text file.
So this is a very basic overview of cURL and there is a lot more to explore if you have a need for it. More information and its documentation can be found here:
https://curl.haxx.se/, but for now I would like to focus on a whole bunch of websites that you can play with safely, so you might have some new sources to use during your work.
cURL for Web Devices
Finding out information on an IP address is easy with a tool like https://ipinfo.io, that was mentioned above, and it still is one of my favorite tools to quickly check whether I am indeed connected to a VPN or not. But you can also use it to retrieve information on an IP address, like this:
sector035@Tuvok:~$ curl https://ipinfo.io/81.169.145.158
{
"ip": "81.169.145.158",
"hostname": "w9e.rzone.de",
"city": "Berlin",
"region": "Land Berlin",
"country": "DE",
"loc": "52.5174,13.3985",
"postal": "12529",
"org": "AS6724 Strato AG"
}
That is it, just add an IP address to the end that you are checking out and it will retrieve basic information about the host name, the geographical location it is attached to it and who owns the block of IP addresses it is part of. Another service that provides similar information is https://ipapi.co, adding a bit of extra information like dial code, time zone and the postal code of the IP address (be aware, this is a guess and usually not very accurate!). You can try it yourself with the following command:
sector035@Tuvok:~$ curl https://ipapi.co/81.169.145.158/json
{
"ip": "81.169.145.158",
"city": "Berlin",
"region": "Land Berlin",
"region_code": "BE",
"country": "DE",
"country_name": "Germany",
"continent_code": "EU",
"in_eu": true,
"postal": "12529",
"latitude": 52.5174,
"longitude": 13.3985,
"timezone": "Europe/Berlin",
"utc_offset": "+0200",
"country_calling_code": "+49",
"currency": "EUR",
"languages": "de",
"asn": "AS6724",
"org": "Strato AG"
}
Some more information on what you can do with ipapi.co can be found in their documentation.
If you want to find the owner of a web site, we use a WHOIS service to request information on the registrar of the domain name. There are multiple websites that offer that service, but one very simple to use website is https://cli.fyi where you only need to add the domain name you want to check:
sector035@Tuvok:~$ curl cli.fyi/cnn.com
{
"type": "Domain Name Information",
"data": {
"dns": [
"cnn.com. 300 IN MX 10 mxa-000c6b02.gslb.pphosted.com.",
[ ... ]
],
"whois": [
"Domain Name: CNN.COM",
"Registry Domain ID: 3269879_DOMAIN_COM-VRSN",
"Registrar WHOIS Server: whois.corporatedomains.com",
[ ... ]
"Registry Registrant ID:",
"Registrant Name: Domain Name Manager",
"Registrant Organization: Turner Broadcasting System, Inc.",
"Registrant Street: One CNN Center",
"Registrant City: Atlanta",
"Registrant State/Province: GA",
"Registrant Postal Code: 30303",
"Registrant Country: US",
"Registrant Phone: +1.4048275000"
, "Registrant Phone Ext:",
"Registrant Fax: +1.4048271995",
"Registrant Fax Ext:",
"Registrant Email: tmgroup@turner.com",
[ ... ]
"Register your domain name at http://www.cscglobal.com"
]
}
}
If you want to go a step further, and you are confident in using the command line or cURL, I also suggest checking out SecurityTrails. They have a large database with DNS and WHOIS information, including historical information on domain name owners. The use of the website itself is free, but by using their API, you are able to query the database and get the result in easy to store and process information. You can register a free account that gives you the ability to run 50 queries a month via their API. With that you can find out what other domain names were registered with a particular email address, for instance: ‘tmgroup@turner.com’.
sector035@Tuvok:~$ curl -X POST "https://api.securitytrails.com/v1/search/list" --header 'apikey:MyPersonalAPIkey' --header 'content-type: application/json' -d '{"filter":{"whois_email":"tmgroup@turner.com"}}' -o tmgroup-domains.txt
Let’s have a closer look at this particular command and break it down into smaller pieces. We have the following items:
| -X POST “url” | This is the ‘HTTP Method’ that was discussed earlier. The API wants us to ‘POST‘ our request. |
| –header apikey | This is the personal API key that is assigned to your account, which is used for authentication. |
| –header content-type | The next part is another header, ‘content-type: application/json‘ lets the server know that the data we will be sending with our request, will be formatted in JSON, a common format to structured text. |
| -d filter | With the ‘-d‘ we specify the data we will be sending, in JSON format. This is the actual search we want to perform. |
| -o tmgroup-domains.txt | The option ‘-o‘ is for output and tells cURL to save the output to a text file. |
The result is a long stream of lines that are thrown back at us, 1732 lines at the time of writing and that is where the last option comes in handy: Saving the output to a file. A part of the result can be seen here:
{
"records": [
{
"whois": {
"registrar": "CSC Corporate Domains, Inc.",
"expiresDate": 1789963200000,
"createdDate": 748670400000
},
"mail_provider": [
"Proofpoint, Inc."
],
"hostname": "cnn.com",
"host_provider": [
"Fastly"
],
"computed": {
"company_name": "Turner Broadcasting System, Inc."
},
"alexa_rank": 135
},
{
"whois": {
"registrar": "CSC Corporate Domains, Inc.",
"expiresDate": 1545411864000,
"createdDate": 945795864000
},
"mail_provider": [
"Google LLC"
],
"hostname": "cnnindonesia.com",
"host_provider": [
"PT. Detik Ini JUga"
],
"computed": {
"company_name": "Turner Broadcasting System, Inc."
},
"alexa_rank": 674
},
[ ... ]
Threat Intelligence
RiskIQ and PassiveTotal are among the leading sites when it comes to threat intelligence. Their API’s provide access to the vast amount of data they collect and analyse, and also provide DNS and Whois data. It is even possible via the endpoint ‘/enrichment/osint’ to gather all information they have about an online threat. Access to their API and documentation can be found here: https://api.riskiq.net/api/concepts.html and here: https://api.passivetotal.org/api/docs/
As an example, searching their database for information on the domain ‘rockradiostream.com’ returns a huge list of information, which looks like this:
sector035@Tuvok:~$ curl -u [account+key] 'https://api.passivetotal.org/v2/enrichment/osint?query=rockradiostream.com' | json_pp | more
{
"success" : true,
"results" : [
{
"tags" : [
"search-engine",
"twitter"
],
"source" : "twitter.com",
"inReport" : [],
"sourceUrl" : "https://twitter.com/hashtag/apt28?lang=en"
},
{
"tags" : [
"search-engine",
"he"
],
"source" : "bgp.he.net",
"sourceUrl" : "https://bgp.he.net/net/91.219.236.0/22",
"inReport" : []
},
{
"inReport" : [],
"sourceUrl" : "https://twitter.com/kyleehmke/status/1113085089909440513",
"source" : "twitter.com",
"tags" : [
"search-engine",
"twitter"
]
},
[ ... ]
Other Useful Web Sites
There are many other websites that can be used with cURL directly, or where an API is provided. Here is a short list of some websites. Some might be more useful than others:
- OpenCageData provides access to geocoding tools, providing coordinates for an address or the other way around: https://opencagedata.com/api
- Shodan provides access to their database via a REST API, which can be easily queried from the command line. More information here: https://developer.shodan.io/api
- Census also has a REST API to dive into the world of online devices like IoT. Information about how to extract the data can be found at: https://censys.io/api
- A very nice and versatile tool that can be used to practice with, is https://cli.fyi. By simple curl commands one can retrieve information about anything you can think of. From prices of Bitcoin to tags of online videos and from country information to hash values of strings.
- A tool to find information about email addresses is provided by https://emailrep.io. It returns information like blacklists, suspicious behaviour, data breaches and credential leaks, ability to be spoofed and connected profiles.
- Have I Been Pwned can also be reached via an API, giving access to breaches, pastes where information was published or unsafe passwords. All information needed to play with it can be found here: https://haveibeenpwned.com/API/v2
- Wigle is known for their WiFi database, but they also provide Bluetooth information, both can be queried using their API. Read about it via: https://api.wigle.net/swagger
- Some low resolution satellite imagery can be retrieved via an API here: https://agromonitoring.com/api/get. The main usage is for farmers to check the crops and weather via automation. A longer list of API’s for satellite imagery can be found over here: https://www.programmableweb.com/category/satellites/api
The command line can be very powerful and at times sending out a query via cURL can be a lot faster than waiting for your web browser to finish loading and displaying the requested information. Depending on your needs or the volume of data you are handling, it can be helpful to practice the usage of simple tools like cURL to retrieve information. If you have any interesting links that people should know about, feel free to share it with the community in a comment.