How I Lost the Securitytrails #ReconMaster Contest, and How You Can Win: Edge-Case Recon Ideas

Note: The audio version doesn't include code or commands. Those parts of the post can be seen in the text version. A while back, Securitytrails announced that they would be running a contest dubbed "Recon Master", the aim of which is to find hostnames that resolve to an IP-v4 address that haven't already been found by Securitytrails. As it had been a while since I flexed my recon muscles, that sounded very interesting to me. These days, the majority of my asset discovery phase is spent literally just using Securitytrails, so this would force me to think outside of the box and stop being so lazy. Also, the whopping $5,000 prize for first place was pretty appealing. At the beginning of the contest, I was in first place by a wide margin for quite some time. Now, the game is in full swing, I'm in seventh place, and I've given up on winning. Frankly, I've spent too much time on it, but during that time I've come up with a few interesting ideas that I believe are useful enough to share in a blog post, so here we are. Much like bug bounties, winning this contest is all about creativity. SecurityTrails is already extremely good at finding hostnames because that is a huge part of their core business. In order to defy the odds and discover hostnames that the SecurityTrails team hasn't, a big chunk of creativity will need to be applied. Let's jump into the techniques I tried, from failures and successes to successes that ended up failing, you'll see what I mean. Streamlining the submission process Amass is great and all, but currently it's the only DN-S enumeration tool I'm aware of that has the Securitytrails submit endpoint built right into it. The contest page actually advertised this, which means that a whole lot of people will simply be using Amass to discover hostnames. I figured that if I wanted to discover hostnames that nobody else has discovered, it would be wise for me to avoid using the same methods and tools. Upgrading haktrails The first thing I did was build the submit endpoint into haktrails, a tool I had already written to query Securitytrails data. I didn't add the submit endpoint to any of the documentation because I thought that this would give me a slight advantage in the contest. Actually, I still haven't added the documentation, but I'll show you how to use it right here: This will take each line in subdomains.txt and submit it in chunks of 1 million lines until there are no lines left to submit. Now that I had an easy way to submit data, I could literally just pipe the output of any subdomain enumeration tool, or any list of hostnames directly into haktrails to submit it. Generating gzip files I figured that there would be times when it would be more efficient to submit gzip files due to the sheer amount of data to upload, so I wrote another quick Golang tool which takes lines of text as input and gzips them into multiple gzip files with x lines each. I called it gzipsplit. Usage is something like this: This command would create multiple gzip files with one million subdomains in each. Note that this is different from splitting a gzip file into multiple files using the split command. If you use the split command, you end up with smaller split files, but they aren't valid gzip files unless you join them back together, so you can't extract the text from them. Submitting bug bounty recon (or not) My guess is that every bug bounty hunter and their dog will be submitting their bug bounty recon data to win this contest. That means that every bug bounty hunter and their dog will be submitting the same data. The ideas I started thinking about things to submit. In particular I wanted to find hostnames that were most likely to be fresh, so that I might beat SecurityTrails to it. The first thing that came to mind was Certstream. Certstream What is Certstream? Their website says it best: Certstream is an intelligence feed that gives you real-time updates from the Certificate Transparency Log network, allowing you to use it ...