Migrated from our earlier blog: “Using AWS Services for password cracking!”

Amazon released its EC2 P3 instances on October 25, 2017. Those instances offered configurations with Nvidia Tesla V100 GPUs. Our purpose was to demonstrate how to create an on-demand password-auditing system and rent it by the hour for authorized engagements after password hashes had been retrieved. The design could also be extended into a cloud cluster using Elcomsoft Distributed Password Recovery instead of purchasing multiple GTX 1080 Ti cards and absorbing their power and cooling costs.

Our on-premises baseline

The local comparison system used the following hardware:

  • 2× Intel Xeon 2699-v3 processors: 36 cores and 72 logical CPUs, both water-cooled
  • 128 GB DDR4 ECC memory
  • Samsung 960 M.2 boot drive
  • RAID 5 volume with four 1 TB SSDs and a PCI-E 3.0 RAID controller
  • One water-cooled MSI Seahawk GTX 1080, or two GTX 1080 Ti cards from MSI and Zotac

Hashcat settings used locally

-OEnable optimized kernels (limits password length)
-oWrite cracked passwords to Found_passwords.txt
-w 3Use the high-performance workload profile
-a 3Use attack mode 3 (brute force)
-m 1000Specify NTLM as the hash type
Hashcat identifies NTLM hashes with mode 1000.

hashes.txt is the source file containing the password hashes. D:\hashcat-4.0\0_PW_LISTS\*.* is the local folder containing the wordlists to test. Our workflow used Hashcat with nearly 400 GB of wordlists before resorting to brute-force masks.

Wordlist attack

Hashcat64.exe -O -o Found_passwords.txt -w 3 -m 1000 hash.txt D:\hashcat-4.0\0_PW_LISTS\*.*

Basic mask variation

Hashcat64.exe -O -o Found_passwords.txt -w 3 -m 1000 hash.txt -a 3 ?u?l?l?l?l?l?a?a?a?a?a?a

Custom policy masks

For targeted brute-force testing, policygen can create masks that reflect an organization’s password requirements:

policygen --minlength=8 --maxlength=14 --mindigit=1 --minlower=1 --minupper=1 --maxspecial=1 -o Client_x.masks
Generating custom masks with policygen.
Verifying the number of generated password masks.

Brute force was treated as a last resort for NTLM hashes:

Hashcat64.exe -O -o Found_passwords.txt -w 3 -m 1000 hashes.txt -a 3
A local Hashcat session showing elapsed time, progress and recovered hashes.

Preparing an Amazon EC2 P3 instance

Before deploying the instance, the original process required an active Amazon EC2 account, enough account quota and Spot capacity in the intended AWS Region. The important request to support was for Spot Requests, not a general instance-count increase.

The original EC2 Limits screen used to verify or request Spot capacity.
Original 2020 pricing note

The article observed that an on-demand P3.16xlarge instance was approximately $25 per hour, while a Spot Request could range from approximately $4–$8 per hour. These figures are historical; confirm current AWS pricing and quota requirements before any deployment.

Creating the Spot Request

The original AWS console workflow selected a one-time request, the desired AMI and the P3 instance type, then set a maximum Spot price before submitting the request.

Spot Request configuration used in the original article.
Selecting the machine image and continuing the instance configuration.
AWS confirmation that the Spot Request was created.
The request appears as active while AWS works to fulfill capacity.

Connecting and checking the GPUs

Once the request was fulfilled and the instance was running, the original workflow connected over SSH, updated the system, installed the matching Linux headers and checked the detected PCI hardware:

sudo su -
sudo apt-get update
sudo apt-get install linux-headers-$(uname -r)
lspci
PCI device output showing the Nvidia GPUs presented to the P3 instance.

Install the CUDA packages

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_6.5-14_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1404_6.5-14_amd64.deb
sudo apt-get install cuda -y

The Nvidia utilities and Hashcat device inventory were then used to validate the driver installation and available GPUs:

nvidia-smi
hashcat -I
nvidia-smi reports the Tesla V100 devices and their driver state.
Hashcat device inventory, part one.
Hashcat device inventory, part two.

Install Hashcat 4.0

sudo apt-get install p7zip-full -y
wget https://hashcat.net/files/hashcat-4.0.0.7z
7z x hashcat-4.0.0.7z
cd hashcat-4.0.0
./hashcat64.bin -O -o Found_passwords.txt -w 3 -m 1000 hashes.txt -a 3
Hashcat recognizing a Tesla V100-SXM2-16GB device.

Performance comparison

The final tests compared an eight-GPU Tesla V100 P3 configuration, a four-GPU Tesla V100 configuration and one local GTX 1080. The screenshots below preserve the original test output, including session status, elapsed time, progress and per-device speed.

Eight Tesla V100 GPUs

Original Hashcat output for the eight-GPU Tesla V100 configuration.

Four Tesla V100 GPUs

Original Hashcat output for the four-GPU Tesla V100 configuration.

One GTX 1080

Original Hashcat output for the local GTX 1080 comparison.

Performance statistics

The original article’s performance comparison chart.

The cloud approach made high-end GPU capacity available only when it was needed, without purchasing and maintaining equivalent hardware locally. As with all credential-auditing work, use an approved scope, protect extracted hashes and terminate cloud resources as soon as testing is complete.

Need an authorized password assessment?

Deep Security can help evaluate password resilience, Active Directory exposure and credential policy in a controlled engagement.

Request an assessment