Yeah, a low profile USB stick will be miles better for installing the operating system on than the old hard drive in the computer. Something like this would do the job and unlike a conventional USB stick, it won’t stick out and be an inconvenience:
https://www.sandisk.com.au/home/usb-flash/cruzer-fit
You would need to change settings in the BIOS to give priority to USB booting first, and then perhaps wipe the internal hard disk to prevent conflicts.