I would reconsider docker because if a specific application leaks some sort of shell access or system file access you’ll be protected out side of container host escalation.

Unrelated to security, I prefer docker because it leaves the server very clean if you remove different apps. Can also save time configuring more complex applications or applications that conflict with system libraries.

Add fail2ban on your list of applications it watches logs for invalid logins and puts them on firewall block rules after so many failed attempts.

I made the jump to a full server a few years ago and there’s some pretty high limits but you can get them spec’ed pretty low. Something like a dell r730 Single 8 core xenon 32gb ram and a couple tb of storage running 4-500$. They can be upgraded over time to be dual 16 core xenon 1tb ram and petabyte of storage.