You can install ISPConfig with Centos on a Pentium III or IV pc with at least 500 MB internal memory, so let's say a normal pc from some years old. For the amount of sites you want to host, you even do not need a huge hdd.
(My first Debian server with ISPConfig was an old IBM Pentium II with 384 MB and with my fast DSL line, i was not disappointed at all). I used that old IBM to teach about Linux and ISPConfig as well.
The speed of your websites does not only depend on the server-hardware. Also the speed of your DSL-connection is important. Especially the upload speed.
You also have to know that some ISP's are blocking port 25, because of their anti-spam policy. So sending email via your own mailserver, sometimes is not possible. For this reason i recommend to choose an ISP, which provides what you need.
If you're disapointed or do not want to have a server at home for any reason...of course i like to host your sites as well for a special price.