I did not skip the postfix setup. My mail would not work at all if it was the case. This is the part of the setup I skipped:
If you do not want to use ISPConfig, configure Postfix to deliver emails to a user's Maildir*:
postconf -e 'home_mailbox = Maildir/'
postconf -e 'mailbox_command ='
As you can specifically see this is only to use the maildir.