I've added soft_bounce = yes
to postfix's /etc/postfix/main.cf
It'll cause postfix not to send any bounces. Which is what I need.
However, instead of dropping the email; postfix will defer it. So the problematic emails will stay in queues.
Clearly I won't be able to use this setting for extended period. However, it helps to avoid being blacklisted by SpamCop for the time being
So I'll keep looking. If anyone knows a more permanent solution, please do share it with us as well.