I simply changed the order of the entries in add-del.ldif. My file looks like this:
olcAccess: to attrs=userPassword,shadowLastChange by dn="cn=admin,dc=orca-central,dc=de" write by anonymous auth by self write by * none
olcAccess: to * by dn="cn=admin,dc=orca-central,dc=de" write by * read
olcAccess: to dn.base="" by * read
You will see, that I simply swapped the last two lines. That solved the problem for me. Using linenumbers didn't do the trick.
I used slapcat to find out the actual order of the acls in the database. It seems the delete command needs them in the same order, as they were entered.