I'm fairly certain a lot of the bonding modes use a gratuitous arp to advise the network which NIC is in control. This would require a switched environment where all NICs see each other. Connecting NIC to NIC removes that. I'm not certain that is the cause of your issues, but I'd be suspicious of it. Maybe you could try mode 1, which, IIRC, is a simple failover system.
To prove your hardware, you could revert to single ethX settings and see if your direct cable works in that configuration.