Regarding Iptables, I am running Bastille, as part of ISPConfig and using that as I always have in the past.
I found something interesting though, if I ifdown eth0 and allow eth1 to use its gateway, eth1 works as it should. If I ifdown eth1 and allow eth0 to use its gateway, it works too as expected. The problem comes when trying to force eth1 to use eth0's gateway.
Another thing I found, although I don't know the exact topology of the external network, is that the external network routing changed recently and that even though the private addresses I have to work with, 192.168.114.2 and 192.168.115.2 appear to be on the same network, they are actually on different networks coming through different switches.
Is it even possible for packets to come in on one network and go out through another??
I think what I am going to do is ask the network admin to route both public IPs through the same internal network and then run both to one network adapter. That seems a LOT easier to work with than what I'm trying to deal with now.
Of course it would help if I knew more than just enough to be dangerous.
One other strange thing, I have another thread here, in the scripting forum, where I was having problems with PHP's "gethostbyname()" method. When I tried the test of only enabling one adapter at a time, that problem went away. When I tried to enable both adapters again, the problem came back!
Some days it just doesn't pay to wake up in the morning!