That was so easy!!!!
To save someone else the headache, three days lost time and fear of losing my job, not to mention my mind, here's how I did it: (Anyone who has corrections to make, just post it and I'll update this)
First, although it is unlikely your 'Nix box does not have the capability to do this, it is a good idea to check to make sure.
At the command prompt execute:
If you end up in the expected directory, this will likely work for you. If not, google "iproute2" and you should find what you need to enable it.
The situation is that you've got two, or more network cards. How do you make sure that the networks connected to each don't clash while at the same time make sure that packets received on network "0" are replied to using network "0" and same for network "1", "2" and so on?
The scripts/files we will be working with are:
The last three are create for each adapter with the "ADAPTER ID" being the number of the adapter.
To begin, we set up configurations for each NIC/IP/eth[ADAPTER ID]
They are found in
They are named "ifcfg-eth0", "ifcfg-eth1" and so on and beginning at 0 and increasing up to the number of NICs you have, minus 1.
First though, you will need to know the MAC addresses of your Network Interface Cards (NIC). Unfortunately, all the common methods for determining the MAC address of the adapters on your system assume the network is already set up.
One cheap and dirty way is to install "NetworkManager" only long enough to to get the MAC address of each of the cards and then uninstall it and forever wipe it from your system.
Note: In this walk-through I use brackets and capitalized names to indicate what the contents of the bracket should be replaced with. Throughout this walk-through, all the bracketed terms are consistent so if you see [GATEWAY] in the ifcfg-eth[ADAPTER ID] section, you can know that the same value is used in the "route" section for the same NIC/IP/eth[ADAPTER ID].
Use whatever editor you prefer although I prefer "nano" and so will use that for this walk-through,
creates the first configuration script and its contents are :
# Adapter id
DEVICE=eth[ADAPTER ID, 0 -> ~, ex. eth0, eth1, eth2 ...]
# Adapter type
# Run at boot time
# Could be DHCP, "static" or "none" but "static" or "none" if you are reading this.
# MAC Address of the adapter.
# Replace the value shown with
# your NIC's MAC address
# "IPADDR", "NETMASK", "GATEWAY", "NETWORK",
# and "BROADCAST" will be provided by your
# ISP or network administrator
Once you have saved that, by hitting [ctrl-x] -> [y] -> [enter] in the nano editor, connect the network cable to the card, if not already connected and type
That initializes the network interface identified as eth0. You may want to wait a couple of minutes before testing it because it can take some time for it to actually come up and definitely wait for a command prompt while it tries to connect to whichever network it was told to.
Then try making a connection to test it out either from within the box you are working on or preferably, from outside so you can test everything at one time.
There are dozens of ways to test the connection including "ping", which I do NOT suggest but instead, prefer trying to telnet to a service I know should be running for a more realistic test. So:
telnet [IP ADDRESS] [SERVICE PORT]
If you use "80" for "SERVICE PORT" it will try to connect to an http service on port 80 at the IP being tested.
Depending on what the server is set up to reply with when an IP based request is made, you may get a page of html, a page of stuff that flies by leaving you with an empty terminal screen or just about anything. But, you will definitely know that it is working because it won't time out.
To terminate the test cleanly, type
and hit enter.
If you want to take that interface down for some reason, just type
ifdown eth[ADAPTER ID]
and it will close. The "I" and "F" in "ifdown" and "ifup" for that matter, stands for "InterFace".
This is useful for testing interfaces individually without having to worry about conflicts. Just make sure that you don't issue an ifdown on whatever IP you are connected to via SSH unless you are near the box you are working on, for obvious reasons.
Similarly, all interfaces can be restarted at the same time by issuing the command:
service network restart
but I don't suggest using that if you are remoting in in case there is an error and you lose the ability to connect at all. Hosting companies usually charge you money to re-setup your network if you take it down and can't get it back up so be careful.
Now, before you start configuring other interfaces, we will configure the routing and rules for that routing for the first NIC. This is MUCH easier than it sounds once you have seen it done.
There are two things we need to tell the system so that it knows what to do and when. The first is the route that traffic to and from a given interface should take and the second, when that route should be used.
Although there are a LOT of routing and rules one can apply, all we care about at this point is to make sure that packets that come in on a given interface for a given IP go out the same interface for the same IP.
To do this next part, we will do it in two steps. The first step, manually execute each instruction on the command line and then after making sure they do what they were supposed to do, we can commit them to initialization scripts so that the will be executed every time the server is restarted or a given interface is brought up after being down.
In practice though, once the first set of scripts are tested for the first interface, the rest can usually be easily copied from the original and then edited to add the interface specific elements.
So now at this point we have a working NIC/IP/Interface and it is time to set routing for it. We need to set two types of routing, to the interface and from the interface.
Also, we need somewhere to tell the system to look for this information and we do that in /etc/iproute2/rt_tables. But, don't edit it directly. We will pipe what we want into it.
What this routing table holds for each contained route is a unique number, [NUMBER] and a table name, [TABLE NAME]. The number indicates to the system which order the tables should be checked for routing a given packet.
If you execute
you will see that some default routing tables already exist and since their numbers are high, they are queried after anything we will be adding here.
Also, in these examples, if you see quotation marks in the "code" section, they must be added on the command line also. Labels in brackets should be replaced, including the brackets, with whatever information the label says it should contain.
To add a table pointer for the NIC we just set up, enter:
echo "[NUMBER] [TABLE NAME]" >> /etc/iproute2/rt_tables
"TABLE NAME" can be whatever you want, just so that it is unique and so that you make sure you use it for the rest of the commands/scripts that apply to a given NIC/IP/eth[ADAPTER ID].
As mentioned, we will be issuing the commands to perform the configuration manually, to test them and then, committing them to a script after testing. (X) indicates the number of the interface we are working with and is the same as the number used above when configuring the NIC itself.
So at the command line type:
ip route add [NETWORK]/24 dev eth[ADAPTER ID] src [IP ADDRESS] table [TABLE NAME]
That tells the system that packets that come from "src" [IP ADDRESS] go to the adapter [ADAPTER ID].
Now, we tell it the reverse, where do packets from that adapter go:
ip route add default via [GATEWAY] dev eth[ADAPTER ID] table [TABLE NAME]
If you think it looks a lot like a default gateway statement, it is. But, only by using iproute2 can one define different gateways
for different NICs and expect things to work.
So we have the routing for the first adapter set up and so now we will set up rules for when that routing should be used and here again, two rules will be create, one for packets to the adapter and one for packets from it.
At the command line type:
ip rule add from [IP ADDRESS]/32 table [TABLE NAME]
The "/32" restricts the rule to just that address. You could change that numer to increase the range of IPs affected but that is out of scope for this walk-through.
What that says is that traffic from [IP ADDRESS] should have the routing defined in [TABLE NAME] applied to it.
That may seem redundant but rules can be used to specify all sorts of cool things, such as types of service which would allow you to route differently depending on what service was being accessed. It is just that we only need a very simple rule but it is a rule we need none the less.
To specify a rule for packets coming from the interface, we execute
ip rule add to [IP ADDRESS] table [TABLE NAME]
To make sure that the commands have taken affect and aren't sitting in a cache somewhere, execute:
ip route flush cache
While we have been entering these commands, they have been taking effect as soon as the command was entered, or the "flush" command executed so whatever method you used to test the interface before, do it again now to make sure it still works.
But, don't issue an ifdown[NUMBER]/ifup[NUMBER] or restart the network because the commands entered as they have been only exist in memory/system and are gone on a restart.
To make them executed automagically on a reboot, network restart or ifdown/ifup, we will create network scripts and include the information into them.
The scripting "engine" is very simple. The routing script engine basically takes each line and adds "ip route add " to the beginning of it and then executes it so, to create the content for the scripts you can simply copy each command after the default part that is added by the scripting engine.
First, the routing script:
nano /etc/sysconfig/network-scripts/route-eth[ADAPTER ID]
[NETWORK] dev eth[ADAPTER ID] src [IP ADDRESS] table [TABLE NAME]
default via [GATEWAY] dev eth[ADAPTER ID] table [TABLE NAME]
And for the "rule" script:
nano /etc/sysconfig/network-scripts/rule-eth[ADAPTER ID]
from [IP ADDRESS]/32 table [TABLE NAME]
to [IP ADDRESS] table [TABLE NAME]
There, the first NIC/IP is fully configured. You can now execute:
and test to make sure everything still works.
From here on out, it is the same process over and over again to configure any other NICs/IPs you have. Whether you choose to do that step-by-step or use copy-n-paste is up to you although step-by-step is more likely to help you remember how to do it again in the future.
Either way, good luck and have fun.