I learned again the value of this configuration – rebooted and my public interface did not come up, but the local one was fine. Simply telnet’d in from another local machine, fixed the issue and rebooted and now all is well. My configs worked for a year – not nor more.. in /etc/sysconfig/networking/devices/ifcfg-eth1 – the issue was the BOOTPROTO – was either giving the mac as 00:00:00:00:00:00 or nothing at all.
GATEWAY=192.168.1.100
TYPE=Ethernet
DEVICE=eth0
HWADDR=00:11:f7:77:34:9e
BOOTPROTO=none < -------- changed to static
NETMASK=255.255.255.0
IPADDR=192.168.1.12
ONBOOT=yes
USERCTL=no
IPV6INIT=no
PEERDNS=yeschanged BOOTPROTO=static:
GATEWAY=192.168.1.100
TYPE=Ethernet
DEVICE=eth0
HWADDR=00:11:f7:77:34:9e
BOOTPROTO=static
NETMASK=255.255.255.0
IPADDR=192.168.1.12
ONBOOT=yes
USERCTL=no
IPV6INIT=no
PEERDNS=yes.. then rebooted and all is well again. It seems a good test is if something survives 2 reboots, its golden. After one reboot the kernel/kudzu/other/undo its still “new” to the system for that boot (hey look, a new card! Lets autoconfigure…), after the second its no longer “new”, then it breaks on the second reboot (say the card did not configure, might be skipped). If some change survives a reboot, great – I test, and if successful reboot right away. If something survives 2 reboots, I generally consider the the fix reliable and can relax.
So why do this? If you have to servers, its easy, what about only one colocated server? What if do not have an IP KVM? Get a $69 Mikrotik router to use as a secondary local access device (LAD)- they use 5W of power and are the size of a small switch – just will need an extra power outlet – I am sure if you explain to your ISP how service calls will be avoided, they’ll let you use an extra outlet. 
I use this as a backup to the backup LAD – works well. Set 2 ports in bridged mode and it will act like a switch, the assign an IP to the bridge (not the individual interfaces).
The config tool system-config-network is good for a single port system – for dual/triple NIC for local, DMZ and advanced networking, you need to do this by hand. The key entry is the GATEWAYDEV – this sets up the routing correctly – as seen in netstat -r
Here is a configuration for dual NIC with eth0 local and eth1 private – your names may be different – tweak as necessary.
/etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=host.example.com
GATEWAYDEV=eth1
MOUNTD_PORT=4002
/etc/sysconfig/networking/devices/eth0 - local network card
# Please read /usr/share/doc/initscripts-*/sysconfig.txt
# for the documentation of these parameters.
GATEWAY=192.168.10.1
TYPE=Ethernet
DEVICE=eth0
HWADDR=00:98:c7:16:77:43
BOOTPROTO=none
NETMASK=255.255.255.0
IPADDR=192.168.10.2
ONBOOT=yes
USERCTL=no
IPV6INIT=no
PEERDNS=yes/etc/sysconfig/networking/devices/eth1 - public network card
# nVidia Corporation MCP51 Ethernet Controller
DEVICE=eth1
BROADCAST=1.1.1.15
HWADDR=00:47:e1:6f:3e:27
IPADDR=1.1.1.4
NETMASK=255.255.255.240
NETWORK=1.1.1.0
ONBOOT=yes
TYPE=Ethernet
GATEWAY=1.1.1.1
USERCTL=no
IPV6INIT=no
PEERDNS=yesReboot your system – though a restart is nice service networking restart; service [firewall software] restart, if your changes survive a couple reboots you can be sure its correct. If this is for a live system in a far away datacenter, test it at home first, then one more time, get some coffee, then test again.
[root@host devices]# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
1.1.1.0 * 255.255.255.240 U 0 0 0 eth1
192.168.10.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth1
default 1.1.1.1 0.0.0.0 UG 0 0 0 eth1
