Should Clustered HA nodes advertise node or pkg IP?

[Matthew Rasmussen]

Should HA (Clustered) nodes their advertise node’s IP, or their package’s IP as their source address?  Ours advertises the node…

Our corporate network-security best-practices, require us to only advertise the package address on our VPN ACLs (access control lists.)  However, our servers (CL 5.8.5.0P on HPUX) advertise their individual node addresses.  Anyway, we have been complying with the policy, and using a bind address for every VPN thread in our enterprise, to work around the issue.  

Has anyone had this problem, or know of an HPUX OS or HA setting, to effectively masquerade each individual host-node’s source IP, as the package/cluster’s IP?

Topic

[Russ Ross]

I just finished performing a fail-over and saw your post this Sunday 5/5/2013 at 3:30 AM CST, so how fitting to answer a question about vitural applicaiton service address that I just got finished dealing with on my HA fail-over.

I’m on AIX and each OS has its own odd behavor.

Doing a hcitcptest along with an IP trace is how I tell what actually goes on.

In my case the default behavor before making any chantges is that inbound traffic can come in on either the application cluster address or the persistant hostname adress.

But all outbound traffic will go out on the persistant hostname that makes it look to the world it is coming from the wrong place and gets blocked by any firewalls we are going thru.

The specific steps may vary but the concept to get around this problem is essentially the same as what I will describe for AIX.

The trick is to reorder the priority of the IPs so the virtual service address (application address) of the HA cluster is treated as the default priority.

In the old days when we used physically assigned NICs, I would put logic in my fail-over scripts to remove the NIC that had the persistant hostname IP and readd it thus changing its priority below the application vritual service address.

Now days on a more virtual envirnment using ether channel I do a alias remove then an alias add of the persistant hostname IP and the virtual service address (appilcation IP) becomes number one and looks transparent to the world for everything used on that box.

I have heard of a quick work around that another hosptial told me they had used on an individual interface at the NetConfig level to quickly get around this issue but I haven’t used it myself because it only impacts a single interface and I need it to work for everything running on the machine.

NetConfig -> thread properties -> Protocol –> Host Name or IP address (which is the choice just below Host & Port in Cloverleaf 5.6)

Russ Ross
RussRoss318@gmail.com

[Matthew Rasmussen]

We’ll give this a shot.  Thanks so much for your help Russ!

[Russ Ross]

Another thing about application cluster IP transparency has to do with the cloverleaf host server.

Since I need to use the host server before I have the IP transparency in place during the early parts of a scratch install like I’m doing with cloverleaf 6.0 right now, and for added safety if the IP transparency were to malfunction, I add the following lines to the end of the $HCIROOT/server/server.ini file

Code:

[firewall]
rmi_exported_server_port=mdahub8sna

where mdahub8sna is the application virutal cluster name, which now allows me to always specify mdahub8sna in my cloverleaf IDE/GUI no matter what physical node my HA cluster is running on.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

When I was dragging this post into my clovertech HACMP folder I see I have 2 others saved you might have an interest in looking at:

https://usspvlclovertch2.infor.com/viewtopic.php?p=24554#24554” class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?p=24554#24554

https://usspvlclovertch2.infor.com/viewtopic.php?p=22642#22642” class=”bbcode_url”>https://usspvlclovertch2.infor.com/viewtopic.php?p=22642#22642

Russ Ross
RussRoss318@gmail.com

[Matthew Rasmussen]

We are currently using the same approach you described in the last option – using the “Local_IP” or bind address at the NetConfig level.  But like you, we need all network activity to be bound to the service address, especially for protocols that do not offer the bind option.

We do have an entry in our server.ini, but it references the IP rather than the DNS name of the service address.  That should serve the same purpose though, right?

[firewall]

rmi_exported_server_port=10.23.33.106

Looks like our NICs are not teamed, so we have separate ether channels for each address:

10.23.33.0         10.23.33.102      UHSb      0         0 en13     –      –   =>

10.23.33.0         10.23.33.106      UHSb      0         0 en12     –      –   =>

106 (en12) is our service address, and 102 (en13) is our persistent address.  Since the addresses are on different ether channels, we couldn’t use aliases – we would have to follow your second suggestion, and drop en13, then bring en13 back up, so en12 would have top priority, right?  

Can we just do this on startup/failover…

ifconfig en13 down

ifconfig en13 up

or do we actually have to detatch/reattach?

I read both of the other posts – will definitely use the core of your debug/audit routine, to test before/after whatever change we use.  Very Nice!

Thanks again Russ

[Russ Ross]

First do you have a HA cluster to test with because this is the sort of thing I work on during my scratch build of the HA cluster and before I run cloverleaf on the box?

To run thru the 3 different approaches you mention would be better done when the you have the machine all to yourself.

I would first learn how to use hcitcptest as a server on the server that you want to configure with a transparent application service IP of the HA cluster.

Then be able to run the hcitcpstest tool on another box as a client and connect the two from with the command prompt on each of the 2 boxes.

Learn how to turn on IP trace and turn it to trace server IP/port you have choosen to test.

Next type at the command prompt for the client hcitcptest prompt a test message and hit enter.

You will see it appear in the server hcitcptest command window, then type your dummy reply in the hictcptest server window.

At this point your IP trace has captured what comes in and goes out so stop the IP trace ahd hcitcptest tools and convert the trace into human readable form if necessary.

Then read the IP trace to see if both the inbound message and outbound reply are using the desired virtual IP.

This is the method I use to confirm I have it working as needed, so be able to perform this before embarking on making changes.

Once you have become comfortable with being able to do this you will be able to proceed trying changes and using this test to determine if you accomplished what you want.

By the way, hcitpstest is a standalone perl script in cloverleaf for AIX and can be run from any other AIX host even if cloverleaf is not installed on it.

Now that you are ready to proceed, I would test the ifconfig alias method first and if not successfull then the ifconfigi down/up method.

One thing to keep in mind that access to your host IP is temporarily not useable via terminal access when running these commands until you add it back so these have to be run from within a script so the add back command can be accomplished while terminal access to the host IP does not exist.

I’m not sure if detach/reattach is applicable when using ehter channel since it is more virtual in nature.

I only scratched the surface of understadning and stopped once the ifconfig alias accomplished the desired outcome of my hcitcptest.

The ifconfig alias I think still has a chance of working in your case and should be tried to find out.

I did not go into this detail earlier but what you really are trying to accomplish is making sure the virtual application HA cluster IP is tied to the default gateway and not the host IP.

When you remove the host IP via ipconfig alias that leaves the remaining virtual application HA cluster IP to become married to the default gateway, then when adding back the host IP it has essentially traded places with the virtual IP.

Your observation is correct that we group more than one physical NIC together in our etherchannel setup to accomplish transparent load balance and increase bandwith even though it shows up as one logical NIC, thus making my etherchannel architecture  a bit different than yours.

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

You asked the question

We do have an entry in our server.ini, but it references the IP rather than the DNS name of the service address.  That should serve the same purpose though, right?

[firewall]

rmi_exported_server_port=10.23.33.106

I would think this would server your purpose and could be tested using the IDe by specifying the vitual application name and see if it talks to the hsotserver.

Russ Ross
RussRoss318@gmail.com

[Matthew Rasmussen]

Hi Ross, just checking back in…

We recently implemented the change you suggested in test, and it looks like we’re now advertising our VIP!  We just had to restart the ether channel that had the proper IP assigned, and when we brought it back up, it took the wheel just like you said it would.  Our midrange guys asked me to pass along their compliments.

THANK YOU so much!!

[Russ Ross]

I got some hopeful news from our sys admin group regarding IBM HA 7.1 that runs under AIX.

Turns out our sys admins have been passing along my complaints to IBM and asking for an enhancment request for about the last 10 years to have IBM HA under AIX have a configurable setting to make all outbound traffic use the service address after a fail-over.

If that is true, the day of playing tricks could be coming to an end once we get to HA 7.1.

Maybe your admins could start making a similar request of HP.

Russ Ross
RussRoss318@gmail.com

[Matthew Rasmussen]

Thanks again for the info Russ.  I’ve reached out to our guys with your find, to see if they can get any additional info.  I’ll post back If I hear more – like my boss always says, iron sharpens iron.  Thank again Russ!

[Author]

Replies