SSH host key on clustered nodes

[Hyunyoung Park]

We have a new SFTP server that connects with the engine. This server forces host key checking for SSH connection and recently we discovered the connection had been failing after our failover since the host key didn’t match with what they previously had.

Server team suggested us to have 1 host key on both nodes instead of having a unique host key on each node. Also was told that this SFTP server can’t accept multiple host keys either.  

Since I wasn’t sure what other impacts the engine might have, I contacted infor support and didn’t get a good answer from them.

“We suggest 1 key per node.

As far as risk go, it is uncertain for this is more of an OS type question.

You may find more info on google.

If doing anonymous SFTp no key needed.”

Does anyone know if there is risk or impacts on the engine for making SSH host key the same on both active and inactive node?

Thank you.

Jen

Topic

[David Barr]

I’m a little confused. Is the SFTP server running on the same box as Cloverleaf? Are you referring to failover of the Cloverleaf system or the SFTP server?

Which software is complaining about the host key mismatch?

[Hyunyoung Park]

No, SFTP server is a different server from cloverleaf box.

Cloverleaf is on clustered env and we did failover Cloverleaf.

SFTP server had the SSH host key of the cloverleaf active node at the time they initially configured the connection with us. After we failed over the cloverleaf box, now the host key of the cloverleaf box doesn’t match with the one SFTP server knows.

For now, SFTP updated SSH host key to our current active node’s but this issue will occur when we failover next time again.

[David Barr]

This still doesn’t make sense. SSH host keys belong to SSH servers, not to SSH clients. Cloverleaf shouldn’t have an SSH host key. Maybe you’re talking about a client key that Cloverleaf is presenting to the SSH server.

I’m not familiar with the SFTP key configuration in Cloverleaf; we use a different product for SFTP transfers. How are you changing the key in Cloverleaf to correct the issue? Is this through the GUI?

[Jason Alexander]

It’s not clear to me which piece is actually acting as the server for the connection but it should not matter.  Regardless, the ssh connection handling should be handled by a protocol layer, just like the TCP layer of the OS handles your TCP connections.

In other words, the SSH keys are in use by the protocol layer in whatever OS you are on and not by the Cloverleaf engine itself.

I’m going to venture a guess that your failover uses name/IP swapping so that your remote hosts don’t have to change the configuration of any of their clients.  That means that they’re going to try connecting to you using exactly the same credentials (i.e. combination of hostname/IP and SSH key) after you fail over.  This will (and *should*) fail unless you take explicit steps to make it work.  From the client’s point of view this looks exactly like a man-in-the-middle attack so you *want* it to fail unless you know exaclty what happened.

In the case of a failover, you know what happened and you want them to be able to connect to this new box so you want to configure it to accept exactly the same credentials.  In a UNIX environment that’s trivial.  You take the entire .ssh directory from the host and account(s) that have been working and you copy them exactly to the new host in the same accounts.  You can do functionally the same thing on a Windows environment and I believe that it is almost as easy but I am not familiar with the process.  This matches the suggestion that your Server group gave.

The risk is fairly straight forward and it has everything to do with what SSH keys are and nothing to do with Cloverleaf.  An SSH key is literally the key to your front door.  You have two machines and if you set them up to use the same keys then if one machine is compromised the other is automatically compromised as well.  Other than that, in the Unix environment at least, SSH doesn’t care if you connect from your primary to your standby even if the keys are the same.  It doesn’t ask “so, is the key I am sending the same as the key I would accept” or anything like that.

[David Barr]

Jason Alexander wrote:

I’m going to venture a guess that your failover uses name/IP swapping so that your remote hosts don’t have to change the configuration of any of their clients.

[Russ Ross]

The issue you are describing is similar or the same as what I’ve have been living with so far and have mixed emotions about.

I also have similar issues not only when Cloverleaf does a fail-over but whenever a foreign system does a fail-over that our cloverleaf server SFTP logs into.

Whenever this happens I’ve been manually editing the ~/.ssh/known_hosts file to remove the offending entry and let it recreate itself.

This undesireable side effect of SSH/sftp is a result of its intended design as a way to prevent someone from pretending to be an authorized foreign system.

So the cup half full version of this is it forces the communication of when a fail-over has occured.

I have also observed this symptom with some OS patches when there isn’t any fail-over.

In other words, SSH/SFTP is diligently on the look out for suspicous changes.

My mixed emotions are that I don’t want the annoyance but I’m not sure how much I reduce the overall SSH/SFTP security when I try to prevent some of the false positives.

Russ Ross
RussRoss318@gmail.com

[David Barr]

Russ Ross wrote:

Whenever this happens I’ve been manually editing the ~/.ssh/known_hosts file to remove the offending entry and let it recreate itself.

If you’re using the openssh client, you can have two entries in your known_hosts file for the same address. I usually comment out the offending entry, let it recreate itself with the new key, then I uncomment the first entry.  From that point forward it will work with either key.

[Russ Ross]

Another challenge with SFTP I thought I would share, is a method I used to restrict all SFTP users to their home directories expect for those I explicitly open up.

By default SFTP gives open access to the cloverleaf server, which is a point I felt was being overlooked when security kept focusing on that SFTP is encrypted.

So for my own comfort level I searched the internet for a solution that gave me control to restirct user access to their home directory or an appropriatly restricted directory tree.

It took a while for me to find a reasonable solution that wasn’t too extensive or required a compile to implement.

Then I located a solution that utilizes a configuration file on AIX called

/etc/ssh/sshd_config

which I have added the following entires thus far, which seems to have done the trick:


########################################################################
# Begin of entries maintained by HUB team and system admins
# to define which SFTP users are not restricted to their $HOME directory
########################################################################
#
# Here are the commands to run as root once making changes to this file
#
# [code]
########################################################################
# Begin of entries maintained by HUB team and system admins
# to define which SFTP users are not restricted to their $HOME directory
########################################################################
#
# Here are the commands to run as root once making changes to this file
#
#

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Code:

If you’re using the openssh client, you can have two entries in your known_hosts file for the same address. I usually comment out the offending entry, let it recreate itself with the new key, then I uncomment the first entry.  From that point forward it will work with either key.

David, in our case we can have multiple entires and the one that works will get used so I went with a similar approach until I learned OS patches will mandate the entries be recreated and I just gave up fighting its intended design for the time being.

Russ Ross
RussRoss318@gmail.com

[Hyunyoung Park]

David, yes you are right, the client key that Cloverleaf is presenting. I probably used an incorrect term since I don’t have much knowledge about SFTP configuration.  🙂

We didn’t make any change in Cloverleaf. SFTP server side updated their key to match the current cloverleaf box key.

There are 2 sftp servers that Cloverleaf interacts. One server ever gives us no problem but this new Sterling server which I have referred as SFTP server so far seems to have somewhat restrictions and limitations.

Thank you everyone for your inputs! I’ll discuss more with our AIX group and SFTP group.

[Author]

Replies