SFTP configuration

[Barb Dozier]

We are attempting to configure a SFTP connection to one of our local physician group clinics.  We are running version 5.4.1 on a AIX server.  I do not a lot of experience with certs and private keys outside of the ones we need for Global Monitor and the GUI.  Can any one point me in the right direction for generating these keys/certs and the associated configuration within my fileset-ftps protocol thread?

Thanks, Barb Dozier

Topic

[Rob Abbott]

Please note the difference between FTPS and SFTP.

FTPS = FTP over SSL (secure socket layer)

SFTP = FTP over SSH (secure shell)

Cloverleaf supports FTPS.  It does not support SFTP.

On unix you will most likely need to use openssl to generate keys.  More info at http://openssl.org

There are many documents out on the internet explaining how to generate keys using openssl.  I suggest you do a google search or ask your FTP partner for more information.

Rob Abbott
Cloverleaf Emeritus

[Barb Dozier]

THanks for the clarification.  I didn’t realize the difference.  I will start googling.

Barb

[Zac Votrain]

Hey Barb

If they are wanting SFTP we were able to do a small script with “expect” to work with those who wanted to.  It is not pretty but it works.

[Russ Ross]

I found it usefull to setup a couple of in-house servers to play with SSH to figure out how it works first.

After you get that working then talking to the vendor would be your next step.

The key exchange is not a requirement and the vendor might just require a password.

When you make your first SSH connection with the vendor do it manually until that works before proceeding to automating it.

Here are some scripts I wrote that are running in production.

This is a real example you can reference to see how to use EXPECT commands within HCITCL to automate an interactive SSH/SFTP session.

Here is the module that does the SSH/SFTP/put/delete and is called _sftp_put_delete.tcl (so far I haven’t written a SFTP get script but the day is approaching I’m sure )

#!/usr/bin/ksh
# this line to escape the next line from the tcl interpreter
exec hcitcl “$0” “$@”

#
# Begin Module Header ==============================================================================
#
#——
# Name:
#——
#
# _sftp_put_delete.tcl
#
#———
# Purpose:
#———
#
# This is a generic argument driven stand alone TCL script.
#
# That uses SFTP to put file(s) in a local directory
# into a remote directory on a foriegn system
# then delete the file(s) from the local directory
#
#——–
# Inputs:
#——–
#
# parameter_file
#
# [code]#!/usr/bin/ksh
# this line to escape the next line from the tcl interpreter
exec hcitcl “$0” “$@”

#
# Begin Module Header ==============================================================================
#
#

---

# Name:
#

---

#
# _sftp_put_delete.tcl
#
#

---

# Purpose:
#

---

#
# This is a generic argument driven stand alone TCL script.
#
# That uses SFTP to put file(s) in a local directory
# into a remote directory on a foriegn system
# then delete the file(s) from the local directory
#
#

---

# Inputs:
#

---

#
# parameter_file
#
#

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

Well the day did come that I had to write some other SFTP scripts prior to upgrading to a version of Cloverleaf that has the SFTP protocol.

#!/usr/bin/ksh
# this line to escape the next line from the tcl interpreter
exec hcitcl “$0” “$@”

#
# Begin Module Header ==============================================================================
#
#——
# Name:
#——
#
# _sftp_put.tcl
#
#———
# Purpose:
#———
#
# This is a generic argument driven stand alone TCL script.
#
# That uses SFTP to put file(s) in a local directory
# into a remote directory on a foriegn system
# but will not delete the file(s) from the local directory
#
#——–
# Inputs:
#——–
#
# parameter_file
#
# [code]#!/usr/bin/ksh
# this line to escape the next line from the tcl interpreter
exec hcitcl “$0” “$@”

#
# Begin Module Header ==============================================================================
#
#

---

# Name:
#

---

#
# _sftp_put.tcl
#
#

---

# Purpose:
#

---

#
# This is a generic argument driven stand alone TCL script.
#
# That uses SFTP to put file(s) in a local directory
# into a remote directory on a foriegn system
# but will not delete the file(s) from the local directory
#
#

---

# Inputs:
#

---

#
# parameter_file
#
#

Russ Ross
RussRoss318@gmail.com

[Russ Ross]

here is the _sftp_get_delete.tcl script:

Code:

#!/usr/bin/ksh
# this line to escape the next line from the tcl interpreter
exec hcitcl “$0” “$@”

#
# Begin Module Header ==============================================================================
#
#——
# Name:
#——
#
# _sftp_get_delete.tcl
#
#———
# Purpose:
#———
#
# This is a generic argument driven stand alone TCL script.
#
# That uses SFTP to get file(s) from a remote directory
# on a foriegn system and place a copy of them into a local directory
# on then delete the file(s) from the remote directory.
#
#——–
# Inputs:
#——–
#
# parameter_file
#
#    the complete file name of the file that will define all the set variable
#    commands that get dynamically executed by this script.
#    Below are the variables that need to be defined in the parameter file:
#
#        host_name
#        user_id
#        pass_word
#        remote_dir
#        remote_files
#        local_dir
#
#        lock_file
#
#        sftp_prompt
#        pass_word_prompt
#        pass_word_prompt_timeout
#        sftp_get_timeout
#
#——-
# Notes:
#——-
#
# All the debug and error output has a unique ID to help locate where they are in this proc!!!
#
#    The ID to be used for new gets debug statements is ##### Dxx
#    The ID to be used for new gets error statements is ##### Exx
#
# sftp only transfers files in binary mode and has no option to transfer files in ascii mode
#    
# Refer to batch_051 for example of normal usage which is as follows:
#
#     nice -20 _sftp_get_delete.tcl $HCISITEDIR/scripts/_051/.${MDA_BATCH_ENV}_051__sftp_get_delete.set_variables
#
# This script can handle multiple files by passing something like “*.charge”
#
# Each of the file(s) will be copied via SFTP/get one at a time to be sure
# each of them were successfully copied via SFTP/get from the remoter server
# before deleting them from the remote directory.
#
# The file(s) will have the same name on the local system that they had on the remote system,
# so if you want the files to have different names then you will have to rename them after this script completes.
#
#——-
#
# Here is a way to run SFTP commands in batch mode but still get prompted for an interactive login which is required by our enterprise security group:
# (if commands ilike get in my_file_of_sftp_commands error out then the batch file will terminate and I can echo $? to determine failure or success)
#
#     sftp   -o “batchmode no”   -b $my_file_of_sftp_commands   $user_id@$hostname
#
#         FYI:
#
#         -b batchfile
#
#         Batch mode reads a series of commands from an input
#         batchfile instead of stdin.  Since it lacks user
#         interaction it typically would be used in conjunction with non-
#         interactive authentication.  However, those places that require a
#         interactive login prompt can get around this by using the switch
#         -o “batchmode no”.  A batchfile of `-‘ may be
#         used to indicate standard input.  sftp will abort if
#         if any of the following commands in my_file_of_sftp_commands fail:
#
#         get
#         put
#         rename
#         ln
#         rm
#         mkdir
#         chdir
#         ls
#         lchdir
#         chmod
#         chown
#         chgrp
#         lpwd
#         df
#         lmkdir
#
#         Termination on error can be
#         suppressed on a command by command basis by prefixing
#         the command with a `-‘ character ( for example, -rm /tmp/blah* )
#
# URL to get exist status code maybe but probably have to use batch mode without interactive login to get this to work
# but I have to use interactive login method due to security policy so I resorted to screen scraping using Expect to check for success
#
#     http://objectmix.com/tcl/246764-how-get-exit-status-spawned-process.html
#
#     spawn ssh root@abc.com
#     expect …
#
#     then how can i get the exit status of ssh?
#
#     Short answer: set status [lindex [wait] 3]
#
#     Longer answer:
#
#     set wait_list [wait]
#
#     foreach {pid spawn_id flag status} $wait_list break
#     set additional_info [lrange $wait_list 4 end]
#
#     if {$flag == 0} {
#         set exit_status $status
#         foreach {errname sig desc} $additional_info break
#
#     } else {
#        set errno $status
#        # read about “errorCode” to parse $additional_info
#        # http://www.tcl.tk/man/tcl8.4/TclCmd/tclvars.htm#M18
#        # and KBK’s contribution to http://wiki.tcl.tk/exec
#     }
#
#     ——————
#     # Expect man page: http://www.tcl.tk/man/expect5.31/expect.1.html
#     ——————
#
#     wait [args]
#     delays until a spawned process (or the current process if none is named) terminates.
#     wait normally returns a list of four integers. The first integer is the pid of the process that was waited upon.
#     The second integer is the corresponding spawn id. The third integer is -1 if an operating system error occurred, or 0 otherwise.
#     If the third integer was 0, the fourth integer is the status returned by the spawned process.
#     If the third integer was -1, the fourth integer is the value of errno set by the operating system. The global variable errorCode is also set.
#
#     Additional elements may appear at the end of the return value from wait. An optional fifth element identifies a class of information.
#     Currently, the only possible value for this element is CHILDKILLED in which case the next two values are the C-style signal name and a short textual description.
#    
#     The -i flag declares the process to wait corresponding to the named spawn_id (NOT the process id).
#     Inside a SIGCHLD handler, it is possible to wait for any spawned process by using the spawn id -1.
#
#     The -nowait flag causes the wait to return immediately with the indication of a successful wait.
#     When the process exits (later), it will automatically disappear without the need for an explicit wait.
#
#     The wait command may also be used wait for a forked process using the arguments “-i -1”.
#     Unlike its use with spawned processes, this command can be executed at any time.
#     There is no control over which process is reaped. However, the return value can be checked for the process id.
#
#———
# History:
#———
#
# 2011.05.10 Russ Ross
#          – wrote initial version
#
#
# 2011.08.26 Russ Ross
#          – removed unecessary parameters that would of been used for alerts
#            because I decided to have the wrapper to this script check the exit status
#            and do the alert if necessary.
#
# End of Module Header =============================================================================

#—————
# initialization
#—————

  set module_name {_sftp_get_delete.tcl}

  set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]

  set host_name                “”
  set user_id                  “”
  set pass_word                “”
  set remote_dir               “”
  set remote_files             “”

  set local_dir                “”

  set lock_file                “”

  set sftp_prompt              “”
  set pass_word_prompt         “”
  set pass_word_prompt_timeout “”
  set sftp_get_timeout         “”

#—————————————————–
# read the parameter file to dynamically set variables
#—————————————————–

  set parameter_file [lindex $argv 0]

  if [catch {open $parameter_file r} parameter_file_handle] {
     puts stderr “”
     puts stderr “( $module_name ) $time_stamp”
     puts stderr “( $module_name ) ***** ERROR *****   can not open parameter_file”
     puts stderr “”
     puts stderr ”     ( $parameter_file )”
     puts stderr “”
     exit 1
  } else {
     while 1 {
        set next_set_command [gets $parameter_file_handle]
        if [eof $parameter_file_handle] {break}
        # puts stdout “( $module_name ) $next_set_command”
        eval $next_set_command
     }
     close $parameter_file_handle
  }

#—————————————
# exit if this script is already running
#—————————————

  set lock_file_handle [open $lock_file w]
  if [flock -nowait $lock_file_handle] {
     # no other copies running
     puts stdout “”
     puts stdout “( $module_name ) started at $time_stamp”
  } else {
     puts stderr “”
     puts stderr “$time_stamp”
     puts stderr “exiting module ( $module_name ) because another copy is aleady running”
     exit 1
  }

#—————————–
# validate the input arguments
#—————————–

# ??? wait and do later

#——————————————–
# evaluate the commands in the parameter file
# which set the necessary input variables
# that control the behavior of this scripts
#——————————————

  if [catch {open $parameter_file r} parameter_file_handle] {
     puts stderr “”
     puts stderr “( $module_name ) $time_stamp”
     puts stderr “( $module_name ) ***** ERROR *****   can not open parameter_file ( $parameter_file )”
     exit 1
  } else {
     while 1 {
        set next_set_command [gets $parameter_file_handle]
        if [eof $parameter_file_handle] {break}
        puts stdout “( $module_name ) $next_set_command”
        eval $next_set_command
     }
     close $parameter_file_handle
  }

#————————————————————————————
# get a list of local directory before picking up any files that might overwrite them
#————————————————————————————

  set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
  if ![file isdirectory $local_dir] {
     puts “nFATAL ERROR E.0R.01 – ( $local_dir ) is not a directory at $time_stampn”
     exit 1
  }

  puts “nn     ##### D.0R.01 ##### Here is a long listing of the local file(s) before doing any SFTP/get from the remote server”
  puts     ”     ##### D.0R.01 ##### that might get overwritten that reside in local directory ( $local_dir )n”
  puts     ”     ##### D.0R.01 ##### ls -l $local_dir/$remote_filesn”
  catch [system ls -l $local_dir/$remote_files 2>/dev/null]
  puts “”

#——————————————
# Use Expect commands to SFTP the next file
#——————————————

  set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
  puts “”
  puts “————————————-”
  puts “$time_stamp”
  puts “About to sftp/connect/login to server ( $host_name )”
  puts “————————————-”
  puts “”

  #—————————-
  # sftp login to remote server
  #—————————-

  ### spawn sftp -v $user_id@$host_name

  spawn sftp $user_id@$host_name
  # set exit_status [lindex [wait] 3]

  #spawn sftp -o “batchmode no” -b /cloverleaf/cis6.0/integrator/test_batch_051/scripts/_051/x.x  $user_id@$host_name
  #spawn sftp -o “batchmode no” -b /cloverleaf/cis6.0/integrator/test_batch_051/scripts/_051/x.x  oftp_000@mdahub8sna >/hcitest/x.x

  # spawn sftp -o “batchmode no” -b – $user_id@$host_name

  match_max 100000

  #——————————–
  # provide pass word when prompted
  #——————————–

  set timeout $prompt_timeout
  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.00.01 – spawn sftp EOF encountered at $time_stamp”
        puts “while while waiting for password prompt from $host_namen”
        exit 1
     }
     “$pass_word_prompt” {
        send “$pass_wordr”
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.00.01 ##### got password prompt from $host_name at $time_stampn”
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.00.02 – timed out at $time_stamp while waiting for password prompt from $host_namen”
        exit 1
     }
  }

  #———————————————————–
  # cd to remote directory where you will get the file(s) from
  #———————————————————–

  set timeout $prompt_timeout
  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0A.01 – spawn sftp EOF encountered at $time_stamp”
        puts “while while waiting for sftp prompt from $host_namen”
        exit 1
     }
     “$sftp_prompt” {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.0A.01 ##### got sftp_prompt at $time_stamp”
        puts     ”     ##### D.0A.01 ##### and will proceed to cd to remote directory ( $remote_dir )”
        puts     ”     ##### D.0A.01 ##### where the file(s) to get ( $remote_files )”
        puts     ”     ##### D.0A.01 ##### are expected to be locatedn”
        send “cd $remote_dirr”
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0A.02 – timed out while waiting for sftp prompt at $time_stampn”
        exit 1
     }
  }

  #————————————————————————————–
  # display a long listing of all the files in the remote directory on the foreign system
  #————————————————————————————–

  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.01 – spawn sftp EOF encountered at $time_stamp”
        puts “while while waiting for sftp prompt from $host_namen”
        exit 1
     }
     “Couldn’t stat remote file: No such file or directory” {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.02 – Couldn’t cd to remote directory ( $remote_dir ) at $time_stampn”
        exit 1
     }
     “$sftp_prompt” {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.0B.01 ##### got sftp_prompt at $time_stamp”
        puts     ”     ##### D.0B.01 ##### and will proceed to get a long listing of ALL the remote file(s)”
        puts     ”     ##### D.0B.01 ##### currently in remote directory ( $remote_dir )n”
        send “ls -alr”
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.03 – timed out while waiting for sftp prompt at $time_stampn”
        exit 1
     }
  }

  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.01 – spawn sftp EOF encountered at $time_stamp”
        puts “while while waiting for sftp prompt from $host_namen”
        exit 1
     }
     “$sftp_prompt” {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.0B.02 ##### got sftp_prompt at $time_stamp”
        puts     ”     ##### D.0B.02 ##### and will proceed to get a list of the remote file(s) speciifed”
        puts     ”     ##### D.0B.02 ( $remote_files )”
        puts     ”     ##### D.0B.02 ##### that need to be picked upn”
        send “ls -1 $remote_filesr”
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.04 – timed out while waiting for sftp prompt at $time_stampn”
        exit 1
     }
  }

  #———————————————————————-
  # get a percise list of the remote file names that need to be picked up
  #———————————————————————-

  set expect_out_buffer “”

  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.05 – spawn sftp EOF encountered at $time_stamp”
        puts “while while waiting for sftp prompt from $host_namen”
        exit 1
     }
     “$sftp_prompt” {
        set expect_out_buffer $expect_out(buffer)
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.0B.03 ##### got sftp_prompt at $time_stamp”
        puts     ”     ##### D.0B.03 ##### and will proceed to prep the expect_out_buffer leaving just the persice file names of interest”
        puts     ”     ##### D.0B.03 ##### that need to be picked upn”
        # normally would send something here to get another sftp_prompt but will do it later”
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.0B.06 – timed out while waiting for sftp prompt at $time_stampn”
        exit 1
     }
  }

  #———————————————————————————————————–
  # prep the expect_out_buffer
  # by getting rid of newlines and creating a list by spliting on carriage returns
  # and getting rid of the first and last element in the list, leaving just the percise file names of interest
  #———————————————————————————————————–

  set files_to_get      “”
  set files_to_get “[split [string map {n {}} $expect_out_buffer] r]”
  lvarpop files_to_get 0
  lvarpop files_to_get end
  if {    [lindex $files_to_get 0] == {.}  
       || [lindex $files_to_get 0] == {./}  
       || [lindex $files_to_get 0] == {..}  
       || [lindex $files_to_get 0] == {../}
     } { lvarpop files_to_get 0 }
  if {    [lindex $files_to_get 0] == {.}  
       || [lindex $files_to_get 0] == {./}  
       || [lindex $files_to_get 0] == {..}  
       || [lindex $files_to_get 0] == {../}
     } { lvarpop files_to_get 0 }
  # check for the case for no matching file names specified
  if { [lindex $files_to_get end] == {} } {
      set files_to_get “”
      puts “nFATAL ERROR E.0B.07 – NO MATCHING FILES FOUNDn”
      exit 1
  }

  puts “n”
  puts “—————————————————–”
  puts “File(s) on the foriegn system that will get picked up”
  puts “—————————————————–”
  puts “n”

  foreach next_file_to_get $files_to_get {
      puts $next_file_to_get
  }

  ##############################################################
  # Now that we are logged in via sftp
  # start the looping logic to sftp get each of the files_to_get
  ##############################################################

  puts “n”
  send “# sent this comment to get another sftp_promptr”
  puts “n”

  foreach next_file_to_get $files_to_get {

     puts “n”
     puts ”     —————————————”
     puts ”     About to loop over the next file to get ( $next_file_to_get )”
     puts ”     —————————————”
     puts “n”

     #————————————————————–
     # show listing of remote directory before getting the next file
     #————————————————————–
 
     set timeout $prompt_timeout
     expect {
        eof {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0C.01 – spawn sftp EOF encountered at $time_stamp”
           puts “while while waiting for sftp prompt from $host_namen”
           exit 1
        }
        “No such file or directory” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0C.02 – Couldn’t cd to remote directory at $time_stampn”
           exit 1
        }
        “$sftp_prompt” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nn     ##### D.0C.01 ##### got sftp_prompt at $time_stamp”
           puts     ”     ##### D.0C.01 ##### and will proceed to show listing of remote directory ( $remote_dir )”
           puts     ”     ##### D.0C.01 ##### before getting the next file ( $next_file_to_get )n”
           send “ls -alr”
        }
        timeout {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0C.03 – timed out while waiting for sftp prompt at $time_stampn”
           exit 1
        }
     }

     #——————————————
     # get the next file from remote sftp server
     #——————————————
 
     set timeout $prompt_timeout
     expect {
        eof {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.01.01 – spawn sftp EOF encountered at $time_stamp”
           puts “while waiting for sftp prompt from $host_namen”
           exit 1
        }
        “No such file or directory” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.01.02 – Couldn’t list remote directory at $time_stampn”
           exit 1
        }
        “$sftp_prompt” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nn     ##### D.01.01 ##### got sftp_prompt at $time_stamp”
           puts     ”     ##### D.01.01 ##### and will proceed to get next file ( $next_file_to_get )”
           puts     ”     ##### D.01.01 ##### and place it in the local directory ( $local_dir )n”
           send “get   $next_file_to_get   $local_dir/$next_file_to_getr”
        }
        timeout {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.01.03 – timed out while waiting for sftp prompt at $time_stampn”
           exit 1
        }
     }
 
     #————————————————————————–
     # if it looks like we got the remote file delete it from the foreign system
     #————————————————————————–
 
     set timeout $sftp_get_timeout
     expect {
        eof {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0D.01 – spawn sftp EOF encountered at $time_stamp”
           puts “while while waiting for sftp prompt from $host_namen”
           exit 1
        }
        “Couldn’t open local file * for writing: No such file or directory” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.02.02 – Couldn’t write file to local directory at $time_stampn”
           exit 1
        }
        “Fetching*$sftp_prompt” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nn     ##### D.0D.01 ##### got sftp_prompt at $time_stamp”
           puts     ”     ##### D.0D.01 ##### and will proceed to delete the remote file ( $next_file_to_get )”
           puts     ”     ##### D.0D.01 ##### from the foriegn systemn”
           # delete the file once we think it was retrieved okay
           send “rm $next_file_to_getr”
        }
        timeout {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0D.03 – timed out while waiting for sftp prompt at $time_stampn”
           exit 1
        }
     }
 
     #————————————————————————————–
     # show listing of remote directory after deleting the next file from the foriegn system
     #————————————————————————————–
 
     set timeout $prompt_timeout
     expect {
        eof {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0E.01 – spawn sftp EOF encountered at $time_stamp”
           puts “while while waiting for sftp prompt from $host_namen”
           exit 1
        }
        “Couldn’t delete file: No such file or directory” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nnFATAL ERROR E.0E.02 – Couldn’t delete remote file ( $next_file_to_get ) at $time_stampn”
           exit 1
        }
        “Removing*$sftp_prompt” {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nn     ##### D.0E.01 ##### got sftp_prompt at $time_stamp”
           puts     ”     ##### D.0E.01 ##### and will proceed to show listing of remote directory ( $remote_dir )”
           puts     ”     ##### D.0E.01 ##### after deleting the next file ( $next_file_to_get )n”
           send “ls -lr”
        }
        timeout {
           set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
           puts “nFATAL ERROR E.0E.03 – timed out while waiting for sftp prompt at $time_stampn”
           exit 1
        }
     }

  ##########################################
  # loop back up to process next_file_to_get
  ##########################################

  }

  #————————————–
  # proceed to terminate the spawned sftp
  #————————————–

  set timeout $prompt_timeout
  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.03.01 – spawn sftp EOF encountered at $time_stamp”
        puts “while while waiting for sftp prompt from $host_namen”
        exit 1
     }
     “No such file or directory” {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.03.03 – Couldn’t list remote directory at $time_stampn”
        exit 1
     }
     “$sftp_prompt” {
        puts “n”
        puts ”     ————————————”
        puts ”     Done looping over the file(s) to get ( $files_to_get )”
        puts ”     ————————————”
        puts “n”
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.03.01 ##### got sftp_prompt at $time_stamp”
        puts     ”     ##### D.03.01 ##### and will proceed to terminate spawned sftpn”
        send “byer”
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.03.04 – timed out while waiting for sftp prompt at $time_stampn”
        exit 1
     }
  }

#———————————————————
# sftp checks for spawn EOF and will terminate prematurely
# unless you add extra expect command
#———————————————————

  set timeout $prompt_timeout
  expect {
     eof {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nn     ##### D.04.01 ##### spawn sftp EOF encountered at $time_stamp”
        puts     ”     ##### D.04.01 ##### which is expected and indicates a successful termination of the spawned sftpn”
        exit 0
     }
     “$sftp_prompt” {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.04.01 – expected EOF but got sftp prompt at $time_stampn”
        exit 1
     }
     timeout {
        set time_stamp [fmtclock [getclock] “%a %b %d %Y %r “]
        puts “nFATAL ERROR E.03.02 – timed out while waiting for sftp prompt at $time_stampn”
        exit 1
     }
  }

puts “DEBUG_x”
exit 0

Here is an example of the calling script ( iftp_051_sftp_get_del.ksh ) that passing the args via a parameter file:

Code:

#!/usr/bin/ksh

ftp_log=/ftp/$MDA_BATCH_ENV/iftp_051/.ftp_log

nice -20 _sftp_get_delete.tcl
        $HCISITEDIR/scripts/_051/.${MDA_BATCH_ENV}_051__sftp_get_delete.set_variables
        > $ftp_log
###         2> /dev/null

exit_status=$?
if [ “$exit_status” != “0” ]
then
 
  ftp_log_copy=”$HCISITEDIR/Alerts/oftp_051_sftp_get_log__`date +%C%y.%m.%d__%H:%M:%S`”
  cp $ftp_log $ftp_log_copy

  alert_message=”     fatal error detected,”
  alert_message=”${alert_message}nn     $HCISITE alert – (iftp_051_sftp_get_del.ksh) fatal error detected,”
  alert_message=”${alert_message}nn     EXIT STATUS ($exit_status) returned by _sftp_get_delete.tcl”
  alert_message=”${alert_message}nn     HUB on-call person needs to check this ftp_log file for details”
  alert_message=”${alert_message}nn     $ftp_log_copy”

  echo “”
  echo “$alert_message”
  echo “”

  send_alert_message.ksh
     “${MDA_BATCH_ENV}_iftp_051_email”
     “$alert_message”

  exit $exit_status

else
  echo “”
  echo “Directory listing of /ftp/$MDA_BATCH_ENV/iftp_051/in”
  ls -l /ftp/$MDA_BATCH_ENV/iftp_051/in
  echo “”
fi

Here is the layout of the parameter file ( .prod_batch_051__sftp_get_delete.set_variables )

Code:

# these are the prod_batch_051 parameters for the SFTP get/delete script
#
# to manually sftp do the following
#
#    sftp 551040@ssh.mytelevox.com
#    or
#    sftp me@10.123.123.123

set   host_name            “10.123.123.123”
set   user_id              “me”
set   pass_word            “secret”
set   remote_dir           “.”
set   remote_files         {HCEXPORT_*_PROD_BATCH_051.TXT}
set   local_dir            “/ftp/$env(MDA_BATCH_ENV)/iftp_051/in”

set   debug_flag           “Y”

set   lock_file            “$HciSiteDir/scripts/_051/.$env(MDA_BATCH_ENV)_051__sftp_get_delete.lock”
set   kill_process_file    “$HciSiteDir/scripts/_051/.$env(MDA_BATCH_ENV)_051__sftp_get_delete.kill”

set   sftp_prompt              “sftp> ”
set   pass_word_prompt         “password: ”
set   prompt_timeout           “120”
set   sftp_get_timeout         “900”

Russ Ross
RussRoss318@gmail.com

[Author]

Replies