Transferring files

Tip

Contact Secretariat System Administrators with HPC Questions!

vshanka@clemson.edu,jopoole@clemson.edu,madonay@clemson.edu

Contact IHG Research Cores with Research Questions!

ihgcores@clemson.edu

When transferring files between external sources and Secretariat, you have a few options.

If you have a small number of small files and you are not very comfortable working from the command line, A software like FileZilla may be a good option for you. (FileZilla facilitates file transfers within a graphical user interface (GUI) that may feel more familiar to novice programmers.) For larger files or larger numbers of files, you should use command line tools like scp and rsync.

Please find instructions on how to use each of the methods, below.

GUI (FileZilla)

Part 1: Connect

Download and install the desired softare (e.g. FileZilla client)
Launch the client
Open the “Site manager” – this is usually the upper left icon on the Toolbar

It resembles a cluster of three rectangular machines
Create a new site:
1. Click “New Site” and name it Secretariat
2. Set “Protocol” as “SFTP - SSH File Transfer Protocol” in the dropdown list
3. Type 130.127.173.136 in the “Host” box
4. Set “Logon Type” as “Interactive” in the dropdown list
  
  The “Logon Type” must be set to “Interactive” because this will allow you to authenticate using Duo.
5. Type your username in the “User” box
6. Click “OK” at the bottom of the window
  
  You will only need to create a new site the first time: After this, start from Step 5
Reopen the “Site manager” and select “Secretariat” in the list of sites in the left window
Click “Connect”, enter your password, and authenticate

First prompt will ask for your password; The second prompt will ask for either your mobile device or key to authenticate via Duo.
Trust the host: An “Unknown host key” window may appear with a message similar to the one below; Click “OK”

The server's host key is unknown. You have no guarantee that the server is the computer you think it is.

[...]

Trust this host and carry on connecting?

Output should print to the ‘Message log’

This is the panel below the ‘Toolbar’ and ‘Quickconnect’ sections at the top of the window.

When output similar to the following appears and files / directories are present in both the ‘Local site’ and the ‘Remote site’ panels, the connection has been successfully established.
```
Status:      Connected to 130.127.173.136
Status:      Retrieving directory listing...
Status:      Listing directory /home/[`username`]
Status:      Directory listing of "/home/[`username`]" successful
```

Connection established!

Part 2: Transfer

Once you have established a connection, you may initiate uploads and downloads between the two sites.

Upload: Local-to-remote
1. Locate the file(s) / folder(s) to upload
  
  Within the ‘Local site’ window, select the file(s) / folder(s)
2. Locate the destination folder
  
  Within the ‘Remote site’ window, select the destination folder
3. Right click the file(s) / folder(s) to upload; Select “Upload” in the drop down list that appears
  1. The transfer should initiate; You may track your progress in the “Transfer queue”
    
    This is the panel at the bottom of the window
  2. Once the files appear in the “Successful transfers” tab of the “Transfer queue”, the file has been successfully transferred from the local site to the remote site

Transfer complete!

Download: Remote-to-local
1. Locate the file(s) / folder(s) to download
  
  Within the ‘Remote site’ window, select the file(s) / folder(s)
2. Locate the destination folder
  
  Within the ‘Local site’ window, select the destination folder
3. Right click the file(s) / folder(s) to download; Select “Download” in the drop down list that appears
  1. The transfer should initiate; You may track your progress in the “Transfer queue”
    
    This is the panel at the bottom of the window
  2. Once the files appear in the “Successful transfers” tab of the “Transfer queue”, the file has been successfully transferred from the remote site to the local site

Transfer complete!

Command line (scp)

scp is a function used to securely copy files and is already installed on Secretariat. Please see the use-cases, below.

Local to remote

scp /path/to/file.txt username@130.127.173.136:/remote/directory/

where

/path/to/file.txt: file (including full path) to upload to Secretariat

username: your username to log into Secretariat

/remote/directory/: where to upload file on Secretariat

Remote to local

scp username@130.127.173.136:/path/to/file.txt /local/directory/

where

/path/to/file.txt: file (including full path) to download from Secretariat

username: your username to log into Secretariat

/local/directory/: where to download file on local machine

Note: rsync follows a similar convention to scp!