Student Sync: Automating Your Student Sync

John Dill -

Once you've uploaded your first student file to Handshake using the Importer you can automate your upload!

Overview

Handshake sets up an isolated location for each school in an amazon s3 bucket for you to upload your student files.

  1. University uploads Data to Handshake's s3 bucket/school directory
  2. Handshake's Importer tool will analyze your student data and return you any feedback via email or in the UI
  3. Handshake importer tool will import your student data into Handshake and it will either update your existing student data or create the new students imported

IMPORTANT NOTES:

  • These s3 directories are write-only so any attempts to GET/LIST contents would result in 'Access Denied' Errors.
  • We support all standard methods of uploading to AWS S3 that only require PUT access
  • We do not support FTP/SFTP due to the required LIST access.
  • Region must be set to us-east-1 for upload to work.
 

Step 1: Setup & Prerequisites

We're going to be using Amazon CLI tools, this is the best tool for interacting with AWS s3 for automating your upload. If you cannot use this tool please let our support know.

If you already have python you can install with the following, see here for more details.

pip install --upgrade --user awscli

If you don't have python, you can install using the bundled installer here

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
# with sudo 
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

# without sudo (assumes ~/bin is in your PATH)
./awscli-bundle/install -b ~/bin/aws

If you don't have python and cannot install it to use the AWS CLI, go here for more information.

 

Step 2: Obtain your Credentials and Configure

Please contact your partnership manager to receive the AWS credentials for uploading data to Handshake. Unless otherwise instructed by the Handshake team, you should use “us-east-1” as your region.

Next run the following and enter in your AWS Access Key and Secret Key

aws configure
AWS Access Key ID [****************JRTA]:
AWS Secret Access Key [****************RZ7O]:
Default region name [us-east-1]:
Default output format [None]:
 

Step 3: Upload Your First File

Now you're ready to upload your file to Handshake! Note that you should have received an email giving you the exact AWS s3 Path to upload your file to, if you have not please reach out to the support team

 

Upload Syntax

aws s3 cp [/path/your_local_file] s3://handshake-importer-uploads/[your folder]/[yyyymmdd]_users.csv
 

Example Upload and Response

aws s3 cp 20140410_users.csv s3://handshake-importer-uploads/importer-production-hudson_university/20140410_users.csv

upload: to s3://handshake-importer-uploads/importer-production-hudson_university/20140410_users.csv

The AWS S3 API will respond with the document ID if the file was successfully transferred, otherwise it will respond with an error. You can also check the command exit code to determine it was successful.

If you wish to do a test run, simply send a file and our data team will verify that it is received and ready for production.

 

Step 4: Validate the file

  1. Login to the Importer and check to see that your file was accepted by the importer.

Here are some common problems you might've encountered if you do not see the file:

The file is not a true CSV
Fix: Double check that the file you are trying to upload can be uploaded manually to the importer and processed successfully

The script you're running isn't successfully uploading the file to your s3 bucket
Fix: Double check your script path and ensure it is returning successfully and giving you back the path of the uploaded object

  1. Ensure the data being uploaded passes all the analyzers, if it does not you'll have to make adjustments to your extract script before we can turn on auto run for you.
 

Step 5: Scheduling your Regular Upload

Afer you've validated everything looks good, we recommend using cron to schedule the upload.

Pick a weekly, monthly, or semesterly date to do your upload, in general the format is:

 +---------------- minute (0 - 59)
 |  +------------- hour (0 - 23)
 |  |  +---------- day of month (1 - 31)
 |  |  |  +------- month (1 - 12)
 |  |  |  |  +---- day of week (0 - 6) (Sunday=0 or 7)
 |  |  |  |  |
 *  *  *  *  *  command to be executed

Example:

5 8 * * Sat   aws s3 cp users.csv s3://handshake-importer-uploads/importer-production-hudson_university/`date +"%Y-%m-%d"`_users.csv  >/dev/null 2>&1
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request