Student Sync: Automation

Teena Blaydes -

 

How to Automate

  1. The best way to do this is through AWS s3.
  2. Once your IT team has configured the above, we request a couple test files to verify that it's working correctly.
    • Please remember to indicate *test* somewhere within the file name
  3. When the test is successful and we've agreed to start processing the incoming files, we'll watch it run for a few cycles (depending on schedule) before we enable auto-run
    • This is to prevent and quickly address any errors that may occur
  4. After the live-file cycles have completed and all errors resolved, we enable auto-run and you're good to go!



If this is automated, why do I have to manually send it through the command line prompt each time we want to upload a student file?

  • Using aws cli allows your IT team the opportunity to set up a script or scheduled task that will automatically feed it through to us on the decided schedule. If you'd like to do this, please reach out to your IT team. 



The following requirements are necessary before we can automatically process your sync:

  • .CSV must meet the basic requirements here
  • Sync has no errors
    • This is confirmed by using the importer analysis feature
      • (directly upload your file to the importer, wait for the analysis to complete, evaluate feedback, and correct as necessary)
      • All headers must be correct
      • All preset values must be correct
  • No duplicates are found
    • auth_identifier
    • email_address
    • username
    • card_id

 

How often can we sync our students?

  • Weekly
  • Monthly
  • Semesterly

We cannot automatically process a student sync multiple times within a 24-hour period. 

 

 

Can we sync our alumni regularly too?

  • If you wish to sync your alumni regularly, we do ask that you do not process this more than once a month and only send the last 6 months of alumni. 

  • If for any reason you have to upload more than that, please reach out to importer@joinhandshake.com for further assistance.



What happens if an error occurs later on, after we've automated our file?

  • We have failsafes in place to ensure no detrimental errors can occur
  • If there are any errors pre-run, autorun will cease and alert us
    • We will investigate and see if we can resolve it on our end before reaching out to your team
  • The email recipients should also receive a regular analysis of the file once it hits our servers; They will get a notification of the file failing or errors occurring


How to get set up on the email recipient list:

This will send you an email with each s3 transfer, once the analysis completes on our side (pre-run)

Send a message to importer@joinhandshake.com requesting that you be put on the recipient list. Our importer team will confirm that you're an acceptable party to be added and inform you of the addition once complete.

    • No sensitive information can be transferred via automatic email analysis



What about post-run feedback - How can I receive any student records that error?

We are working on a response log feature.

In the meantime, our team monitors post-run errors. If there are errors we cannot resolve or need your attention, we'll reach out to get them corrected. If you have any questions or concerns, please reach out to your importer team; We're more than happy to assist!

 

Note: Only student files may be transferred via AWS s3. We also will *not* process any files through s3 until your team has completed testing and informed us that it's okay to enable production; This will allow your files to automatically process.

 

 

Why do you use Amazon S3?

AWS S3 is an industry standard for data storage and transfer. In order to provide the best security of your sensitive data, we use a rigorous access control policy and encryption in transit and at rest. Data can only be uploaded not read from AWS s3 and will only sit there for a short time before it is ingested into Handshake by an automated process.

What setup do I have do on my end?

You do not have to setup an s3 bucket or interact with amazon in any way other then using one of the many tools available to upload to Handshake’s s3 bucket and folder for your team. We will provide you with the folder, access id, secret key, and required bucket and prefix for uploading your file.

What if I can't install python to use the aws cli?

The AWS cli is just interacting with the AWS s3 rest api-- If you'd like, you can use any tool that can talk to a RESTful service such as curl. This isn't recommended because it opens up more opportunities for errors. See here for an example: http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash

What sort of encryption is used with the data?

We use server side 256 bit AES. “Amazon S3 Server Side Encryption employs strong multi-factor encryption. Amazon S3 encrypts each object with a unique key. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 Server Side Encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.” All data sent in transit is transferred over TLS.

 

How do I transfer the data to my secure folder?

The easiest way to transfer data is with the AWS CLI (http://aws.amazon.com/cli/)

  • Install the AWS CLI
  • You should have received an ‘Access Key ID’ and a ‘Secret Access Key’ from the Handshake team.
  • Type “aws configure” in your command line prompt and follow the prompts to enter your Key ID and Secret Key mentioned above.
    • These should be entered precisely as received
  • Unless otherwise instructed by the Handshake team, you should use “us-east-1” as your region.
  • To help alleviate issues with sending your file to us, leave the output field as is
  • For more information see this article.

Once you have the CLI set up, uploading your data is as simple as issuing the following command:

$aws s3 cp [/path/your_local_file] s3://handshake-importer-uploads/[your folder]/[[yyyymmdd]_users.csv]

Handshake has one aws bucket ‘handshake-importer-uploads’ and every school has a unique folder which comes after the bucket which they only have write access to upload files.

Example upload:

aws s3 cp 20140410_users.csv s3://handshake-importer-uploads/importer-production-hudson_university/20140410_users.csv


Example response:

upload: to s3://handshake-importer-uploads/importer-production-hudson_university/20140410_users.csv


If you wish to do a test run, simply send a file and our data team will verify that it is received and ready for production.

 

Can I whitelist s3 though my firewall?

There are a lot of IPs for s3 but they are updated here on a regular basis: https://ip-ranges.amazonaws.com/ip-ranges.json (we only use the us-east-1 region)

If you do want to whitelist by domain:

https://s3.amazonaws.com/*

 

How can I update my previous aws cli setup if new keys are provided?

Just type "aws configure" in your command line prompt, then proceed with the steps as you did during the initial setup.

 

How will I know my file was successfully received by Handshake?

 

Upon request, we can enable your email for receipts of s3 transfers. This just confirms that we've received it and it let's you know if there are any issues (Example: There's a required header that's missing: username). 

 

Can I use s3 to upload other file types? (like majors or contacts)

This type of transfer is only intended for student syncs. If you'd like to upload another file type, please use the Importer App. If you do not have access to the Importer Application, please reach out to your Account Manager.

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request