Multipart Upload

Learn how to upload content into MediaSilo using multipart upload. Scroll down for instructions, along with a full code example that you can try for yourself.

Request a multipart upload ticket to upload files larger than 5GB to MediaSilo

Requesting a multipartUploadTicket returns all information necessary to perform a file upload directly to S3.

Once the file is uploaded to S3, an S3 url can be used to upload the file directly into MediaSilo.

Overview of Multipart Upload

Step 1: Create Multipart Upload Ticket

Create a multipart upload ticket using the endpoint POST /v3/assets/multipart/upload. The request payload takes in a single parameter called fileName You will find an example below. This will have all the information needed to upload the asset to S3. This includes fileName, sessionId, assetUrl, expiration, accessKey, secretKey, sessionToken, objectKey(path in S3 where the file is stored), and bucketName.

Step 2: Upload File into S3

Upload the asset to S3. Once the upload into S3 is complete, the assetUrl will be accessible to create the asset in your MediaSilo account. Although the assetUrl is returned in the multipart upload ticket response, the assetUrl is not accessible until the upload is complete. For uploads that will take longer than one hour, see Step 4 below on Refreshing Credentials.

Step 3: Create Asset

Using the assetUrl and projectId, create the asset inside the desired project using the endpoint POST /v3/assets. The projectId is the UUID of an EXISTING project inside of your MediaSilo account.

(Optional) Step 4: Refreshing Temporary Credentials

The Temporary Credentials consist of accessKey, secretKey, and sessionToken and will expire after 1 hour from the time they were created or refreshed. When requesting a multipart upload ticket, which includes temporary credentials, you only pass in the fileName. When you start uploading the file to S3, the sessionId is the reference to your specific upload session for an individual file. To refresh temporary credentials we use the same endpoint from Step 1 POST /v3/assets/multipart/upload. But this time we pass in the fileName and the sessionId. This way, the refreshed credentials have access to that specific upload session. The new refreshed credentials are interchangeable with the original credentials because they have the same exact access. There is a code example on refreshing credentials using the MediaSilo API further down this page.

Step 1: Create Multipart Upload Ticket

Let's assume we sent the following request to obtain a multipart upload ticket.

POST /v3/assets/multipart/upload
{
    "fileName": "FILE0013.MP4"
}

Then we receive the following payload in response:

//200 OK
{
    "fileName": "FILE0013.MP4",
    "assetUrl": "https://s3.amazonaws.com/s3-bucket/9XXXXXX-7XX9-4XX5-bXX2-b07XXXXXX40b/FILE0013.MP4",
    "expiration": 1507790546000,
    "accessKey": "ASIAJCPLGOLJYYTKUXRQ",
    "secretKey": "TWfwinpU3YRUXzIN/UoSCqsoobOYKBAIpWMC4Si3",
    "sessionId": "9XXXXXX-7XX9-4XX5-bXX2-b07XXXXXX40b",
    "sessionToken": "FQoDYXdzEOf//////////wEaDDwBJ3jkRXXXXXXXXXXXXXXXXXXXXXXXvu8hRdXhhb9KEV3VUTiKxHJGmgkT3nd2FbNRJrWYLVG7acZk+Qr4XXXXXXXXXXXXXXXXXXXXG2g6K2tlh/QgMBGLBQ+/nMli2cKuh/CKGSZ3KibspxO+0t35v5rXXXXXXXXXXXXXXXXXXXXXXpYyhih7u1Q8iRSxmtjMf9uRg7RIZa8usnPRZhuAIU9Kc7Y+pJ8gIjTRZQdx5bdaU6hPJQXXXXXXXXXXXXXXXR9XK8Io4m49KzeXV6vKgAd9OV77f0kUc3mOVK2PMrzvOWCTykAjvYeC9XXXXXXXXXXXXXXXXXdEvuiDPZoMAPZ8cXRy3ts5WEr8eVBIaEj+nEtFQrAww3NCytRSXXXXXXXXXX208pDwsVNSMeKMKB/M4F",
    "objectKey": "9XXXXXX-7XX9-4XX5-bXX2-b07XXXXXX40b/FILE0013.MP4",
    "bucketName": "s3-bucket"
}

This response contains all that we need to start the upload process. It also includes the assetUrl which will be used at the end of this process when we create the asset.

Step 2: Multipart Upload

Multipart Upload can be done in any number of languages (Java, PHP, Python, Node, etc.). In this example we take advantage of the AWS Javascript SDK to accomplish this. The AWS SDK is supported in many languages as well.

For a list of MediaSilo supported file types, take a look at our support documents.

function uploadFileToS3(multipartUploadTicketResponse) {
  let { fileName, assetUrl, expiration, accessKey, secretKey, sessionId, sessionToken, objectKey, bucketName } = multipartUploadTicketResponse;
  console.log('\nmultipartUploadTicketResponse:', multipartUploadTicketResponse); // Logging out the multipart upload ticket response.

  globalSessionId = multipartUploadTicketResponse.sessionId;// Setting the SessionId that will be used to refresh the AWS credentials. Again, this is a UUID that identifies your specific upload session.

  const uploadBody = fs.createReadStream(filePath); // Specify the path to the file on your computer.

  let credentials = new AWS.Credentials({ accessKeyId: multipartUploadTicketResponse.accessKey, secretAccessKey: multipartUploadTicketResponse.secretKey, sessionToken: multipartUploadTicketResponse.sessionToken }); // Credentials Object used to authenitcate the multipart upload requests made to S3. These credentials are used each time an individual part of the file is uploaded.
  let service = new AWS.S3({ credentials }); // Service object that allows us to interact with AWS S3.
  let managedUpload = new AWS.S3.ManagedUpload({ params: { Bucket: multipartUploadTicketResponse.bucketName, Key: multipartUploadTicketResponse.objectKey, Body: uploadBody, }, service }); // This object handles uploading the file to S3 using multipart upload.

  managedUpload.promise().then(function () { // Calling this starts the multipart upload process.
    createAsset(multipartUploadTicketResponse.assetUrl); //We do not want to create the asset until the upload to S3 in finished.
  });

  managedUpload.on("httpUploadProgress", (progress) => {
    console.log(progress); // Logging out each part of the file that is uploaded to S3.
  });
};
  1. The multipartUploadTicketObject allows us to create a credentials object. (Amazon Docs) http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html
  2. The credentials object allows us to create an S3 service object. (Amazon Docs) http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
  3. The S3 service object allows us to create a ManagedUpload object. (Amazon Docs) http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html

The managedUpload.promise().then() call will start and finish the multipart upload.

📘

Data Fields

All data needs to be passed along exactly as it was returned in the ticket request.

Step 3: Create Asset

Once the file is uploaded successfully you have 24 hours to create an asset before the content is deleted. create the asset using the API as usual.

POST /v3/assets
{
    "sourceUrl": "https://s3.amazonaws.com/s3-bucket/26a35865-282f-4aa2-b9af-1999d2c7835b/myfile.mov",
    "projectId": "0XXXCC-014B-2XX0-CF518DXXXX393E"
}

Step 4: Refreshing Credentials

This defines the refresh function for the AWS.Credentials object. It can now be used on any Credentials object in scope.

AWS.Credentials.prototype.refresh = function (callback) {
  var self = this;
  refreshTemporaryCredentials(function (err, response) {
    if (err) {
      callback(err);
    } else {
      console.log('\nCurrent SessionID: ', globalSessionId); // This will never change through out the entire upload of a single file.

      self.accessKeyId = response.data.accessKey; // Updating the accessKeyId.
      console.log('\nNew Access Key: ', response.data.accessKey)

      self.secretAccessKey = response.data.secretKey; // Updating the secretAccessKey.
      console.log('\nNew Secret Access Key: ', response.data.secretKey)

      self.sessionToken = response.data.sessionToken; // Updating the sessionToken.
      console.log('\nNew Session Token : ', response.data.sessionToken)

      callback();
    }
  });
};

This is the function that makes a POST request to the MediaSilo API to get the refreshed credentials. Remember that refreshed credentials contain a new accessKey, secretAccessKey, and sessionToken but retain the same sessionId. The new temporary credentials have the same access to the same upload session.

function refreshTemporaryCredentials(callback) {
  axios.post('https://api.shift.io/v3/assets/multipart/upload', { "fileName": fileName, "sessionId": globalSessionId }, {
    headers: {
      'x-key': apiKey,
      'x-secret': apiSecret,
      'Content-Type': 'application/json',
    },
  })
    .then(response => {
      console.log('\nFinished creating new temporary Credentials');
      callback(null, response);
    })
    .catch(error => {
      console.error('\nError creating multipart upload ticket:', error.response);
    });
}

This is the refresh interval that determines when to call the refresh function. We are refreshing every 30 minutes to ensure that no AWS requests are made with expired credentials.

  const refreshInterval = setInterval(() => {
    credentials.refresh((err) => {
      if (err) {
        console.error('\nError refreshing temporary credentials:', err);
      } else {
        console.log('\nTemporary Credentials have been successfully refreshed.\n');
      }
    });
  }, 1800000); //1,800,000 milliseconds = 30 minutes. Reducing this number will increase the frequency at which your temporary credentials are refreshed.
  //If this number exceeds 3,599,999(59m 59s) then your upload will fail before your temporary credentials are refreshed. 

Here is a full JavaScript example:

Note: This javascript example was created on a Mac Computer. The commands and instructions below reflect that.

  1. First you need an active MediaSilo Account.
  2. Then you need a set of API keys that can be used to authenticate into our API.
  3. Copy and Paste this code below into a file, and name it multipartUpload.js.
  4. Because you already have a MediaSilo Account and a set of API keys, you can fill in the variables apiKey and apiSecret.
  5. Get the project Id from any existing project in your account, and set that as your projectId:
    Follow steps a - c to get a project id if you don't know how:
    a. Log in to your MediaSilo Account. Make sure you have at least 1 project created.
    b. Double click into the project you want to create a new asset in.
    c. Look at the URL and you will notice it ends with something that looks like browse/da55a0d8-8be3-4f01-a828-3edad6604863. The UUID at the end of the URL is your project ID.
  6. Fill in the filePath variable with the path to a file that is stored locally on your computer.
  7. Make sure you have node installed on your computer. Installing node using brew install node will also install npm which is required in the next step.
  8. Open a terminal and navigate to the directory where you saved your multipartUpload.js file.
  9. Run the commandnpm install aws-sdk && npm install axios && npm install fs. This will install the required libraries needed to run this file.
  10. Run the command node multipartUpload.js.

🚧

IMPORTANT NOTE

This example is for teaching purposes. You should never hardcode sensitive data like passwords in live applications.

// This example was written using a Mac Computer.
// I ran `brew install node` to install the latest version of node. Which comes with npm
// Then i ran the commands `npm install aws-sdk && npm install axios && npm install fs`
// Save this file as multipartUpload.js
// Run this file using the latest version of Node with the command `node multipartUpload.js`
// Make sure to fill the variables on lines 15-18!

// Imports
const AWS = require('aws-sdk/global');
const S3 = require('aws-sdk/clients/s3');
const axios = require('axios');
const fs = require('fs');

// Global Variables
const apiKey = 'API_KEY_HERE'; // API Key for a specific ADMIN tied to a specific MediaSilo account. Example 23f9a04f-4a30-48fd-b1d9-0f8e699c2dbf
const apiSecret = 'API_SECRET_HERE'; // Secret Key for a specific ADMIN tied to a specific MediaSilo account. Example 1c975a31f4c6a4d7d7957f2512d514e5
const projectId = 'PROJECT_UUID_HERE'; //This ID references a real project in your Mediasilo account. This will be the final destination for the new asset. Example 550ddaa8-8be3-4f01-a828-6dad63e04863
let filePath = 'FULL_FILE_PATH_HERE'; //The exact path to the file on your computer. Example /Users/Maverick/Desktop/TestMedia/Videos/TopGun.mp4
let fileName = filePath.replace(/^.*?([^\\\/]*)$/, '$1'); //The name of your file which is extracted from the file path.
let globalSessionId; // This is a UUID that is used to identify your upload session. It is used to refresh credentials. This will be set after the createMultipartUploadTicket() function is called.

/*******************************************************************************************************
* Step 1: Create Multipart Upload Ticket
* Create a multipart upload ticket using the endpoint POST /v3/assets/multipart/upload. 
* This will have all the information needed to upload the asset to S3. 
* This includes fileName, sessionId, assetUrl, expiration, accessKey, secretKey,
* sessionToken, objectKey(path in S3 where the file is stored), and bucketName.
*
* Step 2: Upload File into S3
* Upload the asset to S3. Once the upload into S3 is complete, the assetUrl will be
* accessible to create the asset in your Mediasilo account. Although the assetUrl is
* returned in the multipart upload ticket, the assetUrl is not accessible until the upload
* is complete. For uploads that will take longer than one hour,
* see Step 4 below on Refreshing Credentials.
*
* Step 3: Create Asset
* Create the asset. Using the assetUrl and projectId, create the asset inside the
* desired project. The projectId is the UUID of an EXISTING project inside of your
* Mediasilo account.
*
* Step 4(Optional): Refreshing Temporary Credentials
* The Temporary Credentials, accessKey, secretKey, and sessionToken expire after 1 hour 
* from the time they were created or refreshed. When requesting temporary credentials,
* you pass in the file name. When you start uploading the file to S3, the sessionId is the
* reference to your specific upload session for an individual file. To refresh temporary credentials
* we use the same endpoint from Step 1 POST /v3/assets/multipart/upload. But this time we pass in the
* fileName and the sessionId. This way, the refreshed credentials have access to that specific
* upload session. The new refreshed credentials are interchangeable with the original credentials
* because they have the same exact access. There is a code example on refreshing credentials
* using the Mediasilo API further down this page.
*******************************************************************************************************/

// Function call to start the upload process
startAssetCreateProcess();

function startAssetCreateProcess() {
  createMultipartUploadTicket();
}

// ***** STEP 1 *****
function createMultipartUploadTicket() {
  /*
  *This is the POST request that is made to the API that returns the              
  *information needed to uploada file to S3, then later create an asset 
  */
  axios.post('https://api.shift.io/v3/assets/multipart/upload', { fileName }, {
    headers: {
      'x-key': apiKey,
      'x-secret': apiSecret,
      'Content-Type': 'application/json',
    },
  })
    .then(response => {
      console.log('\nFinished creating the multipart upload ticket.');
      uploadFileToS3(response.data);
    })
    .catch(error => {
      console.error('\nError creating multipart upload ticket:', error.response);
    });
}

// ***** STEP 2 *****
function uploadFileToS3(multipartUploadTicketResponse) {
  let { fileName, assetUrl, expiration, accessKey, secretKey, sessionId, sessionToken, objectKey, bucketName } = multipartUploadTicketResponse;
  console.log('\nmultipartUploadTicketResponse:', multipartUploadTicketResponse); // Logging out the multipart upload ticket response.

  globalSessionId = multipartUploadTicketResponse.sessionId;// Setting the SessionId that will be used to refresh the AWS credentials. Again, this is a UUID that identifies your specific upload session.

  const uploadBody = fs.createReadStream(filePath); // Specify the path to the file on your computer.

  let credentials = new AWS.Credentials({ accessKeyId: multipartUploadTicketResponse.accessKey, secretAccessKey: multipartUploadTicketResponse.secretKey, sessionToken: multipartUploadTicketResponse.sessionToken }); // Credentials Object used to authenitcate the multipart upload requests made to S3. These credentials are used each time an individual part of the file is uploaded.
  let service = new AWS.S3({ credentials }); // Service object that allows us to interact with AWS S3.
  let managedUpload = new AWS.S3.ManagedUpload({ params: { Bucket: multipartUploadTicketResponse.bucketName, Key: multipartUploadTicketResponse.objectKey, Body: uploadBody, }, service }); // This object handles uploading the file to S3 using multipart upload.

  managedUpload.promise().then(function () { // Calling this starts the multipart upload process.
    createAsset(multipartUploadTicketResponse.assetUrl); //We do not want to create the asset until the upload to S3 in finished.
  });

  managedUpload.on("httpUploadProgress", (progress) => {
    console.log(progress); // Logging out each part of the file that is uploaded to S3.
  });

  /*
  *Required for Step 4
  *
  *Temporary Credentials expire 1 hour after they are created or refreshed. 
  *This interval will refresh the temporary credentials every 30 minutes.
  *This ensures that uploads needing longer than one hour will complete.
  */
  const refreshInterval = setInterval(() => {
    credentials.refresh((err) => {
      if (err) {
        console.error('\nError refreshing temporary credentials:', err);
      } else {
        console.log('\nTemporary Credentials have been successfully refreshed.\n');
      }
    });
  }, 1800000); //1,800,000 milliseconds = 30 minutes. Reducing this number will increase the frequency at which your temporary credentials are refreshed.
  //If this number exceeds 3,599,999(59m 59s) then your upload will fail before your temporary credentials are refreshed. 
};

// ***** STEP 3 *****
function createAsset(assetPath) {
  /*
  *This is the POST request that is made to the API that creates your asset.
  *This will return the id of the asset that was created. 
  */
  axios.post('https://api.shift.io/v3/assets', { sourceUrl: assetPath, projectId }, {
    headers: {
      'x-key': apiKey,
      'x-secret': apiSecret,
      'Content-Type': 'application/json',
    },
  })
    .then(response => {
      console.log('\nAsset Created! Response:', response.data); // All Done! Your asset has been created.
      process.exit(); // Exit the script after successfully creating the asset in the Mediasilo Project.
    })
    .catch(error => {
      console.error('Error creating asset:', error.response.data);
    });
}

/*
*Required for Step 4
*
*This function is almost identical to the createMultipartUploadTicket. The main difference
*is that we are passing along a sessionId with a fileName when we make the POST request. 
*With the addition of the sessonId, the will API know to refresh your credentials.
* 
*This refresh function actually returns new temporary credentials. The access key
*and secret key will be different than the previous keys that you obtained from Step 1. The two important things to understand
*are that one, the new temporary credentials have the exact same permissions as the previous credentials,
*and two, the expireTime on the new temporary credentials is roughly one hour from the time you refreshed them.
*This allows the credentials to be interchangeable, even in the middle of an upload. 
*/
function refreshTemporaryCredentials(callback) {
  axios.post('https://api.shift.io/v3/assets/multipart/upload', { "fileName": fileName, "sessionId": globalSessionId }, {
    headers: {
      'x-key': apiKey,
      'x-secret': apiSecret,
      'Content-Type': 'application/json',
    },
  })
    .then(response => {
      console.log('\nFinished creating new temporary Credentials');
      callback(null, response);
    })
    .catch(error => {
      console.error('\nError creating multipart upload ticket:', error.response);
    });
}

// ***** STEP 4 *****
/*
*This block of code defines how the refresh() function will work.
*Any AWS.Credentials object that is created may use this refresh function.
*You can see that we using our refreshTemporaryCredentials() function from above.
*Then we take the response from that function and set the new values for our
*accessKeyId, secretAccessKey, and sessionToken.
*/
AWS.Credentials.prototype.refresh = function (callback) {
  var self = this;
  refreshTemporaryCredentials(function (err, response) {
    if (err) {
      callback(err);
    } else {
      console.log('\nCurrent SessionID: ', globalSessionId); // This will never change through out the entire upload of a single file.

      self.accessKeyId = response.data.accessKey; // Updating the accessKeyId.
      console.log('\nNew Access Key: ', response.data.accessKey)

      self.secretAccessKey = response.data.secretKey; // Updating the secretAccessKey.
      console.log('\nNew Secret Access Key: ', response.data.secretKey)

      self.sessionToken = response.data.sessionToken; // Updating the sessionToken.
      console.log('\nNew Session Token : ', response.data.sessionToken)

      callback();
    }
  });
};