To add an additional layer of security, we'd like to use Azure Key Vault to help safeguard cryptographic keys and secrets used by Metric Insights. Is this possible?

Yes, this is definitely possible. Metric Insights recently introduced changes in version 3.3 to leverage Azure Key Vault. In total, there are two keys for Metric Insights: application encryption and file system encryption (for data encryption at rest.)

To enable the use of Azure Key Vault, you will have to make some changes to the Metric Insights configuration. Note that this requires root level access to the Metric Insights server. Here are the steps to enable Azure Key Vault in Metric Insights CentOS systems:

1.) Make sure that the Key Vault is set up in Azure. 

Note that there are some powershell commands necessary to set this up. View the following Azure document for more: http://blogs.technet.com/b/kv/archive/2015/06/02/azure-key-vault-step-by-step.aspx.  

You will need to create a key for the MI application key, we'll refer to this as AppKey in this article. If you'd like to enable file system encrpytion for at rest encryption of data, you will need another key to be used for file system encryption. We'll refer to this as the FileSystemKey

2.) Initialize the AppKey in your Azure Key Vault to be the same value as the key stored in /var/www/aes_password. (You can change this value later)

3.) Now, tell Metric Insights to use your new Azure Key vault by adding the following [encryption] section to /etc/mi/insight.conf in Metric Insights:

[encryption]

key_storage = azure

client_id = your-client-id

client_key = 'your-client-secret-key'

auth_url = https://login.windows.net/<your-auth-code>

vault_name = Vault-Name

key_name = AppKey-Name

You will need to get this information from your Azure Management Portal. The Client-Id/Client-Key, for example, will be the Oauth2 Application Id and Secret for the application you created for MI in Azure AD. The auth_url will be the 'OAuth 2 Authorization Endpoint' for the application. <TODO: screenshot example from Azure Portal>

4.) Once insight.conf is saved with the new [encryption] section, verify that you can connect to one of your data sources. You can do this by testing a data source connection or collecting data from an existing metric. Ex: <SOME URL> 

If this step fails for some reason, you have not configured the [encryption] section of insight.conf correctly.

5.) If step 4 succeeded and you can correctly connect to your data sources, then you can now change the key to your own encryption key. For performance, it's best to pick a key that is somewhere between 16 and 80 bytes, though you can choose larger values for your key if you wish. 

To change the application encryption key, you can use the mi-crypt utility that comes with Metric Insights. For example, to update the key to a new random 64 byte value, run:

/usr/local/lib/mi/bin/mi-crypt setnewkey default $(openssl rand -hex 64)

TODO: NEW SECTION ON FILE SYSTEM ENCRYPTION

This section will explain the steps necessary to setup file encryption:

1.) Get the appropriate ecryptfs rpm (send a request to Metric Insights Support) and save to localhost

2.) Install the ecryptfs rpm from localhost: yum install-local ecryptfs-utils-96-1.el6.x86_64.rpm

3.) Set up your FileSystemKey in Azure Key Vault. This should be a separate key from your AppKey that was created for application encryption. Due to performance reasons, your FileSystemKey value should not exceed 80 bytes. (You can generate a new key by using `openssl rand -hex <n>` where n is the number of bytes you want). Make sure your FileSytemKey is stored in Azure Key Vault before continuing.

4.) Now tell Metric Insights to use at-rest encryption by setting up an [encryption_fs] section in /etc/mi/insight.conf. This will be very similar to the AppKey set up above, just with a different key name. 

8.) Once insight.conf is saved with your new [encryption_fs] section, you can encrypt the file system with the following command:

mi-crypto-mgr enable -p '/usr/local/lib/mi/bin/mi-crypt getkey fs'

Once Azure Key Vault is set, you can update the application key to a new key by running the following command in the Metric Insights instance:

/usr/local/lib/mi/bin/mi-crypt setnewkey default <newkey>

This will take care of decrypting all the secrets stored in the Metric Insights database with the old key, then re-encrypting the secrets with the new key.

To update the file system key, use the following command:

/usr/local/lib/mi/bin/mi-crypt setnewkey fs <newkey>