NetBackup KMS HowTo

Provides support for Advanced Disk and Cloud storage providers.

KMS works with encryption capable tape drives. A SCSI command enables encryption on the encryption capable tape drive. NetBackup accesses this capability through the volume pool name.

Provides support for the T10 / SCSI standard tape drives with embedded (LT04 and equivalent) encryption.
  Drive, tape, and NetBackup capabilities must all match for drive encryption to be successful. A number of drives adhere to the standard. The LT04 is a typical type.
Currently only LT04 drives and LT04 media can be encrypted or decrypted. You can still run LT03 media in LT04 drives for reading and writing but you cannot encrypt the data. If you use LT02 media, that data can be read in LT04 drives but they cannot be written in either unencrypted or encrypted format.
You must keep track of these drive issues and media issues as you run setup encryption. Not only do you need the drives that are capable of encryption but the media needs to be grouped and capable of encryption. For later decryption the tape must be placed in a drive that is capable of decryption.
Following is the interoperatability matrix for the tape drives and media:
¦ LTO4 drives can read LTO2, LT03, and LT04 media
¦ LTO4 drives can write LTO3 and LT04 media
¦ LTO4 drives can only encrypt LTO4 media
¦ LTO4 encrypted and decrypted media only works in LTO4 drives

AES_256 is the default. (AES_128 & AES_192 are also supported).

KMS can be configured for random generated keys or passphrase generated keys. Passphrase generated keys is recommended. (Using a passphrase has definite benefits. It results in keys with better security strength. And if keys are lost, you can regenerate them by providing the passphrase that was used to create the original key. Failing to enter a passphrase will result in a randomly generated key that cannot be re-produced)

The nbkms service is a master-server-based service that provides encryption keys to the media server BPTM processes.

nbkmsutil - KMS configuration utility: For security reasons, the KMS configuration utility can only be run from the master server as root or administrator.

The "ENCR_" prefix on Volume Pools is essential to tell the "BPTM" process that the tapes is to be encrypted. The Volume Pool name is then provided to "KMS", and "KMS" identifies it as an exact match to a Key Group name to pick the active key to use for backups. This requires that Key Group names also follow standard volume pool naming conventions.


Process flow:

- Backup:
  - BPTM receives a request to write to a tape and to use a tape from a volume pool with the ENCR_ name prefix. The ENCR_ prefix is a signal to BPTM that the information to be written to tape is to be encrypted.

  - BPTM contacts KMS and requests an encryption key from the key group with a name that matches the name of the volume pool. KMS hands back to BPTM an encryption key and a key identifier (known as the encryption key tag).

  - BPTM places the drive in encryption mode and registers the key tag and identifier tag with the drive. This process is all done with the SCSI security protocol in or out
command that has been added to the SCSI specification.

  - The backup then proceeds as normal.

  - When the backup is complete, BPTM unregisters the key and tag with the drive and sets the drive back into regular mode.

  - BPTM then records the tag in the NetBackup image record catalog.

- Restore:
When a tape is read and an area of the tape in encountered where an image is encrypted, BPTM determines: what tag is used and KMS loads that record and key
into BPTM. Then BPTM provides the key to the drive and reading the tape proceeds as normal.


Key Records:
- Key records contain many fields but the primary records are the encryption key, the encryption key tag, and the record state. Key records also contain some metadata.
These key records are defined as follows:
¦ Encryption key: This key is given to the tape drive.
¦ Encryption key Tag: This tag is the identifier for the encryption key aind is used by NB to identify which Key a backup image was created with (important for restore functionality)
¦ Record state: Each of the key records has a state. The states are prelive, active, inactive, deprecated, and terminated.
¦ Metadata: Metadata includes logical name, creation date, modification date, and description.
Note: all key records must belong to a key group. Read on...


Key Groups:
Key groups are a logical name and grouping of key records. All key records that are created must belong to a group. A key group can only have one active state
key record at any time.
   - NetBackup 7.5 supports 100 key groups.
   - NetBackup 7.0 supported 20 key groups and NetBackup 6.5.2 supported two key groups.
   - Only 10 encryption keys are allowed per key group.


DR:
To insure the KMS database is recoverable in the event of a loss, copies of the KMS database should: be kept at a designated DR site, and Information Security needs to store a copy of the pass-phrases in a secondary secured location.

In the event of a disaster, the KMS database must be able to be re-created or restored independently of the encrypted tapes, or any other NetBackup involved restores. In other words, do not rely on backups of the KMS database done to encrypted tapes, or you will not be able to restore them without having them in the first place.

NOTE: The KMS database is NOT backed up as part of the catalog backup, and must be backed up separately.

Any time the KMS database is updated, it should be backed up again, and the list of passphrases and related information updated to reflect the new phrase, and to insure the removal of any reference to any keys that have been deleted.

Saving Passphrases:
Keeping a list of all the passphrases alone isn't enough to insure the KMS database can be re-build properly from passphrases.

It is also important to keep a list all of the keys that belong to a Key Group name, and their associated Key Tags. A list can be generated by the following command:

nbkmsutil -listkeys -kgname <key_group_name>

Note: Veritas recommends that you keep a record of the output of the nbkmsutil -listkeys command. The key tag that is listed in the output is necessary if you need to recover keys.

When Information Security stores the passphrases, they will also need to keep a record of each associated Key Tag and Key Name related to the respective passphrase.

Backing up the KMS database is a simple matter of quiescing the database and then coping the files to a DR site or other secure location.

To quiesce the KMS DB, run:

nbkmsutil -quiescedb

This command returns with a quiesce successful statement and an indication of the number of outstanding calls.

After you have copied the files, you can unquiesce the KMS database files by using this command:

nbkmsutil -unquiescedb

A quiesce sets the KMS DB to read-only administrator mode. Quiescing is required to make a consistent backup copy of the KMS DB files.

Key file or key database
Contains the data encryption keys. The key file is located at
/usr/openv/kms/db/KMS_DATA.dat


Host master key contains the encryption key that encrypts and protects the KMS_DATA.dat key file using AES 256. Located at:
/usr/openv/kms/key/KMS_HMKF.dat

Key protection key is the encryption key that encrypts and protects individual records in the KMS_DATA.dat key file using AES 256. The key protection key is located at:
/usr/openv/kms/key/KMS_KPKF.dat

Currently the same key protection key is used to encrypt all of the records.

Options:
- Put the KMS database file on one tape and the HMK files and KPK files on another tape. To gain access to encrypted tapes, someone would then need to obtain both tapes.
              -AND/OR-
- Another alternative is to back up the KMS data files outside of the normal NetBackup process. You can copy these files to a separate CD, DVD, or USB drive.
              -AND/OR-
- You can also rely on passphrase generated encryption keys to manually rebuild KMS. All of the keys can be generated by passphrases. If you have recorded all of the encryption key passphrases you can manually recreate KMS from information you have written down. If you only have a few encryption keys you generate this process could be short.


Recovery Procedures:
NOTE: The KMS data files are not included in the NetBackup catalog backups. Simply restoring the catalog will NOT restore your keys


    To rebuild the database from passphrases, simply follow the instructions listed above, for "Installing the KMS database" and "Create the "ENCR_Offsite" Key Group". Once these are created, you can use the "-recoverkey" option providing the appropriate Key Name and Key Tag (can also be used to migrate a key to a different environment):

nbkmsutil -recoverkey -kgname <key_group_name> -keyname <key_name> -tag <Key Tag>

    This also requires entering the same exact pass-phrase as the original key when prompted.

Example of recovering KMS by regenerating the data encryption key:
1. Create an empty KMS database by running the following command
nbkms -createemptydb
You do not have to use the same host master key and key protection key. You can choose new keys.

2. Run the nbkmsutil -recoverkey command and specify the key group, key name, and tag.
nbkmsutil -recoverkey -kgname ENCR_pool1 -keyname Q1_2008_key -tag d5a2a3df1a32eb61aff9e269ec777b5b9092839c6a75fa17bc2565f725aafe90
Note: If you did not keep an electronic copy of the output of the nbkmsutil -listkey command when you created the key, you must enter all 64 characters manually.

3. Enter the pass phrase at the prompt. It must be an exact match with the original pass phrase you previously provided.
Note: If the tag you enter already exists in the KMS database, you cannot recreate the key.

4. If the recovered key is the key that you want to use for backups, run the following command to make the key active:
nbkmsutil -modifykey -kgname ENCR_pool1 -keyname Q1_2008_key -state active

The -recoverkey option places the key record in the inactive state, and it is brought into the KMS database in the inactive state.

5. If this is a key record that is to be deprecated, run the following command:
nbkmsutil -modifykey -kgname ENCR_pool1 -keyname Q1_2008_key -state deprecated

    To restore the KMS files, simply copy them to the appropriate locations


Considerations:

Only 10 keys are allowed per Key Group (and subsequently per Volume Pool).
- Adding a new key too often may cause old keys to be retired too quickly, such that old data on older tapes will not be able to be recovered, despite not having reached their expiration date.
- Adding new keys too infrequently poses a potential security risk, and may create inconsistencies with the expiration of data on older tapes that are to be retired.
- The KMS data files are NOT included in the NetBackup Catalog backups, and should be backed up separately with special consideration.


Key Rotation and Usage Policy:
- Example:
  - If your maximum retention is 10 years, and each group/pool can have up to 10 keys, the key rotations should occur yearly, with any keys older then 10 years being deleted from the database to make room for newer keys.
  - In environments were the retention is shorter, more frequent rotations can occur.
  - A key must be put into a "Deprecated" or "Terminated" state no later then 12 months after the maximum retention for data/tapes in the environment; based on when the key was last in an "active" state.
  - A key should only be deleted to make room for a new key to be created within the group, or to insure retired media can no longer be read.
  - "Offsite Archive" tapes that are kept "indefinitely" should be created with a key or set of keys that are to NEVER be deleted, nor put into a "Deprecated" or "Terminated" state.
  - Such keys should still be rotated on a yearly basis, but after a maximum of 10 years, the keys will have to be re-used, as they cannot be deleted.
  - In this case the Key's description should contain date ranges that depict the time frames during which the key was used.
  - No other keys should be re-instated as "Active" after having been removed from an "Active" state.
  - Tapes retained on-site, in a secure location, do not need to be encrypted.
  - "Archive" tapes that are stored on-site should NEVER be encrypted to insure recoverability.
  - On-site tapes should be stored in a secured and climate controlled location.
  - Tapes sent off-site, or that are handled by a third party at any time, MUST be encrypted.
  - Catalog Backup tapes are an exception and should not be encrypted to insure the catalog can be easily and quickly restored.
  - Since the Catalog is merely a listing of what was backed up it does not contain any highly sensitive data within it.


Logging:
- The NetBackup master server's nbkms service uses the new unified logging and has been assigned OID 286.
- The nbkmsutil command uses traditional logging and its logs can be found in the file /usr/openv/netbackup/logs/admin/*.log


Configuration:

The KMS service is called nbkms. The service does not run until the data file has been set up, which minimizes the
effect on environments not using KMS.

To install KMS
1. Run the nbkms -createemptydb command.
2. Enter a passphrase for the host master key (HMK). You can also press Enter to create a randomly generated key.
3. Enter an ID for the HMK. This ID can be anything descriptive that you want to use to identify the HMK.
4. Enter a passphrase for the key protection key (KPK).
5. Enter an ID for the KPK. The ID can be anything descriptive that you want to use to identify the KPK.

The KMS service (supposedly) starts when after you enter the ID and press Enter.

6. Start the service by running the following command:
/usr/openv/netbackup/bin/nbkms &
7. Use the grep command to ensure that the service has started, as follows:
ps -ef | grep nbkms | grep -v grep

8. Create the key group. The key group name must be an identical match to the volume pool name. All key group names must have a prefix ENCR_.
Note: When using key management with Cloud storage, the ENCR_ prefix is not required for the key group name.

To create a (non-Cloud storage) key group use the following command syntax.
nbkmsutil -creatkg -kgname ENCR_volumepoolname
  - For tape based encryption, the ENCR_ prefix is essential. When BPTM receives a volume pool request that includes the ENCR_ prefix, it provides that volume pool name to KMS. KMS
identifies it as an exact match of the volume pool and then picks the active key record for backups out of that group.

To create a Cloud storage key group use the following command syntax.
nbkmsutil -creatkg -kgname cloud_provider_URL:volume_name

9. Create a key record by using the -createkey option.
nbkmsutil -createkey -kgname ENCR_volumepool -keyname keyname -activate -desc "message"
  - The key name and message are optional; they can help you identify this key when you display the key.
  - The -activate option skips the prelive state and creates this key as active.

10. Provide the passphrase again when the script prompts you. In the following example the key group is called ENCR_pool1 and the key name is Q1_2008_key. The description explains that this key is for the months January, February, and March.
nbkmsutil -createkey -kgname ENCR_pool1 -keyname Q1_2008_key -activate -desc "key for Jan, Feb, & Mar"

11. You can create another key record using the same command; a different key name and description help you distinguish they key records:
nbkmsutil -createkey -kgname ENCR_pool1 -keyname Q2_2008_key -activate -desc "key for Apr, May, & Jun"
    Note: If you create more than one key record by using the command nbkmsutil -kgname name -activate, only the last key remains active.

12. To list all of the keys that belong to a key group name, use the following command:
nbkmsutil -listkeys -kgname keyname
    Note: Veritas recommends that you keep a record of the output of the nbkmsutil -listkeys command. The key tag that is listed in the output is necessary if you need to recover keys.


Example:

[root@rhel7nbmaster ~]# cd /usr/openv/netbackup/bin

[root@rhel7nbmaster bin]# ./nbkms -createemptydb
Enter the Host Master Key (HMK) passphrase (or hit ENTER to use a randomly
generated HMK). The passphrase will not be displayed on the screen.
Enter passphrase :
Re-enter passphrase :

An ID will be associated with the Host Master Key (HMK) just created. The ID
will assist you in determining the HMK associated with any key store.
Enter HMK ID : rhel7nbmasterhmk

Enter the Key Protection Key (KPK) passphrase (or hit ENTER to use a randomly
generated KPK). The passphrase will not be displayed on the screen.
Enter passphrase :
Re-enter passphrase :

An ID will be associated with the Key Protection Key (KPK) just created. The
ID will assist you in determining the KPK associated with any key store.
Enter KPK ID : rhel7nbmasterkpk

Operation successfully completed
[root@rhel7nbmaster bin]#

[root@rhel7nbmaster bin]# ps -ef | grep nbkms | grep -v grep

[root@rhel7nbmaster bin]# /usr/openv/netbackup/bin/nbkms &
[2] 9776


[root@rhel7nbmaster bin]# ps -ef | grep nbkms | grep -v grep
root      9791     1  0 12:16 ?        00:00:00 /usr/openv/netbackup/bin/nbkms


[root@rhel7nbmaster bin]# nbkmsutil -createkey -kgname ENCR_Offsite -keyname Q3_2014_Q3_2015key -activate -desc "Key for Offsite tapes from Q3 2014 through Q3 2015"

Enter a passphrase:
Re-enter the passphrase:

New Key creation is successful

[root@rhel7nbmaster bin]#


[root@rhel7nbmaster bin]# nbkmsutil -listkeys -kgname ENCR_Offsite

Key Group Name        : ENCR_Offsite
Supported Cipher      : AES_256
Number of Keys        : 1
Has Active Key        : Yes
Creation Time         : Sun Dec 14 12:36:19 2014
Last Modification Time: Sun Dec 14 12:36:19 2014
Description           : -

  Key Tag    : 064ead104a63447dfb2f481e5ba9b2549a496f91b3e3344391972573d28b85ef
  Key Name              : Q3_2014_Q3_2015key
  Current State         : ACTIVE
  Creation Time         : Sun Dec 14 12:48:59 2014
  Last Modification Time: Sun Dec 14 12:48:59 2014
  Description           : Key for Offsite tapes from Q3 2014 through Q3 2015


Number of Keys: 1

[root@rhel7nbmaster bin]#


Advanced Disk:
- To use encryption, you must use the AdvancedDisk_crypt type when you configure the storage servers and the disk pools. You also must use the nbdevconfig command to configure the storage servers and the disk pools. For the AdvancedDisk_crypt type of storage server, you must specify the -st option with a value of 5.

/usr/openv/netbackup/bin/admincmd/nbdevconfig -creatests -storage_server hostname -stype server_type -st 5 -media_server hostname [-setattribute attribute]

Command-line configuration (the only configuration method for Advanced Disk) requires 3 major steps:
  1) if the KMS database has not been configured on the NB master yet, that is step 1 ('nbkms -createemptydb' using passphrase method, start the NetBackup Key Management Service on the master server).
  2) Create key groups for the volumes in the disk pool ('nbkmsutil -createkg -kgname storage_server_name:volume_name' where volume_name is the the last directory name. For example, if the the pathname is /mnt/disk/backups, the volume_name must be backups, as follows:
nbkmsutil -createkg -kgname UX_Host.Veritass.org:backups
   - On the storage that is exposed as a drive letter (such as E:\), nothing appears after the last slash. Therefore, the volume_name is not required, as follows:
nbkmsutil -createkg -kgname Win_Host.Veritass.org:)
  3) Create a key record for each group ('nbkmsutil -createkey -keyname keyname -kgname key_group_name -activate').
Note: save a record of the key names: /usr/openv/netbackup/bin/admincmd/nbkmsutil -listkgs
Note2: For each key group, write all of the keys that belong to the group to a file. Run the command on the master server. The following is the command syntax:
/usr/openv/netbackup/bin/admincmd/nbkmsutil -listkeys -kgname


[root@rhel7nbmaster bptm]# nbdevconfig -creatests -storage_server rhel7nbmaster -stype AdvancedDisk_crypt -st 5 -media_server rhel7nbmaster
Storage server rhel7nbmaster has been successfully created


[root@rhel7nbmaster bptm]# nbdevquery -liststs -stype AdvancedDisk_crypt -U
Storage Server      : rhel7nbmaster
Storage Server Type : AdvancedDisk_crypt
Storage Type        : Formatted Disk, Direct Attached
State               : UP
Flag                : OpenStorage
Flag                : AdminUp
Flag                : InternalUp
Flag                : LifeCycle
Flag                : CapacityMgmt
Flag                : FragmentImages
Flag                : Cpr
Flag                : FT-Transfer
Flag                : OptimizedImage
Flag                : MetaData


Cloud storage providers:
- Veritas recommends that you use the Cloud Storage Server Configuration Wizard and the Disk Pool Configuration Wizard. The wizards include the steps that configure key management and encryption.
- Alternatively, the admin can use commandline (not recommended). Command-line configuration requires 3 major steps:
  1) if the KMS database has not been configured on the NB master yet, that is step 1 ('nbkms -createemptydb' using passphrase method, start the NetBackup Key Management Service on the master server).
  2) Create key groups for the volumes in the disk pool ('nbkmsutil -createkg -kgname storage_server_name:volume_name')
  3) Create a key record for each group ('nbkmsutil -createkey -keyname keyname -kgname key_group_name -activate').
Note: save a record of the key names: /usr/openv/netbackup/bin/admincmd/nbkmsutil -listkgs
Note2: For each key group, write all of the keys that belong to the group to a file. Run the command on the master server. The following is the command syntax:
/usr/openv/netbackup/bin/admincmd/nbkmsutil -listkeys -kgname


bpimagelist, tape copy is copy # 2 in this example:


Copy number:       2
 Fragment:         1
 Kilobytes:        32
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         16
 ID:               3005TA
 Host:             rhel7nbmaster
 Block Size:       65536
 Offset:           22775
 Media Date:       Sun 14 Dec 2014 03:47:59 PM ES (1418590079)
 Dev Written On:   0
 Flags:            0x40  (Tape Encrypted)
 Media Descriptor:        ?
 Expiration Time:  Sat 19 Sep 2015 04:41:25 PM ED (1442695285)
 MPX:              0
 retention_lvl:    9 months (7)
 Try to Keep Time:  Sat 19 Sep 2015 04:41:25 PM ED (1442695285)
 Copy Creation Time:  Sun 14 Dec 2014 03:48:48 PM ES (1418590128)
 Data Format:      Tar
 checkpoint:       0
 resume num:       1
 Key tag:          064ead104a63447dfb2f481e5ba9b2549a496f91b3e3344391972573d28b85ef
 STL tag:          *NULL*
 Copy on hold:     0


==================================================================================

Lab passphrase for both HMK and KPK: youcancountmeout