CloudPoint snapshot is failing with "The host level snapshot of cannot be performed as asset hierarchy is incomplete" due to ebsnvme-id returning /dev/sda when it should just return sda for the boot disk.

Problem

CloudPoint snapshot of AWS instance is failing with "The host level snapshot of cannot be performed as asset hierarchy is incomplete" due to ebsnvme-id returning /dev/sda when it should just return sda for the boot disk.

Error Message

Job Details:
Nov 03, 2021 8:32:31 AM - Error nbcs (pid=19608) Failed to create snapshot for asset: aws-ec2-<region>-<instanceID>
Nov 03, 2021 8:32:31 AM - Error nbcs (pid=19608) The host level snapshot of aws-ec2-<region>-<instanceID> cannot be performed as asset hierarchy is incomplete.

Problem output from ebsnvme-id:
C:\ProgramData\Amazon\Tools>ebsnvme-id
Disk Number: 0
Volume ID: vol-08337faf34b9d0f73
Device Name: /dev/sda1

They are also instances where the instance has NVME, but ebsnvme-id returned no results/output:

C:\ProgramData\Amazon\Tools>ebsnvme-id

flexsnap.log:

flexsnap-coordinator: "Nov 03 03:00:31 flexsnap-coordinator flexsnap-coordinator[9] Thread-2183923 flexsnap.workflows: INFO - create_asset_snap_flow: protection_level host objs [<FlexAsset id=aws-ec2-<region>-<instanceID>>]"
flexsnap-coordinator: "Nov 03 03:00:31 flexsnap-coordinator flexsnap-coordinator[9] Thread-2183923 flexsnap.workflows: ERROR - create_snapshot: The child asset <FlexAsset id=disk-aws-<region>-<instanceID>-/dev/sda1> does not have parent, that means the parent disk is either not discovered or the type of disk is not supported by CloudPoint, making the asset hierarchy incomplete, so the snapshot will fail eventually"

flexsnap-coordinator: "Nov 03 03:00:31 flexsnap-coordinator flexsnap-coordinator[9] Thread-2183923 flexsnap.workflow: ERROR - run: failed to load flow None (The host level snapshot of aws-ec2-<region>-<instanceID> cannot be performed as asset hierarchy is incomplete.)"
flexsnap-coordinator: "Traceback (most recent call last):"
flexsnap-coordinator: "  File ""/opt/VRTScloudpoint/lib/flexsnap/workflow.py"", line 548, in load"
flexsnap-coordinator: "    self._engine = taskflow.engines.load_from_factory("
flexsnap-coordinator: "  File ""/usr/local/lib/python3.8/dist-packages/taskflow/engines/helpers.py"", line 229, in load_from_factory"
flexsnap-coordinator: "    flow = factory_fun(*factory_args, **factory_kwargs)"
flexsnap-coordinator: "  File ""/opt/VRTScloudpoint/lib/flexsnap/workflows.py"", line 3012, in snapshot_factory"
flexsnap-coordinator: "    flow, asset_ids = create_asset_snap_flow(svc, asset_obj, snap_params,"
flexsnap-coordinator: "  File ""/opt/VRTScloudpoint/lib/flexsnap/workflows.py"", line 3391, in create_asset_snap_flow"
flexsnap-coordinator: "    raise flexsnap.OperationFailed(msg)"
flexsnap-coordinator: "flexsnap.OperationFailed: The host level snapshot of aws-ec2-<region>-<instanceID> cannot be performed as asset hierarchy is incomplete."

Cause

We expect ebsnvme-id devices when the instance consist of NVME disk, in addition we expect it to return boot device in the following format:
C:\ProgramData\Amazon\Tools>ebsnvme-id
Disk Number: 0
Volume ID: vol-08337faf34b9d0f73
Device Name: sda1

When it returns the below it is problematic and usually means the instance needs upgraded NVME drivers and rebooting:
C:\ProgramData\Amazon\Tools>ebsnvme-id
Disk Number: 0
Volume ID: vol-08337faf34b9d0f73
Device Name: /dev/sda1

Solution

Review Amazon EBS and NVMe on Windows instances:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/nvme-ebs-volumes.html#identify-nvme-ebs-device

From the above link you can obtain current versions of ebsnvme-id:
https://s3.amazonaws.com/ec2-windows-drivers-downloads/EBSNVMeID/Latest/ebsnvme-id.zip

AND drivers:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/aws-nvme-drivers.html

If drivers were upgraded, don't forget to reboot.

In some cases, after reboot ebsnvme-id still reported /dev/sda1.  In that scenario, it was necessary to Stop and Start the problematic instance via AWS console.

Applicable For

CLOUD-ALL-1