Veeam Backup & Replication 8: RPC error:Access is denied Fix

I recently set up a new Veeam Backup & Replication v8 demo lab, and my intial small job that consisted of two different Linux VMs and one Windows Server 2012 R2 Domain Controller was chugging along nicely. I had one minor from the start though, and that was that file indexing consistently failed for the Windows VM. No big deal, but I thought it was strange at the time. After all, the Linux VMs were indexed just fine. Veeam-1

Fast forward a few days, and all of a sudden Veeam B&R was unable to back up the Windows VM at all, failing with the following error:


18.01.2015 19:48:01 :: Processing Joey Error: Failed to check whether snapshot is in progress (network mode).
RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine: [].
RPC error:Access is denied.
Code: 5

Nothing had changed in my environment, no patches had been installed, no changes made to the backup job or the credentials used. I even tried deleting the job, this is a demo environment after all, and re-creating it, but with the same end result.

As the Access is denied message clearly states, this had to be related to permissions somehow, but I was using domain administrator credentials (again, this is a lab), so all the required permissions should be in place, and the credentials test in the backup job also checked out just fine. It has also worked fine for 5 or 6 days, so I was a bit baffled.

In the end, I tried changing from using User Principle Name (UPN) connotation of administrator@domain.local in the credentials for the VM, to using Down-Level Logon Name aka domain\administrator and retried the job.

Veeam-2That did the trick, and it also fixed the indexing problems I’ve had since setting up the job in the first place.

According to this Veeam Community Forum thread this was also a problem in v7 and has something to do with how the Microsoft API’s work.

So, if your Veeam Backup & Replication jobs fail with access denied messages, and/or can’t index the VM files, check your credentials. They may work, but they might just be entered in the wrong format.

Troubleshooting: Problems connecting HP Insight Control Storage Module to StoreServ 7200 (3Par)

A customer of mine, who runs a pure HP environment based on c7000 and StoreServ 7200, wanted to get the HP Insight Control Storage Module for vCenter up and running. The problem was that while we were able to connect to the older MSA array they run for non-production workloads, we were unable to connect to the newer StoreServ 7200. There is full IP connectivity between the application server that the HP Insight components run on and the storage controllers/VSP (no firewalls between them, they are located in the same subnet).

The only error message we got was an “unable to connect” message, when using the same credentials and ip address used for the 3Par Management Console. After reaching out to quite a few people, including Twitter, we finally found the solution. It turns out that the CIM service on the array was not responding, in fact it was disabled, which naturally resulted in not being able to connect.

A quick ssh session to the array, revealed that the CIM service was disabled.

login as: username
Password: *************

3par-array cli% showcim
-Service- -State-- --SLP-- SLPPort -HTTP-- HTTPPort -HTTPS- HTTPSPort PGVer CIMVer
Disabled  Inactive Enabled     427 Enabled     5988 Enabled      5989 2.9.1 3.1.2
3par-array cli% stopcim
Are you sure you want to stop CIM server?
select q=quit y=yes n=no: y
CIM server stopped successfully.
3par-array cli% startcim
CIM server will start in about 90 seconds
3par-array cli%

Restarting it fixed the issue, and we now have StoreServ data available directly in the vSphere Web (and C#) client. This also fixed the connection problem we had with vCenter Operations Manager and the The HP StoreFront Analytics adapter.

So, if you are unable to connect to your StoreServ, check the CIM service – It might just be disabled.

Troubleshooting: Root Cause of Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize (memsize)

Screenshot 2014-07-12 20.04.31While working on reconfiguring my home lab setup, and migrating all the vSphere resources into a single cluster I ran into a problem powering on one of the VMs which used to run on a single host. The power on operation yielded the following error message:

Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize (memsize)
[showhide type=”errorstack” more_text=”Show error stack (%s More Words)”]

Task Details:
Status: Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize(4096).
Start Time: Jul 12, 2014 4:22:21 PM
Completed Time: Jul 12, 2014 4:22:23 PM
State: error
Error Stack:
An error was received from the ESX host while powering on VM MinecraftServer.
Failed to start the virtual machine.
Module MemSched power on failed.
An error occurred while parsing scheduler-specific configuration parameters.
Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize(4096).
Additional Task Details:
VC Build: 1476327
Error Type: GenericVmConfigFault
Task Id: Task
Cancelable: false
Canceled: false
Description Id: Datacenter.ExecuteVmPowerOnLRO
Event Chain Id: 20017


Clearly there was an issue with memory reservations on the VM, but there was no memory reservation enabled on it at all, nor should there be. The only related errors I found while investigating the issue, was with regards to pass-through devices, which also did not apply in this case. It turns out that the problem was due to the VM was configured to use the latency sensitivity feature introduced in vSphere 5.5.

The Deploying Extremely Latency-Sensitive Applications in VMware vSphere 5.5 whitepaper from VMware clearly states that usage of this feature also demands a memory reservation being set on the VM, and this VM had no reservation.

Enabling the latency-sensitivity feature requires memory reservation.

In the end, the solution was a simple one, revert the latency Screenshot 2014-07-12 21.10.12sensitivity advanced option for the VM to the default value of Normal let me power-on the VM again without issues.

The error message received in vCenter could be a lot clearer though, and a knowledge base article with the exact error message and resolution paths might be in order. It was not immediately obvious that the lack of memory reservation error message was related to the latency sensitivity settings for the given VM. The vSphere Web Client shows a warning that you should check the CPU reservations, but does not mention memory reservations when you enable this feature.

Now I just need to figure out why my home MineCraft server had that setting enabled in the first place…