cloud-init does not appear to be running error after installing WALinuxAgent fix

After a recent migration of a Linux VM (Ubuntu 16.04) from a data centre to Azure using Azure Migrate, it didn’t include the installation of WALinuxAgent

Why is this needed?

This agent is how the Virtual Machine communicates with Azure, it manages the Linux provisioning and VM interaction with the Azure Fabric Controller, pretty much it is used to provide all the functionality between VM and Azure for Linux IaaS Deployments.

If the agent is not installed/configured, you will see agent status within VM properties as below:-

Agent Status: Not ready
Agent version: Unknown

The installation process

Simple installation of this agent , done via apt-get

apt-get install walinuxagent

After this installation I had noticed the Azure Portal had not updated in some time with the Agent Status changing.

Reviewing the walinuxagent log in /var/log/walinuxagent.log , I found

2020/04/15 21:20:07.049590 INFO Daemon Azure Linux Agent Version:2.2.45
2020/04/15 21:20:07.051646 INFO Daemon OS: ubuntu 16.04
2020/04/15 21:20:07.052432 INFO Daemon Python: 3.6.8
2020/04/15 21:20:07.053536 INFO Daemon CGroups Status: The cgroup filesystem is ready to use
2020/04/15 21:20:07.056045 INFO Daemon Run daemon
2020/04/15 21:20:07.061357 INFO Daemon cloud-init is enabled: True
2020/04/15 21:20:07.064921 INFO Daemon Using cloud-init for provisioning
2020/04/15 21:20:07.075452 INFO Daemon Clean protocol
2020/04/15 21:20:07.077083 INFO Daemon Running CloudInit provisioning handler
2020/04/15 21:20:07.078624 INFO Daemon Waiting for cloud-init to copy ovf-env.xml to /var/lib/waagent/ovf-env.xml [1800 retries remaining, sleeping 1s]
2020/04/15 21:20:07.083156 WARNING Daemon cloud-init does not appear to be running
2020/04/15 21:20:08.084520 INFO Daemon Waiting for cloud-init to copy ovf-env.xml to /var/lib/waagent/ovf-env.xml [1799 retries remaining, sleeping 1s]
2020/04/15 21:20:08.094260 WARNING Daemon cloud-init does not appear to be running
2020/04/15 21:20:09.096189 INFO Daemon Waiting for cloud-init to copy ovf-env.xml to /var/lib/waagent/ovf-env.xml [1798 retries remaining, sleeping 1s]
2020/04/15 21:20:09.102416 WARNING Daemon cloud-init does not appear to be running
2020/04/15 21:20:10.104823 INFO Daemon Waiting for cloud-init to copy ovf-env.xml to /var/lib/waagent/ovf-env.xml [1797 retries remaining, sleeping 1s]
2020/04/15 21:20:10.110031 WARNING Daemon cloud-init does not appear to be running

After some investigation and read some blog posts, nothing resolved the issue; with some posts mentioning editing /etc/waagent.conf but didn’t resolve this issue.

Decided to create a blank Linux VM to see the agent configuration of it, within /var/lib/waagent folder there was ovf-env.xml – had a look and it looked very generic apart from a couple of fields that were related to the newly created Virtual Machine

ovf-env.xml file output from newly created VM

<ns0:Environment xmlns:ns0="http://schemas.dmtf.org/ovf/environment/1" xmlns:ns1="http://schemas.microsoft.com/windowsazure" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ns1:ProvisioningSection>
<ns1:Version>1.0</ns1:Version>
<ns1:LinuxProvisioningConfigurationSet>
<ns1:ConfigurationSetType>LinuxProvisioningConfiguration</ns1:ConfigurationSetType>
<ns1:UserName>tamopsuser1</ns1:UserName>
<ns1:DisableSshPasswordAuthentication>false</ns1:DisableSshPasswordAuthentication>
<ns1:HostName>vm-linux</ns1:HostName><ns1:UserPassword>REDACTED</ns1:UserPassword></ns1:LinuxProvisioningConfigurationSet>
</ns1:ProvisioningSection>
<ns1:PlatformSettingsSection>
<ns1:Version>1.0</ns1:Version>
<ns1:PlatformSettings>
<ns1:KmsServerHostname>kms.core.windows.net</ns1:KmsServerHostname>
<ns1:ProvisionGuestAgent>true</ns1:ProvisionGuestAgent>
<ns1:GuestAgentPackageName xsi:nil="true" />
<ns1:RetainWindowsPEPassInUnattend>true</ns1:RetainWindowsPEPassInUnattend>
<ns1:RetainOfflineServicingPassInUnattend>true</ns1:RetainOfflineServicingPassInUnattend>
<ns1:PreprovisionedVm>false</ns1:PreprovisionedVm>
<ns1:EnableTrustedImageIdentifier>false</ns1:EnableTrustedImageIdentifier>
</ns1:PlatformSettings>
</ns1:PlatformSettingsSection>
</ns0:Environment>

Copied this file across and edited the below to correlate with the affected VM
UserName: changed to the current user I was logged into

<ns1:UserName>tamopsuser1</ns1:UserName>
<ns1:HostName>vm-linux</ns1:HostName>

Restarted the service

service walinuxagent restart

Checked log again /var/log/walinuxagent.log

2020/04/15 22:05:02.874146 INFO Daemon Run daemon
2020/04/15 22:05:02.881507 INFO Daemon cloud-init is enabled: True
2020/04/15 22:05:02.881898 INFO Daemon Using cloud-init for provisioning
2020/04/15 22:05:02.882427 INFO Daemon Clean protocol
2020/04/15 22:05:02.882606 INFO Daemon Running CloudInit provisioning handler
2020/04/15 22:05:02.885639 INFO Daemon Detect protocol endpoints
2020/04/15 22:05:02.887207 INFO Daemon Clean protocol
2020/04/15 22:05:02.896640 INFO Daemon WireServer endpoint is not found. Rerun dhcp handler
2020/04/15 22:05:02.907398 INFO Daemon Test for route to 168.63.129.16
2020/04/15 22:05:02.915481 INFO Daemon Route to 168.63.129.16 exists
2020/04/15 22:05:02.917519 INFO Daemon Wire server endpoint:168.63.129.16
2020/04/15 22:05:02.939251 INFO Daemon Fabric preferred wire protocol version:2015-04-05
2020/04/15 22:05:02.940071 INFO Daemon Wire protocol version:2012-11-30
2020/04/15 22:05:02.941771 INFO Daemon Server preferred version:2015-04-05
2020/04/15 22:05:03.105044 INFO Daemon Found private key matching thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
2020/04/15 22:05:03.122504 INFO Daemon ssh host key found at: /etc/ssh/ssh_host_rsa_key.pub
2020/04/15 22:05:03.192874 INFO Daemon Thumbprint obtained from : /etc/ssh/ssh_host_rsa_key.pub
2020/04/15 22:05:03.199286 INFO Daemon Finished provisioning
2020/04/15 22:05:04.217528 INFO Daemon RDMA capabilities are not enabled, skipping
2020/04/15 22:05:04.222667 INFO Daemon Installed Agent WALinuxAgent-2.2.45 is the most current agent
2020/04/15 22:05:04.353251 INFO ExtHandler Agent WALinuxAgent-2.2.45 is running as the goal state agent
2020/04/15 22:05:04.353683 INFO ExtHandler Distro info: ubuntu 16.04, osutil class being used: Ubuntu16OSUtil, agent service name: walinuxagent
2020/04/15 22:05:04.377540 INFO ExtHandler Wire server endpoint:168.63.129.16
2020/04/15 22:05:04.401516 INFO ExtHandler Start env monitor service.
2020/04/15 22:05:04.402254 INFO ExtHandler Configure routes
2020/04/15 22:05:04.403130 INFO ExtHandler Gateway:None
2020/04/15 22:05:04.407057 INFO ExtHandler Routes:None
2020/04/15 22:05:04.408095 INFO ExtHandler Hostname record does not exist, creating [/var/lib/waagent/published_hostname] with hostname [tamops-vm-01]
2020/04/15 22:05:04.414483 INFO ExtHandler Wire server endpoint:168.63.129.16
2020/04/15 22:05:04.433086 INFO ExtHandler WALinuxAgent-2.2.45 running as process 22437
2020/04/15 22:05:04.438408 INFO ExtHandler CGroups Status: The cgroup filesystem is ready to use
2020/04/15 22:05:04.443231 INFO ExtHandler Started tracking new cgroup: walinuxagent.service, path: /sys/fs/cgroup/cpu/system.slice/walinuxagent.service
2020/04/15 22:05:04.449400 INFO ExtHandler Started tracking new cgroup: walinuxagent.service, path: /sys/fs/cgroup/memory/system.slice/walinuxagent.service
2020/04/15 22:05:04.554364 INFO ExtHandler Created slice for walinuxagent extensions system-walinuxagent.extensions.slice
2020/04/15 22:05:04.555469 INFO ExtHandler Wire server endpoint:168.63.129.16

Checking in Azure Portal and agent status is correct

Agent Status: Ready
Agent Version: 2.2.46

2 comments

  1. Thomas I noticed on RHEL7 Linux the Virtual CDROM /dev/sr0 is not mounting, this virtual device contains the ovf-env.xml file which provides provisioning details to waagent and cloud-init.
    The error I receive in waagent.log (if I set Provisioning.UseCloudInit=y in waagent.conf then in cloud-init.log) indicates it is unable to mount sr0 device.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s