Skip to content

Fully implement CloudInit (Ubuntu) #1552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 77 commits into
base: develop
Choose a base branch
from

Conversation

nyanhp
Copy link
Member

@nyanhp nyanhp commented Aug 25, 2023

Description

Fixes #1498

  • - I have tested my changes.
  • - I have updated CHANGELOG.md and added my change to the Unreleased section
  • - The PR has a meaningful title.
  • - I updated my fork/branch and have integrated all changes from AutomatedLab/develop before creating the PR.

Type of change

  • Bug fix
  • New functionality
  • Breaking change
  • Documentation

How was the change tested?

$labName = 'ALLovesLinux'
New-LabDefinition -Name $labName -DefaultVirtualizationEngine HyperV
Add-LabVirtualNetworkDefinition -Name $labName -AddressSpace 192.168.130.0/24
Add-LabVirtualNetworkDefinition -Name 'Default Switch' -HyperVProperties @{ SwitchType = 'External'; AdapterName = 'Ethernet' }
$PSDefaultParameterValues = @{
    'Add-LabMachineDefinition:Network' = $labName
    'Add-LabMachineDefinition:ToolsPath'= "$labSources\Tools"
    'Add-LabMachineDefinition:DnsServer1' = '192.168.130.10'
    'Add-LabMachineDefinition:Memory' = 1.5GB
    'Add-LabMachineDefinition:OperatingSystem' = 'Windows Server 2022 Datacenter'
}
$PSDefaultParameterValues.Add('Add-LabMachineDefinition:Gateway', '192.168.130.10')

Add-LabDomainDefinition -Name contoso.com -AdminUser install -AdminPassword Somepass1
Set-LabInstallationCredential -Username Install -Password Somepass1

$netAdapter = @()
$netAdapter += New-LabNetworkAdapterDefinition -VirtualSwitch $labName -Ipv4Address 192.168.130.10
$netAdapter += New-LabNetworkAdapterDefinition -VirtualSwitch 'Default Switch' -UseDhcp
$postInstallActivity = @()
$postInstallActivity += Get-LabPostInstallationActivity -ScriptFileName PrepareRootDomain.ps1 -DependencyFolder $labSources\PostInstallationActivities\PrepareRootDomain
Add-LabMachineDefinition -Name LINDC1 -Roles RootDC, Routing -Memory 1GB -PostInstallationActivity $postInstallActivity -NetworkAdapter $netAdapter -DomainName contoso.com
Add-LabMachineDefinition -Name UB001 -Memory 8GB -DomainName contoso.com -OperatingSystem 'Ubuntu-Server 22.04.2 LTS "Jammy Jellyfish"' -SshPublicKeyPath $home\.ssh\AzurePubKey.pub -SshPrivateKeyPath $home\.ssh\AzurePubKey

Install-Lab

@nyanhp nyanhp force-pushed the cloudinit branch 8 times, most recently from 7bb5ea6 to 6b771f1 Compare August 30, 2023 15:52
@nyanhp
Copy link
Member Author

nyanhp commented Sep 1, 2023

That was a painful development. I'm not adding any more distributions unless there is at least some standardization 😖

@nyanhp nyanhp marked this pull request as ready for review September 1, 2023 11:41
@nyanhp nyanhp requested a review from raandree September 1, 2023 11:41
@stale
Copy link

stale bot commented Sep 17, 2023

Labeling this pull request (PR) as abandoned since it has gone 14 days or more since the last update. An abandoned PR can be continued by another contributor. The abandoned label will be removed if work on this PR is taken up again.

@stale stale bot added the Abandoned The pull request has been abandoned. label Sep 17, 2023
@nyanhp nyanhp removed the Abandoned The pull request has been abandoned. label Sep 28, 2023
@nyanhp
Copy link
Member Author

nyanhp commented Sep 28, 2023

@raandree ? :)

@raandree
Copy link
Member

raandree commented Sep 29, 2023

Sorry for the delay, @nyanhp. I have just tried the lab you have provided. The deployment wasn't able to reach the Linux VM. Where can I start troubleshooting?

Note: I have used the image Ubuntu 23.04 "Lunar Lobster".

10:58:09|00:02:30|00:00:00.000| - Installing Root Domain Controllers
10:58:09|00:02:30|00:00:00.025|   - Machines with RootDC role to be installed: 'LINDC1'
10:58:09|00:02:30|00:00:00.047|   - Waiting for machines to start up..
10:58:22|00:02:43|00:00:12.776|   - Waiting for Root Domain Controllers to complete installation of Active Directory and restart.........done
11:03:52|00:08:13|00:05:42.800|   - Root Domain Controllers have now restarted. Waiting for Active Directory to start up.........
11:05:32|00:09:53|00:07:23.356|   - Done
11:05:32|00:09:53|00:00:00.000| - Configuring routing
11:05:33|00:09:53|00:00:00.059|   - Waiting for machines with Routing Role to startup..
11:05:44|00:10:05|00:00:11.878|   - Configuring Routing role..................
11:08:49|00:13:10|00:03:16.660|   - Waiting for configuration of routing to complete.......
11:10:11|00:14:31|00:04:38.091|   - Done
11:10:11|00:14:32|00:00:00.000| - Starting remaining machines
11:10:11|00:14:32|00:00:00.010|   - There are 1 Linux hosts in the lab.
                On Windows, those are installed from scratch and do not use differencing disks.

                If you did not connect them to an external switch or deploy a router in your lab,
                AutomatedLab will not be able to reach your VMs, as PowerShell will not be installed.

                The timeout to wait for VMs to be accessible via PowerShell was reduced from 60 to 15
            minutes.
11:10:11|00:14:32|00:00:00.015|   - Waiting for machines to start up.................................Wait-LWLabJob: Timeout while waiting for job 206
Write-LogFunctionExitWithError: Wait-LabVM...leaving: The following machines are not ready: UB001

The Linux VM has trouble reaching the internet. I guess I need a network adapter connected to the internet. I am wondering how your lab worked successfully.

image

@nyanhp
Copy link
Member Author

nyanhp commented Sep 29, 2023

My lab has a router, in case you didn't try it

@nyanhp
Copy link
Member Author

nyanhp commented Sep 29, 2023

@raandree powershell is not yet available for 23.04. Please use an LTS version.
image

@raandree
Copy link
Member

After using Ubuntu 22.04.3 LTS "Jammy Jellyfish" the same happened:

22:55:13|00:07:02|00:04:38.640|   - Root Domain Controllers have now restarted. Waiting for Active Directory to start up.........
22:56:51|00:08:40|00:06:16.249|   - Done
22:56:51|00:08:40|00:00:00.000| - Configuring routing
22:56:51|00:08:40|00:00:00.022|   - Waiting for machines with Routing Role to startup..
22:57:02|00:08:52|00:00:11.631|   - Configuring Routing role...............
22:59:26|00:11:15|00:02:34.986|   - Waiting for configuration of routing to complete......
23:00:49|00:12:38|00:03:58.077|   - Done
23:00:49|00:12:38|00:00:00.000| - Starting remaining machines
23:00:49|00:12:38|00:00:00.004|   - There are 1 Linux hosts in the lab.
                On Windows, those are installed from scratch and do not use differencing disks.

                If you did not connect them to an external switch or deploy a router in your lab,
                AutomatedLab will not be able to reach your VMs, as PowerShell will not be installed.

                The timeout to wait for VMs to be accessible via PowerShell was reduced from 60 to 15
            minutes.
23:00:49|00:12:38|00:00:00.011|   - Waiting for machines to start up.................................Wait-LWLabJob: Timeout while waiting for job 466
Write-LogFunctionExitWithError: Wait-LabVM...leaving: The following machines are not ready: UB001
..............................Wait-LWLabJob: Timeout while waiting for job 468
Write-LogFunctionExitWithError: Wait-LabVM...leaving: The following machines are not ready: UB001
.
23:32:11|00:44:00|00:31:21.960|   - Done
23:32:11|00:44:00|00:00:00.000| - Performing pre/post-installation tasks defined for each machine
23:32:11|00:44:01|00:00:00.365|   - Pre/Post-installations done
23:32:18|00:44:07|00:00:00.000| - Installing RDS certificates of lab machines
23:32:18|00:44:07|00:00:00.181|   - ..
23:32:19|00:44:09|00:00:01.559|   - Done
image

@nyanhp nyanhp marked this pull request as draft October 5, 2023 17:37
@nyanhp
Copy link
Member Author

nyanhp commented Oct 5, 2023

Looks like your VM still boots the installation medium.

@dkettman
Copy link

@nyanhp

If you need/want someone else to test, I am trying to use exactly what you are using (Ubuntu 22.04.03) and run into a similar issue. I downloaded the server installer, and it sits and waits for a cloud-init response, then continues on with the regular install process.

I may be able to explain my issue due to the type of networking I'm doing. I'm using a software NAT (New-NetNat) to separate everything in the lab from everything else on my network. My assumption is, my AutomatedLab setup is not listening on the right interface to know someone is offering up cloud-init data. But I dont know enough about all of it to know for sure.

If you need help testing something I'll do what I can to help. I think I may end up using this with some of my colleagues for some training instances so helping to further the cause is in my best interest. :)

Copy link

stale bot commented Dec 15, 2023

Labeling this pull request (PR) as abandoned since it has gone 14 days or more since the last update. An abandoned PR can be continued by another contributor. The abandoned label will be removed if work on this PR is taken up again.

@stale stale bot added the Abandoned The pull request has been abandoned. label Dec 15, 2023
@stale stale bot removed the Abandoned The pull request has been abandoned. label Feb 21, 2024
@nyanhp
Copy link
Member Author

nyanhp commented Feb 21, 2024

🎉

Sorry for the long hiatus @raandree , being a VP is surprisingly time-consuming.

I've rethought my approach and now think it works better. The machine's Install VHD is detached and removed after it has shut down during the automated setup. This ensure the VM boots properly, without random issues.

I've tested with this beauty as suggested by @dkettman. Remember to use your own SSH keys when testing, @raandree

# Lab Variables

$labName = 'MYLAB'
$labSubnet = '172.0.99' #Just the first three octets
$sql_iso_url = 'https://go.microsoft.com/fwlink/p/?linkid=2215158&clcid=0x409&culture=en-us&country=us'

Import-Module AutomatedLab

#checking if our NAT is created. We want internet access!
$nats = Get-NetNat

if ($nats.Count -eq 0) {
    Write-ScreenInfo -Message "No NATs defined! Defining one for us!"
    New-NetNat -Name $labName -InternalIPInterfaceAddressPrefix $labSubnet'.0/24'
}
else {
    Write-ScreenInfo -Message "There is already a NAT defined. Windows can only handle one NAT at a time. (Weak!)"
    Write-ScreenInfo -Message "Checking if we can use it..." -TaskStart
    if ($nats.InternalIPInterfaceAddressPrefix -like "$labSubnet*") {
        Write-ScreenInfo -Message "As far as I can tell, it looks like this should work. If not, remove the NAT using"
        Write-ScreenInfo -Message "the 'Remove-NetNAT' cmdlet and restart this script." -TaskEnd
    } 
    else {
        Write-ScreenInfo -Message "Looks like this NAT may not work."
        Write-ScreenInfo -Message "Please double-check that the existing NAT will and adjust accordingly."
        Write-ScreenInfo -Message "Without a NAT configured to handle the $labSubnet.0/24 subnet, you will not have Internets Access" -TaskEnd
        break
    }
}

#create an empty lab template and define where the lab XML files and the VMs will be stored
New-LabDefinition -Name $labName -DefaultVirtualizationEngine HyperV

#make the network definition
Add-LabVirtualNetworkDefinition -Name $labName -AddressSpace $labSubnet'.0/24'

# Define the Domain and the Domain Administration acocunt
Add-LabDomainDefinition -Name MYLAB.lab -AdminUser cs_admin -AdminPassword Somepass1

# Create a local user on each box as an administrator of 'cs_admin'
Set-LabInstallationCredential -Username cs_admin -Password Somepass1

# Setting up defaults for all machines. These can be overridden when the machine is defined
$PSDefaultParameterValues = @{
    'Add-LabMachineDefinition:DomainName'      = 'MYLAB.lab'
    'Add-LabMachineDefinition:Network'         = $labName
    'Add-LabMachineDefinition:Gateway'         = $labSubnet + '.1'
    'Add-LabMachineDefinition:DnsServer1'      = $labSubnet + '.10'
    'Add-LabMachineDefinition:ToolsPath'       = "$labSources\Tools"
    'Add-LabMachineDefinition:OperatingSystem' = 'Windows Server 2022 Standard (Desktop Experience)' # GUI by default
}


# Machine Definitions

# DC01
## Roles: RootDC
Add-LabMachineDefinition -Name dc01 `
    -Memory 2GB `
    -OperatingSystem 'Windows Server 2022 Standard' `
    -IpAddress $labSubnet'.10' `
    -DomainName MYLAB.lab `
    -Roles RootDC, CaRoot

Add-LabMachineDefinition -Name ubuntu01 `
    -Memory 8GB `
    -Processors 1 `
    -IpAddress $labSubnet'.90' `
    -SshPublicKeyPath "$home\.ssh\AzurePubKey.pub" `
    -SshPrivateKeyPath "$home\.ssh\AzurePubKey" `
    -OperatingSystem 'Ubuntu-Server 22.04.3 LTS "Jammy Jellyfish"'


# Get to work!
Install-Lab

@nyanhp nyanhp marked this pull request as ready for review February 21, 2024 17:06
@nyanhp
Copy link
Member Author

nyanhp commented Jul 23, 2024

@raandree there is still a slight timing issue that I cannot quite pinpoint, but in general it works:
image
The thing is that the host keys need to be updated after the installation. Maybe you have a good idea when to do that. We can have a chat via Teams on that ;)

@raandree
Copy link
Member

Just tried it and did run into this:

The AL installation session hangs here:

21:46:58|00:07:51|00:00:00.000| Starting remaining machines
21:46:58|00:07:51|00:00:00.000| - There are ubuntu01 Linux hosts in the lab.
                On Windows, those are installed from scratch and do not use differencing disks.
        
                If you did not connect them to an external switch or deploy a router in your lab,
                AutomatedLab will not be able to reach your VMs, as PowerShell will not be installed.

                The timeout to wait for VMs to be accessible via PowerShell was reduced from 60 to 15
            minutes.
21:46:58|00:07:51|00:00:00.078| - Waiting for machines to start up...SSH Transport is not available from within Windows PowerShell.
21:47:13|00:08:05|00:00:14.351| - ...................

The Ubuntu machine shows this (unfortunately I cannot copy the text):

image

I am using that Ubuntu ISO image:

Ubuntu-Server 22.04.3 LTS "Jammy Jellyfish"         0   x64  22.4            08.12.2023 00:00:00 D:\LabSources\ISOs\jammy-live-server-amd64.iso

@nyanhp
Copy link
Member Author

nyanhp commented Jul 25, 2024

Well, in that case I give up. I have no idea why your VM fails partitioning the disk. I'll give it one last try myself with a new VM to install the lab on.
After that, I don't care if we keep Ubuntu in as "don't bother us with issues, if it works, count yourself lucky" or if we close this PR. This ridiculous subiquity gave me so much headache already.

@nyanhp
Copy link
Member Author

nyanhp commented Jul 26, 2024

@raandree further testing:

  • 20.04 ??? Failed me once for no apparent reason
  • 22.04 Works fine, on two separate physical hosts as well as in a VM acting as Hypervisor
  • 24.04 Too new, subiqutiy format changed Testing 24.04 again with slightly updated cloudinit conf

@PhilKeeble
Copy link

I know this is a bit stale but thought I would chime in just in case it is somehow helpful. I am using AL and am building some Linux into my lab environment and tried AL (master) but it failed. I then set out to do it manually outside of AL and have that working, and was going to raise a feature request then saw this PR.

I found with Ubuntu 24.04.1 I could install it unattended in HyperV easiest by modifying the ISO for the user-data. I do also think a PXE boot option could work for AL since it would need no modification of the ISO, but would need a PXE boot server hosting it (maybe in the lab as a windows router / dhcp server / pxe boot server?).

I wrote up my method here:
https://github.com/PhilKeeble/HyperV-Linux-PoC

The powershell works ok in my testing, the part I can't figure out is how we would do the ISO modification within AL. I also can't easily work out how you are sorting the cloudinit file in this repo. If you have already figured out a way of taking the ISO then putting the cloudinit inside then repackaging as part of the base build (similar to how you build the windows bases), then I think this could be a solution for Ubuntu 24.04.

Probably all problems you have already overcome but I wanted to add if it led to an easier way of sorting the cloudinits (since if we can repackage the iso in here then we would just need different user-data config templates for different ubuntu versions going ahead).

Copy link

stale bot commented Apr 26, 2025

Labeling this pull request (PR) as abandoned since it has gone 14 days or more since the last update. An abandoned PR can be continued by another contributor. The abandoned label will be removed if work on this PR is taken up again.

@stale stale bot added the Abandoned The pull request has been abandoned. label Apr 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Abandoned The pull request has been abandoned.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[New Feature]: Support for Kali Linux
4 participants