Skip to content

Allow multiple nodes have templates (use local storage) #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Allow multiple nodes have templates (use local storage) #467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
398c37d
Allow multiple nodes have templates (use local storage)
Mar 25, 2025
51681b3
Add back missing line
Mar 25, 2025
bd53227
Update localstorage support
Atoms Apr 14, 2025
94015d5
Rework how validations are made
Atoms Apr 30, 2025
cffffa8
Rework how validations are made
Atoms Apr 30, 2025
91de687
Rework how validations are made
Atoms Apr 30, 2025
9caa3f2
Rework how validations are made
Atoms Apr 30, 2025
5671658
Rework how validations are made
Atoms Apr 30, 2025
648f5fa
Remove unused error
Atoms Apr 30, 2025
9b57a0e
Remove debugging info
Atoms Apr 30, 2025
694b135
Remove function as it's not needed there
Atoms Apr 30, 2025
7e8ba12
add .spec.Target back and add deprecation notice
Atoms May 15, 2025
86e9f38
Update go.sum and add nolint to test
Atoms May 19, 2025
29b1f09
Update crds
Atoms May 20, 2025
565d6c8
Fix typo
Atoms May 20, 2025
fac81f9
Merge branch 'main' into multiple-node-templates
Atoms May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 25 additions & 9 deletions api/v1alpha1/proxmoxmachine_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,13 @@ type ProxmoxMachineSpec struct {
MetadataSettings *MetadataSettings `json:"metadataSettings,omitempty"`

// AllowedNodes specifies all Proxmox nodes which will be considered
// for operations. This implies that VMs can be cloned on different nodes from
// the node which holds the VM template.
// for operations. This implies that VMs can be cloned on different nodes.
//
// This field is optional and should only be set if you want to restrict
// the nodes where the VM can be cloned.
//
// If not set, the ProxmoxCluster will be used to determine the nodes.
//
// +optional
AllowedNodes []string `json:"allowedNodes,omitempty"`

Expand Down Expand Up @@ -176,8 +177,7 @@ const (
// TemplateSource defines the source of the template VM.
type TemplateSource struct {
// SourceNode is the initially selected proxmox node.
// This node will be used to locate the template VM, which will
// be used for cloning operations.
// SourceNode should be used together with TemplateID.
//
// Cloning will be performed according to the configuration.
// Setting the `Target` field will tell Proxmox to clone the
Expand All @@ -201,6 +201,12 @@ type TemplateSource struct {
// TemplateSelector defines MatchTags for looking up VM templates.
// +optional
TemplateSelector *TemplateSelector `json:"templateSelector,omitempty"`

// LocalStorage defines whether the VM template stored on local storage.
// Combination of (TemplateID, SourceNode) and LocalStorage is mutually exclusive.
// +kubebuilder:default=false
// +optional
LocalStorage *bool `json:"localStorage,omitempty"`
}

// VirtualMachineCloneSpec is information used to clone a virtual machine.
Expand Down Expand Up @@ -239,14 +245,16 @@ type VirtualMachineCloneSpec struct {

// Target node. Only allowed if the original VM is on shared storage.
// +optional
// Deprecated: Use `AllowedNodes` instead.
Target *string `json:"target,omitempty"`
}

// TemplateSelector defines MatchTags for looking up VM templates.
type TemplateSelector struct {
// Specifies all tags to look for, when looking up the VM template.
// Passed tags must be an exact 1:1 match with the tags on the template you want to use.
// If multiple VM templates with the same set of tags are found, provisioning will fail.
// If multiple VM templates on same Proxmox node with the same set of tags are found, provisioning will fail.
// If multiple VM templates on different Proxmox nodes with the same set of tags are found, template on node where vm scheduled will be used.
//
// +listType=set
// +kubebuilder:validation:items:Pattern=`^(?i)[a-z0-9_][a-z0-9_\-\+\.]*$`
Expand Down Expand Up @@ -610,12 +618,20 @@ func (r *ProxmoxMachine) GetVirtualMachineID() int64 {
return -1
}

// GetTemplateID get the Proxmox template "vmid" used to provision this machine.
func (r *ProxmoxMachine) GetTemplateID() int32 {
// GetTemplateMap get the Proxmox template "sourceNode:vmid" used to provision this machine.
func (r *ProxmoxMachine) GetTemplateMap() map[string]int32 {
if r.Spec.TemplateID != nil {
return *r.Spec.TemplateID
return map[string]int32{r.Spec.SourceNode: *r.Spec.TemplateID}
}
return -1
return nil
}

// GetLocalStorage get the Proxmox local storage used to provision this machine.
func (r *ProxmoxMachine) GetLocalStorage() bool {
if r.Spec.LocalStorage != nil {
return *r.Spec.LocalStorage
}
return false
}

// GetTemplateSelectorTags get the tags, the desired vm template should have.
Expand Down
5 changes: 5 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 14 additions & 7 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_proxmoxclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,11 @@ spec:
allowedNodes:
description: |-
AllowedNodes specifies all Proxmox nodes which will be considered
for operations. This implies that VMs can be cloned on different nodes from
the node which holds the VM template.
for operations. This implies that VMs can be cloned on different nodes.

This field is optional and should only be set if you want to restrict
the nodes where the VM can be cloned.

If not set, the ProxmoxCluster will be used to determine the nodes.
items:
type: string
Expand Down Expand Up @@ -149,6 +149,12 @@ spec:
This is always done when you clone a normal VM.
Create a Full clone by default.
type: boolean
localStorage:
default: false
description: |-
LocalStorage defines whether the VM template stored on local storage.
Combination of (TemplateID, SourceNode) and LocalStorage is mutually exclusive.
type: boolean
memoryMiB:
description: |-
MemoryMiB is the size of a virtual machine's memory, in MiB.
Expand Down Expand Up @@ -613,8 +619,7 @@ spec:
sourceNode:
description: |-
SourceNode is the initially selected proxmox node.
This node will be used to locate the template VM, which will
be used for cloning operations.
SourceNode should be used together with TemplateID.

Cloning will be performed according to the configuration.
Setting the `Target` field will tell Proxmox to clone the
Expand All @@ -641,8 +646,9 @@ spec:
type: array
x-kubernetes-list-type: set
target:
description: Target node. Only allowed if the original VM
is on shared storage.
description: |-
Target node. Only allowed if the original VM is on shared storage.
Deprecated: Use `AllowedNodes` instead.
type: string
templateID:
description: TemplateID the vm_template vmid used for cloning
Expand All @@ -657,7 +663,8 @@ spec:
description: |-
Specifies all tags to look for, when looking up the VM template.
Passed tags must be an exact 1:1 match with the tags on the template you want to use.
If multiple VM templates with the same set of tags are found, provisioning will fail.
If multiple VM templates on same Proxmox node with the same set of tags are found, provisioning will fail.
If multiple VM templates on different Proxmox nodes with the same set of tags are found, template on node where vm scheduled will be used.
items:
pattern: ^(?i)[a-z0-9_][a-z0-9_\-\+\.]*$
type: string
Expand Down
21 changes: 14 additions & 7 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_proxmoxclustertemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,11 @@ spec:
allowedNodes:
description: |-
AllowedNodes specifies all Proxmox nodes which will be considered
for operations. This implies that VMs can be cloned on different nodes from
the node which holds the VM template.
for operations. This implies that VMs can be cloned on different nodes.

This field is optional and should only be set if you want to restrict
the nodes where the VM can be cloned.

If not set, the ProxmoxCluster will be used to determine the nodes.
items:
type: string
Expand Down Expand Up @@ -173,6 +173,12 @@ spec:
This is always done when you clone a normal VM.
Create a Full clone by default.
type: boolean
localStorage:
default: false
description: |-
LocalStorage defines whether the VM template stored on local storage.
Combination of (TemplateID, SourceNode) and LocalStorage is mutually exclusive.
type: boolean
memoryMiB:
description: |-
MemoryMiB is the size of a virtual machine's memory, in MiB.
Expand Down Expand Up @@ -654,8 +660,7 @@ spec:
sourceNode:
description: |-
SourceNode is the initially selected proxmox node.
This node will be used to locate the template VM, which will
be used for cloning operations.
SourceNode should be used together with TemplateID.

Cloning will be performed according to the configuration.
Setting the `Target` field will tell Proxmox to clone the
Expand All @@ -682,8 +687,9 @@ spec:
type: array
x-kubernetes-list-type: set
target:
description: Target node. Only allowed if the original
VM is on shared storage.
description: |-
Target node. Only allowed if the original VM is on shared storage.
Deprecated: Use `AllowedNodes` instead.
type: string
templateID:
description: TemplateID the vm_template vmid used
Expand All @@ -698,7 +704,8 @@ spec:
description: |-
Specifies all tags to look for, when looking up the VM template.
Passed tags must be an exact 1:1 match with the tags on the template you want to use.
If multiple VM templates with the same set of tags are found, provisioning will fail.
If multiple VM templates on same Proxmox node with the same set of tags are found, provisioning will fail.
If multiple VM templates on different Proxmox nodes with the same set of tags are found, template on node where vm scheduled will be used.
items:
pattern: ^(?i)[a-z0-9_][a-z0-9_\-\+\.]*$
type: string
Expand Down
21 changes: 14 additions & 7 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_proxmoxmachines.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,11 @@ spec:
allowedNodes:
description: |-
AllowedNodes specifies all Proxmox nodes which will be considered
for operations. This implies that VMs can be cloned on different nodes from
the node which holds the VM template.
for operations. This implies that VMs can be cloned on different nodes.

This field is optional and should only be set if you want to restrict
the nodes where the VM can be cloned.

If not set, the ProxmoxCluster will be used to determine the nodes.
items:
type: string
Expand Down Expand Up @@ -140,6 +140,12 @@ spec:
This is always done when you clone a normal VM.
Create a Full clone by default.
type: boolean
localStorage:
default: false
description: |-
LocalStorage defines whether the VM template stored on local storage.
Combination of (TemplateID, SourceNode) and LocalStorage is mutually exclusive.
type: boolean
memoryMiB:
description: |-
MemoryMiB is the size of a virtual machine's memory, in MiB.
Expand Down Expand Up @@ -571,8 +577,7 @@ spec:
sourceNode:
description: |-
SourceNode is the initially selected proxmox node.
This node will be used to locate the template VM, which will
be used for cloning operations.
SourceNode should be used together with TemplateID.

Cloning will be performed according to the configuration.
Setting the `Target` field will tell Proxmox to clone the
Expand All @@ -598,8 +603,9 @@ spec:
type: array
x-kubernetes-list-type: set
target:
description: Target node. Only allowed if the original VM is on shared
storage.
description: |-
Target node. Only allowed if the original VM is on shared storage.
Deprecated: Use `AllowedNodes` instead.
type: string
templateID:
description: TemplateID the vm_template vmid used for cloning a new
Expand All @@ -614,7 +620,8 @@ spec:
description: |-
Specifies all tags to look for, when looking up the VM template.
Passed tags must be an exact 1:1 match with the tags on the template you want to use.
If multiple VM templates with the same set of tags are found, provisioning will fail.
If multiple VM templates on same Proxmox node with the same set of tags are found, provisioning will fail.
If multiple VM templates on different Proxmox nodes with the same set of tags are found, template on node where vm scheduled will be used.
items:
pattern: ^(?i)[a-z0-9_][a-z0-9_\-\+\.]*$
type: string
Expand Down
21 changes: 14 additions & 7 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_proxmoxmachinetemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,11 @@ spec:
allowedNodes:
description: |-
AllowedNodes specifies all Proxmox nodes which will be considered
for operations. This implies that VMs can be cloned on different nodes from
the node which holds the VM template.
for operations. This implies that VMs can be cloned on different nodes.

This field is optional and should only be set if you want to restrict
the nodes where the VM can be cloned.

If not set, the ProxmoxCluster will be used to determine the nodes.
items:
type: string
Expand Down Expand Up @@ -153,6 +153,12 @@ spec:
This is always done when you clone a normal VM.
Create a Full clone by default.
type: boolean
localStorage:
default: false
description: |-
LocalStorage defines whether the VM template stored on local storage.
Combination of (TemplateID, SourceNode) and LocalStorage is mutually exclusive.
type: boolean
memoryMiB:
description: |-
MemoryMiB is the size of a virtual machine's memory, in MiB.
Expand Down Expand Up @@ -611,8 +617,7 @@ spec:
sourceNode:
description: |-
SourceNode is the initially selected proxmox node.
This node will be used to locate the template VM, which will
be used for cloning operations.
SourceNode should be used together with TemplateID.

Cloning will be performed according to the configuration.
Setting the `Target` field will tell Proxmox to clone the
Expand All @@ -639,8 +644,9 @@ spec:
type: array
x-kubernetes-list-type: set
target:
description: Target node. Only allowed if the original VM
is on shared storage.
description: |-
Target node. Only allowed if the original VM is on shared storage.
Deprecated: Use `AllowedNodes` instead.
type: string
templateID:
description: TemplateID the vm_template vmid used for cloning
Expand All @@ -655,7 +661,8 @@ spec:
description: |-
Specifies all tags to look for, when looking up the VM template.
Passed tags must be an exact 1:1 match with the tags on the template you want to use.
If multiple VM templates with the same set of tags are found, provisioning will fail.
If multiple VM templates on same Proxmox node with the same set of tags are found, provisioning will fail.
If multiple VM templates on different Proxmox nodes with the same set of tags are found, template on node where vm scheduled will be used.
items:
pattern: ^(?i)[a-z0-9_][a-z0-9_\-\+\.]*$
type: string
Expand Down
16 changes: 15 additions & 1 deletion docs/advanced-setups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,21 @@ Our provider is able to look up templates based on their attached tags, for `Pro

For example, you can set the `TEMPLATE_TAGS="tag1,tag2"` environment variable. Your custom image will then be used when using the [auto-image](https://github.com/ionos-cloud/cluster-api-provider-ionoscloud/blob/main/templates/cluster-template-auto-image.yaml) template.

Please note: Passed tags must be an exact 1:1 match with the tags on the template you want to use. The matched result must be unique. If multiple templates are found, provisioning will fail.
**N.B.** Passed tags must be an exact 1:1 match with the tags on the template you want to use.

### localStorage

`false` (default)
* Shared storage is assumed for template.
* The template must be accessible from any node.
* Only one template with the matching tags should exist in the entire cluster.
* If multiple templates found, the operation fails.

`true`
* Local storage is assumed for templates.
* Each node listed in the `allowedNodes` is expected to have a copy of the template stored locally.
* If any of the nodes in `allowedNodes` is missing the template, the operation fails.


## Proxmox RBAC with least privileges

Expand Down
6 changes: 6 additions & 0 deletions internal/controller/proxmoxmachine_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,12 @@ func (r *ProxmoxMachineReconciler) Reconcile(ctx context.Context, req ctrl.Reque
return ctrl.Result{}, err
}

// Emit deprecation warning if .spec.Target is set
if proxmoxMachine.Spec.Target != nil {
logger.Info("DEPRECATION NOTICE: .spec.Target is deprecated and will be removed in a future release. Use .spec.AllowedNodes instead.",
"ProxmoxMachine", req.NamespacedName)
}

// Fetch the Machine.
machine, err := util.GetOwnerMachine(ctx, r.Client, proxmoxMachine.ObjectMeta)
if err != nil {
Expand Down
Loading