Add nodeSelector support in ClusterGroup

[luolanzone]

There is a use case reported from the end user that four clusters are sharing one API service IP (shared one controller plane?) in their env. The user wants to set up ACNPs to make sure only selected Pods are allowed to the K8s API Service, so he/she created a ClusterGroup with pod and namespace selector matching kube-apiserver labels, however, due to the Antrea-controller would omit the hostNetwork Pods in ClusterGroup members, it doesn't meet their requirement.

In order to allow users to create ACNPs to do the right network policy control in such scenarios, it would be good to have nodeSelector support in ClusterGroup.

[luolanzone]

cc @tnqn @edwardbadboy in case I missed any critical information.

[luolanzone]

This is to make the selector consistent on ACNP and ClusterGroup. We now already support nodeSelector in ACNP, but not in ClusterGroup.

[Dyanngg]

I want to clarify the scope for this change here:

1. nodeSelector is supported in ACNP both in ingress/egress rules and in appliedTo. However, the appliedTo case is a gated feature: when an ACNP has nodeSelector in appliedTo, it becomes a NodeNetworkPolicy. Currently this feature is in alpha and not enabled by default. So we need to determine how to deal with nodeSelector CG reference as appliedTo in ACNP: one idea is to check for feature flag enablement in an ACNP webhook if a nodeSelector CG is used in the appliedTo of an ACNP. This however introduces strong dependency of resources: we currently do not validate whether a CG exists at the time when it is referenced in an ACNP being created. I don’t feel this feature should break this… Another idea would be allow this combination to be applied in the cluster in the first place, then when Antrea controller process the ACNP, it marks the policy unrealizable in its status. @antoninbas @tnqn what do you guys think?
2. Also just to confirm that the ask is only to add the nodeSelector in cluster-scope resources, and not namespaced ‘Group’ and ‘ANNP’ references, since it would not make sense for creating a namespaced nodeSelector.
3. Antrea provides the ClusterGroupMember and IPGroupAssociation APIs for ClusterGroups. Should we extend these to support nodeSelector case? I.e. display Node info in ClusterGroupMember queries, and be able to reverse look up CGs by node IP

[antoninbas]

Even today, the NodeNetworkPolicy FeatureGate only exists for the Agent. So a nodeSelector in the appliedTo field is always processed the same way in the Controller: the selector is resolved to a list of Nodes and each Node in the list will get the policy.

@Dyanngg do you want to limit the nodeSelector field to ClusterGroups used as peers (and not allow such a ClusterGroup to be used in an appliedTo)? This is what we do for groups with IPBlocks today. I think this is the use case we are trying to address here. It would greatly simplify the implementation IMO. Otherwise, if we want to allow such groups for appliedTo, we have to have some restrictions, or things can get pretty complicated if a group can have children, with one child selecting Nodes and one child selecting Pods...

[petertran-avgo]

AFter my chat with Yang, I'm going to proceed with these assumptions:

• Like with CNP's appliedTo nodeSelectors, if there is a cluster group with mixed node/pod selector children, the policy will be ignored
• For now, I will not implement supporting a cluster group with nodeSelectors in appliedTo and only in Ingress and Egress rules
  • Verify if node network policy is enabled, check if using a clustergroup with nodeselector in appliedto is already working

Just double check with @antoninbas - do you agree we should start with not supporting a ClusterGroup with mixed node and pod selector group children?

[antoninbas]

do you agree we should start with not supporting a ClusterGroup with mixed node and pod selector group children?

I don't think there is a need for such a restriction as long as a group (with or without children) that selects Nodes cannot be used in an appliedTo.

[petertran-avgo]

do you agree we should start with not supporting a ClusterGroup with mixed node and pod selector group children?

I don't think there is a need for such a restriction as long as a group (with or without children) that selects Nodes cannot be used in an appliedTo.

Seems like a mixed clustergroup for ingress/egress is not going to be a problem? is that right?

[Dyanngg]

Had a chat with @antoninbas and we think that's it's probably easier to just implement nodeSelector support in appliedTo groups.