Networking

Contributing hardware — For institutions and campus IT staff adding servers to the NRP pool.

Our cluster contains several hundreds of nodes around the world, and to make a cluster out of these we place certain connectivity requirements to join a node.

The nodes are mostly connected to a Science DMZ with 10G-100G speeds. To utilize this speed, they should support jumbo frames (9000 MTU) to all other nodes in the cluster. Also, the NRP usually requires the nodes to be on a Science DMZ, outside of traditional firewalls, IDS, and ACLs that significantly lower connection speed and also add restrictions that can make a node unusable to the NRP. See ESNet architecture documents

Kubernetes is building a virtual (overlay) network on top of physical network, which makes all nodes see each other directly via established VXLAN tunnels. Also many services are using ports on nodes. Because we have so many locations, we can’t work with each campus to handle the network policy for us. Instead, we ask the local admins to open the node completely to the world, and use the cluster-wide Network Policies provided by Calico network plugin to handle the firewall.

The nodes are expected to have access to both ScienceDMZ and public Internet (could be at a slower speed) to pull container images from docker hub and google container image repository.

Calico Network Policies

Calico is the network plugin we use for Kubernetes. It provides the GlobalNetworkPolicy resource to manage the preNAT network policy on hosts in a way that allows us to protect the hosts from the world, while still allowing all needed connectivity between the hosts. With some exceptions, hosts will have a few ports and ICMP open to the world, and only allow full connectivity between the cluster hosts defined by our networkset. Because the cluster is constantly changing, it’s not possible to manage the local university policies by sending emails to each admin, and central management is the only way that works.

You can review the current policy applied to the cluster. It adds the IPTables rules to the nodes to filter the incoming traffic in additional to multiple other rules that Kubernetes uses to route packets between hosts. This is often incompatible with manual iptables rules or firewalld enabled on the node, and because of that we ask you to turn those off.

Ports by node type

Most of what a node receives comes from other Nautilus nodes: the pod network, the kubelet, and storage, on many ports. Those nodes number in the hundreds and change constantly, so a campus firewall can’t track them. Leave the node unfiltered and the Calico policy handles it.

From the rest of the internet, a node accepts only a short list of ports, set by the services it runs. Tick them below to see the list, for example for your security office.

This node runs

Every node The minimum to join Nautilus and be administered. Always required.
OSDF cache or origin A Pelican cache or origin serving the Open Science Data Federation.
Globus endpoint A Globus Connect Server data transfer node.

Open to the internet

22/tcp SSH Administrative access for NRP operators.
443/tcp HTTPS Opened on every node by the cluster policy.
icmp ICMP Ping, and path-MTU discovery: jumbo frames only work if "fragmentation needed" messages get through.
8443/tcp Data access Object reads over HTTPS and XRootD. Clients and other caches across the federation fetch from here.
8444/tcp Pelican web Token issuance, director health checks, and metrics.
50000–51000/tcp GridFTP data Data channels for transfers, opened directly to the remote endpoint.

This work was supported in part by National Science Foundation (NSF) awards CNS-1730158, ACI-1540112, ACI-1541349, OAC-1826967, OAC-2112167, CNS-2100237, CNS-2120019.