Top of the rack switch for hadoop cluster năm 2024

Hadoop can consume all available network bandwidth. For this reason, Cloudera recommends that Hadoop be placed in a separate physical network with its own core switches.

Switch per rack

Hadoop supports the concept of rack locality and takes advantage of the network topology to minimize network congestion. Ideally, nodes in one rack should connect to a single physical switch. Two top- of- rack (ToR) switches can be used for high availability. Each ToR switch uplinks to a core switch with a significantly bigger backplane. Cloudera recommends 10 GbE (or faster) connections between the servers and ToR switches. ToR uplink bandwidth to the core switch (two switches in a HA configuration) can often be oversubscribed.

Uplink oversubscription

How much oversubscription is appropriate depends on the workload. Cloudera’s recommendation is that the ratio between the total access port bandwidth and uplink bandwidth be as close to 1:1 as possible. This is especially important for heavy ETL workloads and MapReduce jobs that have a lot of data sent to reducers.

Oversubscription ratios up to 4:1 are generally fine for balanced workloads, but network monitoring is needed to ensure uplink bandwidth is not the bottleneck for Hadoop. The following table provides some examples as a point of reference:

Access Port Bandwidth (In Use) Uplink Port Bandwidth (Bonded) Ratio 48 x 1 GbE = 48 Gbit/s 4 x 10 GbE = 40 Gbit/s 1.2:1 24 x 10 GbE = 240 Gbit/s 2 x 40 Gig CFP = 80 Gbit/s 3:1 48 x 10 GbE = 480 Gbit/s 4 x 40 Gig CFP = 160 Gbit/s 3:1

Redundant network switches

Having redundant core switches in a full mesh configuration allows the cluster to continue operating in the event of a core switch failure. Redundant ToR switches prevent the loss of an entire rack of processing and storage capacity in the event of a ToR switch failure. General cluster availability can still be maintained in the event of the loss of a rack, as long as master nodes are distributed across multiple racks.

Accessibility

The accessibility of your CDP Private Cloud Base cluster is defined by the network configuration and depends on the security requirements and the workload. Typically, there are edge/client nodes that have direct access to the cluster. Users go through these edge nodes through the client applications to interact with the cluster and the data residing there. These edge nodes may be running a web application for real-time serving workloads, BI tools, or simply the Hadoop command-line client used to submit or interact with HDFS.

Cloudera recommends allowing access to the CDP Private Cloud Base cluster through edge nodes only. You can configure this in the security groups for the hosts that you provision. The rest of this document describes the various options in detail.

Internet connectivity

Clusters that do not require heavy data transfer between the internet or services outside of the immediate network and HDFS, might need access to services like software repositories for updates or other low-volume data sources located outside of the immediate network.

If you intend to leverage the multi-cloud/hybrid-cloud functionality in CDP, then you must ensure that adequate network bandwidth is present between your data centers and the public cloud vendors’ networks. Details on this topic are out of scope of this document. Engage with your Cloud vendor’s technical sales team and Cloudera Sales Engineering team to determine the requirements in such scenarios.

If you completely disconnect the cluster from the internet, you block access for software updates which makes maintenance difficult.

Hadoop is very bandwidth-intensive. Use dedicated switches for your Hadoop Cluster. Nodes are connected to a top-of-rack switch ToR. Nodes should be connected at a minimum speed of 1GB/sec . Use 10Gb/sec connections if the jobs produce large amount of intermediate data. For redundancy Consider bonded ethernet to mitigate against failure. Redundant ToR and core switches. Racks are interconnected via core switches. Core switches should connect to top-of-rack switches at 10Gb/sec or faster. Hostname Resolution Use hostnames, not IP addresses, to identify nodes when configuring Hadoop. Use DNS for hostname resolution. Each host must be able to: Perform a forward lookup on its own hostname Perform a reverse lookup using its own ip address. If forward and reverse lookups results do not match, major problems can occur. Rack Topology Awareness To Maximise performance, specify the network locations of hosts and racks appropriately. In CM - Hosts -> RACK -> Pick default -> Actions for selected -> Assign RACK. Rack information can be obtained from DC Teams. Any hosts in /default location needs to be rack assigned. When configuring systems Do not use Linux's LVM (Logical Volume Manager) to make all your disks appear as a single volume. Check the BIOS are optimally configured. Test disk I/O hdparm -t /dev/sda1 Speed should be 70MB/sec or more, anything less is an indication of possible problems. Mount disks with the noatime option. set vm/swappiness to 0 in /etc/sysctl.conf Configure IPTables if required by your security policy. To find out the ports used for communication. CM Cluster page -> configurations -> All port configurations. to see all ports used. Disable Transparent Huge Page compacAon – Can degrade the performance of Hadoop workloads – Open the defrag file of your OS to see if it is enabled – Red Hat/CentOS: /sys/kernel/mm/\ redhat_transparent_hugepage/defrag – Ubuntu/Debian, OEL, SLES: /sys/kernel/mm/\ transparent_hugepage/defrag – A line reading ‘[always] never’ means it is enabled – A line reading ‘[never] always’ means it is disabled – To temporarily disable it – sudo sh -c "echo 'never' > defrag_file_pathname" – Add the following to /etc/rc.local to persist the change – echo never > defrag_file_pathname Use ext4 and ext3 filesystem for performance XFS provides some performance benefits during kickstart XFS formats in 0 seconds, vs several minutes for each disk with ext3/ext4. Increase the nofile ulimit for the cloudera-scm user to at least 32K – Cloudera Manager sets this to 32K in /usr/sbin/cmf-agent by default Disable IPv6 Disable SELinux if possible – Incurs a performance penalty on a Hadoop cluster – ConfiguraGon is non-trivial Install and configure the ntp daemon – Ensures the Gme on all nodes is synchronized – Important for HBase, ZooKeeper, Kerberos – Useful when using logs to debug problems Java Virtual Machine (JVM) Requirements Version 1.7 is required – CDH5 is cerGfied with 1.7.0_55 – Any later maintenance release should be acceptable. RAID (on worker nodes) and virtualizaAon can incur a performance penalty

What are top of rack switches?

Top-of-rack switches provide scalability in managing enclosures and servers by providing higher bandwidth demand for each server with access-layer redundancy. See the HPE OneView Support Matrix for HPE Synergy for the complete list of supported devices.

What is the primary function of the ToR top of rack switch?

The main purpose of ToR switching is to increase network efficiency, reduce cabling complexity, and improve rack space utilization for data centers. ToR switches enable a high-speed connection between servers and storage by providing an aggregation point for communications traffic between them.

What is a switch in a server rack?

Network switches are termed as a network bridge with multiple ports that help connect the devices within the network effectively. These switches act as the brain of the network as they use media access control (MAC) addresses to receive and forward the data to the destination.

What is a rack in Hadoop cluster?

A rack is nothing but a collection of 30-40 DataNodes or machines in a Hadoop cluster located in a single data center or location. These DataNodes in a rack are connected to the NameNode through traditional network design via a network switch. A large Hadoop cluster will have multiple racks.