Transcript
IBM Hortonworks Design Guide 14-Sep-17 v1
Main Benefits
Unique Capability
MAXIMIZE THE NETWORK • Maximize datacenter connectivity ROI through: Density, Scale, Performance OPEN THE NETWORK • Leverage new technologies that increase functionality, investment & ROI • Freedom from vendor lock-in
MELLANOX: THE DIFFERENCE IS IN THE CHIP • Founded as a state-of-the-art silicon chip (ASIC) manufacturer • Intelligence built directly onto our own chip • Other switch vendors are forced to source expensive ASICs from third parties such as Broadcom • Mellanox uses own chip & passes savings to customers 6th Generation
10/40/56GbE & 40/56 Gb IB
Mellanox designs & builds intelligent ASICs that power switches, adapters, & cables
7th Generation 40/56/100 Gb IB
Key Differentiators VALUE & PERFORMANCE The Enterprise Integrated model is familiar to those with traditional SAN deployments. Adding ESS/Spectrum Scale will not only eliminate the data silos, but can also improve performance and reduce data bottleneck. The most common deployment is using Network Shared Disks, whose modular design scales performance and capacity independently. For those familiar with HDFS, or other scale-out software-defined storage, we support shared nothing clusters that provide the native locality APIs for HDFS, but work like centralized parallel storage for other protocols. Using commodity storage rich servers can be the most economical way to scale out your storage needs.
Market Opportunities
SCALE-OUT STORAGE
Combines compute & storage, easier to manage & lowers cost – top of rack switch with density at lower price point most attractive
CLOUD
MEDIA & ENTERTAINMENT
Video streaming & postproduction on 4k/8k workflows – needs extreme high bandwidth to support real-time frame-rates
Create economies of scale through shared services – open switch platform with fairness best for software-defined DC.
BIG DATA
GENOMICS
Extreme scalability using a building - block approach: Capacity, bandwidth and a single name space expand as more building blocks are added, resulting in nearlinear performance gains
Improved analytics for better business decisions – SCALE OUT DATABASE needs non-blocking Scale out of DB2 PureScale, architecture to speed data Oracle RAC, SAP HANA ingestion.
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent
Company Background Established 1999 * NASDAQ: MLNX End-to-end Ethernet Connectivity Solutions – Adapters, Switches, Cables, Software, Support World-class, non-outsourced technical support Trusted as switch manufacturer for every major server OEM
www.mellanox.com
[email protected]
+1 (512) 897-8245
IBM Mellanox Infrastructure for Hortonworks Speed
Switch
Cabling
Adapter
SB7700 – 8828-E36
EDR
SX1410 – 8831-S48
10/40 GbE
See lists on right
EKAU 10/25 EKAL 2@40 EC3L 2@40 LOM
Flat
Internet
Dual Home
Internet
EDW
Firewall
E
S
Private
Internet
“Thin DMZ”
Firewall
M
EB41
2m
40GbE / FDR Copper Cable QSFP
EB42
3m
40GbE / FDR Optical Cable QSFP
EB4A
5m
40GbE / FDR Optical Cable QSFP
EB4B
10m
40GbE / FDR Optical Cable QSFP
EB4C
15m
40GbE / FDR Optical Cable QSFP
EB4D
20m
40GbE / FDR Optical Cable QSFP
EB4E
30m
40GbE / FDR Optical Cable QSFP
EB4F
50m
40GbE / FDR Optical Cable QSFP
EB4G
DMZ
Public
S
S Private
Description
M
E
X
Firewall
S
S
FC EB50
1m
EDR Copper Cable QSFP28
EB51
2m
EDR Copper Cable QSFP28
EB52
1.5m EDR Copper Cabling QSFP28
Firewall
Public
E
S
S
S
Partial Home
Internet
E
EB40
40GbE / FDR Copper Cable QSFP
0.5m EDR Copper Cable QSFP28
M
FC
1m
Length Public
M
Description
EDR Cabling
Public
Firewall
Length
0.5m 40GbE / FDR Copper Cable QSFP
EKAL 2@40 EC3L 2@40
4610-54T – 8831-S52
1/10GbE
Optics*
EKAL FDR/EDR
SX1710 – 8831-NF2
40 GbE
Choice of Cabling
40GbE / FDR Cabling
EB54
3m
EDR Optical Cable QSFP28
EB5A
5m
EDR Optical Cable QSFP28
EB5B
10m
EDR Optical Cable QSFP28
EB5C
15m
EDR Optical Cable QSFP28
EB5D
20m
EDR Optical Cable QSFP28
EB5E
30m
EDR Optical Cable QSFP28
EB5F
50m
EDR Optical Cable QSFP28
EB5G
100m EDR Optical Cable QSFP28
EB5H
Private
* Optics are IBM Parts only FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
IBM Mellanox Infrastructure for Hortonworks As you increase the speed of the network, the topology of the PCI slot becomes important. The two topologies that IBM has for the cards and slots in the servers are 1. PCI Gen 3.0 x8 2. PCI Gen 3.0 x16 The important piece is the x8/x16, what does this mean? This is the width of the PCI bus, how much bandwidth can be passed from Network to the CPU. How much network bandwidth can be passed thru these two PCI Slots. Speed
PCI Gen 3.0 x8 - # Ports
10 GbE
2
25 GbE
2
40 GbE
1
50 GbE
FC#
PCI Gen 3.0 x16 - # Ports
FC#
2
-
2
-
EC3A
2
EC3L / EKAL
1
EKAM*(x16 Card)
2
EC3L / EKAL
56 GbE
1
EC3A
2
EC3L / EKAL
100 GbE
0
-
1
EC3L / EKAM
FDR
1
-
2
EL3D / EKAL
EDR
0
-
1
EC3E / EKAL
EKAU
NOTE: To provide Active/Active redundant network, the PCI Slot must have enough bandwidth to pass the data from the CPU to the Network. IBM FC# EC3A is only a PCI Gen3.0 x8 Card so is limited to max bandwidth of 56Gb To achieve dual 40GbE Active/Active redundant network, the FC# EC3L or EKAL should be used with both ports connected @40GbE on a card with PCI Gen3.0 x16. NOTE: Bonding: The most common mode is Mode 4 LACP/802.3ad, this has an overhead and is originally to bond low speed unreliable links. With the implementation of modern Ethernet networks and enhancements to Linux. Mode 5 – TLB and Mode 6 – ALB. Using Mode 5/6 are good choice as they have less overhead than Mode 4 and they require no configuration on the switches to provide Active / Active redundancy. NOTE: When Mellanox is configured end to end, Adapter, Cable and Switch, there is a free upgrade to Mellanox supported 56GbE. This provides 40% more bandwidth than 40GbE. Activation is a single command on the required interface of “speed 56000” on the switch interface. NOTE: To achieve redundant network for IB - FDR, FC#EC3E / EKAL @ 2x FDR To achieve redundant network for IB - EDR 2x FC#EC3E / EKAL @ EDR Redundancy is provided by Mode 1 Active / Standby, the bond is created the same as normal Linux bond
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
Choice of Cabling
IBM Mellanox Infrastructure for 10 GbE Cluster Hortonworks Switch
Speed 40 GbE
10/40 GbE
Cabling
Adapter
Optics
EC3A
EB27 + EB2J or EB2K
EC37 / EL3X EC2M / EL40 (SR)
EB28 + ECBD or ECBE
SX1710 – 8831-NF2
SX1410 – 8831-S48
See list on right
4610-54T – 8831-S52
1GbE
Sample 192 Node L2 (VMS) Cluster
8831-NF2 36x 40GbE
IPL 4x 56GbE
40GbE / FDR Cabling Length
Description
FC
0.5m 40GbE / FDR Copper Cable QSFP EB40 1m
40GbE / FDR Copper Cable QSFP
EB41
2m
40GbE / FDR Copper Cable QSFP
EB42
3m
40GbE / FDR Optical Cable QSFP
EB4A
5m
40GbE / FDR Optical Cable QSFP
EB4B
10m
40GbE / FDR Optical Cable QSFP
EB4C
15m
40GbE / FDR Optical Cable QSFP
EB4D
20m
40GbE / FDR Optical Cable QSFP
EB4E
30m
40GbE / FDR Optical Cable QSFP
EB4F
50m
40GbE / FDR Optical Cable QSFP
EB4G
Spine
6x 40 GbE Link per Spine
6 x 40 GbE Link per Spine
8831-S48 48x 10GbE + 12x 40GbE Leaf
T T
T
T
10 GbE Client
T
48x 10 GbE Endpoints per Leaf
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
Choice of Cabling
IBM Mellanox Infrastructure for 40GbE Cluster Hortonworks Switch
Speed 40 GbE
Cabling
SX1710 – 8831-NF2
See list on right
40GbE / FDR Cabling
Adapter
Optics
EKAL
EB27 + EB2J or EB2K
Length
FC
0.5m 40GbE / FDR Copper Cable QSFP EB40
Sample 72 Node L2 (VMS) Cluster
8831-NF2 36x 40GbE
Description
IPL 4x 56GbE
1m
40GbE / FDR Copper Cable QSFP
EB41
2m
40GbE / FDR Copper Cable QSFP
EB42
3m
40GbE / FDR Optical Cable QSFP
EB4A
5m
40GbE / FDR Optical Cable QSFP
EB4B
10m
40GbE / FDR Optical Cable QSFP
EB4C
15m
40GbE / FDR Optical Cable QSFP
EB4D
20m
40GbE / FDR Optical Cable QSFP
EB4E
30m
40GbE / FDR Optical Cable QSFP
EB4F
50m
40GbE / FDR Optical Cable QSFP
EB4G
Spine 7x 56 GbE Link per Spine
7x 56 GbE Link per Spine
Leaf Q
4
40 GbE Data Network
4
40 GbE Client
Q
X
X
OR
C
OR
C
C
C
4
4
4
4
18x 40 GbE Endpoints per Leaf
C
40 GbE Customer Network
Q
QSFP to SFP+ Adapter (QSA)
X
* SFP+ DAC or Transceiver
C
10 GbE Customer Network
* 10GBE & Optics are IBM Parts only FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
Choice of Cabling
IBM Mellanox Infrastructure for 40 GbE Cluster Hortonworks Switch
Speed
Cabling
Adapter
Optics
See list on right
EC3A 1@40GbE EC3L 2@40GbE EKAL 2@40GbE
EB27 + EB2J or EB2K
SX1710 – 8831-NF2
40 GbE
40GbE / FDR Cabling Length
Sample 72 (90) Node Redundant L3 (VMS) Cluster Dedicated Storage switches
36port 40GbE
36port 40GbE
6 Ports per TOR
6 Ports per TOR
36port 40GbE
36port 40GbE
36port 40GbE
36port 40GbE
FC
0.5m 40GbE / FDR Copper Cable QSFP EB40
4610-54T – 8831-S52
1 GbE
Description
6 Ports per TOR
1m
40GbE / FDR Copper Cable QSFP
EB41
2m
40GbE / FDR Copper Cable QSFP
EB42
3m
40GbE / FDR Optical Cable QSFP
EB4A
5m
40GbE / FDR Optical Cable QSFP
EB4B
10m
40GbE / FDR Optical Cable QSFP
EB4C
15m
40GbE / FDR Optical Cable QSFP
EB4D
20m
40GbE / FDR Optical Cable QSFP
EB4E
30m
40GbE / FDR Optical Cable QSFP
EB4F
50m
40GbE / FDR Optical Cable QSFP
EB4G
6 Ports per TOR
Layer 3 OSPF/ECMP Network Mellanox VMS
36port 40GbE
18ports
36port 40GbE
18ports
36port 40GbE
18ports
36port 40GbE
36port 40GbE
18ports
36port 40GbE
18ports
36port 40GbE
18ports
18ports
36port 40GbE
18ports
36port 40GbE
18ports
36port 40GbE
36port 40GbE
18ports
Mode 6 - ALB 72x Top Port Dual Port Card
2x EC3L per NSD ESS
ESS
ESS
ESS
72x Bottom Port Dual Port Card
4x ESS with 4x EC3L Cards @ 2x 40Gb 32x 40Gb Ports = ~ 112GB
2x 100Gb Cards x 2 Ports @ 40 = 160Gb per NSD 3x 40Gb Cards x 1 Port @ 40 = 120Gb per NSD Mode 6 - ALB
Compute Node
1x EKAL @ 2x 40Gb per Node
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
Choice of Cabling
IBM Mellanox Infrastructure for IB Cluster Hortonworks Speed FDR / EDR
Switch SB7700 – 8828-E36
Cabling
EDR Cabling
Adapter
Optics
Length
Description
FC
0.5m EDR Copper Cable QSFP28 See list on right
EKAL
NA
EB50
1m
EDR Copper Cable QSFP28
EB51
2m
EDR Copper Cable QSFP28
EB52
1.5m EDR Copper Cabling QSFP28
Sample 72 Node Cluster
8828-E36 36x QDR/FDR/EDR Spine
9x EDR Link per Spine
9x EDR Link per Spine
EB54
3m
EDR Optical Cable QSFP28
EB5A
5m
EDR Optical Cable QSFP28
EB5B
10m
EDR Optical Cable QSFP28
EB5C
15m
EDR Optical Cable QSFP28
EB5D
20m
EDR Optical Cable QSFP28
EB5E
30m
EDR Optical Cable QSFP28
EB5F
50m
EDR Optical Cable QSFP28
EB5G
100m EDR Optical Cable QSFP28
EB5H
Leaf
E
E
E
E
18x IB Endpoints per Leaf
18x IB Endpoints per Leaf
18x IB Endpoints per Leaf
18x IB Endpoints per Leaf
Some Rules: • Links from Leaf to Spine must be Modulo of 18 - 1,2,3,6,9 • Non-Blocking requires as many links down to servers from Leaf as up to Spine from Leaf • Biggest two tier network is 648 Nodes, 18 Spines & 36 Leafs • Think ahead. Add Spines at day 1 for expansion, so extra leafs can be added without re-cabling existing leafs
E
QDR/FDR/EDR Client
# Links to Spine
# Spines
#Leafs
#Ports
1
18
36
648
2
9
18
324
3
6
12
216
6
3
6
108
9
2
4
72
18
1
1
36
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
IBM Mellanox Infrastructure for ESS/Spectrum Scale GB Bandwidth per Port per Speed for Single NSD/IO Node One Port
EDR
8.5
100GbE
8.0
17.0 16.0
FDR
5.0
100GbE@2x 56GbE
4.48
8.96
13.44
56GbE
4.48
8.96
13.44
3.6
40GbE
3.2
2 5 G b E 1.8 3.6
11.0
7.2 6.4 5.4
1 0 G b E 0.8 1.62.4 3.2 4.0
Four Ports
Five Ports
Six Ports
24.0
5.5
16.5
10.0
Three Ports
25.5
EDR@2x FDR
100GbE@2x 40GbE
Two Ports
22.0
15.0
10.8
27.5
20.0
25.0
17.92
14.4
33.0
22.4
18.0
30.0 26.88
21.6
9.6 7.2
9.0
10.8
4.8
SINGLE NSD Port Bandwidth options Ports 10GbE 25GbE 40GbE One Port 0.8 1.8 3.2 Two Ports 1.6 3.6 6.4 Three Ports 2.4 5.4 9.6 Four Ports 3.2 7.2 Five Ports 4.0 9.0 Six Ports 4.8 10.8
100GbE@2x 40GbE 3.6 7.2 10.8 14.4 18.0 21.6
56GbE 4.48 8.96 13.44
100GbE@2x 56GbE 4.48 8.96 13.44 17.92 22.4 26.88
FDR 5.0 10.0 15.0 20.0 25.0 30.0
EDR@2x FDR 5.5 11.0 16.5 22.0 27.5 33.0
100GbE 8.0 16.0 24.0
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
EDR 8.5 17.0 25.5
14-Sep-17 v1
IBM Mellanox Infrastructure for ESS/Spectrum Scale
Max Sequential Throughput (GBytes/s)
Read, IOR, Infiniband+RDMA network, 16MB filesystem blocksize (ESS)
40
Sequential throughput vs. Capacity for selected ESS models
35
GL6S = 34 GB/s
30
GL6 = 25 GB/s
25
GL4S = 23 GB/s
20
GL4 = 17 GB/s
15
GL2S = 11 GB/s
10
GL2 = 8 GB/s 5
10
100
1,000
TB Usable Capacity
Approx max capacity using 8+2P (ESS), combined MD+Data pool. Note logarithmic scale.
Dual NSD Port Bandwidth options Ports per NSD 10GbE 25GbE One Port 1.6 3.6 Two Ports 3.2 7.2 Three Ports 4.8 10.8 Four Ports 6.4 14.4 Five Ports 8.0 18. Six Ports 9.6 21.6
40GbE 6.4 12.8 19.2
100GbE@ 2x 40GbE 7.2 14.4 21.6 28.8 36.0 43.2
56GbE 8.96 17.92 26.88
100GbE@ 2x 56GbE 8.96 17.92 26.88 35.84 44.8 52.76
FDR 10.0 20.0 30.0 40.0 50.0 60.0
EDR@2x FDR 11.0 22.0 33.0 44.0 55.0 66.0
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
100GbE 16.0 32.0 48.0
EDR 17.0 34.0 51.0
14-Sep-17 v1
IBM Support Contacts – Thank You Duane Dial – Director of Sales, IBM WW
[email protected] 512-574-4360 Jim Lonergan – Business Development IBM WW
[email protected] [email protected] Sametime
[email protected] 512-897-8245 Lyn Stockwell-White – North America Channels IBM
[email protected] [email protected] Sametime
[email protected] 602-999-5255 Matthew Sheard - Solutions Architect – IBM WW
[email protected] [email protected] Sametime
[email protected] 919-360-1654 John Biebelhausen – Sr. OEM Marketing
[email protected] 512-770-4991 FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
www.mellanox.com/oem/ibm
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
https://community.mellanox.com/community/solutions
FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
http://academy.mellanox.com/en/ FOR INTERNAL USE ONLY – cannot be posted online or reproduced without Mellanox consent www.mellanox.com
[email protected] +1 (512) 897-8245
14-Sep-17 v1
