Visio by DPTPB: Transitional SAS cables, QSFP to Mini-SAS HD

 

Older generation SAS 6Gbps ports use QSFP form factor connector  while newer generation SAS 12Gbps ports use smaller Mini-SAS HD form factor connector (at least with NetApp gear). These form factors are not compatible, i.e you cannot cable QSFP connector to Mini-SAS HD port or vice-versa.

To make things more confusing, there are also 12Gbps capable ports that are using QSFP form factor connector. In case of NetApp, X2069-R6 adapter is example of such combination.

In order to connect  ports using different form factor connectors, a special transitional cable is required:

  • One cable end with QSFP form factor connector
    • With NetApp onboard ports speed is typically limited to 6 Gbps
    • With NetApp adapter cards speed can be up to 12 Gbps
  • One cable end with Mini-SAS HD form factor connector
    • Typically speed is 12 Gbps
    • But can also operate at 6 Gbps

There are few use cases where transitional SAS cables are required.

6Gbps QSFP controller ports – to – 12 GBps Mini-SAS HD disk shelf ports

QSFP Controller ports

  • Onboard SAS ports on previous generation FAS/AFF controller
    • Operates at 6 Gbps
  • 4-port 6 Gbps SAS adapter card
    • X2065 Operates at 6 Gbps

Mini-SAS HD Disk shelf ports

  • DS212C and DS224C disk shelf
    • Equipped with IOM12 module
    • Operates at 12 Gbps

Typical use case

  • Older generation FAS/AFF controller using new generation disk shelves
  • NetApp P/N for 1 m cable:
    • X66020A-R6

Visio shapes

  • Controller side operates at 6Gbps
  • Disk shelf side operates at 12 Gbps
  • hence name SAS6-SAS12
  • Since “destination” ports are labeled with numbers instead of shapes, the cable shapes are named accordingly.
  • Variants
    • Node1 cables: dark blue
      • A or C ports to “1” port
      • B or D ports to “3” port
    • Node2 cables: light blue
      • A or C ports to “1” port
      • B or D ports to “3” port
  • Cable ends
    • Square and Circle shapes with either 6G or 12G text indicating speed
Screenshot: SAS6-to-SAS12 transitional cable shape thumbnails in Visio Shapes Window

screen-shot-2016-11-16-at-23-46-44

Example: Older generation FAS2552 with new generation SAS DS224C shelves

Screen Shot 2016-11-19 at 09.33.07.png

  • SAS cable shapes have QSFP (“6G”) style cable ends at controller side
  • SAS cable shapes have Mini-SAS HD (“12G”) style cable ends at disk shelf side
  • No ACP cabling since new disk shelves do not have separate ACP ports
  • Effective speed determined by the slowest component
    • Controller ports 6Gbps
    • IOM12 ports 12Gbps
    • Shelf back-plane 12Gbps
    • = Effective speed 6Gbps
  • Instead In-Band Alternate Control Path (IB ACP) is used instead (requires Ontap9)
    • Can operate without ACP with older Ontap versions (with some limitations on admin operations)

12Gbps Mini-SAS HD controller ports – to – 6 GBps QSFP disk shelf ports

Controller ports Mini-SAS HD

  • Onboard ports on new generation controllers
    • FAS26xx / AFF A200
    • FAS8200 / AFF A300
    • FAS9000 / AFF A700
  • 4-port SAS adapter
    • X92071A for FAS9000 / AFF A700
    • No adapter with Mini-SAS HD ports for FAS8200

QSFP Disk shelf ports

  • DS2246 / DS4246 / DS4486 disk shelf
    • Equipped with IOM6 module
    • Operates at 6 Gbps

Typical use case

  • New generation platform connecting to older generation disk shelf
  • Since there is no new 24-bay disk shelf for 3.5″ inch drives (ie no DS424C shelf)
  • I would expect that most common example would be new FAS26xx using older generation DS4246 shelf
  • NetApp P/N for 1 m cable
    • X66020A-R6

Visio shapes

  • Controller side operates at 12Gbps
  • Disk shelf side operates at 6 Gbps
  • hence name SAS12-SAS6
  • Since “destination” ports are labeled with shapes, the cable shapes are named accordingly.
  • Variants
    • Node1 (N1) cables: dark blue
      • A or C ports to Square port
      • B or D ports to Circle port
    • Node2 (N2) cables: light blue
      • A or C ports to Square port
      • B or D ports to Circle port
  • Cable ends
    • Square and Circle shapes with either 6G or 12G text indicating speed
Screenshot: SAS12-to-SAS6 transitional cable shape thumbnails in Visio Shapes Window

screen-shot-2016-11-16-at-23-46-03

Example: New generation FAS2650 with Older generation SAS DS4246 shelves

screen-shot-2017-01-25-at-14-41-00

  • SAS cables shapes have Mini-SAS HD (“12G”) style cable ends at controller side
  • SAS cables shapes have QSFP (“6G”) style cable ends at disk shelf side
  • Effective speed determined by the slowest component
    • Controller ports 12Gbps
    • Shelf back-plane 6 Gbps
    • IOM6 ports 6Gbps
    • = Effective speed 6Gbps
  • No ACP cabling since new FAS platforms do not have “Locked Wrench” port used for ACP cabling – No out-of-band Alternate Control Path (OOB ACP)
  • Instead In-Band Alternate Control Path (IB ACP) is used instead (requires compatible shelf firmware for IOM6 shelves & Ontap9)
  • Minimum Ontap for FAS2650 is 9.1RC1

12Gbps QSFP controller ports – to – 12 GBps Mini-SAS HD disk shelf ports

QSFP Controller ports

  • 4-port SAS Adapter
    • X2069-R6
    • For FAS8200 / AFF A300
    • And older FAS generations
    • Is cabable of 12 Gbps

Mini-SAS HD Disk shelf ports

  • DS224C / DS212C disk shelf
    • Equipped with IOM12 module
    • Operates at 12 Gbps

Typical use case

  • FAS8200 with more than two SAS stacks
  • or
  • FAS8200 with two SAS stacks and quad-path HA cabling
    • Both require additional SAS ports in form of PCI SAS-adapter
  • Four onboard controller ports are using Mini-SAS HD form factor
    • Mini-SAS HD – to – Mini-SAS HD cable
    • NetApp P/N for 1 m cable:
      • X66031A
  • Four SAS adapter (controller) ports are using QSFP form factor
    • QSFP – to – Mini-SAS HD cable
    • NetApp P/N for 1 m cable:
      • X66020A-R6
  • I expect that most FAS8200 / AFF A300 systems will only have max two SAS stacks and no SAS adapter in PCI slot
    • Platform maximum spindle count: 480
    • Can be achieved with only two SAS stacks (when using only spinning disks)
  • Quad-Path HA will most likely be fairly rarely used with FAS8200
    • most likely bottle neck will not be SAS throughput
  • BUT when using more than four SAS ports with FAS8200
    • This is when things can get messy
      • expect to see configurations ordered with wrong cables
    • I hope that NetApp will release 4-port Mini-SAS HD adapter for FAS8200 series, so only one type of SAS cable can be used
      • There is already one available for FAS9000 platform

Visio shapes

  • Not available yet, still figuring out what do
  • With other SAS cable shapes “6G” = QSFP and “12G” = Mini-SAS HD
  • Problem is that we have now QSFP operating at 12 Gbps speed
  • Maybe create a special “Q12G” or similar cable end
    • Very limited use case, might just confuse more
  • Or use existing transitional cable shapes and foot note that it operates at 12 Gbps despite cable label
Example: Quad-Path HA cabled SAS Stack with FAS8200 using both QSFP and Mini-SAS HD ports on controller side

screen-shot-2016-11-18-at-23-40-39

  • Four “Mini-SAS HD -to- Mini-SAS HD” cable shapes
    • Both cable end shapes with “12G” text
  • Four “QSFP -to- Mini-SAS HD” cable shapes
    • QSFP cable end with “6G” text
    • Mini-SAS HD cable end with “12G” text
  • Effective speed determined by slowest component
    • Controller ports 12Gbps
    • Shelf back-plane 12 Gbps
    • IOM12 ports 12Gbps
    • = Effective speed 12 Gbps
  • No ACP cabling since no ACP ports on controller or IOM12 module,  No out-of-band Alternate Control Path (OOB ACP)
  • Instead In-Band Alternate Control Path (IB ACP) is used instead
Advertisements

7 thoughts on “Visio by DPTPB: Transitional SAS cables, QSFP to Mini-SAS HD

  1. Thanks for the excellent Visio templates and examples posted on your site.

    Not directly shown in this article, but I believe the NetApp-Examples-FAS2650-one-DS224C-shelf drawing available from your general downloads page is in contradiction with NetApp’s best practice guide when combining the FAS2650 with DS224C (IOM12) shelves.

    According to Page 29 of:
    https://library.netapp.com/ecm/ecm_get_file/ECMLP2485106

    For FAS2600 series configurations, you must have same-domain connectivity between controller port 0b (the internal storage) and the stack of disk shelves it is connected to.

    When a controller is located in slot A of the chassis (controller 1), its internal storage port (0b) is in domain A (IOM A); therefore, port 0b must connect to domain A (IOM A) in the stack.

    When a controller is located in slot B of the chassis (controller 2), its internal storage port (0b) is in domain B (IOM B); therefore, port 0b must connect to domain B (IOM B) in the stack.

    Like

    1. Hi Rob,

      Thanks for kind words and comments.

      I’ve fixed the example drawing you mentioned, there are few more iterations of the same drawing to be fixed. I will fix these later.

      This is probably the reason why SAS ports on older FAS25xx platform are in “weird” order, counting from left, first 0b port(with square label) and then 0a port (with circle label).

      With the new FAS2600 platform NetApp fixed the “weirdness” with port naming/order and SAS ports are in “normal” order, first 0a port and then 0b port. Since NetApp has removed shape based SAS port labels “Square / Circle”, some of the visual cues are missing and it is easier to make mistakes, like the one I did. I wish that they would have continued using shape based labeling of SAS ports, maybe just adding numbers behind the shapes.

      Like

  2. Update: All FAS2650 example drawings with SAS cabling on downloads page have now been fixed. Similar error was also in one of the drawings in this post “Example: New generation FAS2650 with Older generation SAS DS4246 shelves”, fixed too

    Like

    1. Hi Sami,

      Thanks for the fast response.

      With the recommendation for the SAS 0b ports on the FAS2600 series controllers to connect to an IOM port on the same respective domain (i.e., ctrl-A, 0b connects to IOM-A / ctrl-B, 0b connects to IOM-B), this brings up another consideration:

      Should these 0b connections actually go to port 3 rather than port 1 on the IOM12 modules? This would keep consistency with the general rules of:
      – A and C ports always connect to the first disk shelf in a stack
      – A and C ports always connect to disk shelf IOM ports 1 and 2.
      – B and D ports always connect to the last disk shelf in a stack
      – B and D ports always connect to disk shelf IOM ports 3 and 4

      The FAS2600 cabling worksheet on page 30 from my link above appears to indicate this is the case. Hopefully I’m not becoming a pest, appreciate your input.

      Like

      1. Hi Rob,

        No worries, questions like this was one of the reasons why I started to make Visio drawings, which eventually led to tweaking the original NetApp stencils 🙂

        You are correct with your findings.

        It seems that NetApp has decided to fix some inconsistencies that were present with FAS2500 platform, where 0b port was labeled also with “Square” shape. This indicated that the counter part port on the shelf side would be a “Square” port. To follow other cabling rules that port would reside on IOMA (for controller A). This led to situation where FAS2500 was not following generic SAS cabling rules, because a “B” port was connecting to a “Square” port.

        Even with these modifications FAS2600 platform is still little bit odd-ball. With other platforms Controller A 0a port would connect to IOMA (port 1), instead of IOMB (port 1). But I think that this is “cleaner” cabling sequence than with FAS2500 platform.

        Dropbox link to a drawing using this cabling sequence:
        https://www.dropbox.com/s/s38cqp9szjmas5c/DontPokeThePolarBear-NetApp-Examples-FAS2650-two-DS224C-shelves_v2.pdf?dl=0

        I will fix the other examples later, got to run now

        BR,

        Sami

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s