Document Outline

• Sun StorEdge™ 3900 and 6900 Series Troubleshooting Guide
  • Contents
  • List of Figures
• Preface
  • How This Book Is Organized
  • Using UNIX Commands
  • Typographic Conventions
  • Shell Prompts
  • Related Documentation
  • Accessing Sun Documentation Online
  • Sun Welcomes Your Comments
• Chapter 1. Introduction
  • Introduction
  • Predictive Failure Analysis Capabilities
• Chapter 2. General Troubleshooting
  • Storage Automated Diagnostic Environment Event Grid
    • To Customize an Event Report
  • Fibre Channel Link Diagrams
    • FIGURE21 Sun StorEdge 3900 Series Fibre Channel Link Diagram
    • FIGURE22 Sun StorEdge 6900 Series Fibre Channel Link Diagram
    • Host Side Troubleshooting
    • Storage Service Processor Side Troubleshooting
    • Command Line Test Examples
      • qlctest(1M)
      • switchtest(1M)
  • Fibre Channel Links
  • Multipathing Options in the Sun StorEdge 6900 Series
    • Alternatives to Sun StorEdge Traffic Manager
    • To Quiesce the I/O
      • 1. Determine the path you want to disable.
      • 2. Type:
    • To Unconfigure the c2 Path
      • 1. Type:
      • 2. Using Storage Automated Diagnostic Environment Topology GUI, determine which virtualization en...
      • 3. Use the world wide name (WWN) of the virtualization engine that is in the unconfigure command,...
      • 4. Verify that I/O has halted.
    • To Suspend the I/O
      • 1. Stop all customer applications that are accessing the Sun StorEdge T3+ array.
      • 2. Manually pull the link from the Sun StorEdge T3+ array to the switch and wait for a Sun StorEd...
    • To Return the Path to Production
      • 1. Type cfgadm -c configure device.
      • 2. Verify that I/O has resumed on all paths.
    • To View the VxDisk Properties
      • 1. Type the following:
      • 2. Use the luxadm(1M) command to display further information about the underlying LUN.
    • To Quiesce the I/O on the A3/B3 Link
      • 1. Determine the path you want to disable.
      • 2. Disable the path by typing the following:
      • 3. Verify that the path is disabled:
    • To Suspend the I/O on the A3/B3 Link
      • 1. Stop all customer applications that are accessing the Sun StorEdge T3+ array.
      • 2. Manually pull the link from the Sun StorEdge T3+ array to the switch and wait for a Sun StorEd...
        • a. After the failover occurs, replace the cable and proceed with testing and FRU isolation.
        • b. After testing is complete and any FRU replacement is finished, return the controller state bac...
    • To Return the Path to Production
      • 1. Type:
      • 2. Verify that the path has been re-enabled by typing:
  • General Troubleshooting Procedures
  • Troubleshooting Overview Tasks
    • 1. Discover the error by checking one or more of the following messages or files:
    • 2. Determine the extent of the problem by using one or more of the following methods:
    • 3. Check the status of a Sun StorEdge T3+ array by using one or more of the following methods:
    • 4. Check the status of the Sun StorEdge FC network switch-8 and switch-16 switches using the foll...
    • 5. Check the status of the virtualization engine using one or more of the following methods:
    • 6. Quiesce the I/O along the path to be tested as follows:
    • 7. Test and isolate the FRUs using the following tools:
    • 8. Verify the fix using the following tools:
    • 9. Return the path to service by using one of the following methods:
• Chapter 3. Troubleshooting the Fibre Channel Links
  • A1/B1 Fibre Channel (FC) Link
    • To Verify the Data Host
    • FRU Tests Available for A1/B1 FC Link Segment
    • To Isolate the A1/B1 FC Link
  • A2/B2 Fibre Channel (FC) Link
    • To Verify the Host Side
    • To Verify the A2/B2 FC Link
      • FRU Tests Available for A2/B2 FC Link Segment
    • To Isolate the A2/B2 FC Link
  • A3/B3 Fibre Channel (FC) Link
    • To Verify the Host Side
    • To Verify the Storage Service Processor
    • FRU Tests Available for the A3/B3 FC Link Segment
    • To Isolate the A3/B3 FC Link
  • A4/B4 Fibre Channel (FC) Link
    • To Verify the Data Host
    • FRU tests available for the A4/B4 FC Link Segment
    • To Isolate the A4/B4 FC Link
• Chapter 4. Configuration Settings
  • Verifying Configuration Settings
• Chapter 5. Troubleshooting Host Devices
  • Host Event Grid
    • Using the Host Event Grid
    • To Replace the Master Host
    • To Replace the Alternate Master or Slave Monitoring Host
      • 1. Choose Maintenance -> General Maintenance -> Maintain Hosts.
      • 2. In the Maintain Hosts window, select the host to be replaced from the Existing Hosts list, and...
      • 3. Install the new host.
      • 4. Install the SUNWstade package on the new host.
      • 5. Run /opt/SUNWstade/bin/ras_install.
      • 6. Configure the host as a slave.
      • 7. Choose Maintenance -> General Maintenance -> Maintain Hosts.
      • 8. In the Maintain Hosts window, select the new host.
      • 9. Configure the options as needed.
      • 10. Choose Maintenance -> Topology Maintenance -> Topology Snapshot.
        • a. In the Topology Snapshot window, select the new host.
        • b. Click Create and Retrieve Selected Topologies.
        • c. Click Merge and Push Master Topology.
      • Conclusion
        • 6
        • Troubleshooting Sun StorEdge FC Switch-8 and Switch-16 Devices
• Chapter 6. Troubleshooting Sun StorEdge FC Switches
  • To Diagnose and Troubleshoot Switch Hardware
  • Switch Event Grid
  • Replacing the Master Midplane
• Chapter 7. Troubleshooting Virtualization Engine Devices
  • Virtualization Engine Description
  • Virtualization Engine Diagnostics
    • Service Request Numbers
    • Service and Diagnostic Codes
    • To Retrieve Service Information
  • Virtualization Engine LEDs
    • Power LED Codes
    • Interpreting LED Service and Diagnostic Codes
    • Back Panel Features
    • Ethernet Port LEDs
    • Fibre Channel Link Error Status Report
    • To Check Fibre Channel Link Error Status Manually
  • Sun StorEdge 6900 Series Multipathing Example
    • Virtualization Engine Event Grid
      • Using the Virtualization Engine Event Grid
        • Troubleshooting the Sun StorEdge T3+ Array Devices
    • One Sun StorEdge T3+ array partner pair with 1 500GB RAID 5 LUN per brick (2 LUNs total)
      • FIGURE72 Sun StorEdge 6900 Series Logical View
      • FIGURE73 Primary Data Paths to the Alternate Master
      • FIGURE74 Primary Data Paths to the Master Sun StorEdge T3+ Array
      • FIGURE75 Path Failure—Before the Second Tier of Switches
      • FIGURE76 Path Failure —I/O Routed through Both HBAs
• Chapter 8. Troubleshooting Sun StorEdge T3+ Array Devices
  • Troubleshooting the T1/T2 Data Path
    • T1/T2 Notification Events
    • Sun StorEdge T3+ Array Storage Service Processor Verification
    • T1/T2 FRU Tests Available
    • T1/T2 Isolation Procedures
    • Sun StorEdge T3+ Array Event Grid
      • Using the Sun StorEdge T3+ Array Event Grid
        • 1. From the Storage Automated Diagnostic Environment Help menu, click the Event Grid link.
        • 2. Select the criteria from the Storage Automated Diagnostic Environment event grid, like the one...
          • FIGURE85 Sun StorEdge T3+ array Event Grid
            • 1. Voltage level on power supply and battery have moved out of acceptable thresholds.
            • 2. The internal PCU temp has exceeded acceptable thresholds.
            • 3. A PCU fan has failed.
            • 1. Telnet to affected Sun StorEdge T3+ array
            • 2. Verify disk state in fru stat, fru list, and vol stat.’
            • 1. Telnet to affected Sun StorEdge T3+ array.
            • 2. Verify tje loopcard state with fru stat.
            • 3. Verify the matching firmware with the other loopcard.
            • 4. Re-enable the loopcard if possible (enable u (encid)|[1|2] ). Replace loopcard if necessary.
            • 5. Re-enable the disk if possible
            • 6. Replace the disk, if necessary.
            • 1. Telnet to the affected Sun StorEdge T3+ array.
            • 2. Run refresh -s to verify the battery state.
            • 3. Replace the battery, if necessary
            • 1. Telnet to affected Sun StorEdge T3+ array.
            • 2. Verify the fan state with fru stat.
            • 3. Replace the power cooling unit, if necessary.
            • 1. Telnet to affected Sun StorEdge T3+ array.
            • 2. Verify power cooling unit state in fru stat.
            • 3. Replace PCU, if necessary.
            • 1. Telnet to the affected Sun StorEdge T3+ array.
            • 2. Verify that the power cooling unit state is in ‘fru stat’
            • 3. Replace the PCU if necessary.
            • 1. Verify luxadm via command line (luxadm probe, luxadm display)
            • 2. Verify cables, GBICs and connections along data path.
            • 3. Check the Storage Automated Diagnostic Environment SAN Topology GUI to identify the failing se...
            • 4. Verify the correct FC switch configuration, if applicable.
            • 1. Check Ethernet connectivity to the affected Sun StorEdge T3+ array.
            • 2. Verify the Sun StorEdge T3+ array is booted correctly.
            • 3. Verify the correct TCP/IP settings on the Sun StorEdge T3+ array .
            • 4. Increase the http and/or ping timeout in Utilities-- >System-- >System-- >Timeouts. The curren...
            • 1. Telnet to affected Sun StorEdge T3+ array.
            • 2. Verify the controller state with ‘fru stat’ and ‘sys stat’.
            • 3. Run ‘logger - dmprstlog’ to capture controller information.
            • 4. Re-enable the controller if possible (enable u)
            • 5. Replace the controller, if necessary.
            • 1. Telnet to the affected Sun StorEdge T3+ array
            • 2. Verify that the disk state is in fru stat, fru list, and vol stat.
            • 3. Replace the disk, if necessary.
            • 1. Telnet to the affected Sun StorEdge T3+ array.
            • 2. Verify loopcard state with fru stat
            • 3. Verify matching firmware with other loopcard.
            • 4. Re-enable loopcard if possible (enable u(encid)|[1|2|]
            • 5. Replace the loopcard if necessary.
            • 1. Check the Sun StorEdge T3+ array syslog for battery hold times.
            • 2. If < 6 minutes, replace the battery, or the entire PCU, as required.
            • 1. Telnet to the affected Sun StorEdge T3+ array
            • 2. Check the status of LUNs via vol mode or vol stat.
    • Replacing the Master Midplane
  • Explorer Data Collection Utility
• Chapter 9. Troubleshooting Ethernet Hubs
• Appendix A-Virtualization Engine Reference
  • TABLEA1 SRN and SNMP Reference
  • TABLEA2 SRN/SNMP Single Point of Failure Table
  • TABLEA3 Port Communication
  • TABLEA4 Service Codes
• Appendix B-SUNWsecfg Error Messages
  • TABLEB1 Virtualization Engine SUNWsecfg Error Messages
  • TABLEB2 Sun StorEdge Network FC Switch-8 and Switch-16 Switch SUNWsecfg Error Messages
  • TABLEB3 Sun StorEdge T3+ Array SUNWsecfg Error Messages
  • TABLEB4 Other SUNWsecfg Error Messages
  • setupswitch Exit Values
• Index