Be ware of Warm Reset of the System Control Board on OptiX OSN Equipment

Summary:
The ASON network is unstable. Fiber cut frequently occurs and ASON services are rerouted constantly. In this case, the system control board of an NE is prone to be reset when network resources are insufficient or residual cross connections exist.

[Problem Description]
Trigger conditions:
The problem is triggered if the following conditions are met:

•  The network is an ASON network and fiber cut frequently occurs.
• There are tunnel services and reroute failure events in the ASON domain.
• There are CP_TEL_PATH_MIS and CP_TEL_MSP_MIS alarms in the ASON domain.

Symptom:
The system control board is abnormally reset. Based on the network scale, this problem occasionally occurs on the SSN1GSCC and board, and rarely occurs on the SSN4GSCC board. The NE is transiently unreachable to the NMS when the system control board is being reset.

Identification method:
1. A version that has the problem is used at the site.
2. The preceding trigger conditions are met.
3. The preceding symptom occurs.
4. There are a large number of rerouting failure events in the abnormal events on the NMS. The error code is 40510 (indicating the label allocation failure) and the faulty NE is the NE where the system control board is abnormally reset.

[Root Cause]
When the labels fail to be allocated to the last node of the tunnel service, an error occurs in internal code processing, so the applied memory is not released, resulting in memory leak. The memory of the system control board is used up due to the long-time rerouting label allocation failure. Therefore, the system control board is reset because it fails to apply for memory.

[Impact and Risk]
Fiber cut occurs during ASON NE reset, and services are interrupted.

[Measures and Solutions]
Recovery measures:
None
Preventive measures:

•  Timely handle the CP_TEL_PATH_MIS and CP_TEL_MSP_MIS alarms in daily maintenance.
• If services are frequently rerouted, clear corresponding port alarms and channel alarms as soon as possible to prevent rerouting failures.
• Solution:
  Upgrade the NE to the following versions or install corresponding hot patches.
  NG SDH: V100R008C02SPC200+SPH203
  NG SDH: V100R008C02SPC500+SPH505
  NG SDH: V100R010C03SPC203+SPH209
  NG SDH: V100R010C03SPC208 and later versions
  OCS 9500: V100R006C03SPC200+SPH206 (If the JSCC board is used on the live network, replace it with the ESCC board.)
  OCS 9500: V100R006C05SPC203 and later versions

[Rectification Scope and Time Requirements]

N/A

[Rectification Guide]
N/A
[Attachment]
N/A
[Inspector Applicable or Not]
N/A