Mission Operations Days: 2001/167 to 2001/173
June 16, 2001 00:00:00 GMT to June 22, 2001 00:00:00 GMT
The MODIS Power Supply 2 Shut Off on DOY 2001/166. (details below)
MODIS is currently in Survival Mode.
MODIS is currently NOT producing Science Data.
Terra Spacecraft and MODIS Instrument Status:
| Terra (AM-1) is in Normal Ops Mode MODIS is in Survival Mode |
||
|---|---|---|
| Component | State(s) | Comment |
| Blackbody | A Off; B Off | PS2 Shutdown Anomaly |
| Calibration Electronics | A Off; B Off (Selected) | PS2 Shutdown Anomaly |
| Control Processor | A Off; B Off | PS2 Shutdown Anomaly |
| Nadir Door | Unlatched, Open | PS2 Shutdown Anomaly |
| Space View Door | Unlatched, Open | PS2 Shutdown Anomaly |
| Solar Diffuser Door | Unlatched, Closed | PS2 Shutdown Anomaly |
| FDDI Formatter | A Off; B Off | PS2 Shutdown Anomaly |
| FDDI Port | A Off; B Off | PS2 Shutdown Anomaly |
| FIFO Memory | #1 & #2 Off; #3 & #4 Off | PS2 Shutdown Anomaly |
| Format Processor | A Off; B Off | PS2 Shutdown Anomaly |
| Power Supplies | #1 Off; #2 Off | PS2 Shutdown Anomaly |
| PV VIS FPA | A Off; B Off | PS2 Shutdown Anomaly |
| PV NIR FPA | A Off; B Off | PS2 Shutdown Anomaly |
| PV SMIR FPA | A Off; B Off | PS2 Shutdown Anomaly |
| PV LWIR FPA | A Off; B Off | PS2 Shutdown Anomaly |
| PC LWIR FPA | A Off; B Off | PS2 Shutdown Anomaly |
| RC Outgas Heaters | All Off (CS, IS, OS) | PS2 Shutdown Anomaly |
| RC LWIR FPA Heater | Off | PS2 Shutdown Anomaly |
| RC SMIR FPA Heater | Off | PS2 Shutdown Anomaly |
| Scan Assembly | A Off; B Off | PS2 Shutdown Anomaly |
| SDSM | A Off; B Off | PS2 Shutdown Anomaly |
| SRCA | A Off; B Off | PS2 Shutdown Anomaly |
| PS1 Survival Heater | Enabled | PS2 Shutdown Anomaly |
| PS2 Survival Heater | Enabled | PS2 Shutdown Anomaly |
| Timing Generator | A Off; B Off | PS2 Shutdown Anomaly |
| Flight Software | Rev BD | PS2 Shutdown Anomaly |
| Inhibit Ids Set | None | PS2 Shutdown Anomaly |
| TMONs Enabled | 68 | PS2 Shutdown Anomaly |
This Week's Completed MODIS Activities:
Saturday, June 16, 2001
None
Sunday, June 17, 2001
None
Monday, June 18, 2001
None
Tuesday, June 19, 2001
None
Wednesday, June 20, 2001
None
Thursday, June 21, 2001
None
Friday, June 22, 2001
None
This Week's Scheduled MODIS Activities Not Completed:
None
Upcoming MODIS Events:
No events will be scheduled until PS2 Shut Off Anomaly is resolved.
Saturday, June 23, 2001
None
Sunday, June 24, 2001
None
Monday, June 25, 2001
None
Tuesday, June 26, 2001
None
Wednesday, June 27, 2001
None
Thursday, June 28, 2001
None
Friday, June 29, 2001
None
Maneuvers:
The next Drag Make Up maneuver will occur on DOY 178. It will be a 16 second burn beginning at approximately 19:05.
The next MODIS Lunar Roll Calibration is scheduled for DOY 191 from 15:07:42 to 15:14:12. This roll has the obvious prerequisite that MODIS must successfully from its Power Supply 2 Anomaly (described below).
MODIS Anomalies:
On June 15th, 2001, at 03:56:08 GMT, MODIS experienced an anomaly.
The Flight Operations Team paged the on-call MODIS engineer reporting a trip of TMON #68 (autonomous Telemetry Monitoring software on the spacecraft). TMON #68 is the MODIS double FIFO (First In, First Out data buffer) Write State TMON. In addition, many red limit violations were reported. Inspection of telemetry showed peculiar states including a CP (Control Processor) mode of INVALID and the MODIS model as ENGINEERING. With all telemetry suspect, the MODIS IOT commanded the instrument to low-power mode (using procedure MOD_ALL_OFF) at 05:16:43.
Investigation is still pending, but below is the current understanding of the event:
Continued telemetry analysis showed that the TMON itself did not trip, but rather, it went into an error state because it was receiving stale telemetry. At the anomaly time, all telemetry “pegged out”, and the telemetry headers stopped incrementing (thus explaining the error reported by the TMON).
Science data stopped being recorded on the SSR at the same time the housekeeping telemetry “pegged out”. This suggests that it was NOT a 1553 Bus communication problem between MODIS and the spacecraft, but rather something that would affect both housekeeping and science data.
At the time of the anomaly, the EPS feed current to MODIS (an external spacecraft point) dropped from ~1.5 Amps to 0.07 Amps suggesting that all MODIS components, with the exception of the power supply, powered down at that time.
ANOMALY REVIEW TEAM UPDATE:
At this time, the evidence is very strong that a failure has occurred in PS2 and that the rest of the instrument is okay. There is still the question about whether the failure was a soft failure or a hard failure. If the failure was a hard failure (such as an electrical short), it is possible that powering up PS2 could possibly stress the loads (those components still internally enabled on MODIS from the time of the PS2 shut off) with an over voltage. However, if the failure was a soft failure (such as a Single Event Upset (SEU) caused by a high-energy particle impact), it is possible that powering up PS2 will return it to full operational capability.
WHEN instrument commanding begins for the recovery effort, the electronics will be very cold. The use of Telemetry and Command Processor B (known as TCP B or CP B) would minimize any chance of command processor resets due to the cold (which CP A has a tested history of experiencing).
Until power is supplied to one of the MODIS Telemetry and Command Processors (via PS1 or PS2), only a small set of MODIS health and safety telemetry is available (Space Craft Point To Point (PTP) telemetry). This limits the majority of the analysis of this anomaly to the time prior to the PS2 shut off event.
An Anomaly review/recovery team has been assembled and has begun holding daily meetings to determine the cause, and recovery path from this anomaly. The team is currently digesting the set of possible recovery paths to initially return MODIS to a “Low Power Mode”. Once this mode is reached, greater insight into the health of the instrument will be possible from the telemetry produced by the active CP.
Based on the cumulative results from this week's discussions with the engineering experts, it is hoped that a consensus on the recovery of MODIS may be reached early next week.
General Instrument Comments:
MODIS is currently powered off in Survival Mode. The Survival Heaters are cycling as expected. Return to an operational mode is currently pending the outcome of the MODIS Power Supply 2 Anomaly.
MODIS Telemetry Trends:
Voltage, current and temperature trending over the previous month, in addition to selected past time periods is currently underway to support the resolution of the MODIS Power Supply 2 Shut Down Anomaly.
Non-MODIS Significant Events:
SSR Anomaly:
Late in the evening of June 16, 2001, the Solid State Recorder (SSR) locked up. The SSR was no longer recording data and was not responding to commands uplinked by the Terra Flight Operations Team (FOT) to dump its data. This made it impossible for the ground to acquire any recorded housekeeping telemetry or science data from the Terra spacecraft.
Because housekeeping data could not be dumped, the health of Terra in the “back orbit” (periods outside of direct contact with the spacecraft) was unknown. A spacecraft emergency was declared and near-contiguous real-time TDRSS contacts were scheduled.
The SSR was placed into “Low Power Mode” and its telemetry was analyzed. The decision was made to power up and re-cycle the SSR on June 17th. At 23:13 GMT on June 17, 2001, the SSR successfully dumped housekeeping data. This confirmed the suspicion that the SSR had experienced a multi-buffer lockup across all of its supersets, similar to what had been experienced previously in ground tests and approximately 1 month after launch in January of 2000.
After further testing and analysis, the SSR was restored to full operations (successfully dumping science data) at 11:40:34 GMT on June 20, 2001.
Any interesting byproduct of this event was the functional restoration of supersets 30 and 31 (which, after reallocation of the supersets, they now reside in the MISR buffer). Additionally, ASTER's superset allocation was reduced by one superset to solve a complication with replays of their science data. This free superset has been allocated to the MODIS buffer. With the restoration of the malfunctioning supersets (which are no long allocated to the MODIS buffer), and the additional superset, MODIS will have approximately 3% greater science data storage space than originally allocated.
MOPITT Anomaly:
The MOPITT team is still working to recover their instrument to science operations.
Limited Life Item Status:
SRCA 10W Lamp #1: 186.1 of 500 hours
SRCA 10W Lamp #2: 143.0 of 500 hours
SRCA 10W Lamp #3: 152.2 of 500 hours
SRCA 10W Lamp #4: 61.5 of 500 hours
SRCA 1W Lamp #1: 557.6 of 4000 hours
SRCA 1W Lamp #2: 276.3 of 4000 hours
Solar Diffuser Door: 1633 of 3022 Movements
Nadir Aperture Door: 532 of 1316 Movements
Space View Door: 437 of 1316 Movements