Category: Fault Report

Posted on

Testing data collection with new approach

The 5 second delay introduced by Homin to the end of data collection to flush the buffers does not seem to have worked.
So, I tried something different with data collection to see if it would fix our data collection issues. I modified the rx and tx scripts.

Modifications made
1. To tx script – only enable or disable transmission
2. To rx script – In addition to creating a file for data collection, also send all the spead signals.

I enabled the transmission for both pols. The first dataset was collected for about 75 minutes without any issues. I closed the files and sockets WITHOUT disabling transmission from the correlator. The processed output data had no issues.
The second data set collection went ok for about 75 minutes, when X pol errored out and never recovered. Only a reload fixed it.

I am quite surprised that data collection which went smoothly for about 75 minutes would completely crash, and never recover. This points to something other than buffer issues in the switch, I think.

Posted on

High SEFD problem for this week (problem identified)

This week, our SEFD measurements showed that on both Tuesday night and Wednesday night SEFD measurements were ~ 6 MJy for Rx0 X Pol, and ~ 8MJy for Rx2 X Pol. Last week they were fine. Yesterday night power cycling the interrupt distribution drawer did not change anything. Today Po-Sheng, Peter and I found the problem to be a ROACH walsh misalignment for the aforementioned IF signals ONLY. Running a reset on the value of the interrupt in the ROACH fixed the problem.

The root cause of this problem is still unknown. In ~ 5 years of operations, this is the first time this problem has surfaced. The correlator was setup in the usual way on Tuesday night. Nothing in the setup has been changed for a while. In addition, the interrupt value was set correctly to the internal interrupt. Two ways this could have happened

1. The external interrupt failed to align the internal interrupt.
2. The internal interrupt did get aligned, and then slipped (why??) (only for these 2 IFs)

For now, the problem is fixed.

CT — Could you remind me of the power level of the external interrupt that needs to be seen at the input to the ROACH. Thanks

Posted on

1.6 GHz Y Pol Jupiter Data (corrupted)

We generated 10 minutes of Jupiter data at 6 different LOs as listed in the observing blog this morning Jupiter Observations at 1.6 GHz. Unfortunately, this data is useless since the ROACH walshes are misaligned with the antenna walsh. This is also clearly visible in the oscilloscope image in the same blogpost. I traced the problem to the interrupt register not being written to in the initialize script. Please note that ytla7_init.py has now the ‘walsh_interrupt’ value being written to the  ‘interrupt’ register. We will have to collect the 1.6 GHz data again. With this fix, both the antenna and the demodulation walshes are now synced.

Ranjani

Posted on

Rx4 X pol offline this week

Rx4 X pol was having large (~ 10dB) power fluctuations until the end of observing in 2018. These fluctuations were intermittent. We did not see these fluctuations in 2019, but they reappeared again yesterday, and now seem to be permanent. Peter swapped out the complete IFLO unit, and the problem still persists.

The receiver output power for both pols from the cryostat is fine. The power output from the IFLO module for X pol has a deficit of ~ 12-13 dB. Since Rx4 Y pol is fine, we think the problem is most likely in the power supplies which are different for X and Y pol. This will require time to isolate the problem. So, Rx4 X pol is offline for this week.

Peter, Ranjani