ImageCalibration and Overscan area procedure

T

thutchison

Active member
Hi all:

I have been using PI for some time, but I have been using a CMOS camera that did not include an overscan area. I recently purchased a QHY600m camera that does include overscan. I am confused by the ImageCalibration process as it applies to overscan. I have combed through the other articles on this forum, but I am still quite confused.

The definition of the overscan area is a trivial matter. I completely understand how to define the image area and the overscan area(s). My area of confusion is in the procedure for using it.

With my prior camera, I took no bias frames. I took dark frames, flat frames, and flat dark frames. The dark and flat darks were integrated separately without calibration into master_dark and master_flat_dark images. The master_flat_dark image was used in ImageCalibration with the flat frames to create a master_flat_frame. The master_dark and master_flat were used in ImageCalibration for the light frames. "Calibrate" remained unchecked in all cases, as did optimize on darks because of glow in the sensor.

The QHY600m includes the overscan area and suggests that it should be used to combat bias drift. My questions are many, but include the following:
1. Should bias frames be taken rather than using flat dark frames and building a master_flat_dark?

2. Should I calibrate each set of calibration images (bias, dark, flat) using the overscan area separately, or should I integrate the calibration images as before and only select the "overscan area" option when calibrating the lights? If each should be done separately, I believe that PI will truncate (remove) the overscan areas from the output image, in this case the master... frames? Would this not cause an issue when attempting to calibrate the light frames (incorrect geometry)?

It would seem to me that the correct procedure would be to integrate the bias, dark, and flat frames without calibration and, when integrating lights, select the overscan area and select "calibrate" for each, allowing PI to subtract the average value of the overscan area from each respective master before calibrating the masters with each other and subsequently, the lights. I'm not sure if I have this correct and would GREATLY appreciate any assistance that can be offered.

Thank you.
Tim.
 
Hi Tim,

I have deleted my first post, because it was incorrect. I try it again:

In my view, first of all the manufacturer of the camera (in your case: QHY) or of the sensor (Sony) should be asked for recommendations whether overscan calibration is recommended or not for this camera. If there is no bias drift, the procedure of overscan-calibration is unnecessary. See also https://pixinsight.com/forum/index....bration-overscan-with-optic-black-area.14139/ . Up to now I didn't come to know anything about a feedback given by these companies.

Of course you can give it a trial yourself. Since overscan correction shall correct for a drift of the bias offset over time, it must be applied to each individual calibration frame and also to the individual light frames. During overscan calibration, the overscan area will be cropped. As overscan-calibration is always the first step of the calibration process, all resulting frames (i.e. the Master calibration files as well as the overscan-calibrated light frames) will have the same geometry: the overscan area is cropped. So it should work like this: Continue capturing dark frames, flat frames and flat-darks (and no bias frames).

Calibration of the dark frames:
- the overscan section must be configured correctly,
- the sections 'Master Bias', 'Master Dark' and 'Master Flat' must be disabled.
In order to avoid clipping, an output pedestal has to be applied. Make sure that no clipping occurs in the calibrated dark frames (check with histogram and statistics). The overscan-calibrated dark frames are then integrated to the MasterDark, option 'Subtract pedestals' has to be disabled.

Likewise the flat-darks have to be calibrated and integrated to the MasterFlatDark (same procedure, with output pedestal, checking for clipping and option 'Subtract pedestals' disabled in the integration).

Calibration of the flat frames:
- the overscan section must be configured correctly,
- enable the 'Master Bias' section, select the MasterFlatDark, disable the option 'Calibrate',
- disable sections 'Master Dark' and 'Master Flat'.
An output pedestal will not be necessary. The calibrated flat frames are then integrated to the MasterFlat as usual.

Calibration of the light frames (no dark frame optimization):
- the overscan section must be configured correctly,
- disable the 'Master Bias' section,
- enable the section 'Master Dark', select the MasterDark, disable the options 'Calibrate' and 'Optimize',
- enable the section 'Master Flat', select the MasterFlat, disable the option 'Calibrate'.


Please don't complain if this approach produces weird results. I have no QHY600 camera, so I am not able to try it myself, and I will not do any trouble-shooting for this special case.

Bernd
 
Last edited:
Bernd:

Thank you for your response. I will be trying this as soon as my new scope arrives and I am able to get out under the stars.

I did ask QHY for input regarding the overscan area before I made this post. Here is there response, slightly edited by me to correct some grammar issues related to translation to English from the QHY rep.

At issue is the following problem:

Calibrated image = (L-D)/(F-B)

L=light frame
D= dark frame
F=flat frame
B=bias frame

And it is better to use this instead of the above

Calibrated image = (L-D)/(F-DF)
DF= dark flat frame . It is the same exposure time but with no light coming in.

The on-chip calibration part of the cmos sensor may cause the drift of the whole image when the image is bright. By using the overscan area, you can correct for this drift as follows:

Calibrated image = (L-D+drift of light frame)/(F-DF+drift of flat frame)

The drift of light frame will have little impact. But drift of the flat frame appearing in the denominator can cause over/under calibration.

The best way to handle this is do a overscan calibration. The method is :

(1)Keep the overscan area (in manual we have shown this part)
(2)Stack all flat frame and get a master flat frame, get the overscan area average value , for example: 1000
(3) Stack all dark flat frame and get a master dark frame. get the overscan area average value , for example, 1500

We see a 500 difference accountable to bias drift. Add a constant 500 to the master flat frame. After you add the 500, you will see the master frame overscan area is 1500, matching the dark flat frame.

(4) Do (F-DF) calculation and then you will get the correctly calibrated flat frame.

Normally speaking there is such a function in the stack software to handle this.

As I say, I paraphrased some of the discussion, but did not change the meaning of the rep's text.

The difference that I see with PI's implementation is that PI is subtracting the bias drift from the dark frame, flat frame, and light frame. However, this may be a distinction without a difference in that the end result may well be the same. Until I do some testing I have no reason to believe that PI's solution will not work perfectly well.

As I say, I will test and return with results.

Again, thank you for your response.
Tim.
 
Thank you very much for transmitting QHY's response. This is helpful for understanding why overscan calibration is useful for this type of cameras. For me the actual new information is: "The on-chip calibration part of the cmos sensor may cause the drift of the whole image when the image is bright." and the conclusion: "The drift of light frame will have little impact. But drift of the flat frame appearing in the denominator can cause over/under calibration.".

That means, differing bias offsets are not caused by (thermal) instability over time of some electronic components but by on-chip calibration of the CMOS sensor. This makes a huge difference. If this is the correct explanation, the approach that I described above can be simplified by omitting the overscan-calibration of each individual dark frame and flat-dark. The modified procedure would be:

As usual, the dark frames and likewise the flat-darks are simply integrated to the MasterDark and the MasterFlatDark respectively.

Calibration of the flat frames:
- the overscan section must be configured correctly,
- enable the 'Master Bias' section, select the MasterFlatDark, enable the option 'Calibrate',
- disable sections 'Master Dark' and 'Master Flat'.
An output pedestal will not be necessary. The calibrated flat frames are then integrated to the MasterFlat as usual.

Calibration of the light frames (no dark frame optimization):
- the overscan section must be configured correctly,
- disable the 'Master Bias' section,
- enable the section 'Master Dark', select the MasterDark, enable the option 'Calibrate' and disable the option 'Optimize',
- enable the section 'Master Flat', select the MasterFlat, disable the option 'Calibrate'.

So the flat frames and the MasterFlatDark are overscan-calibrated in the calibration of the flat frames. Likewise the light frames and the MasterDark are overscan-calibrated in the calibration of the light frames. This procedure is much less complex than the one that I outlined above.

Please let me know whether this procedure is producing correct results with your new camera.

Bernd
 
Thank you Bernd. I am waiting on my new scope to put this camera on. As soon as I have some results I will post back to this thread. I appreciate your help.

Tim.
 
Tim,

I've used several QHY cameras (mono and color) and they all have the overscan area left in by default. The simplest thing to do is check the "Remove Overscan Area" in the QHY driver settings where you set gain and offset - then you'll no longer have to worry about any of this. I've noticed no issues with this and everything calibrates fine.

In the past I have left the overscan area in with the QHY247C before I knew about the remove overscan area option and it still calibrated fine with flats, darks, and dark flats. Of course, you'll have to make new masters after you do this. I'm currently using the QHY268C which is the same sensor technology as the QHY600, and I use bias instead of the flat darks now so I'd think you should be able to with the QHY600 as well.

Jarrett
 
bulrichl said:
Thank you very much for transmitting QHY's response. This is helpful for understanding why overscan calibration is useful for this type of cameras. For me the actual new information is: "The on-chip calibration part of the cmos sensor may cause the drift of the whole image when the image is bright." and the conclusion: "The drift of light frame will have little impact. But drift of the flat frame appearing in the denominator can cause over/under calibration.".

That means, differing bias offsets are not caused by (thermal) instability over time of some electronic components but by on-chip calibration of the CMOS sensor. This makes a huge difference. If this is the correct explanation, the approach that I described above can be simplified by omitting the overscan-calibration of each individual dark frame and flat-dark. The modified procedure would be:

As usual, the dark frames and likewise the flat-darks are simply integrated to the MasterDark and the MasterFlatDark respectively.

Calibration of the flat frames:
- the overscan section must be configured correctly,
- enable the 'Master Bias' section, select the MasterFlatDark, enable the option 'Calibrate',
- disable sections 'Master Dark' and 'Master Flat'.
An output pedestal will not be necessary. The calibrated flat frames are then integrated to the MasterFlat as usual.

Calibration of the light frames (no dark frame optimization):
- the overscan section must be configured correctly,
- disable the 'Master Bias' section,
- enable the section 'Master Dark', select the MasterDark, enable the option 'Calibrate' and disable the option 'Optimize',
- enable the section 'Master Flat', select the MasterFlat, disable the option 'Calibrate'.

So the flat frames and the MasterFlatDark are overscan-calibrated in the calibration of the flat frames. Likewise the light frames and the MasterDark are overscan-calibrated in the calibration of the light frames. This procedure is much less complex than the one that I outlined above.

Please let me know whether this procedure is producing correct results with your new camera.

Bernd
Click to expand...
Bernd:

I appreciate your help with this question. I have finally received my telescope and have a first few images. I am having an issue.

The total size of the image from the sensor is 9600x6422 pix.
The optic black area (that is not used) is from the upper left corner (0,0) to (24, 6387).
The overscan area (that should be used for bias) is from (0, 6388) to (9599, 6421).
The image area is from (25, 0) to (9599, 6387)

Here is an image from the manual for the QHY600M:
1590927038612.png


I have defined the overscan area like this:
1590927076432.png


When I attempt to calibrate and apply the overscan area PixInsight says:
***Error: Inconsistant overscan geometry

I do not understand what I have configured incorrectly. Your help would be appreciated

Tim.
 
Hi Tim,

according to the mouse-over help text, the rectangles are defined by the coordinates x, y of the upper left point, and width and height of the rectangle. So if you want to use the region from (0, 6388) to (9599, 6421) as source region and the region from (25, 0) to (9599, 6387) as Image region , the input must be:

Image region: 25 0 9575 6388
Source region: 0 6388 9600 34 <==
Target region: 25 0 9575 6388

A Source region height of 35 would be outside the sensor's dimensions (coordinates start at 0, not at 1). I don't have a frame of the QHY600 any more, so I cannot try these settings.

Bernd
 
Hi Tim,

Out of curiosity (and since I just bought a QHY268C that offers overscan calibration), did you find that the overscan calibration resulted in a visibly improved image quality?

/ralf
 
Ralf:

It really did for the 600M. This sentence from the QHY support team is the key:

The on-chip calibration part of the cmos sensor may cause the drift of the whole image when the image is bright. By using the overscan area, you can correct for this drift as follows:

By using the overscan area when calibrating my flats, and again when calibrating the lights, the images are visibly better. The process was actually trivial once I started doing it. I set up a process icon in PI with the proper values for the overscan area and it is basically no different that NOT using it. I use a master dark and a master flat, no bias. The flats are calibrated with a master dark only when creating the master flat.

Let me know if I can answer any questions...

Tim.

This is how it's set up. Your values will be different if the chip is a different size, but same idea...
1604435728679.png
 
Hi Tim

Thank you for the explanation and screenshot. Very helpful.

Am I correct in assuming that I capture all frame types (dark, flat, dark-flat and light) with overscan enabled in the ASCOM driver to ensure identical image geometry across all frame types?

Looking at the "simplified" process outlined by Bernd and your screenshot, the process steps would appear to be as follows:

1. Dark frames and flat-darks are integrated with the usual settings into MasterDark and MasterFlatDark
--> No overscan calibration takes place to remove the overscan and optical black area, however they will be calibrated in steps 2 and 3 respectively

2. Flat frames are calibrated using the appropriate overscan area for the sensor used
- 'Master Bias' section is enabled, use MasterFlatDark created in steps 1 and enable 'Calibrate'
- 'Master Dark' and 'Master Flat’ sections are disabled

3. Light frames are calibrated using the appropriate overscan area for the sensor used
- 'Master Bias' section is disabled
- 'Master Dark' section is enabled, use the MasterDark created in step 1, enable the option 'Calibrate' and disable the option 'Optimize',
- 'Master Flat' section is enabled, use the MasterFlat created in step 2, disable the option 'Calibrate'.

--> Your screenshot shows 'Optimize' enabled in the 'MasterDark' section, which differs from the settings described by Bernd. Do you use different exposure times for your lights and darks?

Am I getting this right?

/Ralf
 
Last edited:
rakla said:
Hi Tim

Thank you for the explanation and screenshot. Very helpful.

Am I correct in assuming that I capture all frame types (dark, flat, dark-flat and light) with overscan enabled in the ASCOM driver to ensure identical image geometry across all frame types?

Looking at the "simplified" process outlined by Bernd and your screenshot, the process steps would appear to be as follows:

1. Dark frames and flat-darks are integrated with the usual settings into MasterDark and MasterFlatDark
--> No overscan calibration takes place to remove the overscan and optical black area, however they will be calibrated in steps 2 and 3 respectively

2. Flat frames are calibrated using the appropriate overscan area for the sensor used
- 'Master Bias' section is enabled, use MasterFlatDark created in steps 1 and enable 'Calibrate'
- 'Master Dark' and 'Master Flat’ sections are disabled

3. Light frames are calibrated using the appropriate overscan area for the sensor used
- 'Master Bias' section is disabled
- 'Master Dark' section is enabled, use the MasterDark created in step 1, enable the option 'Calibrate' and disable the option 'Optimize',
- 'Master Flat' section is enabled, use the MasterFlat created in step 2, disable the option 'Calibrate'.

--> Your screenshot shows 'Optimize' enabled in the 'MasterDark' section, which differs from the settings described by Bernd. Do you use different exposure times for your lights and darks?

Am I getting this right?

/Ralf
Click to expand...
Hi Ralf:

You have it correct, but I place the master dark in the master dark field in step 2. I don't know what the difference would be as the action taken by PI is the same for both I believe. If you're more comfortable using the master bias field that's fine. It shouldn't matter to my knowledge. You SHOULD uncheck 'Calibrate' in step 3. That was an oversight on my part. My bad. I always use master darks with the same exposure as the flat/light frames that I am calibrating.

I'm glad this was helpful to you. If anything comes up or you have any other questions, I'm happy to help if I can.

Best.
Tim.
 
Hi Tim

Thanks a lot.

Another question: whilst taking test frames, I noticed that the FITS file has another black area on the right side of the frame (4 pixels wide). I could confirm this with PI and FITS Liberator, so not application specific (I had to converted / compressed the file to be able to post it, but you should see the area when you zoom in).
View attachment Target 1_QHYCCD-Cameras-Capture_-15C_gain_60_offset_NA_-15C_7.45sec_2020-11-04_amb_7.4C_1x1_fr...jpg

This is nowhere mentioned on the QHY site / camera manual nor in the senor specs document from Sony.

Do you experience the same with your sensor (I know you have a different sensor, but wonder if this has to do with the QHY ASCOM driver?

Can somebody else confirm the same behaviour?

/Ralf
 
Hi Ralf:

I DO NOT have this black area on the right side of my images. Also, when stretched, I can see brighter and darker pixels in the overscan area that I don't see in yours. That may be because of how you uploaded the image (jpg vs. raw).

Here is a link to a sample file to show you what one of my images looks like:
Sample File

Tim.
 
Hi Tim,

Thanks for that. According to QHY the border on the right side appears to be correct for my sensor... strikes me as somewhat odd.

In case somebody else has the same question / observation, here is the reply from QHY:

"The four pixel widths on the right are black Border.
When you enable the overscan in ASCOM driver, the image size is 6252*4176. If the overscan is not enabled, the image size is 6280* 4210,6280-6252 =28 (24+4 contains 24 pixels of black Border on the left and 4 pixels of black Border on the right)."


/Ralf
 
Last edited:
Hi Ralf,

I've read this thread and a few other related overscan threads you're involved in... thanks for putting your time into these responses. I also have a QHY268C, and have two very specific questions:

1) Do you make any use of the right-side optical black area (the 4px wide vertical strip? Meaning, do you include that as an overscan area used for calibration purposes, or do you exclude that altogether?)

2) Are you able to post your exact QHY268C overscan settings? I see the other sensors' screenshot examples, and the inferences to the QHY268C sensor sizes/areas, but I haven't see a specific screenshot of the QHY268C settings boxes. I think this would help me and others with our same camera when we stumble across this valuable thread.

Thanks!
JC
 
Hi JC,

1/ I don't use the optical black of the right-side and exclude it altogether.

2/ Here are my settings for calibrating flats and lights.

Calibrate Flats.png
Calibrate Lights.png


In case you want to do CC using your Master Dark, make sure you generate a Master Dark that excludes the overscan area, as the geometry would be incompatible otherwise.

I hope this helps.

/Ralf
 
You must log in or register to reply here.
Back
Top