New Script: Batch Debayer

kenpendlebury

Well-known member
Hi All,

I am still on my trial license, but I am very impressed so far.  As part of my trial, it was important to me to experience scripting at least at a very basic level.  As such, I noticed a module named Debayer and while certainly useful, requires opening each file individually in order to apply the module.  Since I have many short exposure images to debayer, I have devised a script to apply this module over multiple images at once.

This script doesn't do anything fancy and I do not claim much of anything here.  The interface is blatantly stolen from Niall Saunders (CMYG Batch Debayer) and the actual Debayer code is from the PCL debayer module.  As such, you will find the same options (Debayer method and Bayer Pattern) in the script dialog.  The chosen method and pattern will be applied to all images in the tree selector.

This is admittedly not tested to an acceptable point, but I only have access to a Mac (and have not even tested all permutations here).  Anyone interested in testing is welcome to do so.  Also on the todo list is better commenting around certain assumptions and methodologies chosen...

batchdebayer.jpg



Regards,
Ken
 

Attachments

Hi Ken,

EXCELLENT

This is exactly the kind of 'newbie' that we are after - obviously you have found our 'deep and cold water' to be just to your liking. There are not many 'newbies' who jump in at the deep end, and then go straight down to the bottom of our 'murky pool' to see what old pennies might be lying around there ;D

But, my code was partly plagiarised from other's efforts, and helped along the way by Juan, and offered for all to use however they wish. So, I am glad that you found it (a) useful and (b) useable enough to allow you to adapt it for you own needs (that actually is my reward for the hundreds of hours I put into writing it in the first place - for someone to be able to use it as the skeleton for their own work - and is why Juan was happy to include it as part of the standard distribution of PI ever since)

Serious congratulations are due to you !

Cheers,
 
Hi Harry (et Bonjour from la deepest France, un fois encore !!),

Yes - it can certainly be used as an example for all those who claim that the PI interface is 'too quirky' to use  ::)

Perhaps we have a future Jedi in the making !!

Cheers,
 
Hi Ken,

Welcome to PixInsight.

Excellent work. I support everything that Niall has said. Please feel like at home on this forum. Don't hesitate to post your work here, and ask us for everything you need regarding development on PixInsight.

By the way, Niall, I think you underestimate the importance of your script. Besides a useful tool, it is a reference work for how you wrote it and the way you documented it.
 
Thanks for the encouragement and feedback.  I am very excited about PI and still wondering how I have never heard of it until recently.  It is possible that I just assumed I could not use it on a Mac (too often in AP, this is a valid assumption).  Its usefulness is mind-boggling and the speed at which I can process data blows me away.

Niall, I certainly agree with Juan.  Without your script (and the others included) this script would have been a long time in the making.  Thank you for your very thoughtful commenting.  I already feel like a have a pretty firm grasp around the UI and general architecture.

I have had a few questions regarding the difference between bilinear interpolation vs. superpixel so I will try to update the "tooltip" boxes to cover this as briefly as I can.

Again, thanks for the compliments and warm welcome.  I look forward to becoming an active PI contributor.

Regards,
Ken
 
Hi Ken

I have had a go with my sxvf m25 OSC camera and the script works brill  8)

Would you know how to introduce weights to the colour channels  ???


Regards Dumb Harry
 
Harry,

I am pretty sure that I do know how to weight the color channels independently.  Unfortunately, I am using the PCL Debayer module to accomplish all the "image" work so I don't have access to the image pixels directly during this process.  I believe, in my limited knowledge of the scripting model, that I would be able to grab the output of the debayer module and apply a "post" weighting to all data via script.  Because of this, the script might turn into a huge time hog.

This is definitely a great idea and is surely worth exploring.  I do have one question though... Would a color weighting during color reconstruction provide any distinct advantage over the Color Calibration module that ships with PI?  This is an honest question... I really don't know.

Regards,
Ken
 
Hi

As you say colour calibration does sort out a lot of problems , but I like to get things as close as possible as not all images have a lot of stars and or
a nice galaxy  ;) for CC

I have used other programs to do this  :'(

Thanks for your time Harry
 
Hi guys,

Ken - as you will have seen from my (DSI-IIC, CMYG) deBayer PJSR script, I used several methods to try and get my data 'colour perfect' at the individual image stage - long before they were aligned and integrated.

However, that script was conceived 'at the same time' as the introduction of the BackgroundNeutralisation and ColourEqualisation modules were being developed, and I would certainly, now, not even waste my time trying to compensate colours until well AFTER I had used ImageIntegration (with the stunning power of Winsorized Sigma Clipping for 'crud' removal) to sort out all of my data calibration, and I would then be invoking DynamicCrop and DBE (perhaps twice) before I felt that my master image was ready for the BackgroundNeutralisation and ColourEqualisation.

I don't know how well I have ever described the ColourCurve adjustment steps that I include in my script, but I figured out a process that just 'worked' - and stunningly well at that, given that my deBayer was of a CMYG 2x4 CFA matrix - which is a devilishly difficult beast to tame when it comes to extracting RGB from the RAW data!

Basically, what i did was to print out a very simple colour chart (included in a post elsewhere on the Forum, but whose link may still be broken pending Juan's final 'tidy up' needed subsequent to his server swap) - essentially three swatches of R, G and B, and three swatches of Cy, Mg and Gn - created as 'perfect colours' and printed out on 'proper' photographic paper by the likes of Costco, WalMart, etc. I then take a LONG EXPOSURE photograph of this (several actually) using EXACTLY the same steps (and equipment) as I would if it were an astronomical target - using natural daylight to illuminate the print, and a simple 50mm lens attached to whichever camera I am using. This will involve SIGNIFICANT 'workarounds' to filter the INTENSITY of the daylight, but NOT at the expense of the SPECTRUM of the incident light. You will have to work this out to suit your available equipment.

Once I have calibrated, aligned and stacked the RAW data, and deBayered where necessary, I have an image that has 'colour' - but not 'pure' colour.

Then I look at the 'Red' swatch, for example, and figure out where I need to place a 'modifying' point on the ColourCurve transformation to 'change' what 'should be' Red (and isn't) into 'actual' Red. I repeat this for Bu and Gn, and then add intermediate points for Cy, Mg and Ye. The LivePreview helps you to see what is happening as you prepare this horribly complex, twisty little curve. Then you apply the curve to the image, see what the result was - and consider repeating the exerciswe again - fine tuning, if you will.

Very importantly - you need to save ProcessIcons for each of the ColourCurve transformations - the idea being that these curves actually 'define' the 'colour response' of your camera, given the specific deBayer routine that you have used. My theory is that the camera has NO IDEA that it wasn't looking at a DSO - it just gathered photons, over a specific period of time. If the Calibration and deBayer process remains (effectively) 'constant', then the application of a 'ColourCurve' correction should also be a 'constant'

What you WILL need though - I forgot to mention this - is a swatch of 'Pure Black' and one of 'Pure White' (effectively the photographic paper itself, of course) in the image as well - to allow you to accurately define the Black and White points. Interestingly, you will notice that the 'greyscale' is NOT affected by your ColourCurve transformation.

You might be able to make sense of the process if you break down the post-deBayering steps that I have placed into my script.

Certainly, when I defined a double-correcting pair of ColourCurves from my 'simulated' colour test-card, and then applied the same Curve processes to RAW data acquired of a general countryside landscape, and of some highly colourful 'still-life' samples, the effect was incredible. And when I then applied the same process to ACTUAL data collected for M81 / M82 (using my DSI-IIC on a Moonfish ED80 APO) the effort had DEFINITELY been worthwhile.

I just haven't really made a big thing about it, because myself and Alex next door are probably the ONLY PixInsight users left who are still trying to image with CMYG one-shot CCDs - and even I have since turned back to my DSI-IIPro mono imager, to try and collect the best photons that I can (Alex may thus be the VERY LAST surviving DSI-IIC imager using PixInsight for post-processing - and he currently doesn't bother with Flats or FlatDarks, and relies entirely on Nebulosity's deBayer routine to create his 'master image', and then just throws very simple BackgroundNeutralisation and ColourEqualisation processes at the data as 'automatic' steps in his initial openeing gambit - and his data ends up quite nicely colour-balanced, thank-you very much - with the minimum of fuss, bother or effort).

It would be worth your while examining different strategies - one will suit you best, and you will know that it may need to be changed as your skill, and the power of PI, improve in the future.

Cheers,
 
Hi

My problem with a fixed rgb weight is that is does not help when using say Light pollution filters and imaging at different dec

As I said before CC works wonders most of the time , but take this image I took at the weekend  Full high res at http://www.harrysastroshed.com/Image%20html/CONE.HTML

I had a devil of a time trying to get the colour right , mainley due to the lack of background and again its nice to start off as close as possible

Of course what I / we want is a total package  calibration inc debayer , registration and stack All wrapped up in a easy to use interface  :P  Well I can hope


Harry
 

Attachments

  • cone600.jpg
    cone600.jpg
    87.6 KB · Views: 129
Hi Folks,

I was going to spend a little time to make debayer decisions easier for folks by including actual camera types in the "Bayer Pattern" drop down (in addition to the 4 patterns).  This should help ease the string of "how do I verify my debayer is correct?" and "What bayer pattern should I use?" questions.

I am in the process of collecting a list of OSC models that use RGB debayer pattern.  If you would like to contribute your "verified by experience" patterns for your camera, I will include them in the next revision.

Also... the debayer module currently supports 2 types of debayer algorithms.  I will try to create a tooltip that summarizes the pros and cons of each type.

While I realize this script is not overwhelmigly popular (with 17 downloads  ;)), I think it offers value with the introduction of the Image Calibration Module.  This makes it easy to do ALL of your OSC pre-processing and calibration without leaving PI.

Thanks for your help,
Ken
 
Hi Ken

As a official script inc ver 1.6 you will not get any further downloads , but I use it all the time and are very grateful of your efforts

As for fixed camera settings , I know you will struggle with my sxvf m25 because depending on firmware versions the settings are different  ???

Harry

 
Thanks Harry.  I think I have a misunderstanding about OSC cameras.  Is the bayer film applied to CCD camera not a property of the chip?  I didn't think the firmware could change the bayer pattern of a CCD chip.

Ken
 
Hi,

I purposely did not include channel weighting in the debayer module because there's no need for it. Color calibration after the fact does a much better job than anyone can hope to do by pre-weighing the colors. Even if you're a genius and the resulting image is 'close' you'll still need to fine tune it afterwards so why not simply tune it the whole way?

Anyway, if you're desperate to add weighting it's easy to add that to the debayer script. Simply use PixelMath to multiply each channel with a constant after the debayer phase.
 
Color calibration after the fact does a much better job than anyone can hope to do by pre-weighing the colors

I have to agree with Sander - I would just leave the 'weighting matrix' out of the equation altogether - set it to a 'unitary transform', which just leaves users the choice of where to 'satrt' the CFA matrix.

Cheers,
 
Hi

Yes CC does a excellent job 90% of the time , but not all the time this is why I would have liked the weight option  :-*


But if it is not in the pipeline :'(  I will have to resort to plan b



Harry
 
Harry the point is that CC will always do a better job than weighting up front. I really don't believe anyone can pick weighting factors *before* seeing the final image that will produce a satisfactory result. LP and atmospheric extinction differences between different nights mean that factors for one night aren't accurate for the next. So you might as well give up on that and use CC or other methods to correct the color afterwards.

In any case, if you *must* apply factors to the channels it isn't even necessary to do that to individual frames. Just do it to the stacked image before further processing. There is no difference since it's multiplicative. If you have fixed ratios it would be trivial to write a PJSR script that does this for you (no GUI needed as the factors don't change) or to keep a PM instance ready to go that does the same.
 
Back
Top