XISF vs FITS

[chris.bailey]

Hello all,
Does it matter when processing in PixInsight whether I capture (in NINA) using Fits or using XISF setting?
Thanks,
Andrew

One advantage of capturing in .xisf is that with lossless compression (available in NINA) file sizes will be appreciably smaller, around 50%. This significantly speeds up data transfer from the capture device, wirelessly (in my case) or via a portable hard drive and saves some hard drive space. If you process fully in PI it’s a pretty convincing case in my view.

 

[cloudbait]

Once more, this time in a more structured way to facilitate the understanding of some essential points:

• This is PixInsight Forum, not a general astronomy/astrophotography forum.
• This forum has rules. This forum is for topics related exclusively to PixInsight.
• The FITS format was deprecated in PixInsight many years ago. Its use is discouraged in PixInsight.
• The XISF format is the native PixInsight file format. Its use is recommended in PixInsight.

The OP asked which format is better, FITS or XISF, for images he will be processing in PixInsight. We know the following facts:

• The user acquires the images using N.I.N.A., a well-known image acquisition and hardware control software.
• N.I.N.A. fully supports the XISF format at the baseline writer level and hence can write standards-compliant XISF files that can be imported directly into PixInsight.

Now, here is the first answer this user has received in PixInsight Forum:



And here is the second answer (yours):



So, not only are you recommending the use of a format we have deprecated, ignoring the fact that we officially recommend and encourage the use of XISF in PixInsight—because remember, this is PixInsight Forum, not a general astronomy forum, and XISF is PixInsight's native format—, but you are also ignoring the work we have been doing during many years to put into value one of the most important contributions we are doing to the astronomy and astrophotography world: a modern, efficient, rigorously defined, versatile, interoperable, open file format for the storage and interchange of astronomical data and images.

But even worse, you have communicated the idea that the XISF specification is "quite fluid." Please tell me if I am wrong, but "fluid" is the opposite of "solid" in my dictionaries. Of course, if a file format has no solid specification, then nobody sane would use it since doing so would put their data at risk. Here is where I couldn't tolerate such a degree of intoxication and felt it necessary to intervene personally in this thread.

XISF has a rigorous specification. It is not fluid but precisely the opposite. Please let me know if you disagree with this statement and, in such a case, enumerate the points where the specification is poorly defined or could be more consistent. That would help us improve it in its next version, which we are planning to fix several errors and extend the format definition to cover essential aspects, such as astrometric solutions, among others.

Besides providing a baseline XISF writer implementation, N.I.N.A. includes all compression algorithms in the official XISF specification, which can be a great advantage for storing raw frames. You say, "It is not clear how closely NINA will be tracking it [the XISF specification]. I expect backward compatibility will be reasonably robust..." What you are saying implies something that cannot happen: the XISF specification guarantees that any existing XISF file generated by a baseline XISF writer will always be readable by any baseline XISF reader, so the possibility of losing data because of format specification changes does not exist by definition, and hence recommending "load and resave ... to ensure my files were to the latest spec" has no sense and generates additional doubts toward the format based on prejudices, not on technically defensible facts.

IMHO, here you are not helping a PixInsight user to use PixInsight proficiently, and undoubtedly, you are not helping the PixInsight project by any means. You are seeding doubts about one of our most important development projects and disseminating the toxic idea that it is best to store raw data as FITS files because the XISF specification is "fluid," and there can be compatibility issues when future versions of the XISF specification are released. In addition, you communicate that using XISF to store raw data that will be processed in PixInsight is not a good option because "only PixInsight uses it", so FITS is better because it allows you to use other applications. And you say this precisely on PixInsight Forum. You are not very respectful toward the hard work we are doing to make astronomy and astrophotography better, let alone the invaluable help you are providing to all who are trying to block the development of XISF by despising it systematically.

Now, let's examine a different element you have introduced in the discussion. Again, under normal circumstances, I wouldn't intervene, but as the owner and CEO of this company, I cannot avoid it. You have written:



Once more, let's put several essential facts in a structured way for better understanding:

• Pleiades Astrophoto S.L., a Spanish software development company registered under EU VAT number ESB97989230, pays for this forum and all its associated infrastructure entirely. The same applies to all corporate servers, websites, hardware, and tools used by me, all our staff, and all our team members and collaborators to build and maintain PixInsight. None of our users pays for anything related to these concepts.
  
  
• Our commercially licensed users pay exclusively for the non-exclusive, non-transferable right to use versions 1.x of our software, PixInsight, under the terms specified in our End User License Agreement.
  
  
• PixInsight Forum exists exclusively to help PixInsight users understand and use the PixInsight software as an image processing and software development platform. This is the only purpose of this forum. This is not and will never be a general astronomy and astrophotography forum. The discussion topics in this forum must be related exclusively to PixInsight. However, discussions about general astronomy, astrophotography, and development topics are also welcome and encouraged, as far as there is no mention of other applications or tools, especially applications or tools that compete with us. This is clearly stated in our rules.
  
  
• We develop and maintain PixInsight without any external dependency or guidance regarding the fulfillment of externally imposed conditions. We make PixInsight the way we consider it necessary, following our criteria exclusively, and offer it to our current and potential users in the hope that it will be useful, beautiful, elegant, and a significant contribution to the worlds of astronomy, astrophotography, science and culture in general. This has always been and will always be our only commitment. We feel no obligation to comply with anybody's expectations.

I have no comments on the other opinions expressed here about me and my behavior. As previously noted, I am unimportant so you can continue if you wish, but don't expect a reaction because that is not my role here.

This continues to miss the point. The discussion is not about comparing two file formats.

Objectively, acquiring an image in XISF format is generally disadvantageous, as it's a non-standard, not widely used format. So the question remains: what advantage is provided by acquiring this format in NINA that offsets this compatibility issue? What information does NINA include in the XISF file that is missing in the FITS file, and in some way gives better results during PI processing? Objectively, why should we prefer our source images to be XISF over FITS? That the former may be a "better" format for a variety of theoretical reasons isn't an answer.

And what does it even mean that the FITS format is "deprecated" in PI? As the term is virtually always used in the context of software, this would suggest that you don't recommend using it and could remove support for it in the future. I hardly see that happening. It's a standard that isn't going away, and you'd be shooting yourself in the foot to remove support. It would be like Photoshop deprecating the TIFF format. It makes no sense. You should not be "deprecating" FITS, you should simply be pointing out that once you've used PI to process an image (and the format that image started as is irrelevant) the appropriate format to use is XISF, PI's native format. In the same way that you would always save an image processed in Photoshop in PSD format, regardless of how it started.

 

[cloudbait]

I think there is a misunderstanding.
Yes, the user asked whether it makes a difference to capture in NINA in FITS or XISF for further processing.
Though that PI supports fully the import of FITS images, it doesn't really matter. Because the FITS files are saved as XISF files.

I don't see in any of the posts that suggesting to do further further processing by saving as FITS files.

Unfortunately, since the release of the official specification 5 years ago, there has been not much adoption of the XISF format, NINA and Ekos are, to my knowledge, the only acquisition software to save the files in XISF format.

This is really an interesting question, I haven't seen an answer to that (though I don't rule out that I have overseen that due lack of caffeine).
Are there more information captured in XISF opposite to FITS?
How about in the future when XISF is not more widely adopted to capture as XISF format?
Like companies for example ZWO with AsiAir only supporting fits format.
Cameras who write only FITS format?
I don't see that the FITS format vanishes in the next years.

On a personal note:
Why always this aggressive tone like users would run a personal attack?
We, the users, try to help that new users don't feel lost with Pixinsight.
It's also in our interest that Pixinsight is successful, but by being harsh to users who help, it's more alienating these "helpers".
A new user doesn't find bugs, most assume the software is the problem and not them.
Honestly, I don't believe that the developers have so much time spending their time on the forum for several hours to help new users, judging by several posts that all your resources are invested in developing Pixinsight.

Cheers
Tom

We also need to consider the lack of symmetry in PI's ability to read and write FITS files. It appears to be able to fully read FITS without missing anything required for subsequent processing. But it does not retain all of the information it doesn't require, so if I subsequently need to produce a FITS version for some other tool, that file will lack many of the keywords present in the original. Which is one reason I would always use FITS at the acquisition stage, even if XISF were an option.

It is interesting to look at Photoshop's PSD file format. It contains a structured metadata section with entries for multiple file types (TIFF, RAW, JPEG, etc). When I open a non-PSD file in Photoshop, the file metadata is retained even if Photoshop doesn't use that information. I can open some rarely used format, like DICOM, process it, save my master as a PSD and also export a new DICOM file, and that latter will have its metadata refilled with the retained data from the source image. This would be a great way for PI to work.

 

[fredvanner]

So, not only are you recommending the use of a format we have deprecated, ignoring the fact that we officially recommend and encourage the use of XISF in PixInsight—because remember, this is PixInsight Forum, not a general astronomy forum, and XISF is PixInsight's native format—, but you are also ignoring the work we have been doing during many years to put into value one of the most important contributions we are doing to the astronomy and astrophotography world: a modern, efficient, rigorously defined, versatile, interoperable, open file format for the storage and interchange of astronomical data and images.

This is simply not true. You keep repeating the same mistake. The OP did not ask for an opinion on the merits of FITS or XISF. He asked "did it matter" in other words "will it make any difference to his results. I still believe the answer is "no".

IMHO, here you are not helping a PixInsight user to use PixInsight proficiently, and undoubtedly, you are not helping the PixInsight project by any means. You are seeding doubts about one of our most important development projects and disseminating the toxic idea that it is best to store raw data as FITS files because the XISF specification is "fluid," and there can be compatibility issues when future versions of the XISF specification are released. In addition, you communicate that using XISF to store raw data that will be processed in PixInsight is not a good option because "only PixInsight uses it", so FITS is better because it allows you to use other applications. And you say this precisely on PixInsight Forum. You are not very respectful toward the hard work we are doing to make astronomy and astrophotography better, let alone the invaluable help you are providing to all who are trying to block the development of XISF by despising it systematically.

This opinion is not "humble". It is arrogant and biased.
We are helping PixInsight users by telling them the truth: that XISF is a format that is only used by PixInsight, while the FITS format is used, and will almost certainly be used for the forseeable future, by virtually every other application and astrophotographic database. Misrepresenting this is not helping users, and will lead to reduced confidence in PixInsight.

 

[tom886]

But it does not retain all of the information it doesn't require, so if I subsequently need to produce a FITS version for some other tool, that file will lack many of the keywords present in the original.

I think that is exactly what @Juan wants to avoid, using any other tool, keep the user in Pixinsight ecosystem.
Like AsiAir only works with Zwo equipment.

Cheers
Tom

 

[fredvanner]

I think that is exactly what @Juan wants to avoid, using any other tool, keep the user in Pixinsight ecosystem.
Like AsiAir only works with Zwo equipment.

That is a strategic decision for PixInsight. However, it makes the case for initial capture in FITS even stronger. If you can never get decent quality FITS files out of PixInsight, and the entire rest of the world recognises FITS files and does not recognise XISF files, then you are well-advised to capture in FITS. This is not a criticism of PixInsight; it is not disseminating "toxic" anti-PixInsight propaganda; it is simply telling the truth.

 

[cloudbait]

I think that is exactly what @Juan wants to avoid, using any other tool, keep the user in Pixinsight ecosystem.
Like AsiAir only works with Zwo equipment.

Cheers
Tom

Well, PI is going to have to create many more internal tools before that can happen! I already gave examples of analysis I perform that cannot be done inside of PI and which requires FITS format files. My workflow there is to preprocess the data in PI (and I'd pay full price for PI just for WBPP!) and then export to FITS for use outside of PI.

BTW, the right way to think of this is to separate "open" and "save" from "import" and "export". Most well designed applications open and save in their native format, but support a rich range of other formats for import and export. That doesn't require "deprecating" any of those other formats!

 

[fredvanner]

Most well designed applications open and save in their native format, but support a rich range of other formats for import and export.

If PixInsight had a half-way reasonable "export" capability for FITS this conversation would not have been needed. Sorry for the blasphemy - of course I understand that FITS format is evil...

 

[tom886]

If PixInsight had a half-way reasonable "export" capability for FITS this conversation would not have been needed.

There was, it has been crippled.
Any other software, keeps backward compatibility with previous file formats, most common examples is Microsoft Office.

Cheers
Tom

 

[bulrichl]

It appears to be able to fully read FITS without missing anything required for subsequent processing. But it does not retain all of the information it doesn't require, so if I subsequently need to produce a FITS version for some other tool, that file will lack many of the keywords present in the original. Which is one reason I would always use FITS at the acquisition stage, even if XISF were an option.

I have just compared two FITS files:
1) saved by NINA,
2) created by PixInsight from a XISF file that was saved by NINA.
Not a single keyword was missing in 2).

So let's get straight to the point: Which keywords are lacking?

Bernd

 

[cloudbait]

I have just compared two FITS files:
1) saved by NINA,
2) created by PixInsight from a XISF file that was saved by NINA.
Not a single keyword was missing in 2).

So let's get straight to the point: Which keywords are lacking?

Bernd

I can't attach my own keywords to an XISF file, AFAIK. And what about a WCS solution present in the FITS data? What about comments?

 

[tom886]

1) saved by NINA,
2) created by PixInsight from a XISF file that was saved by NINA.

This would mean that NINA doesn't save all information in the FITS file, because we know that that PI doesn't export all files to FITS from XISF.
More interesting would be to know, is there a difference in NINA when saving in FITS and XISF.

Cheers
Tom

 

[bulrichl]

This thread goes about NINA, FITS format or XISF.

Did I express myself unclear? Which keywords are lacking?

These keywords are present in 2) just as in 1):

SIMPLE, BITPIX, NAXIS, NAXIS1, NAXIS2 ,EXTEND, BZERO, BSCALE, ROWORDER, IMAGETYP, DATE-OBS, DATE-LOC, DATE-AVG, EXPOSURE,
EXPTIME, INSTRUME, GAIN, OFFSET, EGAIN, XBINNING, YBINNING, SET-TEMP, CCD-TEMP, XPIXSZ, YPIXSZ, BAYERPAT, XBAYROFF, YBAYROFF, SITEELEV, SITELAT, SITELONG, TELESCOP, FOCALLEN, FOCRATIO, RA, DEC, CENTALT, CENTAZ, AIRMASS, PIERSIDE, OBJECT, OBJCTRA, OBJCTDEC, OBJCTROT, FOCNAME, FOCPOS, FOCUSPOS, FOCUSSZ, FOCTEMP, FOCUSTEM, FWHEEL, FILTER, DEWPOINT, HUMIDITY, PRESSURE, AMBTEMP, EQUINOX, END

Bernd

 

[fredvanner]

And what about a WCS solution present in the FITS data?

I'm sure that I have had files with a FITS WCS solution, where the solution was dropped from the header when saved as XISF. I've just tried to reproduce this, and the (full, distortion-corrected) WCS solution was preserved. I wonder if the behaviour of PI has changed?

 

[Andrew]

Well, this turned into quite the discussion. I think I'll stay with capturing in XISF.
Andrew

 

[dom_beau]

Reading this thread is absolutely mind-blowing...

1) Señor Conejero, you jeopardize PI future. Maybe you should step away for a while and think about how you talk to your customers. We, your customers, are the guys who advertise your soft, in good or in bad.

2) You should also listen about your customers' needs. As a customer, I need a software that fully supports FITS and I don't care about XISF. It is not a question of being a good or a bad format. FITS is simply THE standard, that's it. In my workflow, PI is neither at the beginning, nor at the end. So I want it to support fluidly THE format. If you want to develop a new format, free to you to do it. Do it in a freeware. Not a soft one will pay more that 200USD for.

Salut!

 

[Rainer]

Hi,

I am more or less a PixInsight user at a stage of maybe 15% understanding PixInsight.

Now that I have learned to 25% to use NINA and seeing NINA supports *.xisf file saving I asked myself one day ¿should I save in *.xisf? and my first thought was, JEEEZZZ, the files are the double in size then in fits.

As cloudbait asked:

"What advantage does XISF provide at the acquisition stage that FITS can't?"

If somebody has an opinion about this and it convinces me, I would gladly as a beginner save in *.xisf sacrificing disc space. What I can not accept as answer at the moment is "Disc space is dirt cheap, so what?" Such an answer does not tell me as a beginner ¿what advantages do I have saving my RAW material = images in *.xisf format?

As an ignorant about bits I know that my camera is a 12bit sensor, 4096 shades of grey, and the firmware converts my imagen into 16bit = 65536 shades of grey. OK, that is already invented information in my image via interpolation and a guessing algorithm for filling up the gaps between my 4096 shades of grey.

Now so far *.xisf is a 32bit information storing system, which tells me in my ignorance more invented information¿? Sorry is this sounds stupid but this is how I understand it. If I am wrong please correct me.

So, who can convince me to save in *.xisf instead of *.fit?

Have a nice Sunday

Rainer

 

[Geoff]

As someone pointed out on another forum also with a heated ongoing discussion a big problem with the internet forums is that emotions escalate very quickly, and then it becomes difficult to have a constructive conversation. Maybe everyone should just cool it a bit. A semantic argument over “should use FIT” as opposed to “ can use FIT” is not worth getting heated about—this applies to both sides.

 

[monos]

As a recent new member and at the same time recent PI user I was a bit surprised by the outcome of this thread. Don't take too much notice of me. I don't have the knowledge to judge this issue in detail in a technical way. But there are a couple of things that I think need to be taken into account. The first is all the time and effort that I assume the PI team has put into developing a new file format to make up for the shortcomings of the old Fits, and I don't think that all that use of resources was done on the whim of Juan or the rest of the team. My guess is that if such work was done it was because it was necessary to go beyond the limits of Fits and that failure to do so would in some way undermine PI's own progress. So it is an effort made for PI but also for us as users. This has to be appreciated even if we do not see the present or future advantages of xisf development.

The second is that the term standard in computing is in my opinion a meaningless term. MS windows is considered a standard or at least it was for a long time... but not for me. Mp3 is too?, but any Flac or Ogg file beats it. And so on with many other things. And the bad thing is that we often consider or tolerate as a standard a mediocre or outdated format just because the human being tends to save resources and energy and is not usually motivated to change something as long as it works more or less well unless it is imperatively needed or forced. The resistance to change of the human species is an established fact.

In my opinion we should be fair to the effort that has been made in PI to develop something better and we should also be fair to the effort made by the NINA team to implement it. And it is also our responsibility as users to look for the best and demand better developed tools because we are the ones who can motivate the developers to implement those improvements. Not only NINA has implemented it. Also Siril and Graxpert read xisf files and graxpert can even save in xisf because the format is open and documented. And they have not done it for PI's benefit but for the benefit of us, the users of astrophotographic processing tools.

Personally Fits has given me occasional problems such as files being displayed flipped or rotated depending on which tool was displaying them or simply files opening in some tools but not in others. I've also had no success using compression with Fits whereas with xisf it works fine. So there may be trade-offs with either format but as long as PI supports Fits along with xifs who who is worried about compatibility?

Regarding Juan's tone... in written words it is difficult to understand the body language that is usually implied by the use of spoken language. His tone seems to me taxative but not offensive but of course... everyone can interpret it in a different way is inevitable when we only express ourselves through a sad keyboard. However, if the question in the first message had been answered with a simple:

PI always recommends the use of xifs if available so in principle it would be the natural thing to do although there are some who for backward compatibility reasons prefer to keep the Fits format as they use them in other tools. It is up to you as a user to decide.

Probably Juan would not have reacted in such a categorical way. Or at least that's the impression I get.

Regards

 

[cloudbait]

Reading this thread is absolutely mind-blowing...

1) Señor Conejero, you jeopardize PI future. Maybe you should step away for a while and think about how you talk to your customers. We, your customers, are the guys who advertise your soft, in good or in bad.

2) You should also listen about your customers' needs. As a customer, I need a software that fully supports FITS and I don't care about XISF. It is not a question of being a good or a bad format. FITS is simply THE standard, that's it. In my workflow, PI is neither at the beginning, nor at the end. So I want it to support fluidly THE format. If you want to develop a new format, free to you to do it. Do it in a freeware. Not a soft one will pay more that 200USD for.

Salut!

But you should care about XISF. It's the only rational way to save your work in PI. And it would make no sense for PI to try and support FITS as its native format. XISF is perfect for that, and far superior to FITS. What we need, though, is full support for importing and exporting FITS files. Because as much as Juan seems to dislike the idea, there are lots of other useful astronomical apps out there most of us make use of, and they work with FITS files, not XISF. And I really don't see that changing.