[iaufwg] convention or standard

[Lucio Chiappetti]

Miscellaneous answers ...

On Thu, 12 Oct 2006, William Pence wrote:

> > - the other source of confusion is that the second "convention" to be
> >   discussed (XTENSION='FOREIGN') is actually the definition of a new 
> >   conforming extension so it falls in both categories !
> 
> EXTENSION = 'FOREIGN' will be listed in both places.  I don't expect 
> there are many other extension types that need to be registered, but if 
> there are any, then they will also be entered in both places.

OK, so somebody should write a spec for A3DTABLE and IUEIMAGE too (or 
maybe they are too close to BINTABLE and IMAGE that we can give a waiver).

Also, provided there is a link between the extension registry page and the 
convention registry page, that could perhaps solve the problem of 
extension "status".

A registered (namespace reservation or documentation proposal only) 
extension won't have such link. An active (implemented) extension will 
have the link (since the convention registry is for things in actual use).


On Thu, 12 Oct 2006, William Pence wrote:

> The simplest solution with the least impact on existing software and 
> users would be to use the IMAGE extension for this purpose, with NAXIS = 
> 1 and NAXIS1 equal to the size of the byte stream.

On Fri, 13 Oct 2006, Mark Calabretta wrote:

> I would argue that way, but wouldn't it be more natural to put the byte 
> stream in a 1-row, 1-column BINTABLE as a byte-array (or even 
> bit-array).

I would regard both as unelegant workarounds. But I could accept if 
somebody devises a local sub-convention for that ...

I mean, if somebody sends me a file with an IMAGE or BINTABLE extension 
wrapping a PNG or a PDF or whatever *AND* the extension header says 
nothing, I would use ds9 or fv to look at it, probably not with great 
results.

But if a convention on some keywords decides that an IMAGE or BINTABLE 
extension marked in the header with e.g. WRAPPED=TRUE and some other 
keywords is *the* way to attach a preview or a documentation or whatever 
to a FITS file, that's legitimate.

Anyhow how would such a FITS file fit with the FITS MIME types 
(application/fits and image/fits) we have registered recently ? What will 
a MIME reader do with a FITS file which wraps an arbitrary byte stream ?

Back to Bill ...

> If it were decided that IMAGE is not suitable for this purpose, then 
> perhaps we should consider designing a new standard extension type just 
> for storing a stream of bytes, so other groups would be less tempted to 
> invent their own new type of extension for each individual case.

Possibly a new extension would be more elegant (XTENSION='ARBITRARY' ?
or maybe even XTENSION='MIME' see below ?).

I must admit that the FOREIGN extension, despite some portability issues 
in the file naming, some overshooting on directory structure, and the fact 
that Doug Tody would design it differently now :-), has some added value 
with respect to a plain byte-stream encapsulation ... for instance it 
discriminates "text" files (an OS could do conversion of the record 
terminator as the various ftp implementation do for "ascii" transfer) from 
"binary" files.

Since we do not need to reinvent the wheel, and have MIME, I wonder if the 
way to wrap an arbitrary file would be to have something like :

XTENSION='MIME'
MIME0001='Content-Type: text/plain'
MIME0002='Content-Encoding: whatever'
MIME0003='Content-Disposition: whatever'
...

I'm not really a MIME expert, no more than seeing some mail attachments or 
web documents. But if the idea is to put MIME headers in up to 9999 header 
keywords, plus the obligation to support at least two MIME types, 
text/plain and the catch-all application/octet-stream or whatever it is 
called, maybe we are done ... that should also deal with most compression 
encodings...

On Thu, 12 Oct 2006, Don Wells wrote:
 
> A historical note
  
> well as the obvious interchange problem.  In the 1980s there arose a 
> proposal that telemetry formats should be standardized by agreement of 
> the various space agencies of the World.

History for history, that pre-dates the telemetry standardization I've 
followed (BeppoSAX then XMM-Newton) using the "ESA Packet Telemetry 
standard" which should be a reception of the CCSDS inter-agency standard 
(CCSDS being one of those organization having acronyms which expand in the 
same way in English and French ... having not followed it for some years I 
discover now it even has a web site http://public.ccsds.org/default.aspx)

> memory is that I also added a joking remark to my response to the SFDU 
> proposal.  My memory is that I said something like this: "Yes, you 
> (SFDU) can wrap up our data format, but we can declare XTENSION='SFDU' 
> and wrap you up too!"  :-)

Of course telemetry had other needs (ad hoc compression of data streams 
for instance) than data analysis. Anyhow I've seen some earlier proposals 
by ESA in which they wanted to store the XMM data stream into FITS in some 
odd BINTABLE format matching I can't remember whether telemetry frames or 
CCD frames. Luckily we manage to deflect it into a more standard event 
file format ...

... but yes, if MIME can wrap us, we can wrap MIME too !

----------------------------------------------------------------------------
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
----------------------------------------------------------------------------