CTX - Creativyst^® Table Exchange Format

CTX is a more precisely defined and functional alternative to CSV, and a lower overhead alternative to many applications of XML. The CTX exchange format embodies the simplicity of CSV, while permitting, via optional secondary mechanisms, the exchange of data with complex structural hierarchy.

CTX achieves low overhead by permitting data-types to be tagged with type, display, structure, help, and other information. By tagging types instead of data, CTX vastly improves bandwidth-usage over the strategy of tagging every value-instance with two identical nametags. This also permits rich meta-information to be shared about the data being exchanged. The added metadata is optional for writers, and, if present, may be used or ignored at the option of reading applications.

1. Simple - Low complexity entry level, similar to the simplicity of writing and reading a CSV file. The exchange format WILL be easy to understand and implement. It will be specifically designed for the sharing and exchange of flat, relational, or hierarchical tabular data stores.

2. Small - The exchange format WILL take bandwidth costs and restrictions seriously and WILL strive to have relatively low overhead.

3. Field-opaque - The core exchange format itself WILL NOT be required to be concerned with what is represented in or by the data fields it carries.

4. Binary or Text - The transport mechanism WILL BE expected to be able to transport all 256 octet codes by default.

5. Durable - The exchange format writer MAY optionally provide special-case (non-default) encoding extensions to respond to any transport mechanism that is expected to be limited to text-only. CTX readers MUST read and translate such extensions if present in the CTX data.

6. Robust -
   1. Able optionally to exchange meta field information such as labels, names, types, and comments with reading applications.

   2. Able optionally to exchange meta table information such as labels, names, types, and comments with reading applications.

   3. Reading applications may choose to use or ignore as much provided meta-data as they wish

   4. Able optionally to exchange multiple tables, even multiple databases in a single transmission, file, or chunk.

   5. Able optionally to represent hierarchy within data by providing facilities to embed a complete CTX encoded file, record, table, or database into single fields.

7. Orthogonal - The format definition WILL be expressed as a read-function state table in order to be relatively precisely defined, and have predictable attributes and characteristics.

Escape sequences are classified by their function and width:

• Single byte replacement/function sequences:
  An introducer (escape character) followed by a single byte (character) denoting its function or the byte code it is standing in for.

• Optional first-character record-type/function sequences:
  A single-byte escape sequence that is only legal if it appears starting at the first byte position of a record. It denotes that the record that follows is a special function and is not part of plain exchange data.

• Optional multi-byte replacement sequences:
  A multi-byte introducer ('\m'), followed by an optional number, followed by a single byte denoting function, followed by a parameter or set of parameters and terminated by a semi-colon (';');

• Multi-byte terminator replacement sequence:
  A special single-byte replacement sequence that is only legal when it appears inside a multi-byte replacement sequence. Specifically, it is the '\s' sequence, which is replaced with a semi-colon in raw data.

All escape sequences are introduced by a backslash ('\'), which is called an escape sequence introducer. If a file contains a backslash followed by a character that is not defined here, it is an error for this version of CTX.

Case matters. For example, a '\P' (primary type record follows) is not the same as a '\p' (pipe character escape). Also, a '\T' is a legal code at the first character position of a record, while a '\t' is always illegal.

. . . . .
Single byte replacement/function characters.

CR(0x0d), LF (0x0a), '\' (0x5c), and '|' (0x7c) within field data MUST be replaced with \r, \n, \i, and \p sequences respectively. Also, when ';' (0x3b) exists within a multi-byte replacement sequence, it MUST be replaced with \s.

Escape sequences other than those defined here may not appear in CTX encoded data.

. . . . .
Optional first character record-type sequences

First character record-type sequences are only valid when they start in the first character position of a record (the first character of a line, or the first character following CR or LF).

They denote that the record on the line is not data, but a special type that performs a CTX function. These records generally apply to the data records that follow them up to the end of the file, or to the next record-type record of the same type.

. . . . .
Optional multi-byte sequences:

Multi-byte sequences MAY be used by a CTX WRITER to translate difficult byte codes to more transportable representations. This MAY be used to transport a table or set of tables over an ASCII-only email link, for example.

Multi-byte sequences are completely optional for WRITERS, but READERS are REQUIRED to handle them and translate them.

They may be used by a CTX WRITER to accommodate channels that are known to be text-only (non-binary). A binary channel is assumed by default.

The term “header” is a bit of a misnomer when applied to these field type records (everything but \T and \G). They apply to the records within their group, regardless of where they first appear within the group.

Aside from allowing type records to apply to all the fields in their group without regard for their sequential location within the group, this also provides for the use of redundancy in transmissions. This can be used to ensure that types and other field directives are properly applied when long transmissions are broken into multiple chunks (due to time or medium constraints, or transmission errors, for example).

Important considerations regarding backtracking feature:

• Backtracking MUST only apply to the first occurrence of a given header type within a group. — If another header record of the same type occurred earlier in the group, the directives from the earlier occurrence apply to all the fields after it, up to the new header type. Only the first occurrence of a given header type has backtracking applied.

• If used for redundancy, the type headers should be retransmitted at the end of each record group — This will assure the headers are all applied to any remaining preceding records in the event of a transmission interruption near the end of the group.

• Group delimiters are \T, \G, or the beginning or end of the file or transmission — The end of the file is also considered a group delimiter. It may or may not be germane to this behavior. A transmission mode that tracks the last fully committed record to be received may also maintain the headers, or it may expect the headers to be retransmitted by the sender.

Those who've cut their teeth on Object Oriented Programming concepts may find state tables a bit confusing. The key to understanding state tables from an OOP perspective is to understand first, that they take the opposite extreme of the OOP (and functional-programming for that matter) "abstraction through complexity hiding" strategy.

Instead of hiding lesser-used or normally unused constructs, state tables' strength comes from equally disclosing, in a visually graspable way, all trajectories through a problem space. That includes trajectories that would be considered abnormal, unusual, or erroneous.

For many problem types, state table representations are the best choice for bridging specification to finished software code. Use of state tables permits a thorough visualization and understanding of the problem domain that is often not possible with more traditional methods. More importantly, you can code directly off of (from) these state tables. :-)

• READERS:
  • must implement the state tables and bold functions shown below.
  • Records that have fewer fields than expected may be handled by the READER in any of two ways:
    1. Remaining fields may be set to NULL
    2. Remaining fields may be set to blank or zero
  • Records that have more fields than expected may be handled by the READER in any of two ways:
    1. Excess fields may be discarded
    2. An ERROR may be raised.
  • A post-reader process SHOULD also remove or translate any field and byte-values that will cause difficulty to the application being serviced by the READER.

• WRITERS:
  • All CTX tables encoded by WRITER functions must be readable by the Reader represented in these state tables without ERROR, and translated back to the data encode by the writer.
  • The '\l' and '\m...;' functions are optional for WRITERS Both of these functions may be performed anywhere in the route.

Also, in these state tables, i've included the '\l' function directly in the processing stream. Processing of the '\l' function MAY be done in a separate pass prior to performing the main CTX READER translations. In fact, '\l' handling may be performed upstream and in a different process altogether from the final reader. I handled them in a single pass at the reader:

1. to show the entire READER, state table;
2. to produce a READER with slightly better performance and,
3. to provide a more generic READER capable of translating streams as each byte (or 2-byte escape sequence) is received. This is more generic than a two-pass representation because it is capable of handling complete files (like the upstream strategy) as well.

Function mCvrt(Type, N, tmp)
{
    1. if(Type == 'x') {
            ConvertHexToBytes(tmp)
        }
        if(Type == 'b') {
            ConvertBase64ToBytes(tmp)
        }
    2. if(Error during conversion) {
            return(Convert Error)
        }
    3. copy conversion output into OutputBuffer N times
    4. return OutputBuffer
}

Function getFuncRec(F)
{
    1. if(RecursionCheckAlreadyHere()) {
            ERROR
        }
    2. FRec = $ReadRecord();  // call state machine
    3. if(F == '\T') {
            PerformAnyTableRelatedProcessing();
        }
    4. if(F == '\G') {
            PerformAnyGroupRelatedProcessing();
        }

    5. if(IgnoreFuncRec(F)) {
            return
        }
    6. Store FRec
    7. return
}


/* . . . . .
 * These just give an example of some
 * of the kinds of processing you might
 * want to do for table/group boundaries.
 *
 * Of course, you could just ignore
 * the record and return.
 *
*/
Function PerformAnyTableRelatedProcessing()
{
    1. if(!DoTableProcessing) {
            return;
       }
    2. Store Previous Table.
    3. Allocate new table
    4. OtherProcessing()
    5. return; 
}

/* Likewise for Group (Database) records
*/
Function PerformAnyGroupRelatedProcessing()
{
    1. if(!DoGroupProcessing) {
            return;
       }
    2. Store Previous DataBase.
    3. Allocate new DataBase
    4. OtherProcessing()
    5. return; 
}

. . . . .
Basic Types ('N' and 'B')

Aside from the CTX/hierarchy type ('C'), CTX has only two explicit, elemental, primary field types. They are:

• Numbers ('N')
  The Number format is also called 'fixed', because its allowable representations are defined as the output formats specified by the standard C library's fprintf() function when using the format specifiers 'f', 'i', 'x', 'e', or 'd'.

• Non-numbers ('B')
  The Non-number format is also called 'Binary' format, because it can (technically) hold any set of byte values. It MAY be interpreted as "anything but ASCII represented numbers".

. . . . .
The Hierarchy Field Type ('C')

CTX includes a special primary type ('C') that lets you embed CTX encoded records, tables, and even entire databases directly within individual fields of the current-level CTX file.

In my opinion (and it is just my opinion) there is a considerable amount of data in the world for which the type-level hierarchical partitioning supported by this mechanism may be better suited than the value-level hierarchical partitioning specified in XML.

• 'C' The field contains a CTX file
  The relative field holds an entire embedded CTX file. This simply means there is embedded CTX encoded information in the field. The CTX data could be as simple as a single CTX record, or as complex as an entire SET of databases, each with multiple tables, its own type, help, and label identifiers, and anything else that COULD be included within a CTX file.

. . . . .
The Specifically Ambiguous Type ('')

If a \P record is defined but is blank it is specified as ambiguous, a sort of deliberate 'don't care' condition. This is true for trailing fields in records that contain more fields than their respective \P record. Fields in tables for which no \P record is transmitted may also be considered specifically ambiguous.

. . . . .
More About The \P record

1. Each field in the \P record corresponds to the respective field in each data record.

2. A \P record field may contain 'N' for Number', 'B', for 'Binary', or '' for Specifically Ambiguous (which essentially means 'don't care' or 'undefined'), as well as 'C' for CTX encoded data (hierarchy).

3. If a \P record field contains 'N', the respective field MUST be represented as a Number field in the transmission or file.

4. For situations where a numeric field must be interpreted by end applications as a string (such as U.S. zip codes, where leading zeros may have meaning) the field should be defined as 'B' (Binary, non-number)

5. Leaving a particular \P record field blank or not providing a \P record at all MAY be considered ambiguous. In such cases a-priori agreements on how data is interpreted is outside of the scope of this convention.

. . . . .
Implementation Notes

The following box contains some notes about optional primary types that may be important to implementers of CTX writers, readers, as well as applications that employ CTX.

Writer:
    N   Numbers or 'fixed' fields
        Numerically typed fields, if used, MUST be represented in ASCII
        digits and punctuation.  If numeric application types are
        represented in ASCII they MUST be formatted according to the
        format specifications for the standard C library fprintf() function
        for format codes 'f', 'i', 'x', 'e', or 'd' ("fixed").
        No other ASCII formats are permitted.  Leading zeros, and
        leading and trailing spaces may be included only as recommended
        for the standard C printf() function format specifications for
        fixed formatted numbers.

        If there is a \P field definition of 'N', a numeric field must be
        written as a number type as described above.

        Native types SHOULD be exchanged as 'fixed' types, even if primary
        types are not used.


    B   Non-number or 'B'inary fields
        Non-number or Binary fields should be translated byte-for-byte
        into their escaped values for the CTX field.  Each byte in the
        source field must be represented in the CTX field in a way that
        will be translated exactly back into the source byte by the reader.

        The writer MAY choose to escape the bytes of the field so that it
        can be safely transported over restricted channels (such as ASCII
        only email).  The writer may use \mx..;, \mb...;, and \l functions
        to perform this.  The \l function must be applied (effectively) 
        after all other translations have been performed.



Reader:
    See the state-tables.


Application:

    1. If a \P record exists a reading application MAY choose to
       use it or ignore it.

    2. If a \P record exists and the respective \P record field
       value is 'N' a reader application MAY choose (as per 1.) to
       interpret the data-record field as a number in ASCII 
       representation, and convert it into a native numeric
       representation for the application or platform the
       reader is serving.

    3. If a \P record exists and the respective \P record
       field value is 'B' a reader MAY choose (see 1.) to interpret
       the data-record field as a string, even if it contains properly
       formatted ASCII representations of numbers.  This can be useful
       (for example) for representing numbers where leading zeros
       have meaning and should be preserved (such as U.S. zip codes).

. . . . .
A Faux Database

---------           ---------
Persons             Pets
---------           ---------             
Number(P)           Number(P)
LastName            Owner(F:Persons.Number)
FirstName           Name                  
                    Species               
                    Breed                 
. . .
Persons data:
----------------------------------------
Number  LastName            FirstName
----------------------------------------
1       Smythe              Jane
2       Doe                 John
3       Mellonhead          Creg

. . .
Pets data:
---------------------------------------------------------------
Number  Owner   Name        Species         Breed
---------------------------------------------------------------
1       1       Fluffy      Dog             Poodle
2       1       Sharp       Dog             German Shepard
3       2       Silo        Cat             Mix
4       3       Doggie      Dog             Mix

. . . . .
Escence

At its simplest, CTX will transmit just the data records of a given table much like CSV. For example, to share the data from the "Persons" table we could simply send

This is a complete and correct CTX transmission, as are all the following CTX examples.

. . . . .
Field Labels and Names

In CSV the header (column names) are often sent as the first record in the file. Problems arise, however, because reading applications have no way of knowing if the first record is a header row, or the first data record.

CTX solves this problem with special record types that start with '\L' (Labels), and '\N' (Names). Labels should consist of only characters from the set: [A-Za-z0-9_].

There's no reason why you can't WRITE the labels as field NAMES as well, you can even make an '\L' and '\N' record that are exactly the same in every other respect. But, you may want to use the less restricted \Name record to convey slightly more readable (but still short) names for the fields.

CTX provides other meta-record types to describe field Comments ('\R'), display sizes and alignments ('\D'), even fly-over help ('\H') among others.

. . . . .
Field Types

CTX also provides meta-record types to define field types. One example might be to use the SQL-Type record ('\Q') to impart field types for our Persons table.

Other meta-records that impart field TYPE information include application specific type names ('\Y'), primary CTX types ('\P'), and key indicators ('\K').

. . . . .
Tables

Want to convey some information about the table you're sending? Use the '\T' (Table) record type.

. . . . .
Multiple Tables

You can also use '\T' records alone to delimit a group of multiple tables.

Here, our entire database has been sent in a single CTX file.

Note also that this continues to be a legal CTX file. The extra blank line we've put between tables for clarity here is perfectly acceptable in a CTX file because blank lines are ignored. This same behavior means CTX implementations need not be concerned with platform-dependent differences between line endings (CR, LF, CRLF, LFCR).

. . . . .
Databases

You can also add a '\G' (Group) record to impart information about a group of multiple tables (a database).

. . . . .
Multiple Databases

The '\G' records in CTX permit you to send multiple databases in a single CTX file in the same way '\T' records permit multiple tables. Just delimit each database with a '\G' record...

. . . . . . . . . . .