Boundary padding for records and PDUs

[ngjunsiang]

This is a placeholder issue for work relating to boundary padding.

Classes requiring boundary padding

To date, from a quick search in the codebase, the following classes require padding to bit boundaries (Note: not a complete list):

• IFFData
• GridAxisDescriptorVariable
• FundamentalOperationalData
• Environment
• IntercomSignalPdu
• SignalPdu
• MinefieldResponseNackPdu
• VariableDatum
• RecordSpecificationElement

There is some code in VariableDatum for doing this. Ideally the process for doing so should be standardised across all of the above classes. Some helper functions would also help in this area.

Issues affected

This might potentially resolve #12 (since DataPdu composes VariableDatum)

[ngjunsiang]

RecordSpecificationElement from RecordSpecification (from 6.2.73) in particular is tricky because the recordLength is variable and depends on the recordID. recordID is a 32-bit enumeration with lots of potential record types, and DIS7 does not mention any restriction of those record types to a specific subset! 😱

I have managed to narrow down the classes that use RecordSpecification to the following three classes:

• Record-R PDU
• Set Record-R PDU
• Transfer Ownership PDU

Record-R and Set Record-R PDUs do not mention any restriction of recordIDs, though from the purpose of the PDU it may be gathered that this is meant for setting simulation parameters (I am guessing enum values 45000-52530 in [UID 66]).

Transfer Ownership limits the possible records to the following:

• PDU Status record 6.2.67 8 bits
• Munition record 6.2.60 128 bits
• Munition Reload record 6.2.61 192 bits
• Engine Fuel record 6.2.24 64 bits
• Engine Fuel Reload record 6.2.25 160 bits
• Storage Fuel record 6.2.84 64 bits
• Storage Fuel Reload record 6.2.85 160 bits
• Expendable record 6.2.35 128 bits
• Expendable Reload record 6.2.36 192 bits
• Total Record Sets record 6.2.89 32 bits
• Launched Munition record 6.2.50 384 bits
• Association record 6.2.9 256 bits
• Sensor record 6.2.77 96 bits
• Ownership Status record 6.2.65 64 bits

[ngjunsiang]

Hopping back into this again. I looked at the record classes requiring padding again, and at the PDUs that use them. It looks like Transmitter PDU is one of the longest and most complicated ones, so I'll start here and hopefully build out some code infrastructure that will help in fixing the other classes.

Just putting my research here for easier reference.

[ngjunsiang]

From the spec:

7.7.2 Transmitter PDU

Table 175—Crypto Key ID record

Field name Bits Data type
Pseudo Crypto Key 0–14 15-bit unsigned integer
Crypto Mode 15 1-bit enumeration
Total Crypto Key ID record size = 16 bits

Table 176—Transmitter PDU

Field size (bits) Transmitter PDU fields
96 PDU Header

• Protocol Version—8-bit enumeration
• Exercise ID—8-bit unsigned integer
• PDU Type—8-bit enumeration = 25
• Protocol Family—8-bit enumeration = 4
• Timestamp—32-bit unsigned integer
• Length—16-bit unsigned integer
• PDU Status—8-bit record
• Padding—8 bits unused

48 Radio Reference ID

• Site Number—16-bit unsigned integer
• Application Number—16-bit unsigned integer
• Reference Number—16-bit unsigned integer

16 Radio Number 16-bit unsigned integer
64 Radio Type 64-bit record
8 Transmit State 8-bit enumeration
8 Input Source 8-bit enumeration
16 Number of Variable Transmitter Parameters Records (N) 16-bit unsigned integer
192 Antenna Location

• X-component—64-bit floating point
• Y-component—64-bit floating point
• Z-component—64-bit floating point

96 Relative Antenna Location

• x-component—32-bit floating point
• y-component—32-bit floating point
• z-component—32-bit floating point

16 Antenna Pattern Type 16-bit enumeration
16 Antenna Pattern Length (A) 16-bit unsigned integer
64 Frequency 64-bit unsigned integer
32 Transmit Frequency Bandwidth 32-bit floating point
32 Power 32-bit floating point
64 Modulation Type

• Spread spectrum—16-bit record
• Major Modulation—16-bit enumeration
• Detail—16-bit enumeration
• Radio System—16-bit enumeration

16 Crypto System 16-bit enumeration
16 Crypto Key ID 16-bit record
8 Length of Modulation Parameters (M) 8-bit unsigned integer
8 Padding 8 bits unused
16 Padding 16 bits unused

Parameter records section
8M Modulation Parameters Modulation Parameters record—M octets
8A Antenna Pattern Antenna Pattern record—A octets
48 + 8K_1 + 8P_1 Variable Transmitter Parameters record #1

• Record Type—32-bit enumeration
• Record Length—16-bit unsigned integer (6 + K1 + P1)
• Record-Specific fields—K1 octets
• Padding to 64-bit boundary—P1 octets

• • •

48 + 8K_N + 8P_N Variable Transmitter Parameters record #N

• Record Type—32-bit enumeration
• Record Length—16-bit unsigned integer (6 + KN + PN)
• Record-Specific fields—KN octets
• Padding to 64-bit boundary—PN octets

Total Transmitter PDU size = 832 + 8M + 8A + 8 sum[N, i = 1](6 + K_i + P_i) bits

6.2.8 Antenna Pattern record

Table 31—Beam Antenna Pattern record

Field size (bits) Field name Data type
96 Beam Direction

• Psi (ψ)—32-bit floating point
• Theta (θ)—32-bit floating point
• Phi (φ)—32-bit floating point
  32 Azimuth Beamwidth 32-bit floating point
  32 Elevation Beamwidth 32-bit floating point
  8 Reference System 8-bit enumeration
  8 Padding 8 bits unused
  16 Padding 16 bits unused
  32 EZ 32-bit floating point
  32 EX 32-bit floating point
  32 Phase 32-bit floating point
  32 Padding 32 bits unused
  Total Beam Antenna Pattern record size = 320 bits

6.2.59 Modulation Type record

Table 90—Spread spectrum field definition

Field Name Bit Value
Frequency Hopping 0 Enumeration
Pseudo Noise 1 Enumeration
Time Hopping 2 Enumeration
Padding 3–15 13 bits unused
Total Spread Spectrum field size = 16 bits

Table 91—Modulation Type record

Field size (bits) Field name Data type
16 Spread Spectrum 16-bit record
16 Major Modulation 16-bit enumeration
16 Detail 16-bit enumeration
16 Radio System 16-bit enumeration
Total Modulation Type record size = 64 bits

6.2.95 Variable Transmitter Parameters record

Table 130—Variable Transmitter Parameters record

Field size (bits) Field name Data type
32 Record Type 32-bit enumeration
16 Record Length 16-bit unsigned integer (6 + K + P)
8K Record-Specific fields K octets
8P Padding Padding to 64-bit boundary—P octets
Total Variable Transmitter Parameters record size = 48 + 8K + 8P bits

Annex C

Table C.3—Basic HAVE QUICK MP record

Field size (bits) Field name Data type
16 NET ID record 16-bit record
16 MWOD Index 16-bit unsigned integer
16 Reserved 16 bits reserved
8 Reserved 8 bits reserved
8 Reserved 8 bits reserved
32 Time of Day 32-bit unsigned integer
32 Padding 32 bits unused
Total Basic HAVE QUICK MP record size = 128 bits

Table C.4—High-Fidelity HAVE QUICK VTP record

Field size (bits) Field name Data type
32 Record Type = 3000 32-bit enumeration
16 Record Length = 40 16-bit unsigned integer
16 Padding 16 bits unused
16 NET ID record 16-bit record
8 TOD Transmit Indicator 8-bit enumeration
8 Padding 8 bits unused
32 TOD Delta 32-bit signed integer
32 WOD 1 32-bit unsigned integer
32 WOD 2 32-bit unsigned integer
32 WOD 3 32-bit unsigned integer
32 WOD 4 32-bit unsigned integer
32 WOD 5 32-bit unsigned integer
32 WOD 6 32-bit unsigned integer
Total High-Fidelity HAVE QUICK VTP record size = 320 bits

Table C.5—NET ID record

Field Bits Value
Net Number 0 to 9 Unsigned integer
Frequency Table 10 to 11 Enumeration
Mode 12 to 13 Enumeration
Padding 14 to 15 2 bits unused
Total NET ID record size = 16 bits

[ngjunsiang]

Handling Record Types

The tricky part here:

1. Variable Transmitter Parameters record (VTPR) uses the Record Type to determine what the Record-Specific fields are.
2. This means VTPR will have to:
   • read an enum32 value for recordType
   • read a uint16 for record length
   • based on the recordType (the spec only mentions the High-Fidelity HAVE QUICK VTP record as a possibility here), read or parse N bytes
   • based on the record length, determine how many padding bytes to discard (so that any records/PDUs that still need the inputStream after this can continue parsing)

This also implies we need some kind of mapping of the Variable Record Type enum values [UID 66] to the relevant classes.

[ngjunsiang]

Handling bitfields

NET ID Record poses another difficulty, which is echoed in many places in dis7.py: handling bitfields.

The canonical way of doing so in python is using the ctypes module, which enables defining bitfield classes as subclasses of ctypes.Structure. From the documentation:

>>> from ctypes import *
>>> class POINT(Structure):
...     _fields_ = [("x", c_int),
...                 ("y", c_int)]
...
>>> point = POINT(10, 20)
>>> print(point.x, point.y)
10 20
>>> point = POINT(y=5)
>>> print(point.x, point.y)
0 5
>>> POINT(1, 2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: too many initializers
>>>

Instantiating a ctypes.Structure from (buffered) bytes or a stream is tricky, because the only way to do so appears to be through the from_buffer() or from_buffer_copy() methods (which are inherited from ctypes._CData; the _copy() version does not modify the original buffer.

This means, for NET ID, we need to:

1. Read 16 bits / 2 bytes from the input stream as raw bytes
2. Instantiate a bitfield structure using the from_buffer_copy() class method (e.g. NetID.from_buffer_copy(data))
3. Then unpack the fields from NetID into a proper record class; I wouldn't recommend using the ctypes.Structure bitfield object directly unless one is clear from the documentation about how the class works (it is a wrapper around a buffer, and setting object attributes directly modified the underlying buffer)

The DataInputStream class currently has no way to read and return bytes from the stream, except using read_utf() which I am very reluctant to use for this purpose because of the unintuitive naming.

I will likely add a DataInputStream.read(n) method for reading raw bytes, similar to the BufferedIOBase.read() method already provided by python (in fact I will probably just pass the call to the underlying BufferedIOBase).

[ngjunsiang]

Namespacing

Looking at the number of classes in [UID 66] I am disheartened thinking about the number of new classes I will have to dump into dis7.py, which is already hitting 8k LOC. It is already difficult trying to differentiate the various kinds of records.

Very tempted to organise records into a separate module/file, with bitfields separated into their own submodule so that the dependency relation is clearer.

Something like:

dis7
+-- pdu
+-- record
    +-- bitfield

[ngjunsiang]

I will likely add a DataInputStream.read(n) method for reading raw bytes, similar to the BufferedIOBase.read() method already provided by python (in fact I will probably just pass the call to the underlying BufferedIOBase).

Back to this, it implies a call like inputStream.read_bytes(n) where n is the size of the bitfield in bytes. Which further implies a need for bitfields to store their own size (e.g. through an interface like #54).

Adding marshalledSize() to all record and PDU classes in an abstractable way is a huge undertaking and I won't do that here, but bitfields seem like a way to get this interface detail tested incrementally.

[ngjunsiang]

Options for parsing

2-step process using marshalling

Example:

raw = inputStream.read_bytes(NetId.marshalledSize())
record = NetId.from_bytes(raw)

Bitfield.parse(stream) class method

Example:

class NetId(ctypes.Structure):
    _fields_ = [...]

    @classmethod
    def parse(cls, stream):
        raw_bytes = stream.read(cls.marshalledSize())
        return cls.from_bytes(raw_bytes)

...
record = NetId.parse(inputStream)

The second option has one more layer of indirection, but is a simpler call. I might create a Bitfield(ctypes.Structure) base class for bitfields using this.

[ngjunsiang]

The second option has one more layer of indirection, but is a simpler call. I might create a Bitfield(ctypes.Structure) base class for bitfields using this.

A small hiccup: ctypes.Structure only allows integer non-octet fields. But some of the bitfields used by DIS are enums as well.

The fields of a ctypes.Structure are actually descriptors that wrap the underlying buffer. While enums are integer values, which should not pose a problem, I am thinking ahead to future integration work that will make open-dis-python easier to integrate with siso enumerations such as https://github.com/open-dis/dis-enumerations.

This means instead of having records subclass Structure, it might be be safer to use composition instead; bitfield records might hold a non-public reference to an underlying Structure that is used to unpack bytes into values, before the record class wraps those values in appropriate types.

A direct implication is that each bitfield record could require two class definitions: one for the record, and one for the ctypes.Structure subclass. I would prefer not to have such verbose code; I might define a bitfield factory function that creates Structure subclasses dynamically only for the record classes that require them.

[ngjunsiang]

Other bitfield classes required:

• 6.2.12 Beam Status record
• 6.2.13 (B.2.4) Change/Options record
• 6.2.16 Data Filter record
• 6.2.45 Information Layers record
• 6.2.57 Minefield Sensor Type record
• 6.2.59 Modulation Type record
  Table 90 Spread Spectrum field
• 6.2.67 PDU Status record
• 6.2.69 Protocol Mode

[ngjunsiang]

To date, from a quick search in the codebase, the following classes require padding to bit boundaries (Note: not a complete list):

• IFFData
• GridAxisDescriptorVariable
• FundamentalOperationalData
• Environment
• IntercomSignalPdu
• SignalPdu
• MinefieldResponseNackPdu
• VariableDatum
• RecordSpecificationElement

Variable Record types

[UID 66] also maps a few variable record types, with a common recordType and recordLength` field, and subsequent record-dependent attributes based on the specific record type. The ones that can be found in IEEE1278.1-2012 are:

Enum | Class Name

• 0 : ArticulatedPartVP (6.2.94.2)
• 1 : AttachedPartVP (6.2.94.3)
• 2 : SeparationVP (6.2.94.6)
• 3 : EntityTypeVP (6.2.94.5)
• 4 : EntityAssociationVP (6.2.94.4)
• 3000: High-FidelityHAVEQUICKVTP (C.4.2.3)
• 3500: BlankingSectorAttribute (6.2.21.1)
• 3501: AngleDeceptionAttribute (6.2.21.2)
• 3502: FalseTargetsAttribute (6.2.21.3)
• 4000: DEPrecisionAimpoint (6.2.20.3)
• 4001: DEAreaAimpoint (6.2.20.2)
• 4500: DirectedEnergyDamageDescription (6.2.15.2)
• 5000: CryptoControlIFFData (B.2.5)
• 5001: Mode5TransponderLocationIFFData (B.2.30)
• 5002: TransponderLocationErrorIFFData (B.2.54)
• 5003: SquitterAirbornePositionReportIFFData (B.2.47)
• 5004: SquitterAirborneVelocityReportIFFData (B.2.48)
• 5005: SquitterSurfacePositionReportIFFData (B.2.51)
• 5006: SquitterIdentificationReportIFFData (B.2.50)
• 5007: GICBIFFData (B.2.10)
• 5008: SquitterEvent-DrivenReportIFFData (B.2.49)
• 5009: AntennaLocationIFFData (B.2.2)
• 5010: BasicInteractiveIFFData (B.2.3)
• 5011: InteractiveMode4ReplyIFFData (B.2.13)
• 5012: InteractiveMode5ReplyIFFData (B.2.14)
• 5013: InteractiveBasicMode5IFFData (B.2.11)
• 5014: InteractiveBasicModeSIFFData (B.2.12)
• 5500: IOEffect (6.2.48.3)
• 5501: IOCommunicationsNode (6.2.48.2)