Regular Broadcasting of PACSAT File Headers DRAFT 04 November 1991 [All changes and additions in this DRAFT are PROPOSED, and should not be considered final until a subsequent RELEASE version of this document is published.] JW Ward, G0/K8KA HE Price, NK6K DISCUSSION In the past 18 months the PACSAT Broadcast Protocol has proven to be a very efficient means of downloading information from the LEO store-and-forward satellites. It makes good use of the available bandwidth, and especially efficient use of available pass time. The authors always intended to extend this datagram facility to the provision of a "file directory" for the PACSAT server. Since all stations need an up to date directory, this is a prime candidate for point-to-multipoint distribution. A PACSAT file server's "directory entries" are, in fact, PACSAT File Headers (PFHs). The PFH contains all of the important information about a file (name, number, upload time, title, addresses, etc.). This information is present in the PFH form on the server, and routines for interpretation of the PFH are present in the client software. The PFH is the natural data structure in which to present "broadcast directories." The broadcast directory service will, therefore, consist of periodic transmission of PFHs in special Directory Broadcast datagrams. The transmissions will be either automatic or client initiated. This protocol (see Appendix A) specifies the format for Directory Broadcasts and for Directory Broadcast Requests. The protocol does not specify when a server might periodically transmit PFHs without client requests, but such automatic directory broadcasts will be implemented on the existing PACSAT servers. CHOOSING A HANDLE Having decided that a "directory entry" is a PFH and that "directory broadcasting" is PFH broadcasting, we must address the problem of how the client can assure that it has received a complete and up-to-date directory from the server. Although the server uniquely identifies each file with an identification number, this is not necessarily the best way to for the client to request directory entries from the server. A system based on file numbers will not always assure that the client receives directories for all "new" files. The following cases illustrate this: (1) Partial Uploads When a client begins uploading a file, the server assigns a file number. The client may not finish uploading the file for some time. Only when the file upload is finished is the file a "new" file. (2) Non-existent Files Some file numbers are used only briefly (e.g. for temporary files) and may never be associated with a "new" file (as seen by the clients). (3) Non-PACSAT Files Files created by on-board processes, but lacking the PFH. They cannot be downloaded or uploaded. As far as the PACSAT protocol suite is concerned, these are like (2). (4) Daily Log Files The server and its processes also maintain daily logs which, although they may become "new" several times during a day, will cease to grow after midnight. (5) Continuous Log Files The server and processes on the server maintain log files. These files become "new" every time something is written into them. During the design of the directory broadcast protocol, we attempted to solve these problems using simple measures which would still allow file number to be the primary directory request handle. Problems persisted both in the complete specification of the protocol and in a prototype implementation of the groundstation receiving software. Eventually the file-number-based scheme was abandoned in favour of a slightly more complex protocol providing two "time limits" with each broadcast directory entry: an "older limit" , t_old, and a "newer limit", t_new. Thus, as well as containing a PFH, each broadcast directory entry gives the client a pair numbers (t_old, t_new). The proper interpretation of this pair is "There are no files other than this file with UPLOAD_TIME >= t_old and <= t_new." The client maintains a list of holes in its directory time line using the t_old and t_new information. Requests to the server for hole fills use these times, rather than file numbers. EXAMPLES [N.B.: Small times have been used deliberately in the examples; real satellite times are the number of seconds elapsed since 00:00:00 1/1/80. Also, the satellite file system assures that no two files will ever have the same UPLOAD_TIME.] For example, a client with no previous directory information assumes that it has a directory hole from 0 - FOREVER Upon receiving a directory broadcast with (t_old, t_new) (0, 50) the client's directory hole list would look like 51 - FOREVER and the client software is confident that there are no older files. The list can get fragmented, e.g. if the client now receives (120, 150) and the time hole list looks like: 51 - 119 151 - FOREVER Requests from the client to the sever are time hole lists. In effect, they request that the server transmit any directory entries in the hole OR an entry which shows that there are no directories in the hole. FLEETING HOLES The server sets time limits for a particular directory broadcast by examining the next newer and previous older files on the system. These files, however, may move or be deleted, hence the need to show that a hole no longer holds any directories. Starting from the previous example, say that the server creates files at t=151, t=152 and t=153, thus it would transmit [a] (120, 151) (PFH for t=151) [b] (152, 152) (PFH for t=152) [c] (153, 153) (PFH for t=153) A client receiving [a] and [c], but missing [b] would have a hole 152 - 152 in its directory time line, and might assume that there was a file to fill that hole. But the server might delete the file t=152. Thus, when the client requests a fill for (152,152) the server would respond with [c'] (152, 153) (PFH for t=153) which allows the client to fill its hole, but does not contain a directory entry for t=152. MARKING THE LIMITING FILES It is convenient for the client to know when it has received the directory for the current newest file. The server will mark this file with a special flag. The oldest file will always have t_old==0. BACKWARD COMPATIBLE IMPLEMENTATION Unfortunately, adding new data and request frame formats to the existing broadcast protocol in a backward compatible manner required some compromises. On inspection of the existing FRAME_HEADER for broadcast data packets, there is no clear indication of frame type! Strictly speaking, the reserved bits of the FRAME_HEADER.flags byte could be used to indicate that the frame was a directory broadcast frame. Unfortunately, PB at least does not check these bits, and any new directory broadcasts would terminally confuse PB. We expect that clients will try to get updated broadcast receiver software, but we don't want to force the pace of this change by causing the current implementation to crash. We completely avoid this problem, by specifying a separate PID for directory broadcast and request frames. This should allow existing groundstation software to co-exist with the new broadcast directories. IMPLEMENTATION NOTES A single implementation of PACSAT Protocol Suite software runs on UO-14, AO- 16, LO-19 and UO-22. Although there are minor variations from satellite to satellite, these are generally related to ongoing upgrades rippling through the system. There are some implementation peculiarities, and some ad-hoc additions have been made to the standard broadcast protocol. These matters are documented here, but are not added to the formal specification. Queue Management ---------------- The server maintains a queue with 10 entries; each entry is a hole list request or broadcast start request. A particular station (as identified by callsign, not including SSID) can have only one entry on the queue. Entries are removed from the queue: after 10 minutes; after a hole list has been completely transmitted (for hole list); after a file has been completely transmitted (for start request); when a new request is received from a station already in the queue; if the file associated with the entry cannot be opened and read; Broadcast Request Response Messages ----------------------------------- When the server receives a request frame from a client, it will respond immediately indicating the success or failure of the request. This response is intended for use in manually operated stations; because of timing and reliability problems, its use in automated broadcast request systems is discouraged. The request response message is an AX.25 UI frame from the server callsign to the client callsign with the PID 0xf0. [NB - this is a change of the PID.] The data portion of the packet has no header or CRC. The data is in ASCII as follows: <"OK "|"NO ">[] "OK " indicates that the request was successful. There is no . "NO " indicates that the request failed for some reason. In this case, the is an ascii representation of an error number followed by one blank space, e.g. "NO -1 ". The error codes for broadcast start requests and hole list requests should be interpreted as follows: -1 Temporary problem, e.g. queue full or server closed -2 Permanent problem, e.g. file does not exist. Broadcast Server Status Message ------------------------------- The format of the PBLIST message will be altered to include (at the option of spacecraft controllers) the number of the file being broadcast. The format of the PBLIST message will now be one of the following - If file numbers are disabled: PB: ... PB: \P ... PB: \D ... If file numbers are enabled: PB: \ ... PB: \P\ ... PB: \D ... \P indicates that the request is permanent \D indicates that the request is a directory request is the callsign of the requesting station, without SSID is the hexadecimal file number of the requested file If the queue is empty, the message will be : PB: Empty. These messages will be transmitted to PBLIST-0 with PID 0xF0. File Types ---------- There is still some confusion regarding compressed files. The intent was to place all compression information in the compression_type field and all file format information in the file_type field. Thus, as per the FTL0 update released 22 March 1991: Delete file type 0x0a, "Simple ASCII text file, but compressed" from Appendix A, PACSAT FILE TYPES. This type of file should be indicated by file_type 0x00 and a non-zero compression_type item. We propose to fix the conflict between "miscelaneous binary" and "activity log" by changing the activity log file type. Since only Harold and I have to take action to implement this solution, it is more likely to succeed than trying to recall every copy of PFHADD. Suggestions regarding additional, more-informative, file types are solicited. Compression Types ----------------- The PFH protocol was designed to be flexible. We should now add a compression type for the popular LHARC program. Add to APPENDIX B of the document "PACSAT File Header Definition" 0x03 body compressed using LHARC. -------------------------------------------------------------- -------------------------------------------------------------- BROADCAST DIRECTORY PROTOCOL PROPOSED ADDITIONS TO APPENDIX A OF THE PACSAT BROADCAST PROTOCOL SPECIFICATION -------------------------------------------------------------- PACSAT BROADCAST PROTOCOL ADDENDUM - PFH BROADCASTS A.5 Directory Broadcasts The server will transmit directory information automatically and/or as the result of Directory Requests from clients. The directory information provided is the PACSAT File Header (PFH). For a particular file, the directory entry will be transmitted in one or more Directory Broadcast frames, each of which is carried in the data portion of an AX.25 UI frame. Each frame will also contain the DIR_HEADER described below. To aid the client in maintaining its directory database and requesting up-to- date directory entries, Directory Broadcast frames include time limits. These limits allow the client to maintain a "time line" of directory entries and to determine when the time line is complete and up-to-date. Servers will periodically transmit the directories for the most recently uploaded files. Other directory entries can be requested by the client using a Directory Request frame. The Directory Request format allows clients to request that the server transmit directories for all files with UPLOAD_TIME in a particular range. This is equivalent to filling holes in the directory time line. A.5.1 Directory Broadcast frame format A directory entry is transmitted as one or more directory broadcast frames. Directory broadcast frames consist of header, data and CRC. The header is fixed length, and the data is variable length. A directory broadcast frame is wholly contained in the data portion of an AX.25 UI frame. A frame containing a directory broadcast frame uses a PID of 0xbd. A frame containing a directory broadcast frame uses a source address of the transmitting station, and a destination address of QST-1. A.5.1.1 Frame Header Each directory broadcast frame consists of a header occupying the first 17 bytes of the frame. The structure of the frame header is as follows: This structure follows the PACSAT Data Specification Standards as regards field size and byte order; these are defined in a separate document. struct DIR_HEADER { unsigned char flags; unsigned long file_id; unsigned long offset; time_t t_old; time_t t_new; } flags A bit field as follows: 7 6 5 4 3 2 1 0 /----------------------\ |* N E 0 V V T T| \----------------------/ TT Two bit frame type identifier 00 PFH broadcast 01 reserved 10 reserved 11 reserved VV Two bit version identifier. This version is 00. 0 Always 0 indicates a server generated frame. E 1 Last byte of frame is the last byte of the directory PFH. 0 Not the last frame. N 1 This is the newest file on the server. 0 This is not the newest file on the server. * Reserved, always 0. file_id A number which identifies the file. All directory broadcast frames which are part of the same file's PFH are tagged with this number. offset This is the offset from the start of the PFH for the first data byte in this frame. t_old Number of seconds since 00:00:00 1/1/80. See below. t_new Number of seconds since 00:00:00 1/1/80. See below. There are no files other than the file identified by file_id with t_old <= UPLOAD_TIME <= t_new. A.5.1.2 Data The data portion of a directory broadcast frame will contain all or part of the PACSAT File header from the file identified by . The field indicates where the data from the current frame belongs in the PFH. An of 0 and a field with the E bit set to 1 indicates that this directory broadcast frame contains the entire PFH for the identified file. A.5.1.3 CRC The CRC covers the header and data portions of the directory broadcast frame. It is the XMODEM CRC as defined in section A.2.1.3. A.5.2 Directory Broadcast Requests Clients may request that the server transmit directory broadcasts for files with UPLOAD_TIME within a certain range. If entries from several ranges are required, multiple ranges can be included in a single request. A.5.2.1 Request Frame A directory broadcast request frame consists of header and data. The header is fixed length, and the data portion is variable length. The directory broadcast request frame is wholly contained within a standard AX.25 frame. The total length of the request frame may not exceed the length of a legal AX.25 frame. A frame containing a request frame uses the same PID as a directory broadcast frame: 0xbd. The source address is the address of the transmitting station, the destination address is the address of the server to which the request is directed. A.5.2.2 Request Header The format of the directory broadcast request header is: struct DIR_REQ_HEADER{ unsigned char flags; int block_size; } flags - A bit field as follows: 7 6 5 4 3 2 1 /-------------------\ |* * 1 V V C C| \-------------------/ CC Two bit field as follows: 00 directory fill request. 01 reserved 10 reserved 11 reserved VV Two bit version identifier. This version is 00. 1 Always 1 indicates a client-generated frame. * Reserved, must be 0. is the number of data bytes which the server should place in each directory broadcast frame. 5.2.3 Data If the CC field is 00, a list of UPLOAD_TIME ranges is present. The format of the list is pairs times (number of seconds since 00:00:00 1/1/80.) struct PAIR { time_t start time_t end }; For each PAIR, the server will transmit directories for those files with UPLOAD_TIME greater than or equal to and less than or equal to . If there are no files within the range, the directory for the first file with upload time greater than will be transmitted. If there is no such file, the directory for this first file with upload time less than will be transmitted. In either case, the and fields in this directory will indicate to the client that there are no entries between and .