The Serial Link Communication
for
AMS Data Acquisition System
(Preliminary Version)
May, 1996

S.X.Wu  (MIT),  X.Cai (MIT)

1.	Scope
In AMS data acquisition system, serial links are chosen for physics data transfer, sending commands, reporting errors and downloading parameters and software between different boards. This document is to define the communication protocol over the serial links.
2.	Physical Link
Each serial link includes two sets of independent signals in opposite directions. There are two types of serial link in the AMS data acquisition system. Between crates, the link is implemented with differential transmitters and receivers in LVDS (Low Voltage Differential Signalling) standard. In each direction only one differential signal is used. It carries both data and clock. Within a crate, link between different modules uses single-ended TTL signal via backplane. Wherever possible, data and clock are carried by separate lines to reach higher data transfer speed. 
Data can be transferred in both direction simultaneously. The throughput over the link  (both TTL and LVDS) is about 5 - 10 Mbit per second.
3.	Protocol Architecture
The standard communication module consists of two layers, Physical Layer and Data Link Layer. The Physical Layer is implemented by hardware to make physical connection, signal driving/receiving, bit encoding/decoding, serialisation and transfer synchronisation. The Data Link Layer is mainly implemented by software as user interface. It will pack/unpack user data, detect errors and send acknowledges. 
Only the Physical Layer exists in some places since there is no processor in these places. In this case, the packing/unpacking user data is done by hardware and only hardware errors can be detected. Normally there is no acknowledgement will be sent back over this kind of links.
The Data Link Layer provides several services which are named as Command Service, Status Service, Event Service and Download Service. On transmission side, the services will convert different type of user data to a variable length frame format which is called Data Link Frame and will be described in Section 4.1. The Data Link Frames are sent out to the Physical Layer.  On receive side,  the Data Link Frames coming from Physical Layer are deframed and different functions will be performed according to service type of the frame. 
The Physical Layer makes conversion between Data Link Frame (see Section 4.1) and Transfer Frame. In the Transfer Frame, the Data Link Frame is encoded (4B/5B). Start of Frame Mark, End of Frame Mark and Synchronisation Word shall be added on the transmission side according to the transfer length detected from the Data Link Frame. The deframing and decoding are performed in the other side. 
Figure 1 illustrates the architecture of the protocol layers and their functions.
Transmission Side		Receiving Side	
Command Service	Status Service	Event Service	Download Service		Command Service	Status Service	Event Service	Download Service	
Add Data Link Frame Header		Service Type Decoding	
Calculate Checksum for the Long Frame		Error Detection	
Data Link Frame			
			
Transfer Length Detection		Data Link Frame	
4B/5B Encoding			
Add Start/End of  Frame Mark and Synchronisation Word	Transfer Frame	4B/5B Decoding	
Transmitter		Receiver	
Figure 1. Architecture of Protocol Layers
4.	Data Link Layer
4.1	Data Link Frame
There are two kind of data which will be transferred over the links. The command, status and acknowledgement have only a few bytes of information to be sent. The physics events, parameters and programs can contain much more bytes of data and have no fixed length. In order to make efficient for both cases, we define two kind of Data Link Frames which are named as Short Frame and Long Frame.
4.1.1	Frame Header
There are 16 bits frame header at beginning of each frame to describe the frame type, service type, command and status. The format of the frame header is shown in Figure 2.
0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	
Frame Type	Checksum Flag	Service Type	Command	Status	Reserved	
Figure 2. The Header Format of the Data Link Frame
The Frame Type and Service Type are listed in Table 1 and Table 2 respectively. The Checksum Flag is only used for the Long Frame. It is always set to zero in the Short Frame. The Reserved bits can be extended for future uses and should be set to zero currently. The Command and Status can have different meanings in different services. They will be described for each service later.
Table 1. Frame Type
Value	Type	
00	Reserved	
01	Short Frame	
10	Long Frame	
11	Reserved	
Table 2. Service Type
Value	Type	
000	Reserved	
001	Command	
010	Status	
011	Acknowledge	
100	Event	
101	Program	
110	Parameter	
111	Reserved	
4.1.2	Short Frame
The Short Frame is a 32 bit fixed length frame. Besides the 16 bit frame header, it has 16 bits of information followed. The structure is shown in Figure 3. The Information bits can be different for each service and each function. They will be described individually for each service later.
0	15	16	31	
Frame Header	Information	
Figure 3. Short Frame Format
The Short Frame has no checksum provided. But the illegal type and command should be checked on receiver side either in hardware level or in software level. The protections for some important commands should be done inside the services.
4.1.3	Long Frame
The Long Frame is designed for transferring variable length of data in one frame. Therefore, the frame length is the most important information in the frame. In the Long Frame, the frame length is defined in two bytes (16 bits) after the frame header. It will be checked by the Physical Layer and used to generate the End of Frame Mark. The checksum should be calculated for error detection. The checksum word (16 bits) is put at the end of the frame. The frame length defines the length for the whole frame including frame header and checksum word. The frame format is illustrated in Figure 4.
Frame Header	16 bits	
Frame Length	16 bits	
Frame Data	Variable Length (in byte)	
Checksum	16 bits	
Figure 4. Long Frame Format
The frame data will be organised according to the services and can be defined by users. For example, the parameters and programs may need the destination identifier as the data header to be passed through different levels in the system. The format of the frame data will be described later.
4.2	Command Service
The Command Service is designed to send commands from the up level of system to the low level of system. The Short Frame is used for transferring the Command Service data.
The commands are defined by the four Command bits in the frame header. They are listed in Table 3.
Table 3. The List of Commands
Value	Name	Function	Parameter	
0000	Reserved			
0001	Reset	Reset target by hardware	Command conformation	
0010	Initialise	Make software initialisation	Command conformation	
0011	Start	Start a procedure in target	Command conformation	
0100	Stop	Stop a procedure in target	Command conformation	
0101	Status Request	Require status data from target	None	
0110	Event Request	Require an event from target	Event number to be sent	
0111	LV1 Trigger	Send LV1 trigger pattern to target	LV1 trigger pattern	
1000	Retransmission	Require retransmission	(Acknowledgement only)	
1001	Event Sync.	Synchronise the event number	Current event number	
1010	Use Processor 0	Active processor 0 in target	Command conformation	
1011	Use Processor 1	Active processor 1 in target	Command conformation	
1100	User command	TBD	TBD	
1101	User command	TBD	TBD	
1110	Test (NOP)	No actions, reply acknowledge only	Command conformation	
1111	Reserved			
The Information bits are mainly used to pass the parameters along with the command. In some special cases when there is no parameter needed, the Information bits can be used for the conformation of the command. For example, the conformation may be done simply by repeating the frame header in the Information bits. It is useful to detect errors during the transfer. It is used for some important command such as Reset, Initialise etc. This is because there is no checksum in the Short Frame and these commands will cause the low level of system to be restarted. 
The Status bits in the frame header are not used in the Command Service. They should be set to zero always.
There are two kind of commands on receiving side. A few of commands such as Reset command are handled by the Physical Layer directly to perform some hardware actions. There will be no acknowledgement replied for these commands. The status will be checked in other ways. The other commands are mainly handled by processors. An acknowledgement should be sent back to the sender for reporting the transfer status. The same command may have different requirements of the acknowledgement depend on the implementation of the system. The details have to be defined when the software is made for different part of the system.
4.3	Status Service and Acknowledgement
Both the Status Service data and acknowledgement are transferred by the Short Frame. To unify the definition of the Status bits, they are defined in the same format. The acknowledgement are sent back normally when a frame of data is received. The status to be reported are the correction of the transmission. We call it the transfer status. The Status Service is used to report the operation status to the up level of  system. Therefore we define the first bit (in transmission order) of the Status in the frame header is the status type. If the status type is cleared, it means that it is the transfer status. All transfer status are listed on Table 4. If the status type is set, it is the operation status. The operation status are listed on Table 5.
Table 4. List of Transfer Status
Value	Status	Meaning	
00000	Good	No transfer error detect	
00001	Illegal frame type	Invalid frame type found	
00010	Illegal service type	Invalid service type found	
00011	Illegal command 	Invalid command found	
00100	Illegal status	Invalid status found	
00101	Illegal 4B/5B coding	Undefined 4B/5B code found by decoder	
00110	Hardware check error	Hardware CRC check (if exist) error	
00111	Checksum error	Software checksum error	
01000	Command word error	Command conformation error or wrong number of 1’s 	
01001	Status word error	Status conformation error	
01010	Reserved		
01011	Reserved		
01100	Reserved		
01101	Reserved		
01110	Reserved		
01111	Reserved		
Table 5. List of Operation Status
Value	Status	Meaning	Information	
10000	Ready	Subsystem is ready (backup of  READY)	Current event number	
10001	Wait Event	Processor waiting for an event	Event number waiting for	
10010	Wait Trigger	Processor waiting for a LV1 trigger	Event number waiting for	
10011	Wait Transfer	Processor waiting for sending an event	Current event number	
10100	Event mismatch	Event number different from up level	Current event number	
10101	Restarted	The system is restarted	None	
10110	Reserved			
10111	Reserved			
11000	Reserved			
11001	Reserved			
11010	Reserved			
11011	Reserved			
11100	Reserved			
11101	Reserved			
11110	Test	Testing purpose only	None	
11111	Reserved			
4.3.1	Acknowledgement
The acknowledgement will use the command bits to require the retransmission if necessary. The retransmission command is defined in Table 3. If it is used, the retransmission is required. If the Test (NOP) command is replied, there will be no retransmission taken place.
The Information bits is the conformation of the acknowledgement. It is used to detect transfer error for the acknowledgement. If any error found during sending acknowledgement, the last operation will always be performed again.
4.3.2	The Status Service
The Status Service does not use the Command bits. All Command bits should be set to zero. The operation status indicates which action is taken place in the subsystem or what thing the processor is waiting for. In most of cases, the event number is  a very useful information for checking the system. Therefore the Information bits are used to transfer the event number or other information. 
4.4	Event Service
The Event Service is used to transfer physics data from the low level of the system to the up level. The Long Frame format is used in this service. The physics data format is not defined in this document. Please refer to other AMS documents[1].
There are two special cases of using this service. One is that the level-3 processor (JL3) stores to or reads from the data buffer (JBU). The starting address for the data block is needed. The other case is that the fibre optic control (JFO). The JFO has direct link with the data buffer (JBU). It needs only the starting address and number of frames to be transferred.
Considering the above different cases, we design three frame data formats in the Event Service. The first one is called Event Transfer. The second one is named as Buffer Read/Write. The last is called HRDL Output. They are shown in Figure 5, Figure 6 and Figure 7 respectively. We use the Command bits in the frame header to distinguish these different frame data. The used commands are listed in Table 6. The illegal command error will be generated if other commands which are not on the list are received in the Event Service.
Frame Header	16 bits	
Frame Length	16 bits	
Physics Data Block	Variable Length (in byte)	
Checksum	16 bits	
Figure 5. Frame Format for Event Transfer

Frame Header	16 bits	
Frame Length	16 bits	
Starting Address	32 bits	
Buffer Data Block	Variable Length (in byte)	
Checksum	16 bits	
Figure 6. Frame Format for Buffer Read/Write
Frame Header	16 bits	
Frame Length	16 bits	
Starting address	32 bits	
Number of  HRDL frames	16 bits	
Checksum	16 bits	
Figure 7. Frame Format for HRDL Output
Table 6. List of Used Commands for Event Service
Value	Command	Meaning	Frame Format	
0001	Transfer	Make event transfer	Event Transfer	
0010	Read	Read data from buffer	Buffer Read/Write	
0100	Write	Write data to buffer	Buffer Read/Write	
1000	Output	Start HRDL frame transfer	HRDL Output	
The Status bits in the frame header are not used in the Event Service. They should be set to zero always.
There will no acknowledgement required from the Buffer Read/Write and the HRDL Output. In the Event Transfer, the receiving processor should send back a normal acknowledge reply. The retransmission should be performed if there are any errors during transferring.
4.5	Download Service
The Download Service is designed for downloading parameters and programs. The Long Frame is used in this service. We assign the different service types for parameter and program to distinguish these two cases. The frame format is shown in Figure 8.
Frame Header	16 bits	
Frame Length	16 bits	
Destination Identifier	16 bits	
Parameters/Programs	Variable length (in byte)	
Checksum	16 bits	
Figure 8. Frame Format for the Download Service
The destination identifier[1] is very important in this service. It allows the parameters or programs to be passed through the middle levels of system. For example, we can send the parameters or programs from the level-3 processor (JL3) to the tracker data reduction (TDR) processor via the data acquisition board (JDQT). 
It is not clear so far what is the procedure to be done after the destination processor got the new program. How the program is written to RAM or EEPROM? How can we switch to the new program? Are there any additional information needed to be passed along with the programs? These are to be defined.
The parameters are transferred through the system as a black box. The users should have their own definitions about the parameter data block and decode by themselves.
If there are any common requirement about the procedure definition, we can define the different function by using the Command bits in the frame header which are not currently used.
5.	Physical Layer
5.1	Data Length
It is important that the transmitter knows what is the length of the Data Link Frame coming from the Data Link Layer since the End of Frame Mark has to be added in the Transfer Frame. Firstly it need to check the frame type, the first two bits in the Data Link Frame. The frame type is decoded by hardware. The Short Frame has a fixed length, 32 bits. If the Short Frame is detected, the transfer counter will be initialised to a fixed number (for example, 8 or 32). If it is a Long Frame, the transfer counter will be set according to the frame length word in the Data Link Frame.
When counter is reached, the End of Frame Mark should be added and a Transfer Frame is completed.
5.2	Encoding
The 4B/5B encoding method is used in the Physical Layer. It provides the signal level transitions necessary for clock recovering. Even when clock is sent in separate line from the data, which is the case for backplane links, 4B/5B encoding provides a convenient way for framing, byte boundary synchronisation and error check.
For each Transfer Frame, the Synch and the Start of Frame symbols are added at the beginning and  the CRC data and the End of Frame symbol are appended at the end. With the help of 4B/5B encoding, receiver could easily recover the data bytes boundary and detecting any possible data transfer errors. The 4B/5B codes are listed in Table 7 and the control symbols are listed in Table 8.
Table 7. The 4B/5B Code Table
Hex Data	Binary Data	Encoded Data	
0	0000	11110	
1	0001	01001	
2	0010	10100	
3	0011	10101	
4	0100	01010	
5	0101	01011	
6	0110	01110	
7	0111	01111	
8	1000	10010	
9	1001	10011	
A	1010	10110	
B	1011	10111	
C	1100	11010	
D	1101	11011	
E	1110	11100	
F	1111	11101	
Table 8. Control Symbols
Symbol	4B/5B Encoded Data	Note	
Sync	11000 10001		
Start of Frame	11001 00111	as 1 byte binary data	
End of Frame	00111 11001	as 1 byte binary data	
5.3	Synchronisation
The synchronisation is guaranteed by both the clock and the Sync symbol. The Sync symbol is sent at the beginning of the Transfer Frame. More Syncs can be sent if one is not enough.
At the receiver side, if no separate clock is available, the clock must be recovered from the data stream. In this case, the baud rate of the transmitter must be known. The receiver samples the incoming data stream at five times of the baud rate. Every level transition is monitored and used to synchronise the data sampling. Every bit will be sampled three times between the expected transitions. The bit value is determined by a majority logic. If a separate clock comes with the data, bit value is taken at every clock edge, both rising and falling edges.
5.4	Interrupt
The interrupt will be used on the receiving side in order to notify the processor in the destination to complete the Data Link Layer protocol.
6.	Protocol without Processor
6.1	General Problem
6.2	JL1 Board
6.3	JFO Board
6.4	JBU Board
7.	Software Libraries
There is no description yet.
7.1	Description of Library
There is no description yet.
7.2	Command Service Routine
There is no description yet.
7.3	Status Service Routine
There is no description yet.
7.4	Event Service Routine
There is no description yet.
7.5	Download Service Routine
There is no description yet.