Inside a transistorized shift register box, built in 1965 for Apollo testing

One of the under-appreciated aspects of the Apollo launches to the Moon is how much testing was required. I recently came across an item that was part of this testing: the Computer Buffer Unit. It is essentially a 16-bit shift register that interfaced test equipment to the Apollo Guidance Computer. While a shift register is a trivial circuit nowadays, back then it took a box full of transistors that weighed about 5 pounds. In this blog post, I look inside this unit, describe its unusual packaging and circuitry, and explain how it works.

The Computer Buffer Unit is a 4"×6"×6" box. The three electrical connectors on the left are covered by protective covers. It has a humidity indicator and pressurization valve at the bottom.

Testing for the Apollo missions

The Apollo spacecraft required extensive testing even while it was sitting on the launch pad. Thousands of different spacecraft components needed to be activated and analyzed for various tests. Since the control room was miles away from the launch pad, it wasn't practical to run separate wires to each component. Instead, NASA invented (and patented) a complex digital test system that communicated efficiently between the control room and the rocket. This test system sent digital commands to the launch site, where racks of control and interface units were wired to the spacecraft components. These units decoded the commands and performed the specified operation. Massive quantities of measurement data from the spacecraft were encoded digitally and serialized for communication back to the control room.

The complexity of testing is illustrated by the control room below.2 This is not Mission Control, but a separate control room specifically for testing, called ACE-S/C (Acceptance Checkout Equipment-Spacecraft). These consoles were crammed with control switches, tape readers, CRT displays, chart recorders, and status panels for conducting tests and recording results. The ACE-S/C system supported manual, semiautomatic, and automatic testing, driven by two minicomputers1.

ACE control room. From Applicability of Apollo Checkout Equipment.

All parts of the spacecraft were tested, including the fuel cells, cryogenic fuel storage, communications, and environmental control. For this blog post, the relevant subsystem is "Guidance and Navigation", responsible for determining the Apollo spacecraft's position in space using inertial navigation and guiding it on the proper trajectory including the landing on the Moon's surface. The key to Guidance and Navigation was the Apollo Guidance Computer, 70-pound computers onboard the Lunar Module and the Command Module.

The Apollo Guidance Computer that we restored, next to a replica DSKY.

In space, astronauts operated the Apollo Guidance Computer through the Display/Keyboard (DSKY), a box (above) with keys, indicator lights, and numeric displays. But for ground testing, there needed to be a way to feed commands into the Apollo Guidance Computer from the testing system. The solution was the Computer Buffer Unit, the box that I'm examining. To operate the Apollo Guidance Computer remotely, the ACE test system encoded each DSKY keypress as a 16-bit command3 and sent it to the Buffer Unit. The Buffer Unit converted the message to serial, transferring one bit at a time to the Apollo Guidance Computer, which then processed the desired keypress.4 Thus, the Apollo Guidance Computer could be controlled remotely for testing, providing control over the Guidance and Navigation system, and the Computer Buffer Unit was the interface with the Apollo Guidance Computer.

Inside the Computer Buffer Unit

Next, I'll discuss the physical construction of the Computer Buffer Unit. Removing the lid reveals the components inside.5 The main circuitry consists of six horizontal circuit boards wired into a vertical backplane board; the top board is visible below. One unusual feature is the bag of desiccant inside the unit, zip-tied to the right side of the case. The designers of the unit were worried about Florida humidity and the risk of corrosion.6 To guard against damp air, the unit has a valve on the front so it can be pressurized with dry nitrogen. On the front of the unit, you can see a humidity sensor that changes color to indicate 10%, 20%, and 30% humidity. If the internal humidity exceeded 30%, the desiccant needed to be replaced, as described by the warning label.

The Buffer Unit with the lid removed.

I removed the circuit boards with some difficulty, as they fit tightly. The photo below shows the stack of six printed circuit boards wired into the vertical backplane. The wires from the connectors are soldered directly to the backplane.

With the circuit boards pulled out of the unit, the wiring to the backplane is visible.

The circuit boards can be opened up like a book to provide access to the inner boards. The boards are not soldered directly to the backplane, but are connected by short, flexible wires, allowing them to swing apart. To prevent short circuits between the boards, they are separated by white sheets of (probably) silicone.

After removing six screws, the boards can be unfolded like a book.

The circuitry is constructed in a very unusual way that I haven't seen before. Instead of mounting components directly on the circuit boards, components are mounted on small boards, each forming a module with a logic gate or two. These smaller modules are then soldered on pins above the main circuit boards, forming two-layer boards. Essentially they built pseudo-integrated-circuits on small boards, and then constructed the circuitry from these modules.

Closeup of logic modules mounted on the circuit board. A blue resistor is visible on the underside of the module.

It is difficult to see the components sandwiched between the main board and the smaller modules, but the side view below shows some of the components. The two boards are connected by the vertical pins. A tiny glass diode is visible towards the left. The longer components are resistors. The shiny metal-can transistors are in the middle of the module and harder to see.

This side view shows a latch module (bottom) attached to the circuit board (top). The diodes, resistors, and transistors of the latch module are visible.

One question is why the circuitry is implemented with small circuit boards attached to the larger circuit board, instead of mounting the components directly on the circuit board. This approach seems overly complex and makes the boards twice as thick. One advantage, though, is that the separate logic modules could be manufactured, testing, and repaired separately, important in an era when semiconductors were less reliable. Second, the main boards and the logic modules are different types of printed circuit boards: four-layer circuit boards with widely-spaced traces versus single-sided but dense boards.

Logic gates

The circuitry is implemented with a logic family called Diode-Transistor Logic (DTL). This type of logic was used in the early 1960s as it only required one (expensive) transistor per gate, using cheaper diodes where possible. As transistor prices dropped, Transistor-Transistor Logic (TTL) became more popular because of its better performance. Nowadays fast, low-power CMOS logic is used in most integrated circuits.

I reverse-engineered the schematic below, which shows a NOR gate from this unit. This gate has two inputs, as well as two outputs (for reasons that will be explained below). If both inputs are low (0), the transistor will be turned off. As a result, the resistors pull the outputs high, producing 1 outputs.

The NOR gate with both inputs low, outputs high.

If an input is high, the circuit behaves as shown below. Current flows from the input pull-up resistor through the diodes and the transistor's base, turning the transistor on. As a result, current flows from the outputs, through the transistor to ground, pulling the outputs low. Thus, the circuit implements a NOR gate: the output is 1 if all inputs are low, and 0 otherwise.

The NOR gate with a high input, outputs high.

The reason for multiple outputs is clever. If you connect the outputs from multiple gates together, this combined output will be pulled low if any output is low (i.e. the transistor is turned on), and otherwise will be pulled high by the resistor.7 This logic is equivalent to an AND gate. Note that the AND gate is implemented "for free" by wiring outputs together, without requiring additional logic; this is called wired-AND. However, you can't use a gate's output in two different wired-AND gates, since everything will be shorted together. Instead, a gate provides multiple outputs that can be wired independently; the diodes keep the outputs isolated from each other.

The board I examined has 5 different types of logic module8, from an inverter with 1 input and 8 outputs to a module with two 2-input, 5-output NOR gates. These modules follow the circuit above, but with different numbers of inputs and outputs.

Implementation of the shift register

The idea behind a shift register is to store multiple bits in a row. Each time a clock signal is activated, the bits are shifted by one position. Shift registers can be used to store data, convert parallel data to serial, or convert serial data to parallel. In this Buffer Unit, the shift register converted a 16-bit parallel value from the test equipment into a serial stream of bits for the Apollo Guidance Computer.9

The board implements four bits of the 16-bit shift register. The schematic below shows the circuitry for a one-bit stage of the shift register. There's a lot going on, but I'll try to explain it. The heart of the stage consists of two latches, which store one bit. A bit is stored by first updating the primary latch, and then the secondary latch. (Each latch consists of two cross-coupled NOR gates, and can hold either a 0 or a 1.) The shift out lines are the outputs from the shift register stage, a regular output and an inverted output.

One stage of the shift register. It can read the bits in parallel, or shift a bit from one stage to the next.

Each shift out line is fed to the shift in lines of the next stage, allowing the bits to be transferred from stage to stage through the shift register. The shift and load control lines, along with the AND gates, select the input to each stage. With shift high, the input will be the shift out from the previous stage. With load high, the input reads the external bit in lines. This allows a 16-bit data word to be read into the shift register in parallel. (I'm not sure what the clear bit function is used for.) After a bit has been loaded into the primary latch, the clock line is activated to load the bit into the secondary latch, completing the shift or load cycle.

An interesting function of the unit is that after loading, the value in the latch is compared to the input value, to make sure that the circuit is operating correctly. If there is a mismatch, a compare AND gate will activate, clearing the match line. (A compare AND gate will activate if the input bit is 1 and the latch bit is 0, or vice versa.) This circuit also detects a fault in the bit input wires. Each bit is provided over two wires: one with the bit value and one with the inverted bit value. If a wire is broken or affected by noise, the comparison will fail.10

This diagram shows the functions of the gates. Note that the circular golden transistors are faintly visible through the circuit boards.

The board above11 contains four of these shift-register stages. The photo above shows how these stages map onto the hardware. The external signals (4 pairs of bit lines) enter at the bottom of the board, and pass through the input inverters. The 8 primary latch NOR gates implement four primary latches. Four secondary latch modules implement the four secondary latches, since each module contains two NOR gates. The clock driver, load driver, and shift driver provide 8 copies of the clock, load, and shift signals for the circuitry. Finally, the two match NOR gates combine the 8 match signals. (Note that since the AND gates are implemented with wired-AND, they don't use additional circuitry and do not appear in this diagram.)

I/O and power

I'll wrap up with a few comments on the I/O and power supply for the Buffer Unit. The unit has three military-style connectors on the front. At the top is a 61-pin connector for receiving the parallel data and control signals from ground equipment. (The pin count is larger than you might expect because each bit uses two wires as discussed earlier. Also, many of the 61 pins are unused.)

The unit has three connectors. The unit receives parallel data from ground equipment through the 61-pin connector at the top. The middle connector communicates the serial data to the Apollo Guidance Computer. The unit is powered with 28 volts through the bottom connector, which has larger pins for the high-current supply.

The middle connector has four pins that provide the serial data stream to the Apollo Guidance Computer. The wiring is a bit unusual. Instead of transmitting data over one serial line, the unit uses two pairs of lines: one to transmit "0" bits and one to transmit "1" bits. To provide electrical isolation between the unit and the Apollo Guidance Computer, these signals are transmitted via two small pulse transformers, shown below. When a pulse is fed into a pulse transformer, a similar pulse is produced on the output. (In modern equipment, an optoisolator provides similar functionality.)

Two pulse transformers on the top circuit board. Each small transformer is about 1 cm in diameter.

The bottom connector on the unit has two thick pins to provide 28 volts to the unit. This view inside the unit shows the power converter, a sealed black box. I believe this is a switching power supply module that converted the 28-volt input into the lower voltage required by the logic circuitry. It also provided electrical isolation from the power supply. The smaller black box on the right is an EMI filter on the power input; the Apollo ground test equipment encountered faults from voltage transients and electrical noise, so they added filtering.

The power supply components are sealed in black plastic.

Conclusion

This Computer Buffer Unit was built in 1965, a time when the industry was shifting from transistors to integrated circuits. This may explain the Unit's unusual construction technique, small circuit-board modules that are like integrated circuits built from discrete components.12 Interestingly, Motorola built a similar Buffer Unit for NASA that used integrated circuits (but was just as large),13 illustrating that transistors and integrated circuits were both viable approaches in 1965.

This box also illustrates the rapid pace of integrated circuit technology since the 1960s. The first commercial MOS integrated circuit was a 20-bit shift register introduced in 1964 and by 1970, Intel was producing a 512-bit shift register. In 1971, Western Digital was selling a UART chip, putting a complete parallel-to-serial and serial-to-parallel communication system onto a chip. Thus, it took 6 years to shrink the complex shift-register box down to a single chip (more or less). Nowadays, this functionality forms a tiny part of a complex chip. Coincidentally, Moore's Law, describing the exponential growth of integrated circuits, was published in 1965, the same year this box was manufactured.

I announce my latest blog posts on Twitter, so follow me @kenshirriff. (The Twitter thread corresponding to this blog post is here.) I also have an RSS feed. Thanks to Steve Jurvetson for letting me examine this artifact. A video tour of his space museum is here. Thanks to Mike Stewart for providing documents and extensive information on this box.

Notes and references

1. The photo below shows the ACE computer room that supported ACE testing. The system was controlled by two 13-bit CDC 160-G minicomputers. Strangely, the CDC 160-G minicomputers were 13-bit computers, with 13-bit addresses, 13-bit registers, and 13-bit arithmetic. The earlier CDC 160 computer was 12 bits, and CDC improved the 160-G model by adding one more bit. The CDC 160 was designed by Seymour Cray, reportedly over a weekend.

   

   "An ACE Station with twin Control Data computers." From Computers in Spaceflight.

    ↩

2. There were about 10 ACE installations for testing at various sites. ACE testing was performed at contractor sites, as well as at the launch pad. ↩

3. To send a DSKY keypress through the testing system, each keypress was encoded as 5 bits as shown below. The 16-bit message consisted of a 1 bit followed by three copies of the 5-bit keypress, with the middle copy inverted. (Sending the keypress in triplicate detected communication errors.)

   

   The encoding of keys when communicating with the Apollo Guidance Computer. From ACE-S/C Operator's Manual.

    ↩

4. The serial protocol used by the Apollo Guidance Computer is a bit unusual compared to modern serial protocols. Instead of a single serial line, it used two pairs of wires: one to receive a 1 bit and one to receive a 0 bit. This worked well with the Apollo Guidance Computer hardware, which included a feature for incrementing and decrementing counters in response to interrupts. In particular, a serial input 0 triggers a SHINC instruction (shift left), while a serial input 1 triggers a SHANC (shift and increment by 1) instruction.

   (The interrupt-triggered counter mechanism worked well except during the Apollo 11 landing, when the power supply for the Apollo Guidance Computer and the power supply for the rendezvous radar had a phase difference. For complex reasons, this resulted in a high rate of interrupts, overloading the Apollo Guidance Computer and causing restarts. This was indicated by the famous 1201 and 1202 program alarms during the landing.)

   

   The K-START (Keyboard - Selections To Actuate Random Testing) panel is used to send commands to the Apollo Guidance Computer. From ACE-S/C Operator's Manual.

   In the ACE testing control room, DSKY keypresses were entered on a panel called K-START (Keyboard - Selections To Actuate Random Testing), shown above. The keyboard corresponds to the keyboard on the DSKY, while it has other switches specific to testing. These key entries could also be recorded on perforated tape and played back at high speed. ↩

5. Another interesting feature of the unit is how it is mounted on a rack. The back of the unit has two Teflon-lined holes. Two "dagger pins" from the rack fit into these holes. On the front, the unit has two small hold-down hooks; a knob on the rack engages with the hook to hold the unit in place. The mounting hooks are type NAS 622, an aerospace standard. The hold-down mechanism is described here.

   

   Back of the Buffer Unit with identifying label and two holes for dagger pins. The labels say "Unit, Computer Buffer Guidance & Navigation. NAA/S & ID Control No. ME901-0271-0002. Stock No. Contract No. M5H3XA-450001. NAA/S & ID Inspection Serial No. Control Data Corporation MFGR Part No. 106068-0002. Mfgr Serial No. 10136SA08185. US Nov 19 1965.

    ↩

6. The document Acceptance Checkout Equipment for the Apollo Spacecraft discusses the corrosion problems encountered by the test equipment due to humidity and insufficient air conditioning. The specifications don't discuss pressurization of the unit, but I'm assuming they used nitrogen based on other items I've studied. ↩

7. One subtlety with the wired-AND gate is that connecting multiple outputs together will result in multiple pull-up resistors in parallel, which may provide too much pull-up current. The solution is that some gates have outputs without pull-up resistors, so each wired-AND output has a single pull-up. The wired-AND isn't entirely free, since the multiple outputs require multiple diodes, but diodes are inexpensive compared to transistors. I should admit that I'm not 100% sure of the circuitry. Since the components are all hidden underneath the module, I had to deduce the circuitry by probing it from above. There were a few inputs that didn't seem to have connectivity; perhaps there are capacitors to make these inputs pulse-based. ↩

8. The board I examined uses the following types of modules:
   2304: 1-in, 8-out inverter
   2309: 3-in, 4 out NOR
   2311: 4-in, 2-out NOR
   2319: 1-in, 4-out inverter
   2314: dual 2-in, 5-out NOR (larger than the other modules) ↩

9. The specifications for the Buffer Unit describe its purpose: "This specification covers the requirements for a Guidance and Navigation Computer Buffer Unit, hereinafter referred to as the G&N buffer. The G&N buffer shall form a part of the Digital Test Command System (DTCS) which is the up-link portion of the Automatic Checkout Equipment (ACE). The ACE will be used as ground support equipment for the Apollo space craft. The G&N buffer shall receive remotely generated digital test commands from the control room via the DTCS and shall store, verify, and shift out G&N data in appropriate format to the G&N on-board computer."

   

   Functional diagram of the Buffer Unit. Image from Specification MC 901-0666 courtesy of Mike Stewart.

   The specifications for the Computer Buffer Unit can be viewed online: MC901-0666, ME901-0666, ME901-0271, ME476-0070.

   The unit includes more functionality than just a shift register (but not much more). As shown in the functional diagram above, the unit also includes the clock oscillator that controls the timing of the serial pulses. Second, it contains a control circuit to handle loading the bits in parallel and then shifting them out serially. Third, for reliability reasons, it has a comparator circuit to check that the bits loaded into the shift register match the input bits. ↩

10. Modern systems often use differential signaling, using two complementary signals for a bit. Looking at the difference between the two signals provides noise immunity, since electrical noise will often affect both signals equally, and thus will be canceled out. Although the Buffer Unit uses two complementary signals, it doesn't provide this noise immunity, since the two signals are processed independently rather than differentially. ↩

11. I only reverse-engineered one of the boards, since I didn't want to risk more disassembly, and one board is enough to understand the basic logic. I studied board 6 of the unit, which implements bits 15 through 18 of the shift register. Board 3 implements bits 3-6, board 4 implements bits 7-10, as well as mode bits 1 and 2, board 5 implements bits 11 through 14, and board 6 (the one I examined) implements bits 15 through 18. Boards 2 and 4 implement control logic, while board 1 has the output driver transformers.

    

    With board 6 folded down, board 5 is visible.

    The photo above shows board 5. Note that the circuit layout is entirely different from board 6. I thought that the unit might consist of four identical 4-bit shift register boards, but it turns out that the boards are optimized for particular roles. ↩

12. In the context of "not-quite-integrated circuits", I should mention IBM's use of hybrid modules (called SLT) for the System/360 mainframes. These small aluminum-cased modules contained a few transistor or diodes as silicon dies, mounted on a ceramic substrate along with thick-film resistors. These modules were not quite integrated circuits, since they were built from discrete (but unpackaged) components. But they were closer to integrated circuits than the modules in the Buffer Unit, which used packaged transistors, resistors, and diodes on a printed circuit board. ↩

13. Motorola made a similar Buffer Unit, but they used integrated circuits, specifically Motorola's line of high-speed ECL chips, introduced in 1962. Since each chip is a few gates, it still took multiple boards to build the unit. Apollo Guidance Computer expert Mike Stewart has photos of the Motorola box here, as well as reverse-engineered schematics. The functionality of the Motorola box is nearly identical, except it has separate inputs for the 16-bit compare value. It is built with chips such as the MC308 flip flop and MC 309 dual NOR gate, described here.

    

    A board from the Motorola version of the Buffer Unit. Each metal can is an integrated circuit. Photo courtesy of Mike Stewart.

     ↩

Reverse-engineering a vintage OR/NOR chip

Recently, I received a die photo of a mystery integrated circuit, the OQ100,1 from EvilMonkeyDesignz. I analyzed the die photo and found that it is a logic chip implemented with fast ECL (Emitter-Coupled Logic) circuitry, probably from the early 1970s. The chip contains three logic gates, two with 2 inputs and one with 4 inputs. Each gate has non-inverted and inverted outputs, acting as both an OR gate and a NOR gate. This blog post summarizes my investigation. (I also recently analyzed the OQ104, a different chip in this series.)

Die photo of the Philips QC100 chip. Click this photo (or any other) for a larger version. Photo courtesy of EvilMonkeyDesignz.

The die photo above shows the chip under the microscope. Most of the silicon appears bright pink in this image. Regions of silicon with different doping appear green or yellowish and form the transistors and resistors of the chip. The speckled regions are the metal layer on top of the silicon, wiring the circuitry together. Around the edges, the black bond wires connect the chip to the external pins.

The chip's components

Transistors are the key components in a chip. This chip uses a type of transistor called an NPN transistor. The photo below shows a transistor as it appears on the chip. Underneath the photo is a cross-section drawing showing approximately how the transistor is constructed. The transistor is more complicated than the N-P-N sandwich you see in books, but if you look carefully at the vertical cross-section below the 'E', you can find the N-P-N layers that form the transistor. The emitter (E) wire is connected to N+ silicon. Below that is a P layer connected to the base contact (B). And below that is an N layer connected to the collector (C).

Structure of an NPN transistor. Top: transistor as it appears on the die. Bottom: cross-section diagram.

The chip also uses a few PNP transistors. Although you might expect a PNP transistor to simply be the reverse of an NPN transistor, it has a different structure, with the regions arranged laterally instead of vertically. The collector and base form concentric square rings around the emitter. The base wire is not connected to the base region directly. Instead, the wire is at a distance, and the base signal travels underneath through the N layer.

Structure of a PNP transistor. Top: transistor as it appears on the die. Bottom: cross-section diagram.

The PNP transistors in this chip have another complication. The collector is split, so the transistor has two collectors. Moreover, one of the collectors is wired directly to the base. In the photo above, you can see how the collector region is split vertically, so there is one collector on the left and one on the right, with collector on the right connected to the base. This construction may seem bizarre, but it is common in integrated circuits. The motivation is to build a current mirror, where both collectors pass the same current.

The other key components of this chip are the resistors. The photo below shows two resistors as they appear on the die. The resistors are formed from strips of higher-resistance P silicon, which appears pink in the die photos. Each end of a resistor is connected to the metal layer; the metal in the middle connects the two resistors together in series. (A metal wire also passes over the resistor but is not connected.) A resistor has higher resistance if it is longer and narrower, so these resistors have relatively high resistance.2 Resistors are relatively large on an integrated circuit and fairly inaccurate.

Two resistors as they appear on the die.

The circuitry

Once the components can be recognized on the die, the circuit can be traced out and reverse-engineered. But before I describe the complete circuit, I'll explain how ECL (Emitter-Coupled Logic) works.3 The schematic below shows a differential pair, or long-tailed pair, which amplifies the difference between its two inputs. (This circuit is also common in analog circuits, forming the heart of an op-amp.) The basic idea is that a current sink (the circle at the bottom) generates a fixed current I. This current gets split between the left path (I1) and the right path (I2). If the transistor on the left has a higher input voltage than the transistor on the right, most of the current will go to the left. But if the transistor on the right has a higher input, most of the current will go to the right. This circuit amplifies the voltage difference: even a small difference between the two inputs will switch most of the current from one side to the other.

Schematic of a simple differential pair circuit. The current sink sends a fixed current I through the differential pair. If the two inputs are equal, the current is split equally between the two branches. Otherwise, the branch with the higher input voltage gets most of the current.

To make this into an OR gate, we can put multiple transistors on the left. If any input is high, the current will be switched to the left, otherwise the current will be switched to the right. Since the current pulls that side low, the left branch will be the NOR output while the right branch will be the OR output. (With ECL, you get both the complemented and uncomplemented outputs "for free".) The schematic below shows how one logic gate is implemented; this is a two-input gate. The gate uses a second differential pair to buffer and amplify the outputs. The current sink circuit is discussed in a footnote.4

Schematic of one logic gate.

The diagram below shows how the four-input gate is implemented on the die. The majority of the area is occupied by the current sink and the associated resistors. The NPN and PNP transistors are relatively compact, but the resistors occupy a lot of space. At the bottom, the four input transistors implement the OR function, along with the reference transistor on the other branch. The output transistors are larger so they can provide more current.

One gate with functional blocks labeled.

The diagram below shows how the three gates are arranged on the die. (The gate described above is at the right.) The voltage divider resistors provide a voltage reference for the current sources.

The die with major functional blocks labeled.

Putting everything together, the diagram below shows how the circuitry of the chip maps onto its 16 pins. The three OR gates are represented by the OR symbols; the gate on the right has four inputs. Each gate has a non-inverted output and an inverted output, which is indicated by a bubble. I don't know what voltages the chip takes, so I've indicated the power pins with + and -.

Reverse-engineered pinout of the chip.

I announce my latest blog posts on Twitter, so follow me @kenshirriff. Many thanks to EvilMonkeyz Designs for providing the photos; follow on Instagram or Twitter for more interesting die photos.

Notes

1. A reader said that Philips used the OQ designation for their custom integrated circuits. That would explain why I was unable to find these chips in a databook. ↩

2. The resistance of a resistor is proportional to the length divided by the width. To understand this, note that a region twice as long is the same as two resistors in series, so it has twice the resistance. A region twice as wide is the same as two resistors in parallel, so it has half the resistance. ↩

3. The logic circuit on this chip has a couple of differences from standard ECL gates. A typical ECL gate has inputs on one branch and a reference voltage connected to the transistor on the other branch. Thus, an input higher than the reference voltage is a logic 1, and an input lower than the reference voltage in a logic 0. This gate, however, uses the output of the first branch as the input to the second branch. If an input is high, it pulls this output low, shutting off the other branch. Conversely, if the input is low, the output goes high, turning on the other branch.

   I'm not sure what the motivation is for this design. It looks a bit like NTL (Non-Threshold Logic), since there isn't a threshold set by a reference voltage. One possibility is that the circuit implements a Schmitt-trigger, a circuit with hysteresis, where once it turns on, the input must drop significantly lower to turn it off.

   The second difference between this circuit and a typical ECL gate is the output buffer. ECL gates typically use an emitter follower, rather than a second differential pair. ↩

4. I'll just briefly describe the current sink circuit, shown below. Two large resistors form a voltage divider that produces a reference voltage midway between the two supply voltages (maybe 0 volts). Due to the behavior of transistors, V_BE will be one diode drop (~0.7 V). The rest of the circuit generates the "correct" current through the lower-right resistor to achieve this voltage drop. On the chip, the two PNP transistors at the top are one transistor with two collectors. They implement a current mirror, where the current through the right transistor matches the current through the left transistor.

   

   The current sink circuit used in the chip. The divider is shared by all the current sinks on the chip.

    ↩

Reverse-engineering a vintage comparator chip

I recently saw an interesting die photo of an unknown chip on Twitter, so I did some analysis of it. Looking at the circuitry inside, the chip appears to be four comparators, probably in the ECL (Emitter Coupled Logic) family. This is a quick blog post to summarize my investigation.

The die photo below shows the chip under the microscope. Regions of the silicon appear pink, blue, or yellow, depending on how the silicon was doped. The speckled regions are the metal layer on top of the silicon, wiring the circuitry together. Around the edges, the black bond wires connect the chip to the external pins. These wires are attached to the square bond pads. The die has four blocks of circuitry, one for each of its four comparators. Much of the die is unused, especially the large metal area in the middle. Because this chip's circuitry is relatively simple, it only uses a fraction of the available space.

The chip with pins labeled. Click this photo (or any other) for a larger version. Photo courtesy of EvilMonkeyDesignz.

The photo below shows the chip with its metal lid removed. Modern chips are usually in a black epoxy package, but this chip has a white ceramic package. The tiny silicon die is visible in the middle, with bond wires connecting the die to the lead frame, the metal connections to the chip's gold-plated pins. The metal layer on top of the die is visible, and can be matched with the die photo above. The semi-circular notch on the left indicates the orientation of the chip; the "P" is pin 1.

The chip with the metal lid removed, showing the tiny silicon die inside. Photo courtesy of EvilMonkeyDesignz.

The chip's components

Transistors are the key components in a chip. This chip uses a type of transistor called the NPN transistor. The photo below shows a transistor as it appears on the chip. The orange and blue colors are regions of silicon that have been doped differently, forming N and P regions. The speckled areas are the metal layer of the chip on top of the silicon—these form the wires connecting to the transistor's collector, emitter, and base.

Underneath the photo is a cross-section drawing showing approximately how the transistor is constructed. There's a lot more than just the N-P-N sandwich you see in books, but if you look carefully at the vertical cross-section below the 'E', you can find the N-P-N that forms the transistor. The emitter (E) wire is connected to N+ silicon. Below that is a P layer connected to the base contact (B). And below that is an N+ layer connected (indirectly) to the collector (C). The emitter, base, and collector can be distinguished on the die with careful examination. The base's region surrounds the emitter, forming a blue rectangle. The collector contact is larger and off to the side.

Structure of an NPN transistor. Top: transistor as it appears on the die. Bottom: cross-section diagram.

The other key components of this chip are the resistors. The photo below shows two resistors as they appear on the die. The resistors are formed from strips of higher-resistance P silicon, which appears blue in the die photos. The two ends of each resistor are connected to the metal layer. A resistor has higher resistance if it is longer and narrower. While one resistor below is a simple rectangle, the other has a complex zig-zag shape to fit more length into the available space. (The blue rectangle in the lower right is a transistor, not a resistor. To distinguish it from a resistor, note that it has three contacts, and the two contacts on the rectangle are not symmetrical.)

Two resistors as they appear on the die.

The circuitry

Once the components can be recognized on the die, the circuit can be traced out and reverse-engineered. But before I describe the complete circuit, I'll explain one important functional block.

The schematic below shows a differential pair, or long-tailed pair, which amplifies the difference between its two inputs. This circuit is common in analog circuits, forming the heart of an op-amp. It is also the basis of Emitter-Coupled Logic (ECL). The basic idea is that a current sink (the two circles at the bottom) generates a fixed current I. This current gets split between the left path (I1) and the right path (I2). If the transistor on the left has a higher input voltage than the transistor on the right, most of the current will go to the left. But if the transistor on the right has a higher input, most of the current will go to the right. This circuit amplifies the voltage difference: even a small difference between the two inputs will switch most of the current from one side to the other.

Schematic of a simple differential pair circuit. The current sink sends a fixed current I through the differential pair. If the two inputs are equal, the current is split equally between the two branches. Otherwise, the branch with the higher input voltage gets most of the current.

In this chip, the circuit is used as a comparator, a circuit that compares the two inputs and generates a logic output that indicates which input is higher. The side with the current will get pulled low, while the other side is pulled high by the resistor. Thus, the output can be treated as a logic signal.

The schematic below shows the circuitry for one of the four comparators on the chip. Note that the differential amplifier circuit above is used twice. For better performance, the output from the first differential amplifier is fed into a second differential amplifier. This sharpens the output: if the inputs are close together, the outputs from the first stage may not be fully "0" or "1". The second stage amplifies this difference, providing solid "0" and "1" outputs. The current sinks provide a relatively constant current for the amplifiers, but I won't explain them in detail.

Schematic of one comparator.

The diagram below shows the circuitry on the die, corresponding to the schematic above. The red boxes indicate the transistors for the two amplifier stages and their associated current sinks. The other components are the resistors, which appear as blue rectangles. The wiring in the metal layer connects the components together, as well as the inputs on the left and the output at the top.

One comparator, as it appears on the die.

The diagram below shows how the circuitry of the chip maps onto its 16 pins. Each triangle represents a comparator with its positive and negative inputs. Two of the comparators have a single output, while two also have an inverted output (indicated by a bubble).

Reverse-engineered pinout of the chip.

Conclusion

So what is this chip? Maybe it's simply four comparators, but they could have a specific purpose. The chip could be a converter for four differential input signals, e.g. DCS (Differential Current Switch) logic. Another chip in the family seems to be Emitter-Coupled Logic, so this chip could be four ECL inverters (but it doesn't make sense to have four pins for the reference voltage). It's a bit puzzling that two comparators have inverted and noninverted outputs, while two have single outputs.

Based on the circuitry and ceramic packaging of this chip, I estimate that it is from around 1970. The chip is labeled "OQ104", so I did a bunch of searching through old databooks, but I couldn't find anything that matches it. Since the chip looks like ECL and has the part number 104, it's tempting to think that the chip might be the 10,104 part in the MECL 10,000 series. Unfortunately, the 10,104 is a quad AND gate, and the internal circuitry is different. The "P" on the chip might indicate Philips, but I couldn't find any matching Philips components. For now, the exact identity of this chip is a mystery.

I announce my latest blog posts on Twitter, so follow me @kenshirriff. Many thanks to EvilMonkeyz Designs for providing the photos; follow on Instagram or Twitter for more interesting die photos.

Teardown of a logic chip from a vintage IBM ES/9000 mainframe

IBM and its large mainframe computers ruled the computer industry for decades. But during the 1980s, mainframes faced increasing competition from microprocessors, workstations, and super-minicomputers. To meet this challenge, IBM pushed technology to the limit to create the ES/9000 in 1991, a family of powerful mainframes with a price tag to match, from $70,500 up to $22 million. The processor of the ES/9000 wasn't a single chip, but a metal and ceramic package called a Thermal Conduction Module (TCM) that held 121 chips. Recently, Dave Jones of EEVBlog created a popular teardown video of a TCM, showing its complex construction. After disassembling the module, he kindly sent me some of these cutting-edge chips to analyze. In this blog post, I examine the circuitry inside one of these logic chips from the ES/9000.

Detail of a bipolar logic chip from the ES/9000 computer. This closeup of the die shows the four layers of metal and the transistors underneath. Click this photo (or any other) for a larger version.

The ES/9000

The ES/9000 family of computers consisted of three lines with performance spanning two orders of magnitude: small entry-level systems for an office, mid-range air-cooled systems (below), and high-end water-cooled systems that could fill a room. The technology of the ES/9000 was very advanced for its time in many ways. Along with the ceramic thermal conduction modules, IBM created new high-speed integrated circuits with state-of-the-art transistors. At the system level, IBM introduced new operating systems as well as ESCON (Enterprise Systems Connection), a high-speed fiber-optic connection between the mainframe and peripherals. An optional cryptographic feature provided high-speed encryption in tamper-resistant hardware. Even the power supplies were innovative; the water-cooled power supplies could be swapped while the computer was running. The innovations of the ES/9000 generated numerous journal articles and patents.1

The ES/9000 type 9121, from Hampage.

In this article, I'm focusing on the mid-range systems, known as the 9121 processors.2 This system (above) was packaged in a drab frame the size of a large refrigerator.3 It used 7.4 KVA of power, occupied 14.7 square feet of floor space, and weighed 2000 pounds. It could hold up to 1 gigabyte of memory, a large capacity at a time when personal computers typically had 1 to 4 megabytes of RAM. A typical 9121 system cost $1.5 million and had about twice the performance of a contemporary Intel 80486 computer that cost $10,000. This is a bit of an apples-and-oranges comparison, since the mainframe gave you high-speed I/O channels, fast memory access, and an advanced operating system, but it shows the dramatic price/performance advantage of microprocessors.

The TCM (Thermal Conduction Module)

One of the most interesting features of the ES/9000 was the Thermal Conduction Module (TCM) that held the integrated circuits. The high-performance bipolar chips generated a lot of heat, so IBM developed new cooling mechanisms so this computer could function without water cooling. The cut-away photo below shows a TCM with its large heat sink attached. At the bottom, some of the integrated circuit dies are visible along with the copper cooling pistons. The computer's main circuitry consists of five different TCMs.4

Diagram of the TCM with the heat sink on top. Photo from Dr. Chu / IBM, diagram from TCM paper.

The TCM is surprisingly small, 5 inches (127.5mm) on a side, yet it holds 121 integrated circuits. Each integrated circuit has a spring-loaded copper piston on it to remove the heat. These pistons transfer the heat into the TCM's metal case, where the heat passes into the heat sink and then the air flow. The pistons are precision-machined to maximize contact and thus heat transfer. The module is filled with oil (visible below), which also increases heat transfer. The design of the TCM allows it to dissipate 600 watts of heat—imagine holding six 100-watt light bulbs in your hand.

Closeup of the TCM showing the copper cooling pistons on top of the silicon dies. Courtesy of Dave Jones.

The integrated circuits in the TCM are not packaged like regular integrated circuits, but consist of a silicon die soldered upside-down to the ceramic substrate, flip-chip style. This ceramic substrate is an incredible feat of engineering. It's essentially a printed-circuit board made out of ceramic, with 63 layers of wiring inside. It has over 80,000 connections on the top to the integrated circuits, 2 million vias, 400 meters of internal wiring, and 2772 pins on the bottom.

The TCM opened up, showing the chips inside. Most of the chips are the bipolar logic chips described in this blog post. The two slightly-smaller dies on the left are probably also logic chips. The 16 reddish rectangular chips are 128-kilobit static RAM chips. Six of the 121 positions are unused. The small reddish components between the chips are decoupling capacitors. From EEVBlog Flickr album, © Dave Jones, used with permission.

The manufacturing process for the ceramic substrate was very complex. Each ceramic sheet, the thickness of two sheets of paper (0.2mm), has tens of thousands of via holes punched in it. Next, the wiring was applied in the form of a molybdenum metal paste, forming wires just 100µm wide. The stack of 63 sheets was then laminated under heat and pressure. Next, the stack was sintered at 600°C to decompose the polymer binder, followed by hydrogen treatment at 1560°C for densification. During this process, the substrate shrank by 17%, but the millions of vias must remain aligned. After trimming and polishing, two layers of thin-film wiring were placed on top of the substrate. (The thin-film wiring allowed wiring changes to be made to the module for bug fixes.)5 Finally, the module was protected with a layer of polyimide film, with thousands of openings burned in it with a laser for the chip's connections.

The bipolar logic chip

Most of the chips on the TCM are bipolar logic chips; these are the square black chips in the previous photo. The die photo below shows one of these logic chips, 6.5mm on a side.8 This chip has an unusual appearance because it was connected directly to the substrate instead of the typical approach of putting pads around the perimeter with bond wires attached. The black circles are the 549 solder balls in a 27×27 grid that connect the chip to the substrate. Of these connections, 228 of these are used for signals, while 321 are used for power. The chip is covered with metal conductors that connect the solder balls to the circuitry underneath.

Die photo of a bipolar logic chip, showing the solder balls. (Click for a larger version.)

The chip is built from a type of transistor called the bipolar transistor, an older type of transistor than the MOS transistors in modern processors. The transistors in this chip used a cutting-edge design with a complex internal structure.6 IBM used bipolar transistors because they provided higher performance at the time, but they had the disadvantages of using higher power and taking up more area on a chip. (This is why the chip needed 321 connections for power and why the ES/9000 required multi-chip modules with a complex cooling system.) The chip contains approximately 85,000 transistors, 40,000 resistors, 10,000 capacitors, and 1000 Schottky diodes. While this may seem like a large number, contemporary CMOS microprocessors (such as the Intel 486) contained over a million transistors, illustrating the much higher density of MOS transistors.7

As shown in the closeup photo below, the chip has four layers of metal wiring on top of the silicon, a lot of layers for the time. The metal layer on top of the chip (called M4) provides power and signal distribution from the solder bumps. Underneath, layer M3 provides horizontal wiring: thick lines to distribute power across the chip and thin lines for signals. Layer M2 provides vertical wiring for both power and signals. The bottom layer (M1) implements the local wiring of the gate circuitry, connecting the transistors and resistors together. The narrowest metal lines are 1.6µm wide. Power distribution uses a hierarchy: the numerous solder balls feed power into the very wide power lines in the top metal layer. These are interconnected with the wide horizontal lines, which connect to the thinner vertical lines, which connect to the circuitry. This hierarchy ensures that voltage drop is minimized across the chip, while providing the multi-amp current it requires.

The chip has four layers of metal. The silicon circuitry is visible underneath, somewhat obscured by the multiple layers of insulating silicon dioxide and silicon nitride on top.

The architecture of the chip is IBM's "master slice" approach, building the chip from a gate array of identical cells. To avoid the expense of creating fully-custom chips, IBM built the various logic chips from a common grid of cells that was customized by the wiring on top. In the photo above, you can see some of these cells underneath the metal. The master cell approach has the disadvantage of being less dense than a custom chip. It turns out that roughly half of the cells in each logic chip went unused because the number of I/O pins on the chip was too small.12 You can see that most of the cells are unused in the photo above; while the transistors and resistors are present, they aren't connected to anything.

The chip contains 5240 cells, capable of implementing 2620 DCS logic gates. The structure of a cell is shown below. The cells are very flexible: each cell can implement one gate in the ECL (Emitter-Coupled Logic) family,9 two gates in the NTL (Non-Threshold Logic) family,10 or half a gate in the DCS (Differential Current Switch) family (which this chip uses). The key components are the transistors, which I've colored blue. The resistors are colored yellow.11 At the top are two large capacitors (red). The capacitors are unused in this DCS circuitry, but can be used to speed up ECL gates.

Diagram of the cell layout used by the chip. From patent EP0493989A1.

The image below shows six of the chip's 5240 cells after removing the metal layers from the chip. You can see how the layout matches the diagram above. (The cells in the middle are upside down.)

Closeup of the logic cells. I stacked multiple photos after removing the metal layers to get this image.

The logic chips are fabricated with a special technique that allows hundreds of different types of logic chips to be produced from a single set of masks. The transistors and other components in the silicon "master slice" are constructed using masks and photolithography as in most integrated circuits. However, the metal layers are patterned using direct-write electron beam lithography, rather than masks. This electron beam is steered to "write" the desired metal layer patterns on the die to produce the desired type of chip. In other words, the basic pattern of the chip is created using masks, but then the different chip types are manufactured directly from the design files, providing flexibility.

The photo below shows the entire die after dissolving the metal layers. This image shows the grid of cells, as well as three vertical rows holding 360 I/O cells.13 The grid pattern is most clear in the upper-right corner, where I sanded the die down. (Due to the difficulty of removing four layers of metal as well as layers of silicon nitride, I couldn't get the die as clean as I like.)

Die after removing the metal. The rounded corners are from my mechanical planarization processing (by which I mean sanding with 600-grit sandpaper). The original die was not rounded.

Differential Current Switch logic (DCS)

The chip is built with an uncommon logic family called DCS (Differential Current Switch).15 As the name suggests, DCS operates on differential signals: each input signal is expressed by two wires carrying both the signal and its complement. The voltage difference between the two wires represents a 0 or 1. Thus, a three-input logic gate will have six input wires, as well as two output wires.

Most logic families implement a NAND or NOR gate as their basic gate. The basic DCS gate, however, is the SELECT operation: it outputs either input A or input B, selected by the S input. In other words, SELECT implements the function if S then A else B, or in Boolean logic, SA+S'B. The SELECT operation is surprisingly flexible; with appropriate inputs, it can implement AND, XOR, or even a latch.14

A SELECT gate is shown below at the conceptual level. Three toggle switches are controlled by the S, A, and B inputs. These switches will pull one output to ground, while the other output will be pulled high by a resistor. Starting at the bottom, the S switch will direct the ground current to either the "A" side or the "B" side. With the switches in the indicated positions, the output will be pulled to ground, while the complemented output remains high. But if input A is set to 1, the output levels reverse, with the output pulled high. Now, suppose input S is set to 0, so the current is directed to the B side. In this case, the output is controlled by switch B. You can verify that the output matches A if S is 1 and matches B if S is 0. In other words, the circuit selects between inputs A and B, depending on the value of S. Note that this circuit generates differential outputs: both the output and its complement.

Conceptually, a DCS gate consists of toggle switches that pull one output high and the other low.

Next, I'll describe how the current switch is implemented with a pair of transistors. At the bottom, a current sink generates a fixed current, which can be switched to either the left side or the right side of the circuit. The idea is that the transistor with a higher input voltage will direct the current to that side, pulling that output low. Thus, the circuit acts like a toggle switch. An important feature of the circuit is that it provides a high degree of amplification: a slight difference in voltages is enough to switch most of the current to one side. (This circuit is essentially the same as the differential amplifier used in an op-amp.) As a result, a voltage swing of just 200 millivolts is enough to distinguish a logical 0 and 1, reducing power consumption. Another important feature of this circuit is that it is activated by the difference between the input voltages, so it is relatively insensitive to electrical noise. In other words, a voltage fluctuation that affects both inputs will cancel out, rather than causing an erroneous 0 or 1.

A 1 input switches the current through the transistor on the left. A 0 input switches the current through the transistor on the right.

The schematic below shows the implementation of a DCS gate. The three green boxes are current switches, using transistor pairs as described above. The yellow boxes are buffer circuits, called emitter followers. Two emitter followers buffer the outputs, while two more are used on the select inputs. Finally, the blue box is the current sink circuit, providing the fixed current that gets switched by the circuit.

Components of a DCS gate.

The diagram below shows this circuit in action. Starting at the bottom, the S input switches the current to the left. The A input then switches the current to the right. This current pulls the complemented output low, while the pull-up resistor pulls the output high. Note that a 0 input on A would switch the current to the other side, and thus switch the output. The B input has no effect since the current bypasses the B side of the circuit. Pulling the S input low, however, would switch the current to the B side, causing the B input to control the output. Thus, this circuit implements the SELECT operation.

Schematic of a SELECT gate, showing how the current is steered.

Reverse-engineering a DCS gate

In this section, I'll look at how a SELECT gate is implemented on the chip. The diagram below zooms in on a corner of the die, and then zooms again on one logic gate, the rectangle at the bottom. As you can see, each logic gate is very small on the die. Because this gate is at the edge of the die, it has less wiring over it so it is easier to see. Even so, the wiring layers on top partially obscure it. A DCS gate is created from four half-cells; I've highlighted the one I will discuss.

Starting from the die, zooming in on a corner and then a cell logic gate.

The components on the die can be matched against the diagram below. As before, the transistors are colored blue, the resistors yellow, and the unused capacitor red.

A half-cell as shown in the patent.

Below, I've indicated some of the components in the previously-highlighted half-cell. The wiring on the bottom metal layer customizes this cell for a particular function. Looking at this wiring, you can see that the emitters (E) of transistors T-5 and T-6 are connected, as are the emitters of transistors T-7 and T-8. The collectors (C) of transistors T-6 and T-8 are connected to the base of the output transistor T-12. The collector of transistor T-7 is connected to resistor R3. The wiring in the upper metal layers is shadowy and less clear. The vertical wiring along the sides provides power to the circuit. Other faint vertical wires are connected to the bases of transistors T-7 and T-8.

A half-cell as it appears on the die, with components labeled. "B" is a transistor base, "E" emitter, and "C" collector.

By studying the die closely, I traced out the circuitry for the gate and found it was a SELECT gate. The schematic below is from the patent; I modified it to match the gate I traced out. Note that IBM used its own symbol for a transistor as I've indicated at the bottom. I've marked the transistors and resistors from the photo above in red. The circuit has six transistors for testing, in the blue box.16 As you can see, one DCS gate takes a lot of components: 17 transistors and 18 resistors. This is one reason the density of the bipolar logic chips is so low.

Schematic of the DCS logic gate, as implemented on the chip. Vcc and Vee are the power supplies for the collector and emitter respectively. Vx controls the current sink. Vt is the pull-down voltage for the emitter-followers, but I'm not sure what Vt stands for. The original schematic was for an AND gate; I modified it to show a SELECT gate.

This shows the circuitry of one logic gate. Larger functional blocks such as adders were constructed by combining multiple gates. The full computer contains hundreds of thousands of these gates, implementing the processor and its control circuitry.

Conclusion

This bipolar logic chip illustrates the advanced technology of the ES/9000 mainframe.17 IBM pushed the limits of technology in everything from integrated circuit construction to ceramic modules to cooling systems. After all this effort, however, sales of the ES/9000 were underwhelming and couldn't slow the advance of microcomputers. Two years after the announcement, IBM had installed about 3600 of them, largely the lower-end models.18 In comparison, about 20 million personal computers were being sold per year, about 10,000 times the volume. Mainframes were 21.6% of computer industry revenue and dropping, less than half of personal computer revenue (44.5% of the industry). In 1997, IBM's bipolar processors reached the end of the road as IBM fully moved to CMOS processors.

I announce my latest blog posts on Twitter, so follow me @kenshirriff. I also have an RSS feed.

If you're interested in the TCM, you should definitely watch Dave Jones' teardown video below, as well as the videos where he attempts to remove the chips with hot air and a heating plate before finally succeeding. Thanks to Dave for sending me the chips as well as letting me use his photos.

Notes and references

1. For more information, the IBM ES/9000 type 9121 was described in detail in a series of articles in the IBM Journal of Research and Development, May 1991. The most relevant articles: The IBM Enterprise System/9000 Type 9121 air-cooled processor describes how the processor was implemented, Differential current switch—High performance at low power describes the Differential Current Switch logic, IBM System/390 air-cooled alumina thermal conduction module describes the structure and manufacturing of the TCM in detail, IBM Enterprise System/9000 Type 9121 Model 320 air-cooled processor technology. The Sept 1992 issue has other relevant articles, including A four-level VLSI bipolar metallization design with chemical-mechanical planarization, Improved performance of IBM Enterprise System/9000 bipolar logic chips. Also see The Design of the ES/9000 module and High performance packaged electronics for the IBM ES9000 mainframe. IBM's announcement of the ES/9000 provides a good summary. ↩

2. It's a bit tricky to keep track of IBM's naming and numbering schemes. The first distinction is between the architecture and the computers that implement the architecture. Enterprise Systems Architecture/390 (ESA/390) was IBM's mainframe architecture for the 1990s, continuing the path from System/360 and System/370. The ESA/390 architecture was implemented by several families of computers, including ES/9000, the CMOS-based 9672 Parallel Enterprise Server, the microprocessor-based Enhanced S/390 MicroProcessor Complex, S/390 Integrated Server, and S/390 Multiprise. The ES/9000 had three main processor types: the low-end CMOS 9221 in an air-cooled rack, the midrange 9121 in an air-cooled frame, and the large water-cooled 9021. (Confusingly, bigger numbers indicate a smaller system.) The 9121, the processor type in the middle, is the one I'm discussing in this blog post. Each processor type had several model numbers, as described below.

   

   The different ES/9000 models, from the reference guide. The two-way and three-way multiprocessors are called "dyadic" and "triadic".

   The ES/9000 family covered an enormous range of performance levels; the largest model provided over 100 times the performance of the smallest. The sizes varied widely too. The rack-mounted 9221 was designed for an office and took about 6 square feet of floor space, while the 9121 in the middle was roughly refrigerator-sized, occupying 15 to 24 square feet. The water-cooled 9021 was the classic room-filling mainframe, sized at 88 to 180 square feet. Roughly speaking the low-end ES/9000 9221 was a replacement for the IBM 9370 office-environment "super-mini computer", the air-cooled ES/9000 9121 was a replacement for the IBM 4381, while the water-cooled ES/9000 9021 was a replacement for the larger IBM 3090 systems. ↩

3. IBM was a leader in industrial design, from their computers to the architecture of their buildings and even their logo, as discussed in the book The Interface: IBM and the Transformation of Corporate Design. In the 1950s and 1960s, the design for IBM's computers concealed the internal circuitry, rather than showing it off like many other systems. Instead, IBM expressed the "inherent drama" of computing through spinning tape drives and other peripherals.

   

   A large ES/9000 installation with the water-cooled 9021 processor. From IBM ESCON brochure.

   My opinion is that IBM's design style fell apart in the 1980s with the loss of dramatic consoles and tape drives, leaving just the featureless boxes. To make things worse, these boxes were stripped of subtle detailing such as their pedestal bases and accent trim, leaving units that wouldn't look out of place in a Soviet paper mill. The ES/9000 won a 1991 design award, however, so some people must like the design more than I do. ↩

4. The ES/9000 had five TCMs. The Central Processor Element (CPE) is the microcoded CPU, the module that executes the instructions. The Buffer Control Element (BCE) implements a 64- or 128-kilobyte high-speed cache with error correction, and also handles virtual memory. The System Control Element (SCE) manages the flow of data between the different parts of the computer. (The System Control Element is especially important in a two- or three-processor system.) The Channel Control Element (CCE) controls the I/O channels and is essentially a separate I/O processor. The system can also have an optional Vector Control Element (VCE) for vector arithmetic.

   I was unable to conclusively determine the function of Dave's TCM. The large number (16) of memory chips suggests the cache in the Buffer Control Element (BCE), but this paper says the BCE has 26 memory chips. Possibly the chips are holding the microcode for the Central Processor Element (CPE). ↩

5. The ceramic module has two layers of complex thin-film wiring visible on top. This wiring has a surprising purpose: it allows modifications and bug fixes to be made to the module. By cutting wires with a laser and attaching new wires, signals can be re-routed.

   

   Closeup of an IC location, showing the thin-film wiring on top. Courtesy of Dave Jones.

   IBM calls the modification of computer wiring an Engineering Change or EC. Back in the 1950s, an engineer could easily perform an engineering change by adding and removing wires from a mainframe's wire-wrapped backplane. The printed-circuit boards of the System/360 made changes more difficult, but IBM developed a special "delete" tool to drill out a trace on the circuit board, allowing modification.

   

   This diagram shows how an Engineering Change is made to an EC/9000 TCM. Parts of the thin-film wiring are cut with a laser, and a wire is attached to the special EC pads. From this paper.

   The introduction of the ceramic TCM raised the issue of how could engineering changes be made when the wiring was encased in ceramic. (Discarding the expensive module wasn't an attractive choice.) The solution was to put exposed wiring on the surface of the module, wiring that could be modified as necessary. This consisted of two layers of polyimide plastic (Kapton) with thin-film wiring. Instead of connecting the IC to the ceramic wiring directly, each chip signal went to an EC pad on the surface. The original trace could be vaporized with a laser, and a modification wire (gold-plated cadmium-copper alloy) ultrasonically bonded to the EC pad. The photo below shows a chip with some EC wires.

   

   Closeup of the module showing Engineering Change wires next to the die. The smaller reddish-brown objects are capacitors. Courtesy of Dave Jones.

   In some cases it was necessary to remove a chip from the TCM. As Dave Jones found, unsoldering a chip is very difficult due to the thermal mass of the TCM. IBM invented a focused infrared machine to unsolder a chip. It combined a vacuum chip pick-up tool and infrared heater, along with a bias heater underneath the substrate to heat the whole TCM. A special prism ensured alignment of the new chip while a "mirror substrate" provided temperature feedback. This illustrates how the development of the ES/9000 required the invention of new, specialized tools. ↩

6. These bipolar chips were created using an IBM technology called ATX-4 that achieved almost five times the density of IBM's earlier ATX-1 chips. IBM described three advanced features of these transistors. First, they used a polysilicon base contact self-aligned with the emitter, reducing stray capacitance by a factor of 3. Second, the transistors were surrounded by deep trenches that allowed transistors to be closely packed. Third, they used a very thin implant for the base and optimized doping for the collector. These features improved the density and performance of the transistors. ↩

7. It's interesting to compare the complexity of the bipolar chip with a CMOS microprocessor at the same time. I did some rough estimates of transistor and gate counts, comparing the ES/9000 to a contemporary microprocessor. Each bipolar chip has 85,000 transistors. A CMOS processor from 1991, such as the MIPS R4000, has 1,350,000 transistors, almost 16 times as many, showing the huge density advantage of MOS over bipolar.

   Looking at gates shows an even larger advantage for CMOS. The bipolar chip implements 2620 DCS gates, of which about half are used. For the CMOS processor, I'll estimate 6 transistors for a 3-input gate; subtracting the 16-kilobyte cache in the MIPS R4000 yields about 100,000 gates, a factor of 70 more than the bipolar chip.

   Comparing a 121-chip TCM to a microprocessor yields a different story, with a TCM a bit more complex than a microprocessor. The TCM has roughly 144,000 gates and 256 kilobytes of cache, compared to 100,000 gates and 16 kilobytes of cache for the microprocessor. Thus, my estimate is that a TCM has 44% more gates than a contemporary microprocessor. Taking into account the R4000's external, off-chip cache, the cache sizes are comparable. The ES/9000 uses five TCMs for the processor, which works out to about 7 times the gates of the R4000.

   

   Cross-section of a transistor in the IC. From Advancing the state of the art in high performance logic and array technology.

   The four metal layers of the chip are also highly advanced. The wiring in the chips is made from aluminum-copper alloy sandwiched with titanium to support high current density. The wiring layers are double-insulated with silicon dioxide and silicon nitride to prevent shorts from developing over time. Each layer of the chip is polished flat (planarized) with chemical-mechanical polishing. Even the vias between wiring layers are complex, created by a "damascene stud" method. The vias are constructed by creating holes with reactive ion etching, filling them with metal, and then polishing away excess metal. ↩

8. Here's a summary of the chip's parameters from IBM Enterprise System/9000 Type 9121 Model 320 air-cooled processor technology.

   

   Design parameters of the chip.

    ↩

9. If you're familiar with ECL (Emitter-Coupled Logic), DCS is similar except it uses differential inputs instead of reference-controlled inputs. Although ECL and DCS both use a current-switching differential amplifier, ECL inputs are compared to a reference voltage, rather than the complemented input. (In IBM's ECL circuitry, the reference voltage is ground, so a negative signal is a logic 0 and a positive signal is a logic 1.)

   The key performance benefit of ECL and DCS logic is that transistors are never completely turned on, i.e. saturated. A transistor is relatively slow to get out of saturation, so a logic family such as TTL that saturates transistors is slower. ↩

10. One fairly obscure logic family supported by the chip is NTL, Non-Threshold Logic. NTL is similar to ECL, but without the reference voltage and reference transistors. As a result, NTL gates don't switch on and off sharply, but change in a more analog fashion with the input voltage. One advantage of NTL is that it uses one half-cell instead of the two used by ECL, so you can fit more NTL gates on a chip. NTL also consumes less power than ECL. However, its performance was poorer and it was more sensitive to noise, so it was rarely used in the ES/9000. NTL is described in more detail in this patent. ↩

11. Each resistor has multiple taps (gray boxes) allowing 15 different resistance values to be obtained. Gates with various speed/power tradeoffs can be constructed by using different resistances: the DCS family supports high, medium, low, and ultra-low power gates. (Most of the circuitry is low and ultra-low power.) The 0.2 pf capacitors were used for ECL speedup and for delay elements. ↩

12. Why do these chips require so many solder balls? There's some theory behind it. In the 1960s, E. F. Rent at IBM noticed a relationship between the number of components in an integrated circuit and the number of pins it required. Specifically, as the number of components increased, the number of pins required also increased, according to a power law. This became known as Rent's rule. As IBM increased the complexity of the logic chips, the number of solder bumps increased correspondingly. Chips in the IBM 3080 computer (1980) had an 11×11 grid of solder balls, while the IBM 3090's chips (1985) had a 17×17 grid. (Numbers from this paper.) The chip I examined has a 27×27 grid, but since the chips were limited by the number of I/O connections and half the gate were unused, it seems that this was insufficient. ↩

13. The image below shows some cells from the chip's I/O circuitry. These cells have a different structure from the cells for the logic gates. These cells include larger transistors to provide the necessary output current.

    

    Die photo showing I/O cells. This die photo was formed from a stack of images.

     ↩

14. The SELECT operation (SA+S'B) can implement multiple operations. For instance, setting B=0 implements S AND A. (For this gate, the redundant transistors can be omitted.) Setting A=B' implements S XOR A. (XOR is inconvenient to implement in most logic families but simple with DCS). Wiring the output back to A results in a latch: when S is high, the output value is held, but when S is low, the latch is loaded from B. An inverter is trivial with DCS: because of the differential signaling, a signal can be inverted simply by switching the two lines. ↩

15. Curiously, IBM's articles about the ES/9000 expand the DCS acronym as both Differential Current Switch and Differential Cascode Current Switch. The term cascode refers to "a two-stage amplifier that consists of a common-emitter stage feeding into a common-base stage." Essentially, it refers to how DCS has two layers of switching transistors, compared to the single layer in a typical ECL gate. ↩

16. Each DCS gate has about 6 additional transistors for test purposes. The problem is how to detect a faulty logic gate. In most logic families, a faulty gate will typically have the output stuck at 0 or 1. By running various test sequences through the circuit, this stuck bit can be detected. However, since a DCS gate uses differential logic, it can end up with a fault where both differential outputs are approximately the same, for instance, if the current sink fails. This is difficult to detect with tests since it is unpredictable how this signal will be interpreted by other gates. This non-determinism makes it hard to detect a faulty gate. The solution is to add test circuitry to each gate. The test circuitry will force an indeterminate output to a 0 or a 1, depending on which test circuit is activated. This makes the tests deterministic and a faulty gate can be detected. This seems like a weird corner case, but it was important enough for IBM to add a substantial amount of circuitry to each gate. The test circuit is described in more detail in this patent. ↩

17. In the 1980s, IBM faced the problem that it was the reigning computer company with its advanced mainframes, but it was encountering competition from microcomputers. Although microcomputers were technically inferior and much less powerful, they were much cheaper and rapidly increasing in power. The book The Innovator's Dilemma is the classic guide to this sort of problem. Incumbents often ignore the risk from disruptive technologies but IBM took the "right" approach and developed the IBM PC (1981) to take advantage of microprocessors. Although the IBM PC was extremely successful, IBM lost control of the PC architecture and personal computers devoured the mainframe market. It will be interesting to see what happens to Intel in the analogous situation as ARM processors gain functionality and cut into the market for technologically-advanced x86 chips. ↩

18. According to Computerworld, the adoption of ES/9000 was slow, with 3600 installed almost two years after introduction. Of the installations, 47% were low-end rack-mounted systems, 36% were air-cooled frame systems, and 17% were high-end water-cooled systems. IBM had over half the mainframe market, well ahead of Fujitsu, Hitachi, and NEC. ↩