Showing posts with label electronics. Show all posts
Showing posts with label electronics. Show all posts

Apollo Guidance Computer: Dipstiks and reverse engineering the core rope simulator

Onboard the Apollo spacecraft, the revolutionary Apollo Guidance Computer helped navigate to the Moon and land on its surface. The AGC's software was physically woven into permanent storage called core rope memory. We1 are restoring an Apollo Guidance Computer (below), which is missing the core ropes, but instead has core rope simulator boxes. These boxes were used during development and ground testing to avoid constantly manufacturing ropes. The core rope simulator is undocumented, so I reverse-engineered it, built an interface, and we used the simulator to run programs on our Apollo Guidance Computer. But we ran into some complications along the way.

The Apollo Guidance Computer with the cover removed, showing the wire-wrapped backplane. At the back, rope simulator boxes are visible in the core rope slots. The interface boards at the front are modern.

The Apollo Guidance Computer with the cover removed, showing the wire-wrapped backplane. At the back, rope simulator boxes are visible in the core rope slots. The interface boards at the front are modern.

The AGC's core ropes

The Apollo Guidance Computer held six core rope modules, each storing just 6 kilowords of program information (about 12 kilobytes).2 Core rope modules were a bit like a video game ROM cartridge, holding software in a permanent yet removable format. Programs were hard-wired into core rope by weaving wires through magnetic cores. A wire passed through a core for a 1 bit, while a wire going around a core was a 0 bit. By weaving 192 wires through or around each core, each core stored 192 bits, achieving much higher density than read/write core memory that held 1 bit per core.

Detail of core rope memory wiring from an early (Block I) Apollo Guidance Computer. Photo from Raytheon.

Detail of core rope memory wiring from an early (Block I) Apollo Guidance Computer. Photo from Raytheon.

Manufacturing a core rope was a tedious process that took about 8 weeks and cost $15,000 per module. Skilled women wove the rope by hand, threading a hollow wire-filled needle back and forth through the cores, as shown below3. They had the assistance of an automated system that read the program from a punched tape and positioned an aperture over the matrix of cores. The weaver threaded the needle through the aperture to install the wire in the right location. Once completed, the core rope was mounted in a module along with hundreds of resistors and diodes and encased in epoxy to make it solid for flight. (See my earlier article on core rope for details.)

A woman weaving a core rope memory, wiring software into read-only memory.

A woman weaving a core rope memory, wiring software into read-only memory.

The core rope simulator

Since weaving a core rope was a time-consuming and expensive process, an alternative was required during development and ground testing. In place of the core ropes, NASA used rope simulators4 that allowed the AGC to load data from an external system. Our Apollo Guidance Computer was used for ground testing so it didn't have core ropes but instead had a core rope simulator. The simulator consists of two boxes that plugged into the AGC's core rope slots, each box filling three rope slots. These boxes are visible in the upper-left side of the AGC below, with round military-style connectors for connection to the external computer.

The core rope simulators are installed in the left side of the AGC in place of the real core ropes. Two round connectors on the left allowed the simulators to be connected to an external computer that provided the data.

The core rope simulators are installed in the left side of the AGC in place of the real core ropes. Two round connectors on the left allowed the simulators to be connected to an external computer that provided the data.

Although we have extensive documentation for the Apollo Guidance Computer, I couldn't find any documentation on the simulator boxes. Thus, I had to reverse engineer the boxes by tracing out all the circuitry and then figuring out what the boxes were doing. From the outside, the boxes didn't reveal much. One end of each box has a round MIL-Spec plug for connection to an external system. The other end has three groups of 96 pins that plugged into the AGC. Each group of pins took the place of one core rope module.

Each core rope box communicated with the external system via a round 39-pin connector.
Each box had three sets of 96 pins that plugged into the AGC, replacing three rope modules.

Each core rope box communicated with the external system via a round 39-pin connector. Each box had three sets of 96 pins that plugged into the AGC, replacing three rope modules.

Opening up the boxes showed their unusual construction techniques. Part of the circuitry used high-density cordwood construction which mounted components vertically through holes in a metal block. On either side of the block, the component leads were welded to point-to-point wiring. Other circuitry in the boxes used standard integrated circuits (7400-series TTL). But unlike modern printed circuit boards, the chips were mounted inside plastic units called Dipstiks and wire-wrapped together.

A rope simulator box, partially disassembled. The round external connector is visible at the right, and the pins to connect to the AGC at the left. Analog circuitry with cordwood construction is center-left. To the right, several Dipstik modules are visible, white with rows of pins.

A rope simulator box, partially disassembled. The round external connector is visible at the right, and the pins to connect to the AGC at the left. Analog circuitry with cordwood construction is center-left. To the right, several Dipstik modules are visible, white with rows of pins.

Cordwood construction

Cordwood construction was extensively used in the Apollo Guidance Computer for analog circuitry, and the cordwood construction in the rope simulators is similar (below). The white circles in the center are the ends of resistors and diodes mounted vertically through the module, with connections welded on either side. These components are stacked together densely, like wood logs, giving cordwood construction its name. Pulse transformers are under the large gray circles. Similar pulse transformers are on the other side of the module with their orange, yellow, red, and brown wires emerging from the holes. The black wires connect the cordwood circuitry to the digital logic. At the top of the photo, the posts have diodes and resistors mounted behind them, along with connections to the pins that plug into the AGC.

A closeup of the cordwood circuitry in a rope simulator box.

A closeup of the cordwood circuitry in a rope simulator box.

The main purpose of the cordwood circuitry was to provide electrical isolation between the Apollo Guidance Computer's circuitry and the rope simulator boxes. In modern circuitry, this function would be implemented with optoisolators but the rope simulator used small pulse transformers instead. Because each box receives signals directed to three different rope modules, numerous diodes merge the three signals into one. Resistors control the current through the pulse transformers.

Reverse-engineering the analog cordwood circuitry was a pain. First, none of the components were visible since they are embedded in the module. I had to use a multimeter to try to figure out what the components were. Second, since cordwood construction has connections on both sides, I spent a lot of time flipping the box back and forth to find the connection I wanted. Finally, I couldn't come up with a good way of drawing a diagram of cordwood construction without ending up in a maze of lines.

Digital logic and the Dipstiks

The Dipstik was a plug-in module introduced in 1968 to simplify prototyping with integrated circuits. It replaced printed circuit boards with a packaging system that provided twice the density. (See this vintage Dipstik ad.) The idea of the Dipstik was a plastic connector block with wire-wrap pins on the bottom for wiring up the circuit. The integrated circuits were clipped into a carrier that fit into the connector block. The carrier had solder lugs on top for additional components, such as decoupling capacitors. (The photo below shows Dipstik modules with one IC carrier removed. Each carrier held 5 integrated circuits.) The pins of the integrated circuit were sandwiched between contacts on the carrier and contacts on the connector block. It seemed like a great idea but turned out to be unreliable. As the plastic flexed and bowed out, the contacts with the pins became unreliable. (This was a problem both for contemporary Dipstik users and for us decades later.) The Dipstik was a stock market failure.

A Dipstik package opened up. The carrier (left) holds the ICs, and is inserted into the connector block (right). Photo courtesy of Marc Verdiell.

A Dipstik package opened up. The carrier (left) holds the ICs, and is inserted into the connector block (right). Photo courtesy of Marc Verdiell.

The photo below shows the wire-wrapped connections on the underside of the Dipstiks. Tracing this was extremely tedious since I couldn't follow a wire through the sea of identical blue wires. Instead, I had to beep everything out with a multimeter to find what was connected to what. Then I could construct a schematic diagram of the logic circuitry and ponder what it was doing.5 In total, the rope simulator used about 50 ICs.

Each rope simulator box contains complex circuitry. On the left, wire-wrapped wiring connects the TTL integrated circuits. On the left, the analog components are mounted using cordwood construction. At the back, two voltage regulators in large metal TO-3 packages are mounted on a heat sink.

Each rope simulator box contains complex circuitry. On the left, wire-wrapped wiring connects the TTL integrated circuits. On the left, the analog components are mounted using cordwood construction. At the back, two voltage regulators in large metal TO-3 packages are mounted on a heat sink.

Based on the dates on the components, the simulator boxes were built in 1971. Even though this is just a few years after the design of the AGC, the technology in the simulator boxes is much more advanced, illustrating the rapid changes in IC design between the mid-1960s and the early 1970s. The AGC was built with simple integrated circuits, each containing two NOR gates and built with primitive resistor-transistor logic (RTL). The simulator boxes, on the other hand, were built from more complex 7400-series chips6 containing up to a dozen TTL (transistor-transistor logic) gates. Unlike the obsolete flat-pack integrated circuits in the AGC, the simulator boxes used DIP (dual in-line package) ICs, a packaging style that is still in use.

Results of reverse engineering

After tracing out all the circuitry, I figured out how the rope simulator worked and created a schematic.

Essentially, one box decodes the address being accessed, while the second box sends the desired data to the AGC. (I'll call these the "address box" and "data box".)7

The address box takes the rope signals and converts them into a binary address. This task is not straightforward because the signals it receives are 14-volt high-current pulses designed to flip the rope cores. These pulses are also separated in time since some flip the core and others flip the core back. Finally, the pulses sent to the ropes are not a simple address, but also signals to select one of the 6 rope modules, signals to select one of 12 strands in a module.

The address box uses pulse transformers to convert the 14-volt pulses into TTL signals. It has a bunch of AND-OR logic to convert the signals into a binary address. (This is not trivial because each module holds 6 kilowords, not a power of 2, so a lot of bit manipulation is required.) A flip flop latches the address when it is available. Finally, resistor-capacitor one-shots control the timing, determining from the various signals when the address is ready and when the result should be sent to the AGC.

The data box is simpler. It receives 16 bits of data from the external system and sends signals to the AGC's sense amplifiers simulating the millivolt output from a core. These signals are generated via pulse transformers. The address box and data box communicate with each other via wires on the AGC backplane.

The boxes communicate with the external system via differential signals, to avoid picking up noise on long cables. The boxes contain LM109 5-volt regulators to power their TTL circuits. One box receives unregulated DC through the external connector, and sends unregulated DC to the other box through the AGC's backplane wiring. (This seems strange to me.)

With the address box opened, the wire-wrapped circuitry is visible.

With the address box opened, the wire-wrapped circuitry is visible.

The BeagleBone interface

Once I had reverse-engineered the core rope simulator, the next step was to build an interface that could provide program data to the simulator. I used a BeagleBone, a tiny single-board Linux system. The advantage of the BeagleBone is that it includes fast microcontrollers that could respond to the AGC's memory requests quickly, in real time. (I've written about the BeagleBone's PRU microcontrollers before, and used them to make a Xerox Alto Ethernet interface.)

The interface to the rope simulator consists of a board plugged into a BeagleBone. The two cables from the interface board go to the two simulator boxes.

The interface to the rope simulator consists of a board plugged into a BeagleBone. The two cables from the interface board go to the two simulator boxes.

I designed an interface board that plugged into the BeagleBone. The board is pretty straightforward: some AM26C32 differential line receivers to convert the differential signals from the simulator into 3.3V logic signals for the BeagleBone, and some AM26C31 differential line drivers to send signals to the simulator.8 I designed the board in KiCad and PCBWay manufactured it. They are a sponsor of our AGC restoration, so send them some business :-)

I wrote some software that runs on the PRU, the BeagleBone's microcontroller. This software is basically a state machine that waits for an address from the simulator box, waits for the timing signal, reads the word from BeagleBone RAM, and sends the word to the simulator box. The software is on Github.

Problems with the core rope simulator

The core rope simulator boxes were not built to the standard of the Apollo Guidance Computer, and I ended up spending a lot of time debugging them.10 Many welds in the cordwood circuitry were broken and needed to be soldered. (I don't know if the welds went bad over time or if we broke connections while reverse engineering.) We also found a short circuit on a Dipstik and a bad IC.

Debugging the simulator boxes. We're used Marc's vintage Tektronix 7854 scope (1980) to examine the pulse transformer's differential signals. The problem turned out to be a broken connection in the cordwood circuitry. Behind the AGC are the power supplies for the AGC and the rope simulator.

Debugging the simulator boxes. We're used Marc's vintage Tektronix 7854 scope (1980) to examine the pulse transformer's differential signals. The problem turned out to be a broken connection in the cordwood circuitry. Behind the AGC are the power supplies for the AGC and the rope simulator.

The Dipstiks were the worst problem, as many of the contacts between a Dipstik and the IC were intermittent. The problem was that the IC pins are sandwiched between contacts in the Dipstik carrier and contacts in the Dipstik connector block. The plastic Dipstiks tended to bow outwards, resulting in intermittent bad contacts. By bending the IC pins into S curves, Marc was able to keep the pins in contact with both sides, at least for a little while. But after a few hours, the soft IC pins would bend back and connections became unreliable again, so we don't have a good long-term fix.

The most interesting problem was a race condition between two signals from the AGC that should have dropped simultaneously. They fed two ends of a pulse transformer coil, so the transformer should have produced no signal. However, one signal dropped a bit slower than the other, causing a glitch pulse from the pulse transformer.5 Unfortunately, the digital logic in the simulator box was asynchronous, so the glitch latched a bad address bit into the box's flip flops, causing an access to the wrong memory location. Eventually, we tracked the problem down and put capacitors across the offending signals to filter out the glitch. Unfortunately, we used capacitors that were a bit too large and delayed the address signal too much in other circumstances, causing different errors. We put in smaller capacitors and we were finally able to successfully run programs on the AGC, using the vintage core rope simulator.

Conclusion

The Apollo Guidance Computer used core ropes for program storage. Since it wasn't practical to constantly manufacture core ropes during development, rope simulators were used in place of the core rope modules. I reverse-engineered the rope simulator and built a BeagleBone-based interface to drive it. We successfully ran programs on the AGC through the rope simulator. The rope simulator, however, had many problems and wasn't very reliable.

We will be demonstrating the Apollo Guidance Computer next week to celebrate the 50th anniversary of the Moon landing. Come see these historic demos at the Cradle Of Aviation Museum (Long Island) on July 18 and the MIT Museum on July 20.

@CuriousMarc made a video (below) showing our work with the core rope simulator. I announce my latest blog posts on Twitter, so follow me @kenshirriff for future articles. I also have an RSS feed. My rope simulator files are on Github. Thanks to PCBWay for sponsoring this board.

Notes and references

  1. The AGC restoration team consists of Mike Stewart (creator of FPGA AGC), Carl Claunch, Marc Verdiell (CuriousMarc on YouTube) and myself. The AGC that we're restoring belongs to a private owner who picked it up at a scrapyard in the 1970s after NASA scrapped it. For simplicity, I refer to the AGC we're restoring as "our AGC". 

  2. The AGC was a 15-bit machine: each word consisted of 15 data bits and a parity bit. While a word that isn't a power of two may seem bizarre now, computers in the 1960s were designed with whatever word size fit the problem. 

  3. The original caption on the photo was: "Space age needleworker 'weaves' core rope memory for guidance computers used in Apollo missions. Memory modules will permanently store mission profile data on which critical maneuvers in space are based. Core rope memories are fabricated by passing needle-like, hollow rod containing a length of fine wire through cores in the module frame. Module frame is moved automatically by computer-controlled machinery to position proper cores for weaving operation. Apollo guidance computer and associated display keyboard are produced at Raytheon Company plant in Waltham, Massachusetts." Caption and photo are from a Raytheon document, courtesy of Transistor Museum

  4. Several different core rope simulators were built for the AGC. The AGC monitor provided a debugging console with lights and switches along with the rope simulator. The Portafam was a rope simulator that could load programs from magnetic tape. While these rope simulators had some documentation, unfortunately, I couldn't find any documentation on the Raytheon rope simulator that we had, so I had to reverse engineer everything. 

  5. I figured out all the circuitry except for two mysteries. The first mystery is the parity circuit, using two uncommon 74180 parity generator chips. These are not used for the memory parity bit, which is supplied externally. They do not check the address parity supplied by the AGC. It appears that based on an external switch they will optionally replace address bit 7 with the parity of the other address bits. Lacking an address bit, the system would then be unusable. We left the parity switched off and everything worked fine.

    The second mystery is a transistor that looks like it would amplify the strand select signal. The problem is that the transistor's collector isn't connected to anything, so the transistor is unpowered and doesn't function. When we encountered timing issues with the strand select signal, we powered up this transistor to see if it helped, but it made things worse. In addition, the transistor appears to have been added to the cordwood circuitry at a later date. We ended up ignoring the transistor. 

  6. The TTL chips in the interface boxes were 5400-series chips. These are the military version of the well-known 7400-series, identical except operating over a wider temperature range. 

  7. The split into "address box" and "data box" isn't exact. Due to the way the rope slots are wired, the data box determines which of the 6 modules are being addressed and provides this information to the address box. 

  8. I had some difficulty getting the round MIL-spec 20-39S connectors to attach to the simulator boxes, since they have unusual keying. There are a zillion slightly-different variants of these connectors, many costing hundreds of dollars. I ended up getting connectors off eBay and Marc milled the keying off so the connectors worked. I used 25-pair Amphenol telco cables between the BeagleBone and the simulator. Soldering the wires to the connectors was more of a pain than I expected. 

  9. Viewing the pulse transformer signals on a scope was difficult because you need to see the small differential signal across the transformer. Since we didn't have a differential probe for the modern scopes, we used Marc's vintage scope that did have a differential probe. (You might think you could subtract the two signals with a modern scope, but the problem was that the difference was much smaller than the common-mode voltage, so you essentially get zero when you subtract.) 

  10. We built some infrastructure to help debug the simulator boxes. I tested the boxes extensively outside the AGC, using an Arduino and a power transistor to generate test signals. I added debugging code to the BeagleBone to detect when something went wrong, and illuminated a status LED on my interface board. I also used the BeagleBone to generate an oscilloscope trigger signal at a known-bad address, letting us see the analog signals at that specific point. Mike wrote some FPGA code to check the data from the simulator box against the data the AGC should have read, detecting whenever something went wrong. Finally, I logged the addresses that the BeagleBone saw, while Mike logged that addresses that the AGC was sending. By comparing the addresses, we could see which addresses were bad.

    I learned a few lessons from my interface board that I'll apply to future boards. Putting an RGB status LED on the board was my best idea, since it made it much easier to tell what was happening. I should have exposed the BeagleBone's serial port connector on my interface board. As it was, if I ran into any problems booting the BeagleBone, I had to pull off the interface board, attach the FTDI serial adapter, fix the problem via the serial console, remove the serial adapter, and then reinstall the interface board. Finally, I should have exposed a couple generic I/O pins for functions such as oscilloscope triggering. Instead, I had to solder temporary wires onto my interface board. 

Software woven into wire: Core rope and the Apollo Guidance Computer

Onboard the Apollo spacecraft, the revolutionary Apollo Guidance Computer helped navigate to the Moon and land on its surface. One of the first computers to use integrated circuits, the Apollo Guidance Computer was lightweight enough and small enough (70 pounds and under a cubic foot) to fly in space. An unusual feature that contributed to its small size was core rope memory, a technique of physically weaving software into high-density storage. In this blog post, I take a close look at core rope and the circuitry that made it work.1

Detail of core rope memory wiring from an early (Block I) Apollo Guidance Computer. Photo from Raytheon.

Detail of core rope memory wiring from an early (Block I) Apollo Guidance Computer. Photo from Raytheon.

The Apollo Guidance Computer (AGC) had very little memory by modern standards: 2048 words of RAM in erasable core memory and 36,864 words of ROM in core rope memory. In the 1960s, most computers (including the AGC) used magnetic core memory for RAM storage, but core ropes were unusual and operated differently. Erasable core memory and core rope both used magnetic cores, small magnetizable rings. But while erasable core memory used one core for each bit, core rope stored an incredible 192 bits per core, achieving much higher density.2 The trick was to put many wires through each core (as shown above), hardwiring the data: a 1 bit was stored by threading a wire through a core, while the wire bypassed the core for a 0 bit. Thus, once a core rope was carefully manufactured, using a half-mile of wire, data was permanently stored in the core rope.

The Apollo Guidance Computer. The empty space on the left held the core rope modules. The connectors on the right linked the AGC to the rest of the spacecraft.

The Apollo Guidance Computer. The empty space on the left held the core rope modules. The connectors on the right linked the AGC to the rest of the spacecraft.

We3 are restoring the Apollo Guidance Computer shown above. The core rope modules (which we don't have)4 would be installed in the empty space on the left. On the right of the AGC, you can see the two connectors that connected the AGC to other parts of the spacecraft, including the DSKY (Display/Keyboard). By removing the bolts holding the two trays together, we could disassemble the AGC. Pulling the two halves apart takes a surprising amount of force because of the three connectors in the middle that join the two trays. The tray on the left is the "A" tray, which holds the logic and interface modules. The tray on the right is the "B" tray, which holds the memory circuitry, oscillator, and alarm. The six core rope modules go under the metal cover in the upper right. Note that the core ropes took up roughly a quarter of the computer's volume.

The AGC is implemented with dozens of modules in two trays. The trays are connected through the three connectors in the middle.

The AGC is implemented with dozens of modules in two trays. The trays are connected through the three connectors in the middle.

How core rope works

At a high level, core rope is simple: sense wires go through cores to indicate 1's, or bypass cores to indicate 0's. By selecting a particular core, the sense wires through that core were activated to provide the desired data bits.

Magnetic cores have a few properties that made core memory work.7 By passing a strong current along a wire through the core, the core becomes magnetized, either clockwise or counterclockwise depending on the direction of the current. Normally the cores were all magnetized in one direction, called the "reset" state, and when a core was magnetized the opposite direction, this is called the "set" state. When a core flips from one state to another, the changing magnetic field induces a small voltage in any sense wires through the core. A sense amplifier detects this signal and produces a binary output.

The key advantage of core rope is that many sense wires pass through a single core, so you can store multiple bits per core and achieve higher-density storage. (In the case of the AGC, each core has 192 sense wires passing through (or around) it5, so each core stored 12 words of data.) This is in contrast to regular read/write core memory, where each core held one bit.

Core rope used an unusual technique to select a particular core to flip and read. Instead of directly selecting the desired core, inhibit lines blocked the flipping of every core except the desired one. In the diagram below, the current on the set line (green) would potentially flip all the cores. However, various inhibit lines (red) have a current in the opposite direction. This cancels out the set current in all the cores except #2, so only core #2 flips.

This diagram illustrates how core rope memory worked. Simplified diagram from MIT's Role in Project Apollo vol III, Fig. 3-12

This diagram illustrates how core rope memory worked. Simplified diagram from MIT's Role in Project Apollo vol III, Fig. 3-12

In the diagram above, only the sense lines (blue) passing through core #2 pick up an induced voltage. Thus, the weaving pattern of the sense lines controls what data is read from core #2. To summarize, the inhibit lines control which core is selected, and the sense wires woven through that core control what data value is read.

The inhibit lines are driven from the address lines and arranged so that all inhibit lines will be inactive for just the desired core. For any other address, at least one inhibit line will be activated, preventing the core from flipping and being read. 6

The AGC's core ropes

The Apollo Guidance Computer contained six core rope modules, each storing 6 kilowords of program information. The AGC was a 15-bit machine: each word consisted of 15 data bits and a parity bit. (While a word that isn't a power of two may seem bizarre now, computers in the 1960s were designed with whatever word size fit the problem.8) Each module contained 512 cores, each storing 12 words of data. That is, each module had 192 (12×16) sense wires going either through or around each core. Each group of 16 sense wires for a word was called a "strand", so there were 12 strands.

AGC with rope modules. A rope module is partially inserted into one of the 6 slots in the AGC. Photo © Draper Labs via Caltech.

AGC with rope modules. A rope module is partially inserted into one of the 6 slots in the AGC. Photo © Draper Labs via Caltech.

The photo above shows how the core rope modules slid into the Apollo Guidance Computer; the pins on the end of each module meshed with connectors in the AGC. The core rope module below (and its companion) held an early Lunar Module program called Retread 50. We took our AGC to the Computer History Museum to read the data from these modules and we put the results online.

This core rope module held the Retread 50 software for the Apollo Guidance Computer. This module is from the Computer History Museum.

This core rope module held the Retread 50 software for the Apollo Guidance Computer. This module is from the Computer History Museum.

The 512 cores in each module were arranged physically as two layers of 256 cores (but electrically as four planes of 128 cores).9 A set and reset line went through all the cores in a plane, allowing a particular plane in the module to be selected.6 The photo below shows the interior of a core rope module. One layer of 256 cores is visible, with the tiny wires threaded through them. (The second layer of 256 cores is underneath.) Note that the cores only take up about half the module space. Surrounding the cores are hundreds of resistors and diodes that were used to select the desired word.10: These components were mounted with cordwood construction, with the components installed vertically through holes in the module.

Inside a core rope. The 32×8 cores visible form a layer of 256 cores; a second layer is underneath. Photo from CHM/Raytheon CN-4-421-C.

Inside a core rope. The 32×8 cores visible form a layer of 256 cores; a second layer is underneath. Photo from CHM/Raytheon CN-4-421-C.

The photo below shows one of the Retread 50 modules from the Computer History Museum with the cover removed. The cores were encased (potted) in protective epoxy to protect them during flight, so the cores are not visible.

A core rope module with the cover removed.  This module is at the Computer History Museum.

A core rope module with the cover removed. This module is at the Computer History Museum.

Manufacturing the core rope

Wiring of the core rope was a tedious process that took about 8 weeks and cost $15,000 per module. As a result, the computer code needed to be frozen months in advance and last-minute patches to the code were not possible.11 The core ropes (and the AGC) were manufactured by Raytheon in Waltham, Massachusetts. Many of the women building the ropes were hired from the local textile industry for their sewing skills; other skilled women came from the Waltham Watch Company, a company that also helped with the high-precision gyroscopes used on the Apollo missions.12

Much of the core rope wiring (address inhibit wires, set, reset, etc.) was the same for all core rope modules. Two women passed a needle back and forth through the cores to create this wiring. The needle was hollow and contained the necessary length of wire. The clip above (from Computers for Apollo via xpascal) shows this process.

"Space age needleworker 'weaves' core rope memory for guidance computers used in Apollo missions.
Memory modules will permanently store mission profile data on which critical maneuvers in space are based.
Core rope memories are fabricated by passing needle-like, hollow rod containing a length of fine wire through cores in the module frame.
Module frame is moved automatically by computer-controlled machinery to position proper cores for weaving operation. Apollo guidance computer and associated display keyboard are produced at Raytheon Company plant in Waltham, Massachusetts."
Caption and photo are from a Raytheon document, courtesy of 
Transistor Museum.

"Space age needleworker 'weaves' core rope memory for guidance computers used in Apollo missions. Memory modules will permanently store mission profile data on which critical maneuvers in space are based. Core rope memories are fabricated by passing needle-like, hollow rod containing a length of fine wire through cores in the module frame. Module frame is moved automatically by computer-controlled machinery to position proper cores for weaving operation. Apollo guidance computer and associated display keyboard are produced at Raytheon Company plant in Waltham, Massachusetts." Caption and photo are from a Raytheon document, courtesy of Transistor Museum.

To store the desired binary data, the core rope's sense lines were threaded through or around cores in the proper sequence. Originally, this wiring was done entirely manually, which was slow and error-prone. Raytheon improved the process by combining automated positioning with manual threading. First, the program's assembly code was fed into an assembler called YUL that produced a Mylar punched tape. An automated system (above, below) read this tape and step-by-step moved the proper core into position. A woman manually threaded the sense line through an aperture into the indicated core. The aperture then jogged down to pull the wire around a nylon pin, moving the wire out of the way for the next sense wire to be threaded. Once all the cores were threaded, the nylon pins were removed and the final core rope module was tested by an automated system, again controlled by punched tape.

A woman wiring sense lines in a core rope. She is threading the wire through a white circular aperture that indicates the core to wire. Source: Raytheon CN-4-20C / Smithsonian Institution WEB15435-2016.

A woman wiring sense lines in a core rope. She is threading the wire through a white circular aperture that indicates the core to wire. Source: Raytheon CN-4-20C / Smithsonian Institution WEB15435-2016.

Core rope circuitry

Core rope required a lot of digital and analog circuitry to drive and read the ropes. This section briefly describes this circuitry and shows the modules that implemented it. The bottom four modules in the picture below (Sense Amplifier, Strand Select, and two Rope Drivers) implemented the analog circuitry. Logic modules (in the "A" tray shown earlier) decoded the address into rope, module, and strand select signals. We carefully tested the analog and digital modules individually before powering up the AGC.

Tray B of the Apollo Guidance Computer. The erasable memory module is the large black module. Most of the other modules on the left are support for the erasable (RAM) and fixed (core rope) memory. The core rope modules would slide into the right hand side under the metal cover.

Tray B of the Apollo Guidance Computer. The erasable memory module is the large black module. Most of the other modules on the left are support for the erasable (RAM) and fixed (core rope) memory. The core rope modules would slide into the right hand side under the metal cover.

Sense Amplifier Modules

When a core flipped (either in fixed memory or erasable memory), the changing magnetic field induced a weak signal in a sense line, one sense line for each bit in the word. This signal needed to be amplified and converted to a logic signal; this was the job of the sense amplifiers. The sense amplifiers were implemented using a custom sense amplifier IC. (The AGC used only two different types of integrated circuits, the sense amplifier and a dual NOR gate.) The AGC had two identical sense amplifier modules; one (in slot B13) was used by the erasable core memory, while the other (B14) was used by the fixed core rope memory.

The photo below shows a sense amp module. Eight sense amplifiers are visible and eight other sense amplifiers are on the other side of the module. The sense amplifiers required carefully-tuned voltage levels for bias and thresholds so the modules included voltage regulation circuitry (center and right in photo). On top of the module (front in the photo), you can see the horizontal lines of the nickel ribbon that connected the circuits; it is somewhat similar to a printed circuit board.

Sense amplifier module with the top removed. Note the nickel ribbon interconnect at the top of the module.

Sense amplifier module with the top removed. Note the nickel ribbon interconnect at the top of the module.

The closeup photo below shows the module's cordwood construction. In this high-density construction technique, components were inserted into holes in the module. Resistors and capacitors passed through from one side of the module to the other, with one lead on either side. On each side of the module, components were connected by point-to-point wiring. This wiring was welded, not soldered. White insulating sleeves were placed over the wires to prevent short circuits.

Closeup of the sense amplifier module for the AGC. The sense amplifier integrated circuits are at the top and the reddish pulse transformers are below. The pins are at the bottom and the wires at the top go to the nickel ribbon, which is like a printed circuit board.

Closeup of the sense amplifier module for the AGC. The sense amplifier integrated circuits are at the top and the reddish pulse transformers are below. The pins are at the bottom and the wires at the top go to the nickel ribbon, which is like a printed circuit board.

Near the top of the photo are two amplifier integrated circuits in metal cans. Below are two reddish pulse transformers. An output driver transistor is between the pulse transformers.13 Only the ends of resistors are visible, due to the cordwood construction. At the top of the module are connections to the nickel ribbon interconnect. Modules that were flown on spacecraft were potted in plastic so the components were protected against vibration. Since our AGC was used on the ground, most modules were unpotted and the components are visible.

Address decoding

Address decoding for the core rope required a fair amount of logic for two reasons. The first problem was the AGC's instructions used 12-bit addresses, which could only address 4 kilowords of storage. Since the AGC had 36 kilowords of fixed memory and 2 kilowords of erasable memory, it used a complex system of bank registers and mapping logic to convert a 12-bit address into the correct physical memory location. The second problem was that each core rope module held 6 kilowords, which is not a power of two. Thus, moderately complex decoding circuitry was required to generate module and strand select signals from the address.

The core rope addressing, control, and timing logic is spread across several logic modules. Most of the address decoding was implemented in logic module A15. Other core rope logic is in modules A6, A8, and A14. Photo courtesy of Mike Stewart.

The core rope addressing, control, and timing logic is spread across several logic modules. Most of the address decoding was implemented in logic module A15. Other core rope logic is in modules A6, A8, and A14. Photo courtesy of Mike Stewart.

The AGC's logic circuitry (including the processor) was implemented with NOR gates. Each integrated circuit implemented two NOR gates using RTL (resistor-transistor logic), an early logic family. These ICs were costly; they cost $20-$30 each (around $150 in current dollars). There wasn't much inside each IC, just six transistors and eight resistors. Even so, the ICs provided a density improvement over the planned core-transistor logic, making the AGC practical. The decision to use ICs in the AGC was made in 1962, amazingly just four years after the IC was invented. The AGC was the largest consumer of ICs from 1962 to 1965 and ended up being a major driver of the integrated circuit industry.

Each IC contained two NOR gates implemented with resistor-transistor logic. From SCD 2005011.

Each IC contained two NOR gates implemented with resistor-transistor logic. From SCD 2005011.

Strand select

The strand select module in position B15. Photo from Mike Stewart.

The strand select module in position B15. Photo from Mike Stewart.

The address decoding logic described above produced signals indicating the desired strand and module. These logic-level signals needed to be converted to 14-volt pulses to drive the core rope modules. This task was performed by the strand select module, which consisted of transistor driver circuits using NPN and PNP transistors. The resistors in these circuits were individually selected to produce exactly the desired currents.

A closeup of the strand select module, showing its cordwood construction.

A closeup of the strand select module, showing its cordwood construction.

Rope driver

The rope driver modules generated the high-current pulses (up to 450mA) necessary to flip the cores. Like the strand select module, the rope driver modules used NPN and PNP transistor driver circuits with carefully-selected resistors to ensure the desired current. They also used inductors to control the pulse shape, necessary to keep switching noise from overwhelming the small signals generated by the cores. The modules generated 16 inhibit signals (7 address bits and parity, along with complements) as well as two core set signals and four core reset signals.

The rope driver modules in B16 and B17 provided high-current pulses to the rope module. Photo from Mike Stewart.

The rope driver modules in B16 and B17 provided high-current pulses to the rope module. Photo from Mike Stewart.

The core rope simulator

Unfortunately, the core ropes were missing from the Apollo Guidance Computer we are restoring. Instead, this AGC has core rope simulator boxes in place of the ropes. The purpose of these simulator boxes was to feed code into an AGC for development and ground testing without requiring a new core rope to be manufactured every time. The simulator allowed an external computer to supply data words in place of the core rope, allowing the AGC to run arbitrary programs.

The core rope simulators are installed in the left side of the AGC in place of the real core ropes. Two round connectors on the left allowed the simulators to be connected to an external computer that provided the data.

The core rope simulators are installed in the left side of the AGC in place of the real core ropes. Two round connectors on the left allowed the simulators to be connected to an external computer that provided the data.

The simulator consists of two boxes that plugged into the AGC's core rope slots. These boxes are visible in the upper-left side of the AGC above, with round military-style connectors for connection to the external computer. One box exported address information each time the AGC performed a core rope read, while the other box fed 16 data bits into the AGC, "tricking" the AGC into thinking it had read from core rope. I built an interface from the core rope simulator boxes to a Beaglebone and will write more about that project later.

Conclusion

Core memory was the most common storage technology for computers in the 1960s. However, the Apollo Guidance Computer also used core ropes for read-only storage, an uncommon storage technique that achieved high density but required considerable labor. Core ropes made it possible to fit complex software into the compact physical space of the AGC. While 36K of code seems ludicrously small by modern standards, it held enough code to navigate and land on the Moon. And now, decades later, we can recover this code from core rope modules and learn more about it.

Marc has a series of AGC videos; the video below discusses the core rope simulators. I announce my latest blog posts on Twitter, so follow me @kenshirriff for future articles. I also have an RSS feed. See the footnotes for more information sources15. Thanks to Mike Stewart for supplying images and extensive information.

Notes and references

  1. Prototype core rope memories had a single bundle of wire that looked similar to a rope. The final core rope memories didn't look as rope-like, but kept the name. 

  2. The core rope memory achieved a density of 1500 bits per cubic inch, including the driving hardware and packaging. This was about 5 times the density of erasable core memory. See MIT's Role in Project Apollo vol III pages 91 and 274.

    The cores in core rope were not regular ferrite cores, but permalloy ribbon wound around a non-magnetic steel bobbin. If you're used to ferrite cores, winding a metallic ribbon around a bobbin may seem like a strange way to make a core, but that was how the earliest cores were built, until ferrite cores started to be used in the early 1950s. The ferromagnetic materials used in wound-ribbon cores were developed by Germany in World War II and used by the German navy in magnetic amplifiers. After the war, American personnel brought back the material along with a rolling mill, and started US manufacturing under the name Deltamax. In other words, core memory had its origin in Nazi technology captured by the US. See Memories that shaped an industry, pages 39-40, 52, 87, 90.

    The cores used in the core rope were rather large, .249" in diameter, about 5 times the diameter of the cores used in the erasable core memory. (In comparison, some IBM 360 mainframes of the same era used tiny cores that were .021" in diameter.) When a rope core flipped, it yielded a fairly large voltage pulse of 215-430 mV. Properties of the cores were defined in NASA specification control drawing SCD-1006320

  3. The AGC restoration team consists of Mike Stewart (creator of FPGA AGC), Carl Claunch, Marc Verdiell (CuriousMarc on YouTube) and myself. The AGC that we're restoring belongs to a private owner who picked it up at a scrap yard in the 1970s after NASA scrapped it. For simplicity, I refer to the AGC we're restoring as "our AGC".

    The Apollo flights had one AGC in the command module (the capsule that returned to Earth) and one AGC in the lunar module. In 1968, before the Moon missions, NASA tested a lunar module with astronauts aboard in a giant vacuum chamber in Houston to ensure that everything worked in space-like conditions. We believe our AGC was installed in that lunar module (LTA-8). Since this AGC was never flown, most of the modules were not potted with epoxy. 

  4. Yes, we know about Francois and the rope modules he read; those are ropes for the earlier Block I Apollo Guidance Computer and are not compatible with our Block II AGC. Also, many people have asked if we talked to Fran about the DSKY. Yes, we have. 

  5. Mike Stewart pointed out that a core wouldn't have all 192 sense wires passing through it. Because the system used odd parity, at most 15 of the 16 bits can be high. Thus, at most 180 sense wires would pass through a core. 

  6. The Apollo Guidance Computer had one additional pair of inhibit lines for address parity so all non-matching cores will have at least two inhibit lines activated. (If a core was one line away from being activated, the parity would also be different, yielding two active inhibit lines.) The purpose of this was to ensure that every non-selected core received two inhibit signals and was solidly inhibited. Otherwise, a core with just one inhibit line high might receive a bit of net set current, changing its magnetism slightly and introducing noise.

    Note that 7 address lines select one of 128 cores. Each module consists of four (logical) planes of 128 cores, yielding 512 cores. To select one of the four planes, four different reset lines are used, to reset just the core in the desired plane. Thus, only that plane is read. Two set lines are used, one to set cores in planes A and B, and one for planes C and D. To avoid setting cores in two planes, the reset line in the undesired plane was activated at the same time as the set line, blocking the set in that plane. The obvious approaches would be to use four set lines (one per plane), or one set line (and use reset to block the others). I don't know why they used two set lines.

    Each module also had a "clear" line that passed through all the cores. This was similar to reset, but was needed due to a complexity of the AGC's opcode decoding. To fit more opcodes into a 15-bit instruction, the AGC used "quartercode" instructions. The idea was that some instructions didn't make sense on a fixed memory address, such as increment. The AGC would perform an entirely different instruction in that case, allowing a larger instruction set. The problem was that by the time instruction decoding had decided that the instruction didn't apply to the specified address, the read of fixed memory had already started, and the core was set. The "clear" line allowed this core to be reset so it wouldn't interfere with the desired read. I don't know why the existing reset lines couldn't be used for this purpose. (Summary and details.) The schematic is here

  7. If you are familiar with regular core memory (i.e. erasable RAM core memory), there are many similarities, but also many important differences. First, erasable core memory was arranged in a grid, and a particular core was selected by energizing an X line and a Y line in the grid. Second, erasable core memory stored a single bit per core, with the direction of magnetization indicating a 0 or 1. Core rope, on the other hand, stored many bits per core, with a 0 or 1 depending on if a sense wire goes through the core or not. The cores in core rope were much larger, since about 200 wires went through each core, while erasable core memory typically had 3 wires through each core. Finally, erasable core memory used the inhibit line for writing a 0, while core rope used inhibit lines for addressing. 

  8. For more information on why the Apollo Guidance Computer used 15-bit words, see MIT's Role in Project Apollo vol III page 32. The short answer is they required about 27-32 bits accuracy for navigation computations, and about 15 bits for control variables. Using a 15-bit word for small values and double-precision values for navigation provided sufficient accuracy. A 14-bit word was too small. A 17- or 18- bit word would simplify some things but increase costs. 

  9. Core rope memory in the earlier Block I Apollo Guidance Computer was configured slightly differently. The memory was folded into 4 layers of 128 cores instead of 2 layers of 256 cores. As a result, the Block I rope modules had a different shape: roughly square in cross-section, unlike the flat Block II modules. In early Block I modules, each core had 128 sense wires (8 words) threaded through it, yielding 4K words per module. With 6 modules, an early Block I AGC had 24K words of rope storage. In later Block I AGCs, the rope modules provided more storage: 192 sense wires (12 words) per core, yielding 6K words per module. Thus, 24K of rope storage required just 4 modules. For the Block II computers used for the Moon missions, 6 modules × 512 cores per module × 192 bits per core ÷ 16 bits per word = 36864 (36K) words in total. 

  10. Selection of a particular module and strand was done through a resistor/diode biasing network using hundreds of resistors and diodes in each module. The diagram below shows how this network operated. The selected strand line was pulled high, while the selected module line was pulled low. This resulted in current (red) through the selected strand and through the sense amplifier transformer. The remaining diodes were reverse-biased, so no current flowed. A voltage pulse on the sense wire through the selected core perturbed the current flow, resulting in a voltage imbalance across the sense amplifier transformer. This signal was detected by the sense amplifier, resulting in a 1 bit. (This circuit is rather confusing; you might expect the circuit to be a loop through the sense wire and the sense amplifier transformer, but instead, the currents are flowing towards the 0-volt module line in the middle. Study the directional arrows carefully to see how the current flows. The current from a flipped core is essentially superimposed on this current flow.)

    A particular strand and module were selected by a resistor/diode network. The non-selected diodes were reverse-biased, blocking those signals from the sense amplifiers.  Based on MIT's Role in Project Apollo vol III, Fig. 3-13

    A particular strand and module were selected by a resistor/diode network. The non-selected diodes were reverse-biased, blocking those signals from the sense amplifiers. Based on MIT's Role in Project Apollo vol III, Fig. 3-13

    The resistor and diodes in the left green box were repeated once for each strand (i.e. 192 times in each module), while the resistor and diodes in the right box were repeated once for each sense line (i.e. 16 times in each module). 

  11. The need to freeze the software design weeks in advance was viewed as a feature: "The inability to change the program without rebuilding one or more modules provides an effective management tool for the control of software changes. It also provides another incentive to make the software error free." See MIT's Role in Project Apollo vol III page 274. Much of the information in this section on core rope manufacturing is from One Giant Leap. The 1965 video Computers For Apollo shows AGC manufacturing process (including core ropes) in detail. 

  12. There are interesting gender issues behind the manufacture of core rope and core memory that I'll only mention briefly. The core rope has been referred to as LOL memory, referencing the "Little Old Ladies" who assembled it, but this name erases the women of color who also assembled core ropes. (I'm also not convinced that the LOL name was used at the time.) The software for a particular flight was managed by "rope mother" who was generally male (although the famous Margaret Hamilton was rope mother on LUMINARY). Also see Making Core Memory: Design Inquiry into Gendered Legacies of Engineering and Craftwork.

    Women weaving a core rope. Raytheon photo, via BBC.

    Women weaving a core rope. Raytheon photo, via BBC.

  13. The sense amplifier output circuitry is a bit confusing because the erasable core memory (RAM) and fixed rope core memory (ROM) sense amp outputs were wired together to connect to the CPU. The RAM had one sense amp module with 16 amplifiers in slot B13, and the ROM had its own identical sense amp module in slot B14. However, each module only had 8 output transistors. The two modules were wired together so 8 output bits are driven by transistors in the RAM's sense amp module and 8 output bits are driven by transistors in the ROM's sense amp module. (The motivation behind this was to use identical sense amp modules for RAM and ROM, but only needing 16 output transistors in total. Thus, the transistors are split up 8 to a module.) 

  14. I'll give a bit more detail on the sense amps here. The key challenge with the sense amps is that the signal from flipping a core is small and there are multiple sources of noise that the sense line can pick up. By using a differential signal (i.e. looking at the difference between the two inputs), noise that is picked up by both ends of the sense line (common-mode noise) can be rejected. The differential transformer improved the common-mode noise rejection by a factor of 30. (See page 9-16 of the Design Review.) The other factor is that the sense line goes through some cores in the same direction as the select lines, and through some cores the opposite direction. This helps cancel out noise from the select lines. However, the consequence is that the pulse on the select line may be positive or may be negative. Thus, the sense amp needed to handle pulses of either polarity; the threshold stage converted the bipolar signal to a binary output.

    Schematic of the circuitry inside a sense amplifier IC.

    Schematic of the circuitry inside a sense amplifier IC.

    The sense amplifier (above) was a custom integrated circuit designed by Sperry Rand in 1962. This chip pushed the state of the art for analog ICs and it may be the first integrated circuit amplifier. The sense amp chips initially cost $200 each, equivalent to $1700 now. 

Hammer time: fixing the printer on a vintage IBM 1401 mainframe

The Computer History Museum has two operational IBM 1401 computers used for demos but one of the printers stopped working a few weeks ago. This blog post describes how the 1401 restoration team diagnosed and repaired the printer. After a lot of tricky debugging (as well as smoke coming out of the printer) we fixed a broken trace on a circuit board. (This printer repair might sound vaguely familiar because I wrote in September about an entirely different printer failure due to a failed transistor.)

The IBM 1401 business computer was announced in 1959, and went on to become the best-selling computer of the mid-1960s, with more than 10,000 systems in use. A key selling point of the IBM 1401 was its high-speed line printer, the IBM 1403. It printed 10 lines per second with excellent print quality, said to be the best printing until laser printers were introduced in the 1970s.

The IBM 1401 mainframe computer (left) at the Computer History Museum printing the Mandelbrot fractal on the 1403 printer (right).

The IBM 1401 mainframe computer (left) at the Computer History Museum printing the Mandelbrot fractal on the 1403 printer (right).

To print characters, the printer used a chain of type slugs (below) that rotated at high speed in front of the paper, with an inked ribbon between the paper and the chain. Each of the 132 print columns had a hammer and an electromagnet. At the right moment, when the desired character passed the hammer, an electromagnet drove the hammer against the back of the paper, causing the paper and ribbon to hit the type slug, printing the character.1

The type chain from the IBM 1401's printer. The chain has 48 different characters, repeated five times.

The type chain from the IBM 1401's printer. The chain has 48 different characters, repeated five times.

The printer required careful timing to make this process work. The chain spun around at 7.5 feet per second and each hammer had to fire at exactly the right time to print the right character perfectly aligned without smearing. Every 11.1 µs, a print slug lined up with a hammer, and the control circuitry checked if the slug matches the character to be printed. If so, the electromagnet was energized for 1.5 ms, printing the character.

Printing mechanism of the IBM 1401 line printer. From 1401 Reference Manual, p11.

Printing mechanism of the IBM 1401 line printer. From 1401 Reference Manual, p11.

While the printer is usually reliable, a few weeks ago the printer stopped working and displayed a "sync check" error on the console. The computer needs to know the exact position of the chain in order to fire the hammers at the right time. If something goes wrong with this synchronization, the computer stops with "sync check" rather than printing the wrong characters.

When the sync check light on the printer is illuminated, you have a problem.

When the sync check light on the printer is illuminated, you have a problem.

To track the chain position, the computer receives a sequence of pulses from the printer: a pulse when the first hammer is lined up with a type slug2 and a double pulse when the chain is in its "home" position with the first character lined up. The pulses are created by a slotted metallic timing disk inside the printer. A magnetic pickup detects these slots and produces a small 100 millivolt signal.7 This signal is amplified inside the printer by two differential amplifier cards to produce a stronger square wave signal. (This is the only electronic part of the printer. Everything else inside the printer is electromechanical or hydraulic; a high-speed hydraulic motor feeds paper through the printer and drips oil on the floor.)

The computer receives these pulses from the printer and generates a logic signal that increments counters to keep track of the chain's position. The schematic below shows part of the circuitry inside the computer, starting with the sense amplifier signal from the printer at the left. Don't try to understand this circuit; I just want to show the strange schematic symbols that IBM used in the 1960s. The box with "I" is an inverter. The triangle is an AND gate. The semicircle that looks like an AND gate is actually an OR gate. The large box with a "T" is a trigger, IBM's name for a flip-flop. The "SS" box is a "single shot" that creates a 400µs pulse; this detects the double pulse that indicates the chain's "home" position.

Excerpt from the 1401 Intermedia Level Diagrams (ILD) showing the chain detection circuitry.

Excerpt from the 1401 Intermedia Level Diagrams (ILD) showing the chain detection circuitry.

To track down the problem, we removed the printer's side panel to access the two amplifier circuit boards, which are visible below. We probed the boards with an oscilloscope. The first amplifier stage (on the right) looked okay, but the second stage (on the left) had problems. In the photo below, the computer is at the back, mostly hidden by the printer.

We took the side panel off the 1403 printer to reveal the circuit boards. We hooked an oscilloscope up to the front board to test it.

We took the side panel off the 1403 printer to reveal the circuit boards. We hooked an oscilloscope up to the front board to test it.

The trace below shows what should happen. The board receives a differential signal at the bottom, with alternating cyan and pink signals. The difference between these signals (middle) is amplified to produce the clean, uniform pulse train at the top. Note the double pulse in the middle indicating the chain's home position.

Oscilloscope trace from a working printer.

Oscilloscope trace from a working printer.

But when we measured the signal, we saw signals that were entirely garbled. The differential signals at the bottom are a mess, and track each other rather than alternating. The output signal (top) is basically random. With this signal from the printer, the computer couldn't keep track of the chain position and the sync check error resulted.

Oscilloscope trace from the faulty printer.

Oscilloscope trace from the faulty printer.

We swapped the board with the board from the other, working printer, and verified that the board was the problem. The museum has a filing cabinet full of replacement circuit boards, but unfortunately not a replacement for this "WW" amplifier board. Instead, we had to diagnose the problem with this card and repair it. On the board below you can see the diodes (small gray cylinders), capacitors (silver cylinders), resistors (striped cylinders and large tan cylinders), and germanium transistors (round metal cans). The transistors are germanium transistors as the 1401 predated silicon transistors.

The "WW" differential amplifier card used by the printer.

The "WW" differential amplifier card used by the printer.

We suspected a failed transistor, so we used Marc's vintage Hewlett-Packard Tektronix curve tracer (below) to test the transistors. One transistor was much weaker than the others. Since the performance of a differential amplifier depends on having transistors with closely matched characteristics, we searched through a couple dozen transistors to find a matching pair and replaced the transistors. (We later determined that these transistors were not part of the differential pair—they were "emitter follower" buffers, so our effort was wasted.)3

We used a vintage HP transistor curve tracer to test the transistors.

We used a vintage HP transistor curve tracer to test the transistors.

Back at the Computer History Museum, we tested out the repaired board and the printer still didn't work. Even worse, smoke started coming from the back of the printer! I quickly shut off the system as an acrid smell surrounded the printer. I expected to see a blackened transistor on the board, but it was fine. I examined the printer but couldn't find the source of the smoke.

I decided to test the board outside the printer by feeding in a 2kHz test signal, but the measurements didn't make sense. The board seemed to be ignoring one of the inputs, so I tested that input transistor but it was fine. Next I checked the diodes, capacitors and resistors again; all the components tested okay, but the board still mysteriously failed. I started carefully measuring voltages at various points in the circuit but the signals didn't make sense and weren't consistent. Since all the components were fine but the board didn't work, I was starting to losing confidence in electronics. Eventually, I nailed down a signal that randomly jumped between 10 volts and 1 volt. After wiggling all the components, I finally noticed that the voltage jumped if I flexed the board. Finally, I had an answer: a cracked trace on the circuit board between the input and the transistor was making intermittent contact.

The board had a cracked trace in the upper left, connecting the upper gold contact. Carl put a wire jumper across the bad section.

The board had a cracked trace in the upper left, connecting the upper gold contact. Carl put a wire jumper across the bad section.

To fix the board, Carl put a wire bridge across the bad trace6. We put the board in the printer, and the printer mostly worked. However, when the printer tried to print in column 85, the column failed to print and the printer stopped with an error.5 More testing revealed four columns of the printer were failing to print due to hammer problems. Each electromagnetic hammer coil is driven by a 60 volt, 5 amp pulse for 1.5 milliseconds. This is a lot of power (300 watts), so if anything goes wrong, hammer coils can easily burn up.

We swung open one of the computer's "gates" (lower left), revealing the cards that drive the printer.

We swung open one of the computer's "gates" (lower left), revealing the cards that drive the printer.

We looked at the printer driver cards inside the computer. Each card generates pulses for two hammers, so there are 66 of these cards. In the photo below, you can see the two large high-current transistors at the left that generate the pulses. (Note the felt insulators on top of these transistors. Due to their height, the transistors pose a risk of shorting against the bottom of the neighboring card.) Just to the right of these transistors are two colorful purple and yellow fuses. In the event of a fault, these fuses are supposed to burn out and protect the hammer coils. We checked the cards associated with the four bad columns on the printer and found four burnt-out fuses.

The "AEC" Alloy-Hammer Driver Latch card produces high-current pulses to drive the printer hammer coils.

The "AEC" Alloy-Hammer Driver Latch card produces high-current pulses to drive the printer hammer coils.

Why did the fuses blow? The circuit to drive the hammer coils is a bit tricky. Every 11 microseconds, a hammer lines up with a character slug and can be fired. But when a hammer is fired, the coil needs to be activated for about 1.5 ms, a much longer time interval. To accomplish this, the hammer driver latches on when a hammer is fired. Later in the print cycle, the hammer driver is turned off. This process is controlled by the chain position counters, which are driven by the pulses from the chain sensor, the same pulses that were intermittent. Thus, if the computer received enough pulses to start printing a line, but then the pulses dropped out in the middle of the line, hammer drivers could be left in the on state until the fuse blows. This explained the problem that we saw.

After Carl replaced the fuses, the printer worked fine except for two problems. First, characters in column 85 were shifted slightly so the text was slightly crooked. Frank explained that the hammer in this column must be moving a bit too slow, hitting the chain after it had moved past its position. This explained the smoke: in the time it took the fuse to blow, the coil must have overheated and been slightly damaged. We'll look into replacing this coil next week. The second problem was that the printer's Ready light didn't go on. This turned out to be simply a bad light bulb, unrelated to the rest of the problems. In any case, the printer was working well enough for demos so the repair was a success.

Closeup of the type chain (upside down) for an IBM 1403 line printer.

Closeup of the type chain (upside down) for an IBM 1403 line printer.

I announce my latest blog posts on Twitter, so follow me at @kenshirriff for future articles. I also have an RSS feed. The Computer History Museum in Mountain View runs demonstrations of the IBM 1401 on Wednesdays and Saturdays so if you're in the area you should definitely check it out (schedule).

Notes and references

  1. You might expect that the 132 hammers align with 132 type slugs, so the matching hammers all fire at once, but that's not what happens. Instead, the hammers and type slugs are spaced slightly differently, so only one hammer is aligned at a time, and a tiny movement of the chain lines up a different hammer and type slug. (Essentially they form a vernier.) Specifically, every 11.1 microseconds, the chain moves 0.001 inches. This causes a new hammer / type slug alignment. For mechanical reasons, every third hammer lines up in sequence (1, 4, 7, ...) until the end of the line is reached; this is called a "subscan" and takes 555 microseconds. Two more subscans give each hammer in the line an option to fire, forming a print scan of 1.665 milliseconds. If you want more information on how the print chain works, I have an animation here

  2. To be precise, the printer generates a pulse if hammer 1, 2, or 3 lines up with a type slug. This is due to the three "subscans", each using every third hammer. 

  3. I'll explain how the differential amplifier works in this footnote, since most readers may not want this much detail. The computer uses two differential amplifier boards in series, first a WV board and then a WW board. They use similar principles, except the WV uses NPN transistors and the WW uses PNP. The differential output from the WW board is transmitted to the computer where a third differential amplifier (an NT card) converts the signal to a logic output. Each board is a differential amplifier, which takes two inputs and amplifies the difference, essentially an op amp with two outputs.4

    A differential pair circuit.

    A differential pair circuit.

    The basic differential pair circuit for a differential amplifier is shown above. (Op amps contain a similar differential pair.) The resistor at the top sets a fixed current I. If the two inputs are equal, the current will be split, with half going through each transistor and branch resistor. But if one if the inputs is slightly lower, that transistor will conduct more and most of the current will go through that branch. Thus, the difference between the inputs steers the current down one side or the other, yielding an amplified signal across the lower resistors.

    Schematic of the WW amplifier board from the SMS documentation.

    Schematic of the WW amplifier board from the SMS documentation.

    The IBM 1401 documentation provides the schematic above for the board, but it's hard to follow what's happening. (Note the unusual transistor symbol, three boxes with an emitter arrow in or out.) I redrew the main part of the circuit below, so it resembles the simple differential pair. It has the same resistors at top and bottom as the differential pair, but there is an R-C circuit in each branch. To simplify, if there is a DC offset or low-frequency input, the capacitor will charge and counteract this offset. Thus, the amplifier operates as a high-pass amplifier; it cuts out low-frequency noise while amplifying the 1800 Hz sync pulses. The diodes clip the output, yielding a square wave. The differential output goes through emitter-follower buffers (omitted below) so the signal is strong enough to be transmitted through an under-floor cable from the printer to the computer.

    The differential amplifier circuit of the WW card.

    The differential amplifier circuit of the WW card.

     

  4. An op amp with positive and negative outputs is known as a "fully differential op amp". 

  5. The IBM 1403 printer has multiple error checks to avoid printing incorrect data. For a business machine, it would be bad to drop digits in, say, payroll checks or tax records. To detect hammer failures, the printer has 132 wires from the hammers back to the computer, to verify that each hammer fired when it was supposed to. If the computer doesn't get a pulse back from a hammer, the computer stops immediately, as we saw. 

  6. We noticed that there was solder smeared across the broken part of the trace. My suspicion is that the same problem happened a few years ago and was repaired by bridging the broken trace with solder. Eventually the heavy vibrations inside the printer caused a hairline crack in the solder, causing the problem to recur. By bridging the break with wire rather than just solder, we hope we have fixed the problem permanently. We also noticed the transistor connected to the broken trace had been replaced, so they must have tried that first in the previous repair. 

  7. The 1403 printer is documented in IBM 1403 Printer Component Description and 1403 Printers Field Engineering Maintenance Manual. See also this brief article about the 1403 printer in the IEEE Spectrum. For details on how the timing pulses work, see the 1403 Manual of Instruction, page 42.