Repairing the card reader for a 1960s mainframe: cams, relays and a clutch

I recently helped repair the card reader for the Computer History Museum's vintage IBM 1401 mainframe. In the process, I learned a lot about the archaic but interesting electromechanical systems used in the card reader. Most of the card reader is mechanical, with belts, gears, and clutches controlling the movement of cards through the mechanism. The reader has a small amount of logic, but instead of transistorized circuits, the logic is implemented with electromechanical relays.1 Timing signals are generated by spinning electromechanical cams that generate pulses at the proper rotation angles. This post explains how these different pieces work together, and how a subtle timing problem caused the card reader to fail.

The IBM 1402 card reader/punch. The 1401 computer is in the background (left) and a tape drive is at the right.

The IBM 1401 was a popular business computer of the early 1960s, used for applications such as payroll or inventory. Data records were stored on 80-column punch cards, which were read into the computer by the card reader and printed on the high-speed line printer. The IBM 1401's card reader (above) could read 800 cards per minute—over 13 cards per second—which is a remarkable speed for a mechanical device. Cards whizzed through the card reader, 80 metal brushes read the holes in each column, and then the cards were stacked in the hoppers in the middle of the reader.3 If anything went wrong, from a misread card to a card jam, the card reader would slam to a halt (hopefully before bent cards started piling up) and an error light would illuminate on the card reader's control panel. At this point, the operator would fix the problem and processing could continue.2

80-column punch cards. Each column of the card holds a character, represented by the holes punched in the column. THE text at the top of the card shows what is in each column. Black paper behind the first card makes the holes more visible.

The photo below shows the IBM card reader/punch with the front doors opened up. The right half is the reader and the left half is the punch. (The punch records data on cards; I will ignore it in this article.) The blue panel has the operator controls and indicators. Cards enter through the feeder in the upper right, travel right-to-left through the brushes (behind the blue panel) and fall into the stacker slots (center). To the right of the stacker is the service panel with the dynamic timer (circle with knob); this will be relevant later. The relays, resistors, capacitors and diodes that make up the reader's circuitry are below the service panel. Converting the hole pattern to characters was done by the 1401 computer, not the reader, so this circuitry is fairly basic.3 The garbage can on the lower left collects "chips", the bits punched out of cards by the punch, and triggers a warning light when it is full.

With the front doors of the card reader/punch removed, the circuitry is visible. The left half is the punch and the right half is the reader. The *Reader Stop* light is illuminated on the panel, indicating a malfunction.

A few months ago a problem with the card reader kept showing up under certain circumstances: the card reader would halt with a "Reader Stop" error, indicating a problem with card feeding and travel.2 Given the age of the card reader at the Computer History Museum, it's not surprising that Reader Stop problems arise. However, in this case, cards were feeding fine and there was nothing visibly wrong. Eventually the 1401 maintenance team determined that the problem occurred if you do multiple cycles of reading a card, printing a line on the printer, reading a card, printing a line, and so on. The first three cards read fine, but the fourth card read always failed with a Reader Stop. But during normal processing, hundreds of cards could be read without a problem. What was different about repeatedly reading and printing?

The card reader can read 800 cards per minute, but the line printer can only print 600 lines per minute. Thus, if you continuously read and print, the printer will eventually fall behind. The consequence is that the card reader will occasionally be blocked while the printer finishes a line. In particular, the system can read and print three cards in a row, but there is a delay before reading the fourth card.4 This is normal behavior for the system, but for some reason, this delay was triggering the Reader Stop error.

The question at this point of the investigation was why the delayed read triggered a Reader Stop. The logic diagram below shows the many different things can cause a Reader Stop.5 (This logic was implemented in the reader by relay circuits of Byzantine complexity.) Usually a card jam or card feed problem is responsible for a stop, but there weren't any card movement problem. And the card levers (microswitches) that detect card movement were operating correctly. Eventually, the team discovered that removing relay R-10 (Read Clutch Impulse near the bottom of the diagram) stopped the Reader Check from happening. Removing this relay prevented the clutch status from being checked8 so it prevented the error from being reported but didn't fix the underlying problems. It did, however, indicate that the problem was with the clutch.

Diagram showing the logic conditions that yield a Reader Stop error in the card reader. From the manual.

At this point, I'll explain how the clutch works in the card reader. The card reader is powered by a 1/4 horsepower electric motor. This motor powers a variety of shafts, belts, rollers and cams inside the card reader. However, the reader needs a mechanism to rapidly start and stop card reading under computer control. This is accomplished by a clutch that powers the card feed rollers. With the clutch engaged, cards will be fed through the card reader. With the clutch disengaged, cards will not move (although other parts of the reader will keep running). The GIF below shows the clutch in action: it is disengaged for one cycle and engaged for one cycle. Although the silver pulley (driven by the motor) turns continuously, the black gear is started and stopped by the clutch, which is just behind the black gear.

The clutch in the 1402 card reader. The clutch is disengaged for one cycle and engaged for one cycle.

The diagram below shows the main parts of the clutch. The ratchet (green) continuously rotates, powered by the motor. When the coil (orange) pulls the latch (purple), the dog (blue) will fall into the ratchet (green), engaging the clutch. This will cause the outer wheel to rotate in sync with the ratchet, feeding a card. After one cycle, the clutch can be disengaged by the latch, or can remain clutched for another cycle. Above, you can see the latch move just before the clutch engages. The dogs and ratchet are hidden behind the black gear. For more details, see the footnotes.67

The card reader's clutch mechanism. From the 1402 manual.

The clutch requires careful timing. If the coil pulls the latch too late, the dog won't engage with the ratchet until the next cycle and no card will be fed. And if the latch is released too late, the clutch won't disengage until the next cycle, feeding an extra card. Either way, this would cause a serious problem if you are, for instance, printing payroll checks. To catch this, the card reader includes a clutch check circuit that ensures that a clutch happens only when requested.8 This circuit is what triggered the Reader Stop issue.

Timing in the card reader is controlled by the angular position of the main drive shaft; there isn't a clock signal. A 360° rotation of the drive shaft corresponds to one card read cycle.9 The mechanical and electrical components must be precisely adjusted so they activate at the correct rotational angles. For instance, a card's first row of holes reaches the brushes at 8° and card lever #1 is triggered at 227°. And the clutch is engaged or disengaged at exactly 315°. Electrical signals are generated by rotating cams that open and close microswitches at the appropriate degree angles. For example the read clutch is controlled by cam RC6 that closes at 272° and opens at 342°. The photo below shows a cam on a drive shaft, with the microswitch above it; when the cam rotates to a high point, the switch closes.

One of the cams and microswitches in the card reader.

To examine timings, the card reader has an interesting maintenance feature called the Dynamic Timer, which is sort of a mechanical oscilloscope that visually shows the angle timing of signals. The dynamic timer consists of a neon bulb that spins behind a plastic disk labeled with the angles from 0° to 359°. If a signal is selected with a control panel switch, the bulb lights up when the signal is active.1 Because the neon bulb is spun by the main driveshaft, its position is synchronized to the rotation angle of the card reader's mechanisms. Thus, the neon bulb traces out sectors of a circle, indicating when the signal is active. For example, the image below shows the read brush signal, which is activated as each card row passes under the brushes. Since there are 12 rows on a punch card, the signal is activated 12 times in a card cycle. By reading the angle labels on the clear plastic dial, the signal timings can be checked for correctness. In addition, the knob in the middle of the dynamic timer can be rotated manually to move the card reader's cycle to any desired angle.

The dynamic timer of the 1402 card reader. The 12 signals correspond to the 12 rows on the card, indicating when the row passes through the brushes.

We measured the timing of read clutch control cam RC6 and found it was off by about 5 degrees. The manual called the timing of this cam "critical" saying it needed to be within 2° earlier or 0° later from the correct angle.10 Since the cam was well outside the allowed range, we tried to adjust it. The first problem was that the cam was frozen to the shaft; apparently after decades of storage in a a damp garage it had rusted to the shaft. Carl eventually hit the cam hard enough (yet carefully!) with a hammer to loosen it from the shaft so it could be adjusted.

The photo below shows how we manually adjusted cam timings. The dynamic timing wheel (left) can be turned by hand until the cam closes, and then the angle can be read from the dial. The adjustment process is rather tedious, requiring trial and error. First, the duration that contact is closed is corrected by moving the contact closer or farther from the cam. After each adjustment, the duration must be manually checked by slowly rotating the dynamic timing wheel 360°. Once the duration is correct, the next step is to rotate the cam a few degrees at a time until it closes and opens at the right angle. Again, the timing must be carefully checked after each adjustment until trial-and-error yields the right cam setting.

By manually rotating the dynamic timer wheel, the cam timing (center) can be checked. The light (upper right) shows when the cam has closed. This photo shows cam RL10 being adjusted.

Once cam RC6 had the proper timing, we tested the card reader. Surprisingly, it failed just as before. We checked the timing of RC6 with the dynamic timer, and strangely, the timer still showed that it closed 5° too late, even though we had measured it as closing correctly. After puzzling over the schematics, I found that cam RC6 was powered through cam RL10, so a bad timing on cam RL10 would cause the dynamic timer to show the wrong time for RC6.

We continued the next week and measurement of RL10 showed it was closing 5° too late. So we repeated the cam adjustment process with RL10. Carl loosened RL10's setscrews, whacked it with a hammer, and was surprised when it spun around on the shaft—apparently it hadn't rusted onto the shaft the way RC6 had. Since the timing of RL10 was now thoroughly off, it took some extra effort to find the right position for it. One complication is that RL cams only rotate while the clutch is tripped, so we had to manually trip the clutch for each test. (This video shows how the clutch can be tripped manually.) After a couple hours of adjustment, the cam had the right duration and then the right start angle.

We powered up the card reader and this time it ran all the tests successfully! We had found the elusive problem. Apparently cam RL10 was open a bit too much, so the signal to read a card after a pause in reading wasn't quite able to engage the clutch. Then the clutch check circuitry would detect that the clutch had failed to engage, triggering the Read Stop.

Conclusion

Tracking down and fixing the mysterious Reader Stop errors on the card reader was tricky and more time-consuming than most problems. But it provided an interesting look back in time at the obsolete technology used inside the card reader: relay logic, spinning cams and a mechanical clutch. Debugging the card reader is a different experience from tracking down software problems or even normal hardware problems. Just understanding the schematics with their obsolete symbols can be a challenge. Hopefully what we learned in this repair will help us the next time the card reader malfunctions.

Many members of the 1401 restoration team were involved in fixing this problem, including Carl, Frank, Alexy, Ron, George, Glenn, Jim and Grant. The Computer History Museum in Mountain View runs demonstrations of the IBM 1401 on Wednesdays and Saturdays so check it out if you're in the area; the demo schedule is here.

Follow me on Twitter or RSS to find out about my latest blog posts.

Notes and references

1. While IBM's computers were supposed to be transistorized by the time of the 1401, two vacuum tubes are hidden deep inside the card reader. These tubes power the neon bulbs for the dynamic timer used for diagnostics. The photo below shows the tubes and the transformer that powers them.

   ↩

   The 1402 card reader contains two vacuum tubes to drive the neon bulbs in the dynamic timer.

2. The 1402 card reader reports three types of errors. A Reader Stop indicates a jam or card feed problem. (This is the error we encountered.) A Validity error indicates an invalid character caused by a bad hole pattern. A Reader Check indicates that the card was read incorrectly. An interesting technique was used to detect a bad read. The first set of brushes was used to count the number of holes in each column. The second set of brushes performed the actual read of the characters. If the number of holes in a column was different between the first and second reads, the system knew that the card had been read incorrectly. (This was typically caused by a brush that was slightly out of alignment or making bad contact.) You might wonder why the holes were counted, instead of just checking the characters. Counting holes required fewer bits of expensive core memory. ↩

3. The card reader itself didn't convert the hole pattern into characters. Instead, there were 80 parallel wires directly from the 80 hole-sensing brushes to the computer, which did the hole-to-character conversion. The reader had one set of brushes to count holes (for the error check) and a second set to actually read the characters. The punch had a set of brushes to check the punched holes and optionally a set of brushes to read before punching. Thus, there could be up to four sets of brushes in the reader/punch. In addition, the punch had 80 wires from the computer to the 80 hole punch coils. As a result, two massive 200-wire cables connected the card reader and the computer. ↩

4. The 1401 computer has a core-memory print buffer that allows computation to continue while printing a line. However, a complex interlock circuit will block the computer if it tries to print a line while the buffer is still in use. (This buffering and locking is all done in hardware; there is no operating system.) The print buffer was an optional feature for the computer, renting for an extra $386 per month. ↩

5. The Reader Stop diagram shown earlier is somewhat confusing, but I'll give an overview. As cards pass through the card reader, they trip microswitches (called Card Levers or CLC). The logic tests if a microswitch isn't triggered, indicating if a card got jammed before reaching the microswitch. For example, the second triangle triggers an error if there are cards in the hopper, but no card reached card lever 1 after a feed signal. (Note that the logic symbols are backwards compared to modern usage; IBM used a semicircle for OR, and a triangle for AND. Thus, each set of inputs into a triangle shows a set of conditions that will trigger a reader stop.)

   A label like R-6 indicates that the signal comes from relay 6. Relay R-13 is the Reader Stop Control relay; it gets triggered if there was a card at microswitch 1 or 2 during angle 158°-188°. During this time, the card should have already passed the microswitch, so if it is still closed, there must be a jam. "CLC 1 DELAY" indicates that there was a card at CLC 1 in the previous cycle. If there is no card at CLC 2 this cycle, the card must have been jammed along the way. ↩

6. Several documents on the 1402 card reader/punch are available on bitsavers. The IBM 1402 Card Read-Punch Manual is an operator's manual for the card reader. The Customer Engineering Manual of Instruction explains the implementation of the card reader for the service engineer, and is probably the best guide to how it operates. The Customer Engineering Instruction Reference Manual explains the internals of the card reader, with a focus on maintenance. The Wiring Diagram provides schematics of the card reader. The Service Index provides a summary of information for servicing. ↩

7. The operation of the clutch mechanism is a bit tricky. In the diagram below, the ratchet (green) is constantly spinning. The remaining parts of the clutch are connected together and rotate only when the clutch is engaged. The key idea is that when the intermediate arm (middle) is latched, the control studs push the dogs up and out of the ratchet disengaging the clutch. (This happens because the intermediate arm can pivot slightly relative to the drive arm (left), so the dog pivot and control stud will move relative to each other, swinging the dogs upwards.) When the intermediate arm is unlatched, the dogs are no longer raised and they will engage with one of the ratchet's teeth. At this point, the mechanism is clutched and everything rotates in synchronization, until the intermediate arm hits the latch again, disengaging the dogs from the ratchet. The video of a simpler ratchet clutch may clarify the dog and ratchet mechanism.

   ↩

   Diagram of the card reader's ratchet clutch, from the service manual.

8. In case anyone (most likely me, in the future) needs to understand the clutch check circuit, I'll give an explanation. The reader sends a "Proc Feed" signal to the computer each time a card could be fed. If the computer wants to read a card, it sends a "Read Clutch" signal back. This signal is gated by cams RL10 and RC6 and cam RC5 and does two things. It triggers the read clutch magnet, the coil that pulls the clutch latch. It also energizes the clutch check relay R10. (See the schematic for details.) A bit later in the cycle, the clutch status is checked using the circuit below. If the clutch check relay is activated but the clutch did not latch, cam RL10 (not shown) will not be rotating and will remain closed. RL10 will trigger the Read Stop relay R4 through RC7 and R10's normally-open contacts. On the other hand, if the clutch check relay is not activated, but the clutch is latched, cam RL2 will be rotating and will trigger the Read Stop relay through the Read Stop relay R4's normally-closed contacts.

   ↩

   The clutch check circuit in the card reader uses cams and relays to trigger a Read Stop if the clutch fails to latch. From the 1402 schematic.

9. Although one card can be read every 360° cycle of the reader, it takes three cycles to process a particular card. During the first cycle, the card is moved out of the hopper into the reader and triggers card lever 1. During the second cycle, the card passes through the first set of brushes (which counts the holes in each column), and triggers card lever 2. During the third cycle, the card passes through the second set of brushes (which actually reads the card), and is ejected into a hopper. Thus, there are typically three cards in flight in the reader at any time. (This is one reason the reader has so much circuitry to detect jams quickly, to prevent multiple cards from piling up in a crumpled heap.) ↩

10. Many cams (including RC6) actually have three lobes, so they open and close three times every 360°, spaced 120° apart. This is due to an optional feature called Early Read (which cost an extra $10 per month). On the basic card reader, if the computer requests a card too late in the cycle, it has to wait for up to an entire revolution (75ms) until the clutch can latch, slowing down the system. With Early Read, there are three clutch points per revolution, cutting down the wasted time to at most 25ms. (One complication: the clutch is geared to half the speed of the regular shaft and turns 180° per cycle. This is why there are 6 teeth on the clutch ratchet.) ↩

An 8-tube module from a 1954 IBM mainframe examined: it's a key debouncer

IBM's vacuum tube computers of the 1950s were built from pluggable modules, each holding eight tubes and the associated components. I recently came across one of these modules so I studied its circuitry. This particular module implements five contact debouncing circuits, used to clean up input from a key or relay. When you press a key, the metal contacts tend to bounce a bit before closing, so you end up with multiple open/closed signals, rather than a nice, clean signal. The signal needs to be "debounced" to remove the extra transitions before being processed by a computer. (Perhaps you have used a cheap keyboard that sometimes gives you duplicated letters; excessive key bounce causes this.)

Front of an IBM tube module. This module contains five key debouncing circuits.

The module is apparently from an IBM 705, a powerful business computer introduced in 1954.1 The 705 was a big computer, weighing 16 tons. It contained 1700 vacuum tubes and consumed 70 kilowatts of power, with 40 tons2 of air conditioning to take away the heat from the tubes. A typical system cost $1,640,000 ($15 million in 2017 dollars), but was normally rented monthly for $33,500 ($300,000 in 2017 dollars). A few dozen 705 systems were built, mostly used by large companies and the US government. 8

The CPU of an IBM 705 was a massive unit. Photo from Farmers Insurance, showing their first computer.

IBM introduced multi-tube pluggable modules in 1953 as part of the 700 series of computers.3 Pluggable modules were an innovation that simplified manufacturing and maintenance, as well as providing a way to pack circuitry densely. Computers were built from hundreds of pluggable modules of many different types. The photo below shows how the 8-tube modules were packed in the IBM 709.

Closeup of tube modules in the IBM 709. Note that the modules are installed "sideways" with the tubes horizontal.

IBM hoped that they could manufacture a small set of common pluggable modules, but this didn't work out since the computers required many different types of modules. Instead, IBM settled for modules that were built from standardized design rules and circuits. The main circuit classes were the inverter, the flip flop (which IBM called a "trigger"), diode logic,6 the multivibrator (pulse generator), and the cathode follower (buffer).75 Unfortunately even this level of standardization didn't work too well. For instance, they ended up with dozens of different types of inverters for special cases, everything from an "Open-filament neon inverter" to a "Line capacity yanker". Thus, each computer ended up with hundreds of different types of tube modules, built from a multitude of circuits.4

A single tube module could contain a couple dozen of these simple circuits. You might wonder how eight tubes could support so many circuits, but there were two factors. First, the tubes were typically dual triodes in one package, so the module had the equivalent of 16 simple tubes. Also, AND and OR logic gates were implemented in the module with compact semiconductor diodes, so complex Boolean logic could be implemented almost for free.

I've seen claims that an 8-tube module represents an 8-bit byte, but that's not the case. The eight tubes don't generally map onto eight of anything since functions can take more or less than one tube. For instance, the module I examined contains five circuits. I have another module that stored three bits. Also, the byte wasn't a thing back then. These computers used 36-bit words (scientific models), or 6-bit characters (business models) so 8 bits was irrelevant.

Vacuum tubes

This module was built from a type of vacuum tube known as a triode. In a triode, electrons flow from the cathode to the plate (or anode), under the control of the grid. The heater, similar to a light bulb filament, heats the cathode to around 1000°F, causing electrons to "boil" off the cathode. The anode has a large positive voltage (e.g. +140V), which attracts the negatively-charged electrons. The grid is placed between the cathode and the anode. If the grid is negative, it repels the electrons, blocking the flow to the plate. Thus, the triode can act as a switch, with the grid turning on and off the flow of electrons. The module I examined used dual triode tubes, combining two triodes into one tube for compactness. 9

Schematic symbol for a triode tube

Vacuum tubes required inconveniently large voltages; this module used -130V. -60V, +140V and 6.3VAC. Tubes also had high power consumption—note the many large (1 watt) resistors in the module. The filament was hot enough to glow, using a couple watts per tube. In total, each tube module probably used dozens of watts of power.

A 5965 vacuum tube from the module. The pins plug into a socket on top of the tube module. This tube is a dual-triode; the two plate structures are visible.

The module's circuitry

I closely examined the tube module and found that it consisted of five copies of the same circuit. I traced out one of these circuits, which was a bit inconvenient because the module has components on both the front and the back. I came across a manual of 700-series circuits, and found a circuit that was an exact match, down to the values of the resistors. The circuit is called a "Contact-Operated Trigger" and was used to interface a mechanical input (key, relay, or cam) to electronic circuits.1

Schematic of one "trigger" circuit of the tube module.

The schematic above shows one of the trigger stages. The basic idea of this circuit is a resistor-capacitor filter (left) smooths out the input, removing any short glitches. This signal goes through two inverter circuits, creating a sharp output. Both inverters are part of the same 6211 tube and are wired as a Schmitt trigger.12 To understand how the inverter works, when the input (pin 2) is high, the current through the tube pulls the plate (pin 1) low. Conversely, when the input is low, the electron flow is blocked and the resistors pull the plate high.10 The output from the first inverter (plate, pin 1) is fed into the second inverter (grid, pin 7). This inverter works similarly, with its output on pin 6. The final output is buffered by a "cathode follower" circuit (not shown) to drive other modules. It also drives a neon bulb for a status indicator,

The tube module showing how the five debounce triggers are divided among the tubes.

The tube module contains five of the above debounce circuits. The diagram above shows how these circuits map onto the eight tubes. Each cathode follower (CF) uses half a tube so one tube implements two cathode followers.11

One amusing component I found in the tube module was "Vitamin Q" capacitors. Presumably this is related to the quality factor or Q factor in radio, and these capacitors were designed to improve the Q.

A Vitamin Q capacitor in an IBM 705 tube module.

A brief history of vacuum tube computing, with a focus on IBM

It took longer to get from the invention of the vacuum tube to the development of electronic computer than you might expect. The triode vacuum tube was invented in 1907 and started being used for radio in 1912. The vacuum tube flip-flop, able to store one bit of data was invented by Eccles and Jordan in 1918, but it wasn't until about 1937 that flip flops were connected together into binary counters.

IBM started investigating vacuum tube counters for calculations in 1941 and built an experimental vacuum tube multiplier in 1942. In 1948, IBM introduced the IBM 604 "Electronic Calculating Punch", which implemented high-speed arithmetic operations for punch-card systems by using vacuum tubes. It was IBM's first product with pluggable tube units, using simple modules with one or two vacuum tubes in each module (as seen below).14

Two tube modules from the IBM 604. By Jud McCranie CC BY-SA 4.0, via Wikimedia Commons

Meanwhile, vacuum tube computers were being invented, with Atanasoff's linear equation solver built in 1942 and the Colossus codebreaking system in 1943. But it wasn't until the programmable ENIAC computer was announced in 1946 that the vacuum tube computer revolution really started, leading to the development of numerous computers by the early 1950s.

In 1952, announced the 701, IBM's first commercial scientific computer. The 701 introduced multi-tube pluggable modules, the focus of this article. These modules were used in the 700-series computers until they were obsoleted in 1958 by IBM's transistorized 7000-series computers, which used germanium transistors. The transistorized computers were built from printed circuit boards the size of a playing card, called SMS cards.13 Thus, although vacuum tube computers were highly influential, their lifespan was short. (Transistorized computers also had a short lifespan of a bit over a decade. Integrated circuits, on the other hand, have been powering computers for more than 50 years, with no signs of being replaced.)

A tube module implements more logic than an SMS card, but the cards are much smaller and use less power. Photo shows a "DGT" SMS card implementing four AND gates on top of a different tube module.

How the modules are built

One important advantage of the tube module is the circuitry is somewhat three-dimensional, compared to components soldered on a circuit board. This allows the circuitry to more efficiently fill the volume of the computer. The module has four layers of contact boards with terminal strips attached, allowing four vertical layers of components (resistors, capacitors, etc) to be soldered to the terminal strips.5 The solder terminals are connected together by thin horizontal metal strips, which can be cut. Thus, neighboring tabs can be electrically connected or not, as required by the circuit. Jumper wires are used to provide additional connections as needed.

The module has four levels of components between the tubes and the connector. In addition, there are components on the front and back of the module.

The tube module has 64 metal contact tabs at the bottom, providing connections for power and signals. The tube module doesn't simply plug into the socket as you might expect, but uses a two step mechanism. First, the module is placed in the socket but the pins don't make contact. You may have noticed the rod sticking up from the top of the module. To lock the module into place, the rod is rotated 180 degrees with a special tool, causing a circular cam at the bottom of the module to rotate. The force of the cam against the socket causes the module to slide sideways so the pins mesh with the socket contacts. This reduces the force required to insert the module and minimizes the risk of the module coming loose.

The module has 64 metal tabs that plug into the socket. The circular cam (rusted) rotates to slide the module (and the tabs) sideways into the socket.

The contacts are carefully positioned to establish the ground connection first, then the negative-voltage connections and finally the positive connections. This allows hot-swapping, replacing a tube module while the system is powered up. (The manual does say, "However, it is wise to drop DC voltages before removing or installing pluggable units.") Interestingly, USB connectors use the same idea. If you look at a USB connector, you'll see that the middle two contacts (data) are shorter than the outside contacts (power and ground), so power and ground connections are established first.

Patent 2,754,454 describes the tube module mechanism in detail.

The tube module has locking tabs that holds the module into place after it slides sideways. I noticed that these tabs are broken off on the module I examined; compare with the intact module below. Probably the rusted cam couldn't be moved, so the module had to be forced out of the socket, breaking the tabs.

When the tube module slides sideways into the connectors, the locking tabs engage to keep it from being pulled out. In this tube module, the locking tabs are broken off.

More evidence of the module's hard life is there are at least four resistors missing from the module, and two resistors with broken solder joints. (This made it considerably harder to figure out what the module did.) Positions where two components are soldered to the same tabs seem especially prone to having a component knocked off. The photo below illustrates one of the missing resistors; one trigger circuit has a 91K (white-brown-orange) resistor soldered across the capacitor, while it is missing in the second circuit.

The arrow shows one of the missing resistors in the tube module.

Conclusion

The tube module is an interesting artifact from the era of vacuum tube computers. Studying it revealed its function, five contact debouncing circuits for use with keys or relays. Part II of this blog post will describe powering up the module and getting it to work. For a sneak preview, you can watch CuriousMarc's video:

Thanks to Carl Claunch for providing the module.

Follow me on Twitter or RSS to find out about my latest blog posts.

Notes and references

1. I found the circuit for this module in a manual of circuits for IBM's 700-series computers (page C35). Chapter C has information on circuits for the 702 and 705 business systems (which use different voltages from the scientific 701, 704 and 709 systems described in chapter B). Since the 705 was much more popular than the 702, it's more likely that the module is from a 705. ↩

2. Strangely, a ton of refrigeration means cooling equal to one ton of ice per day, not an air conditioner that weights one ton. This unit probably made more sense in the 1900s when people were switching from ice-based cooling to refrigeration. In more sensible units, one ton of refrigeration is 12,000 BTU/hour or about 3500 Watts. For comparison, a typical window air conditioner is one ton of cooling, while a central home air conditioner is 1.5 to 5 tons of cooling. ↩

3. Several detailed articles on the IBM 701 computer, written by the designers in 1953 are here.  ↩

4. IBM went through a similar failed standardization process in the 1960s with their transistorized Standard Modular System (SMS) cards. They hoped to build computers from a small number of standard cards but ended up with thousands of different cards. ↩

5. IBM's 700 Series Data Processing System Component Circuits provides extremely detailed information on the circuits used in tube modules. If you want to know all about tube modules, this is the document to read. A couple other pages of interest are the IBM 709 CPU Diagrams with schematics of the 709's tube modules and IBM 705 Drawings with a few 705 tube modules. The 1952 patent, Multiple Pluggable Unit, describes the mechanical structure of the tube modules in detail. A few drawings of 705 circuits are here, and 709 drawings are here. Bitsavers has some information on the 704 including details of the circuitry, but not to the tube level. Bitsavers' information on the 701, 702, 705 and 709 is less relevant. ↩

6. The development of the semiconductor diode significantly reduced the size of computers, as small diodes could perform logic functions (AND, OR) that previously required vacuum tubes. The IBM 706 contained many more diodes than vacuum tubes: 4600 germanium diodes versus 1700 vacuum tubes. I think the importance of diodes is underestimated, with people focusing on tubes versus transistors. ↩

7. The tube modules are described in more detail in the book IBM's Early Computers, pages 147-151. ↩

8. The BRL Report has details on most computer systems as of 1961, including the 705. Take a look at this report if you want to know the size, price, features, or number of components in old computers. ↩

9. The module uses two types of vacuum tubes: five 6211 tubes and three 5965 tubes. Both tubes were designed for computer applications, able to handle the stress of constantly turning signals on and off (unlike radio applications, where tubes handled analog signals). They are both dual triodes, but the 5965 is a higher power tube. The 5965 tube is IBM number 317261. The 6211 tube is IBM number 252551. The 5965 tube handles up to 300V, while the 6211 only 200V. The 5965 also handles more current and has higher gain, which is probably why it is used in the cathode follower driver circuit. For full details on the tubes, see the 5965 datasheet and the 6211 datasheet. These tubes are both similar to the more common 12AT7 tube. ↩

10. The various resistors adjust (biasing) the voltage levels from the high plate voltage to the lower voltages used by the grid. (That's why the -130V supply is needed, to pull the voltage down enough.) ↩

11. The triggers are implemented with type 6211 tubes, while the cathode followers are implemented with the more powerful 5965 tubes. From the photo, it may appear that the tubes don't match the locations since there are four tubes with metallized tops and four tubes with metallized sides. However, the tube markings indicate that all tubes were in the right locations. The location of the shiny getter is independent of the tube type. ↩

12. The tube is wired as a Schmitt trigger. When the input becomes sufficiently high, it will rapidly switch to the on state. The input will need to fall to a lower level, at which point it will rapidly switch to the off state. The key to the Schmitt trigger is the resistor connected to the cathodes. When one tube turns on, the voltage drop across the resistor will raise the the cathode voltage close to the grid voltage. This will force the other tube off. You can also look at it as a differential amplifier, where the higher grid "wins". ↩

13. If you want a quick summary of IBM's machines from the 701 through the 370, see The Architecture of IBM's Early Computers and "System/360 and Beyond". ↩

14. The book IBM's Early Computers has extremely detailed information on IBM's pre-360 machines. Also see IBM's web page on the 603. ↩

Repairing a 1960s mainframe: Fixing the IBM 1401's core memory and power supply

A few weeks ago, I wanted to use one of the vintage IBM 1401 mainframe computers at the Computer History Museum, but the computer wasn't working.1 This article describes the multi-week repair process to get the computer working again.

The problem started when the machine was powered up at the same time someone shut down the main power, apparently causing some sort of destructive power transient. The computer's core memory completely stopped working, making the computer unusable. To fix this we had to delve into the depths of the computer's core memory circuitry and the power supplies.

The IBM 1401 computer. The card reader/punch is in the foreground. The 12K memory expansion box is partially visible to the right behind the 1401.

Debugging the core memory

The IBM 1401 was a popular business computer of the early 1960s. It had 4000 characters of internal core memory with additional 12000 characters in an external expansion box.2 Core memory was a popular form of storage in this era as it was relatively fast and inexpensive. Each bit is stored in a tiny magnetized ferrite ring called a core. (If you've ever heard of a "core dump", this is what the term originally referred to.) The photo below is a magnified view of the cores, along with the red wires used to select, read and write the cores.4 The cores are wired in an X-Y grid; to access a particular address, one of the X lines is pulsed and one of the Y lines is pulsed, selecting the core where they intersect.3

Detail of the core memory in the IBM 1401. Each toroidal ferrite core stores one bit.

In the 1401, there are 4000 cores in each grid, forming a core plane that stores 4000 bits. Planes are then stacked up, one for each bit in the word, to form the complete core module, as shown below.

The 4000 character core memory module from an IBM 1401 computer. Tiny ferrite cores are strung on the red wires.

To diagnose the memory problem, the team started probing the 1401 with an oscilloscope. They checked the signals that select the core module, the memory control signals, the incoming addresses, the clock signals and so forth, but everything looked okay.

The next step was to see if the X and Y select signals were being generated properly. These pulses are generated by two boards called "matrix switches", one for the X pulse and one for the Y pulse.5 Some address lines are decoded and fed into the X matrix switch, while the other address lines are decoded and fed into the Y matrix switch. The matrix switches then create pulses on the appropriate X and Y select lines to access the desired address in the core planes.

The photo below shows the core memory module and its supporting circuitry inside the 1401. The core memory module itself is at the bottom, with the two matrix switch boards mounted on it. Above it, three rows of circuit boards (each the size of a playing card) provide the electronics. The top row consists of inhibit drivers (used for writing memory) and the current source and current driver boards (providing current to the matrix switches). The middle row has 17 boards to decode the memory addresses. At the bottom 19 sense amplifier boards read the data signals from the cores. As you can see, core memory requires a lot of supporting electronics and wiring. Also note the heat sinks on most of these boards due to the high currents required by core memory.

Inside the IBM 1401 computer, showing the key components of the core memory system.

After some oscilloscope measurements, we found that one of the matrix switches wasn't generating pulses, which explained why the memory wasn't working. We started checking the signals going into the matrix switch and found one matrix switch input line showed some ringing, apparently enough to keep the matrix switch from functioning.

Since the CHM has two 1401 computers, we decided to swap cards with the good machine to track down the fault. First we tried swapping the thermal switch board (below). One problem with core memory is that the properties of ferrite cores change with temperature. Some computers avoid this problem by heating the core memory to a constant temperature in air (as in the IBM 1620 computer) or an oil bath (as in the IBM 7090). The 1401 on the other hand uses temperature-controlled switches to adjust the current based on the ambient temperature. We swapped the "AKB" thermal switch board (below) and the associated "AKC" resistor board, with no effect.

The core memory uses a thermal switch board to adjust the current through core memory as temperature changes. The switches open at 35°C, 29°C and 22°C. The type of the board (AKB) is stamped into the lower left of the board.

Next we tried swapping the "AQW" current source boards that control current through the matrix switches.6 We swapped these board and the 1401's memory started working. Replacing the original boards one at a time, we found the bad board, shown below.

The IBM 1401 has four "AQW" cards that generate currents for the core memory switches. This card had a faulty inductor (the upper green cylinder), preventing core memory from working.

I examined the bad board and tested its components with an multimeter. There were two 1.2mH inductors on the board (the large green cylinders). I measured 3 ohms across one and 3 megaohms across the other, indicating that the second inductor had failed. With an open inductor, the board would only provide half the current. This explained why the matrix switch wasn't generating pulses, and thus why the core memory didn't work.

I gave the bad inductor to Robert Baruch of Project 5474 for analysis. He found that the connection between the lead and the inductor wire was intermittent. He dissolved the inductor's package in acid and took photographs of the winding inside the inductor.7

The faulty inductor from the IBM 1401 showing the failed connection.

We looked in the spare board cabinet for an AQW board to replace the bad one and found several. However, the replacement boards were different from the original—they had one power transistor instead of two. (Compare the photo below with the photo of the failed card from the computer.)

The replacement AQW card had one transistor instead of two, but was supposedly compatible with the old board.

Despite misgivings from some team members, the bad AQW card was replaced with a one-transistor AQW card and we attempted to power the system back up. Relays clicked and fans spun, but the computer refused to power up. We put the old card back (after replacing the inductor), and the computer still wouldn't start. So now we had a bigger problem. Apparently something had gone wrong with the computer's power supplies so the debugging effort switched focus.

Diagnosing the power supply problem

The power supply system for the IBM 1401 is more complex than you might expect. Curiously, the main power supplies for the system are inside the card reader; a 1250W ferro-resonant transformer in the card reader regulates the line input AC to 130V AC, which is fed to the 1401 computer itself through a thick cable under the floor. Smaller power supplies inside the 1401 then produce the necessary voltages.

Since it was built before switching power supplies became popular, the IBM 1401 uses bulky linear power supplies. The photo below shows (left to right) the +30V, -6V, +6V and -12V supplies.8 In the lower left, under the +30V supply, you can see eight relays for power sequencing. The circuit board to the right of the relays is one of the "sense cards" that checks for proper voltages. Under the +6V supply is a small "+18V differential" supply for the core memory. Foreshadowing: these components will all be important later.9

Power supplies in the IBM 1401.

After measuring voltages on the multiple power supplies, the team concluded that the -6V power supply wasn't working right. This was a bit puzzling because the AQW card (the one we replaced) only uses +12 and +30 volts. Since it doesn't use -6 volts at all, I didn't see how it could mess up the -6 volt supply.

Inside the IBM 1401's -6V power supply.

The team removed the -6V supply and took it to the lab. In the photo above, you can see the heavy AC transformer and large electrolytic capacitors inside the power supply. Measuring the output transistors, they found one bad transistor and some weak transistors and decided to replace all six transistors. In the photo below, you can see the new transistors, mounted on the power supply's large heat sink. These are germanium power transistors; the whole computer is pre-silicon.

The -6V power supply from the IBM 1401 uses six power transistors on a large heat sink.

The -6V power supply tested okay in the lab with the new transistors, so it was installed back in the 1401. We hit the "Power On" button on the console and... it still didn't work. We still weren't getting -6V and the computer wouldn't power up.

In the next repair session, we tried to determine why the computer wasn't powering up. Recall the eight relays mentioned earlier; these relays provide AC power to the power supplies in sequence to ensure that the supplies start up in the right order. If there is a problem with a voltage, the next relay in the sequence won't close and the power-up process will be blocked. We looked at which relays were closing and which weren't, and measured the voltages from the various power supplies. Eventually we determined that about halfway through the power-up process, relay #1 was not closing when it should, stopping the power-up sequence.

Relay #1 was driven by the +30V supply and was activated by a "sense card" that checked the +6V supply. But the +30V and +6V supplies were powering up fine and the sense card was switching on properly. Thus, the problem seemed to be a failure with the relay itself. Just before we pulled out the relay for testing, someone found an updated schematic showing the relay didn't use the regular +30V supply but instead obtained its 30 volts through the "18V differential supply".11 And the schematic for the 18V differential supply had a pencilled-in fuse.10

Could the power problem be as simple as a burnt-out fuse? We opened up the 18V differential supply, and sure enough, there was a fuse and it was burnt out. After replacing the fuse, the system powered up fine and we were back in business.

The 18V differential power supply in the IBM 1401 provides 12 volts to the core memory. The fuse is under the large electrolytic filter capacitors.

With the computer operational, I could finally run my program. After a few bug fixes, my program used the computers's reader/punch to punch a card with a special hole pattern:

A punch card with "Merry Xmas" and a tree punched into it.

Happy holidays everyone!12

Conclusion

After all this debugging, what was the root cause of the problems? As far as we can tell, the original problem was the inductor failure and it's just a coincidence that the problem occurred after the power loss during system startup. The new AQW card must have caused the fuse to blow, although we don't have a smoking gun.13 The reason the -6V power supply wasn't showing any voltage is because it was sequenced by relay #1, which didn't close because of the fuse. The bad transistors in the -6V power supply problem were apparently a pre-existing and non-critical problem; the good transistors handled enough load to keep the power supply working. The moral from all this is that keeping an old computer running is challenging and takes a talented team.

Thanks to Robert Baruch for the inductor photos. Thanks to Carl Claunch for providing analysis. The Computer History Museum in Mountain View runs demonstrations of the IBM 1401 on Wednesdays and Saturdays so check it out if you're in the area; the demo schedule is here.

Follow me on Twitter or RSS to find out about my latest blog posts.

Notes and references

1. Although there are two IBM 1401 computers at the CHM, only one of them has the "column binary punch" feature that I needed. "Column binary" lets you punch arbitrary patterns on a punch card (to store binary) rather than being limited to the standard punch card character set of 64 characters. ↩

2. Note that the 1401 has 4000 characters of memory and not 4096 because it is a decimal machine. Also, the memory stores 6-bit characters plus a (metadata) word mark and not bytes. ↩

3. If you want to know more about the 1401's core memory, I've written in detail about core memory and described a core memory fix . ↩

4. The trick that makes core memory work is that the cores have extremely nonlinear magnetic characteristics. If you pass a current (call it I) through a wire through a core, the core will become magnetized in that direction. But if you pass a smaller current (I/2) through a wire, the core doesn't change magnetization at all. The result is that you can put cores on a grid of X and Y wires. If you put current I/2 through an X wire and current I/2 through a Y wire, the core at their intersection will get enough current to change state, while the rest of the cores will remain unchanged. Thus, individual cores can be selected. ↩

5. The matrix switch is another set of cores in a grid, but used to generate pulses rather than store data. The 1401's memory has 50 X lines and 80 Y lines (yielding 4000 addresses), so generating the X and Y pulses with transistors would require 50 + 80 expensive, high-current transistors. The X matrix switch has 5 row inputs and 10 column inputs, and 50 outputs—one from each core. The address is decoded to generate the current pulses for these 15 inputs. Thus, instead of using transistor circuits to decode and drive 50 lines, just 15 lines need to be decoded and driven, and the matrix switch generates the final 50 lines from these. The Y lines are similar, using a second matrix switch to drive the 80 Y lines. ↩

6. Each matrix switch has two current inputs (for the row select and the column select), so there are four current source boards and four current driver boards in total. ↩

7. Strangely, half the inductor is nicely wound while the winding in the other half is kind of a mess.

   ↩

   The faulty inductor from the IBM 1401.

8. The 1401 has more power supplies that aren't visible in the picture. They are behind the power supplies in the photo and slide out from the side for maintenance. ↩

9. If you want to see the original schematics and diagrams of the 1401's power supplies, you can find them here. Core memory schematics are here. ↩

10. The pencilled-in fused on the schematic also had a note about an IBM "engineering change". In IBM lingo, an engineering change is a modification to the design to fix a problem. Thus, it appears the the 1401 originally didn't have the fuse, but it was added later. Perhaps we weren't the first installation to have this problem, and the fuse was added to prevent more serious damage. ↩

11. The 18V differential supply provides 12 volts. This seemed contradictory, but there's an explanation. The core memory circuitry is referenced to +30 volts. It needs a supply 18 volts lower, which is provided by the 18V differential supply. Thus, the voltage is +12V above ground. Unlike the regular +12V power supply, however, the differential power supply's output will move with any changes to the +30V supply, ensuring the difference is a steady 18 volts. ↩

12. The "Merry Xmas" card was inspired by a tweet from @rrragan. (I had also designed a card with a menorah, but unfortunately encountered keypunch problems and couldn't get it completed in time. Maybe next year.) Punch cards normally encode characters by punching up to three holes per column. Since this decorative card required many holes per column, I needed to use the 1401's column binary feature, which allows arbitrary binary data to be punched. I ended up punching the card upside down to simplify the program:

    ↩

    Front of my "Merry Xmas" punch card.

13. After carefully examining the AQW boards, we determined that one- and two-transistor cards should be compatible. The two-transistor board had the two transistors in parallel, probably using earlier transistors that couldn't handle as much current. It's possible that the filter capacitor between +30V and ground was shorted in the replacement AQW board, blowing the fuse. ↩