Hammer time: fixing the printer on a vintage IBM 1401 mainframe

The Computer History Museum has two operational IBM 1401 computers used for demos but one of the printers stopped working a few weeks ago. This blog post describes how the 1401 restoration team diagnosed and repaired the printer. After a lot of tricky debugging (as well as smoke coming out of the printer) we fixed a broken trace on a circuit board. (This printer repair might sound vaguely familiar because I wrote in September about an entirely different printer failure due to a failed transistor.)

The IBM 1401 business computer was announced in 1959, and went on to become the best-selling computer of the mid-1960s, with more than 10,000 systems in use. A key selling point of the IBM 1401 was its high-speed line printer, the IBM 1403. It printed 10 lines per second with excellent print quality, said to be the best printing until laser printers were introduced in the 1970s.

The IBM 1401 mainframe computer (left) at the Computer History Museum printing the Mandelbrot fractal on the 1403 printer (right).

The IBM 1401 mainframe computer (left) at the Computer History Museum printing the Mandelbrot fractal on the 1403 printer (right).

To print characters, the printer used a chain of type slugs (below) that rotated at high speed in front of the paper, with an inked ribbon between the paper and the chain. Each of the 132 print columns had a hammer and an electromagnet. At the right moment, when the desired character passed the hammer, an electromagnet drove the hammer against the back of the paper, causing the paper and ribbon to hit the type slug, printing the character.1

The type chain from the IBM 1401's printer. The chain has 48 different characters, repeated five times.

The type chain from the IBM 1401's printer. The chain has 48 different characters, repeated five times.

The printer required careful timing to make this process work. The chain spun around at 7.5 feet per second and each hammer had to fire at exactly the right time to print the right character perfectly aligned without smearing. Every 11.1 µs, a print slug lined up with a hammer, and the control circuitry checked if the slug matches the character to be printed. If so, the electromagnet was energized for 1.5 ms, printing the character.

Printing mechanism of the IBM 1401 line printer. From 1401 Reference Manual, p11.

Printing mechanism of the IBM 1401 line printer. From 1401 Reference Manual, p11.

While the printer is usually reliable, a few weeks ago the printer stopped working and displayed a "sync check" error on the console. The computer needs to know the exact position of the chain in order to fire the hammers at the right time. If something goes wrong with this synchronization, the computer stops with "sync check" rather than printing the wrong characters.

When the sync check light on the printer is illuminated, you have a problem.

When the sync check light on the printer is illuminated, you have a problem.

To track the chain position, the computer receives a sequence of pulses from the printer: a pulse when the first hammer is lined up with a type slug2 and a double pulse when the chain is in its "home" position with the first character lined up. The pulses are created by a slotted metallic timing disk inside the printer. A magnetic pickup detects these slots and produces a small 100 millivolt signal.7 This signal is amplified inside the printer by two differential amplifier cards to produce a stronger square wave signal. (This is the only electronic part of the printer. Everything else inside the printer is electromechanical or hydraulic; a high-speed hydraulic motor feeds paper through the printer and drips oil on the floor.)

The computer receives these pulses from the printer and generates a logic signal that increments counters to keep track of the chain's position. The schematic below shows part of the circuitry inside the computer, starting with the sense amplifier signal from the printer at the left. Don't try to understand this circuit; I just want to show the strange schematic symbols that IBM used in the 1960s. The box with "I" is an inverter. The triangle is an AND gate. The semicircle that looks like an AND gate is actually an OR gate. The large box with a "T" is a trigger, IBM's name for a flip-flop. The "SS" box is a "single shot" that creates a 400µs pulse; this detects the double pulse that indicates the chain's "home" position.

Excerpt from the 1401 Intermedia Level Diagrams (ILD) showing the chain detection circuitry.

Excerpt from the 1401 Intermedia Level Diagrams (ILD) showing the chain detection circuitry.

To track down the problem, we removed the printer's side panel to access the two amplifier circuit boards, which are visible below. We probed the boards with an oscilloscope. The first amplifier stage (on the right) looked okay, but the second stage (on the left) had problems. In the photo below, the computer is at the back, mostly hidden by the printer.

We took the side panel off the 1403 printer to reveal the circuit boards. We hooked an oscilloscope up to the front board to test it.

We took the side panel off the 1403 printer to reveal the circuit boards. We hooked an oscilloscope up to the front board to test it.

The trace below shows what should happen. The board receives a differential signal at the bottom, with alternating cyan and pink signals. The difference between these signals (middle) is amplified to produce the clean, uniform pulse train at the top. Note the double pulse in the middle indicating the chain's home position.

Oscilloscope trace from a working printer.

Oscilloscope trace from a working printer.

But when we measured the signal, we saw signals that were entirely garbled. The differential signals at the bottom are a mess, and track each other rather than alternating. The output signal (top) is basically random. With this signal from the printer, the computer couldn't keep track of the chain position and the sync check error resulted.

Oscilloscope trace from the faulty printer.

Oscilloscope trace from the faulty printer.

We swapped the board with the board from the other, working printer, and verified that the board was the problem. The museum has a filing cabinet full of replacement circuit boards, but unfortunately not a replacement for this "WW" amplifier board. Instead, we had to diagnose the problem with this card and repair it. On the board below you can see the diodes (small gray cylinders), capacitors (silver cylinders), resistors (striped cylinders and large tan cylinders), and germanium transistors (round metal cans). The transistors are germanium transistors as the 1401 predated silicon transistors.

The "WW" differential amplifier card used by the printer.

The "WW" differential amplifier card used by the printer.

We suspected a failed transistor, so we used Marc's vintage Hewlett-Packard Tektronix curve tracer (below) to test the transistors. One transistor was much weaker than the others. Since the performance of a differential amplifier depends on having transistors with closely matched characteristics, we searched through a couple dozen transistors to find a matching pair and replaced the transistors. (We later determined that these transistors were not part of the differential pair—they were "emitter follower" buffers, so our effort was wasted.)3

We used a vintage HP transistor curve tracer to test the transistors.

We used a vintage HP transistor curve tracer to test the transistors.

Back at the Computer History Museum, we tested out the repaired board and the printer still didn't work. Even worse, smoke started coming from the back of the printer! I quickly shut off the system as an acrid smell surrounded the printer. I expected to see a blackened transistor on the board, but it was fine. I examined the printer but couldn't find the source of the smoke.

I decided to test the board outside the printer by feeding in a 2kHz test signal, but the measurements didn't make sense. The board seemed to be ignoring one of the inputs, so I tested that input transistor but it was fine. Next I checked the diodes, capacitors and resistors again; all the components tested okay, but the board still mysteriously failed. I started carefully measuring voltages at various points in the circuit but the signals didn't make sense and weren't consistent. Since all the components were fine but the board didn't work, I was starting to losing confidence in electronics. Eventually, I nailed down a signal that randomly jumped between 10 volts and 1 volt. After wiggling all the components, I finally noticed that the voltage jumped if I flexed the board. Finally, I had an answer: a cracked trace on the circuit board between the input and the transistor was making intermittent contact.

The board had a cracked trace in the upper left, connecting the upper gold contact. Carl put a wire jumper across the bad section.

The board had a cracked trace in the upper left, connecting the upper gold contact. Carl put a wire jumper across the bad section.

To fix the board, Carl put a wire bridge across the bad trace6. We put the board in the printer, and the printer mostly worked. However, when the printer tried to print in column 85, the column failed to print and the printer stopped with an error.5 More testing revealed four columns of the printer were failing to print due to hammer problems. Each electromagnetic hammer coil is driven by a 60 volt, 5 amp pulse for 1.5 milliseconds. This is a lot of power (300 watts), so if anything goes wrong, hammer coils can easily burn up.

We swung open one of the computer's "gates" (lower left), revealing the cards that drive the printer.

We swung open one of the computer's "gates" (lower left), revealing the cards that drive the printer.

We looked at the printer driver cards inside the computer. Each card generates pulses for two hammers, so there are 66 of these cards. In the photo below, you can see the two large high-current transistors at the left that generate the pulses. (Note the felt insulators on top of these transistors. Due to their height, the transistors pose a risk of shorting against the bottom of the neighboring card.) Just to the right of these transistors are two colorful purple and yellow fuses. In the event of a fault, these fuses are supposed to burn out and protect the hammer coils. We checked the cards associated with the four bad columns on the printer and found four burnt-out fuses.

The "AEC" Alloy-Hammer Driver Latch card produces high-current pulses to drive the printer hammer coils.

The "AEC" Alloy-Hammer Driver Latch card produces high-current pulses to drive the printer hammer coils.

Why did the fuses blow? The circuit to drive the hammer coils is a bit tricky. Every 11 microseconds, a hammer lines up with a character slug and can be fired. But when a hammer is fired, the coil needs to be activated for about 1.5 ms, a much longer time interval. To accomplish this, the hammer driver latches on when a hammer is fired. Later in the print cycle, the hammer driver is turned off. This process is controlled by the chain position counters, which are driven by the pulses from the chain sensor, the same pulses that were intermittent. Thus, if the computer received enough pulses to start printing a line, but then the pulses dropped out in the middle of the line, hammer drivers could be left in the on state until the fuse blows. This explained the problem that we saw.

After Carl replaced the fuses, the printer worked fine except for two problems. First, characters in column 85 were shifted slightly so the text was slightly crooked. Frank explained that the hammer in this column must be moving a bit too slow, hitting the chain after it had moved past its position. This explained the smoke: in the time it took the fuse to blow, the coil must have overheated and been slightly damaged. We'll look into replacing this coil next week. The second problem was that the printer's Ready light didn't go on. This turned out to be simply a bad light bulb, unrelated to the rest of the problems. In any case, the printer was working well enough for demos so the repair was a success.

Closeup of the type chain (upside down) for an IBM 1403 line printer.

Closeup of the type chain (upside down) for an IBM 1403 line printer.

I announce my latest blog posts on Twitter, so follow me at @kenshirriff for future articles. I also have an RSS feed. The Computer History Museum in Mountain View runs demonstrations of the IBM 1401 on Wednesdays and Saturdays so if you're in the area you should definitely check it out (schedule).

Notes and references

  1. You might expect that the 132 hammers align with 132 type slugs, so the matching hammers all fire at once, but that's not what happens. Instead, the hammers and type slugs are spaced slightly differently, so only one hammer is aligned at a time, and a tiny movement of the chain lines up a different hammer and type slug. (Essentially they form a vernier.) Specifically, every 11.1 microseconds, the chain moves 0.001 inches. This causes a new hammer / type slug alignment. For mechanical reasons, every third hammer lines up in sequence (1, 4, 7, ...) until the end of the line is reached; this is called a "subscan" and takes 555 microseconds. Two more subscans give each hammer in the line an option to fire, forming a print scan of 1.665 milliseconds. If you want more information on how the print chain works, I have an animation here

  2. To be precise, the printer generates a pulse if hammer 1, 2, or 3 lines up with a type slug. This is due to the three "subscans", each using every third hammer. 

  3. I'll explain how the differential amplifier works in this footnote, since most readers may not want this much detail. The computer uses two differential amplifier boards in series, first a WV board and then a WW board. They use similar principles, except the WV uses NPN transistors and the WW uses PNP. The differential output from the WW board is transmitted to the computer where a third differential amplifier (an NT card) converts the signal to a logic output. Each board is a differential amplifier, which takes two inputs and amplifies the difference, essentially an op amp with two outputs.4

    A differential pair circuit.

    A differential pair circuit.

    The basic differential pair circuit for a differential amplifier is shown above. (Op amps contain a similar differential pair.) The resistor at the top sets a fixed current I. If the two inputs are equal, the current will be split, with half going through each transistor and branch resistor. But if one if the inputs is slightly lower, that transistor will conduct more and most of the current will go through that branch. Thus, the difference between the inputs steers the current down one side or the other, yielding an amplified signal across the lower resistors.

    Schematic of the WW amplifier board from the SMS documentation.

    Schematic of the WW amplifier board from the SMS documentation.

    The IBM 1401 documentation provides the schematic above for the board, but it's hard to follow what's happening. (Note the unusual transistor symbol, three boxes with an emitter arrow in or out.) I redrew the main part of the circuit below, so it resembles the simple differential pair. It has the same resistors at top and bottom as the differential pair, but there is an R-C circuit in each branch. To simplify, if there is a DC offset or low-frequency input, the capacitor will charge and counteract this offset. Thus, the amplifier operates as a high-pass amplifier; it cuts out low-frequency noise while amplifying the 1800 Hz sync pulses. The diodes clip the output, yielding a square wave. The differential output goes through emitter-follower buffers (omitted below) so the signal is strong enough to be transmitted through an under-floor cable from the printer to the computer.

    The differential amplifier circuit of the WW card.

    The differential amplifier circuit of the WW card.

     

  4. An op amp with positive and negative outputs is known as a "fully differential op amp". 

  5. The IBM 1403 printer has multiple error checks to avoid printing incorrect data. For a business machine, it would be bad to drop digits in, say, payroll checks or tax records. To detect hammer failures, the printer has 132 wires from the hammers back to the computer, to verify that each hammer fired when it was supposed to. If the computer doesn't get a pulse back from a hammer, the computer stops immediately, as we saw. 

  6. We noticed that there was solder smeared across the broken part of the trace. My suspicion is that the same problem happened a few years ago and was repaired by bridging the broken trace with solder. Eventually the heavy vibrations inside the printer caused a hairline crack in the solder, causing the problem to recur. By bridging the break with wire rather than just solder, we hope we have fixed the problem permanently. We also noticed the transistor connected to the broken trace had been replaced, so they must have tried that first in the previous repair. 

  7. The 1403 printer is documented in IBM 1403 Printer Component Description and 1403 Printers Field Engineering Maintenance Manual. See also this brief article about the 1403 printer in the IEEE Spectrum. For details on how the timing pulses work, see the 1403 Manual of Instruction, page 42. 

15 comments:

Cole Johnson said...

Wonderful job as always!

Since you seem to be good at this, could you fix my (modern) printer?

DHess said...

Weird, I have never seen an HP curve tracer marked Tektronix before.

Ken Shirriff said...

DHess: Ha ha, I guess I should have looked more closely :-)

Russel said...
This comment has been removed by the author.
Russel said...

Ah the good old days. I remember wearing IBM's required dress suit while I was a CE and working on 1403s and trying to avoid oil stains. It was a great career though; ask me about using a little hammer to 'tap' the heads of a 3330 into alignment ... while the drive was running and the heads were flying above the disk platters!

Ah, the good old days. :-)

Franc said...

Hi,

Why did they use hydraulics to run the printer, is that to move the paper one line at a time?

Anonymous said...

Hi Ken, will you post some of your work about restoring Apollo AGC?

Ed said...

About the odd conventions in the schematics... Lynn Conway's wonderful memoir "IBM-ACS: Reminiscences and Lessons Learned From a 1960’s Supercomputer Project" contains this remarkable observation:
"And then, Boom! I figured it out: the breakthrough came when several designers admitted they could not initially follow other designers’ work either, and it had taken them quite a while to become ‘experts.’ It turned out that different designers were using different ad-hoc symbolic representations for the same things. Thus all logic equations and logic circuit sketches were in the particular ‘shorthand’ of the originating designer, right up until entry into DRKS. Thus newcomers to ACS engineering could easily become confused while attempting to decipher other designers’ work."

I do recommend reading the memoir, for the technical and social history, and the massive misstep IBM took due to not understanding what they had - Dynamic Instruction Scheduling invented in the 60s and very nearly lost, and reinvented in the 80s. ACS could have been the fastest machine on the planet, and was cancelled. The DIS invention, a means to great performance, was lost by IBM.

J.Gustavo A. Murta said...

Hi Ken,
I´m a retired IBM brazilian Technician. I worked for many years fixing IBM 1403 Printer.
That model is the oldest of 1403 Printers ( Chicken´s feet). May be I can help you. I dont worked with 1401 System. I worked with 2821 Controller.
I need remember some information.

John S said...

I had a really enjoyable time at the Computer History Museum over the '18 Memorial Day weekend and I was really happy to get my 1403 printouts. What a great system and such a great set of debugging descriptions. It's amazing how intricate the systems were at the time, and how today alot of this is handled in much different ways.

Please keep up your work, both at the museum and documenting the work you do to keep these pieces of history working.

Cheers,
John

Tom Hargrave said...

Great article! One of my hobbies is much older tech - vacuum tube radios from the 1920 through the 1940's.

More printer related - I worked at Browns Ferry Nuclear Power Plant in the very late 80's and the Nuclear Physicists used a PRIME computer for some of their calculations. In 1989 they replaced their high speed line printers with higher speed laser printers. The high speed line printers were already fast - green bar paper would literally fly out the back of the printers when reports ran. The new printers were even faster, so fast that the poor guy responsible for keeping them on line would stage reams of paper around the new printers before any print job, and even then he could not keep up.

I don't remember the printer model. Also, any time he had an issue with one of the new printers he'd pick up the phone and two IBM folks in grey suits would be on site within two hours!

DonH said...

Regarding the question about why the use of hydraulics, it was probably the best way at the time to get the combination of power, speed and accuracy needed. The disk drives of the period used hydraulics to drive the heads in and out too.

The 1403 paper drive used 3 hydraulic gear pumps. (I think they were Cessna aircraft pumps.) Each pump consisted of 2 meshed gears in a figure-8 housing. Oil entered on one side, was moved around the outside of the gears in the spaces between the teeth, and exited on the other side. Two pumps were driven by an electric motor, and the third was used as the motor to drive the paper feed. The whole assembly sat in an oil bath formed by a finned cover over the pumps.

Spool valves controlled the oil flow. At rest, the oil from the pumps was returned to the sump. To drive the paper, oil was switched from one or both of the pumps to lift a detent and drive the motor. Line-to-line feed was done with one pump, longer feeds were done with both (called High-speed skip).

As well as single line feeds, the printer responded to "skip to n" commands, where n was a number from 1 to 12. An endless loop paper tape driven by the paper feed had 12 columns, each of which could be punched to indicate a "skip to" position. When a skip command was received, the paper would feed until the corresponding hole was detected. The tape was cut to a length corresponding to the length of the forms being printed on. If the tape was incorrectly punched or if it jammed, paper could feed continuously...

Anonymous said...

As an IBM Customer Engineer in Dallas, TX, in the 1960's I worked on a lot of 1403 printers along with 2401 tape drives, 2540 card/punch, and 2311 disk drives. A great time to be involved in the computer revolution.

WH

--- said...

"the massive misstep IBM took due to not understanding what they had - Dynamic Instruction Scheduling"

It was a good thing, and a bad thing. Modern out-of-order CPUs spend insane amounts of power and die area on dealing with scheduling stuff in the instruction window. Stuff that should have been done by compilers with a hardware-specific specializer that IBM had for eons (that's what made all the old 360 code run on all those disparate platforms).

These days, dynamic instruction scheduling (reordering), register rename buffers, and all the associated junk is a massive investment into supporting obsolete designs. Tomasulo's algorithm is a great thing as long as you don't try to have the hardware doing it over and over and over and over again at runtime. Yuck.

DHess said...

You can get a real earful from Linus Torvalds on that subject over on RWT.

Static instruction scheduling by the compiler continuously fails because the compiler lacks the needed information. This is what Itanium attempted and it failed at. Meanwhile dynamic instruction scheduling at execution time works.