Nyquist Plot Fundamentals for Signal Processing Filter Design

The Nyquist plot remains one of the most powerful graphical tools in control theory and signal processing. By mapping the open-loop frequency response of a system onto the complex plane, engineers gain immediate visual insight into stability margins, resonant behavior, and overall filter performance. When applied to filter design, the Nyquist plot reveals precisely how modifications to component values or algorithm coefficients affect system behavior across the entire frequency spectrum. This expanded guide explains the core principles of Nyquist analysis and demonstrates how to leverage those insights to create more robust, higher-performing filters.

Understanding the Nyquist Plot: A Core Tool for Engineers

A Nyquist plot represents the complex frequency response G(jω)H(jω) of a transfer function. As frequency ω sweeps from 0 to +∞, the plot traces a curve in the complex plane where the x-axis is the real part (gain) and the y-axis is the imaginary part (phase). Unlike Bode plots that separate magnitude and phase, the Nyquist plot preserves the relationship between gain and phase at every frequency, making it uniquely suited for stability analysis.

The Polar Representation of Frequency Response

Every point on the Nyquist plot corresponds to a specific frequency. The distance from the origin represents the magnitude of the response at that frequency, while the angle from the positive real axis represents the phase shift. As frequency increases, the locus of points reveals how the system’s gain and phase evolve. A simple low-pass filter, for example, traces a semicircle in the left half-plane as phase lags from 0° to -90° and magnitude rolls off.

Nyquist Criterion for Stability

The Nyquist stability criterion uses the plot to determine whether a closed-loop system will remain stable. The key is the point (-1, 0) in the complex plane. If the Nyquist plot of the open-loop transfer function encircles this critical point in a clockwise direction, the closed-loop system has unstable poles. The number of encirclements of -1 minus the number of open-loop unstable poles (poles in the right half-plane) gives the number of closed-loop poles in the right half-plane. A zero net encirclement indicates stability, assuming the open-loop system is stable.

Key Nyquist Plot Insights for Filter Design

Filters are inherently frequency-selective systems. Whether you are designing analog active filters, digital IIR filters, or linear-phase FIR filters, the Nyquist plot provides actionable insights that go beyond simple magnitude and phase response.

Stability Analysis and Gain/Phase Margins

The Nyquist plot directly reveals the gain margin and phase margin. The gain margin is the distance (in dB) from the plot’s crossing of the negative real axis to the critical point -1. The phase margin is the amount of additional phase lag required to bring the plot through -1. In filter design, inadequate margins produce ringing, overshoot, or outright instability when the filter is cascaded or placed inside a feedback loop. For example, a low-pass filter with insufficient phase margin in a feedback control system can cause oscillations at the cutoff frequency.

Resonance and Peak Detection

Peaks in the Nyquist plot correspond to frequencies where the magnitude response exceeds unity. These peaks indicate potential resonance, which can degrade filter performance by introducing unwanted amplification at specific frequencies. In active filter designs like Sallen-Key or multiple-feedback topologies, the Nyquist plot helps engineers identify Q-factor values that create peaking. By adjusting component ratios or damping elements, the plot can be shifted away from the -1 point, flattening the passband and improving transient response.

Phase Lag Compensation Insights

All filters introduce phase lag, but the Nyquist plot quantifies how that lag accumulates with frequency. For cascaded filter stages, the total phase shift may exceed 180°, pushing the loop gain toward instability. By examining the Nyquist plot of a multistage filter, designers can decide where to insert lead compensation networks. The plot shows exactly which frequencies need extra phase boost to keep the response clear of the -1 point.

Practical Steps for Applying Nyquist Insights to Filter Improvement

Using Nyquist analysis to improve a filter design follows a systematic process. The steps below assume a continuous-time analog filter, but the same logic applies to discrete-time digital filters when transformed via the bilinear transform.

Step 1: Generate the Open-Loop Nyquist Plot

Whether you use an analog circuit simulator (SPICE) or a mathematical tool (MATLAB, Python with control libraries), compute the open-loop transfer function G(s)H(s) for your filter cascaded with any downstream components or feedback paths. Plot the frequency response from DC to well above the cutoff frequency, capturing at least two decades above the highest pole frequency.

Step 2: Identify Critical Encirclements and Margins

Look for any encirclement of (-1, 0). If the plot encircles this point clockwise, the closed-loop system has right-half-plane poles and is unstable. If the plot approaches but does not encircle -1, measure the gain margin: find the frequency where the phase is -180°, then identify the magnitude at that point. The gain margin in dB is -20 log₁₀(|G(jω_pH180)|). Similarly, find the phase margin at the frequency where the magnitude crosses 0 dB (unity gain). The phase margin is 180° plus the phase angle at that gain crossover.

Step 3: Adjust Filter Parameters to Improve Margins

Use the Nyquist plot to guide parameter changes. To increase gain margin, reduce the low-frequency gain of the filter or move poles further away from the imaginary axis. To increase phase margin, reduce the Q-factor of any resonant poles, or add a zero in the left half-plane to provide phase lead. For example, increasing the damping ratio of a second-order low-pass section reduces the peak on the Nyquist plot near the resonant frequency, pulling the plot away from -1 and improving phase margin.

Step 4: Validate Changes with Simulation

Before committing to hardware or firmware changes, re-simulate the Nyquist plot with the adjusted parameters. Verify that the plot no longer approaches the -1 point within unsafe margins. Then simulate the closed-loop step response to confirm that overshoot, settling time, and ringing meet specifications. A well-tuned filter will exhibit a step response with minimal overshoot (typically less than 10%) and a clean, monotonic decay.

Advanced Nyquist Considerations in Filter Design

Beyond basic stability margins, Nyquist plots reveal subtle behaviors that can affect real-world filter performance.

Non-Minimum Phase Systems

Filters with zeros in the right half-plane are non-minimum phase. Their Nyquist plots exhibit additional phase lag and may produce a “loop” that can encircle -1 even if the magnitude response appears benign. These systems are often found in feedforward equalizers and certain communication filters. The Nyquist plot makes these hidden stability issues visible, allowing designers to add pre-compensation or choose alternative filter topologies.

Time Delays and Transport Lag

Every real filter introduces some delay, but in digital filters, the sampling delay is a fixed additional phase shift that rotates the Nyquist plot clockwise as frequency increases. A long delay relative to the filter’s time constants can cause the plot to wrap around and encircle -1 at high frequencies. The Nyquist plot shows this effect plainly, enabling designers to either increase sampling rate, reduce filter order, or add delay compensation.

Multi-Input Multi-Output (MIMO) Filters

For systems with multiple inputs and outputs, the Nyquist plot generalizes to the characteristic loci of the return ratio matrix. Each eigenvalue trace provides its own Nyquist plot. Instability occurs if any one of these plots encircles the critical point. This approach is essential when designing MIMO feedback filters for audio crossovers, antenna array beamformers, or active noise cancellation systems.

Case Study: Designing a Stable Active Low-Pass Filter

Consider a fourth-order Sallen-Key low-pass filter with a cutoff of 1 kHz. The initial design uses two second-order sections, each with Q = 0.707 (Butterworth response). The Nyquist plot of the open-loop filter (with the filter in the forward path and a unity feedback loop) shows a smooth, semicircular shape that remains well within the right half-plane. Gain margin is infinite, and phase margin is approximately 60°, which is excellent.

Now the designer decides to increase the Q of the second section to 1.3 to achieve a Chebyshev response with a faster roll-off. The Nyquist plot immediately reveals a problem: the magnitude at the phase crossover frequency exceeds 0 dB, and the plot swings perilously close to (-1, 0). The phase margin drops to 25°, and the gain margin is only 3 dB. The closed-loop step response shows 40% overshoot and sustained ringing.

Using Nyquist feedback, the designer reduces the Q of the second section to 0.9 and adds a small lead capacitor across the feedback resistor of the first op-amp. The new Nyquist plot shows improved margins: gain margin becomes 8 dB and phase margin rises to 50°. The step response now has under 15% overshoot, and the filter meets the original specifications without sacrificing roll-off performance.

Conclusion: Making the Nyquist Plot an Everyday Design Tool

The Nyquist plot is not just a classroom exercise — it is a practical, action-oriented tool for engineers who build filters for real systems. By learning to read the plot for stability margins, resonant peaks, and delay effects, designers can iterate quickly, avoid costly prototype re-spins, and ensure filters behave reliably under all operating conditions. Whether you are fine-tuning an active analog filter or optimizing a digital compensation network, the Nyquist plot gives you a direct visual of how your changes affect system stability. Incorporate this tool into your regular design workflow, and you will produce filters that are not only theoretically correct but truly robust in practice.

For further study, refer to the canonical Nyquist stability criterion explanation on Wikipedia, a deep dive into Nyquist plots in MATLAB, and a practical guide on All About Circuits that walks through filter design examples.