Few tools in control engineering are as revealing—or as easily misinterpreted—as the Nyquist plot. In principle, this graphical representation of a system’s frequency response allows engineers to quickly assess stability, gain margins, and phase margins. In practice, however, the real world is rarely linear. Saturation, dead zones, hysteresis, and other nonlinearities can warp the Nyquist curve, leading to stability assessments that are either overly optimistic or dangerously pessimistic. Understanding how system nonlinearities distort Nyquist plots—and how to compensate for that distortion—is essential for designing robust control systems that perform reliably under all operating conditions.

Nyquist Plots in Control Engineering

Fundamentals of Frequency Response

Every linear time-invariant (LTI) system can be characterized by its transfer function G(s). When a sinusoidal input of frequency ω is applied, the steady-state output is another sinusoid of the same frequency, but with a magnitude change and a phase shift. The frequency response is the complex-valued function G(jω), which encodes both magnitude and phase for every ω from zero to infinity. For linear systems, this response is independent of input amplitude—a key property that simplifies analysis.

The Nyquist plot is a polar diagram of G(jω) in the complex plane. As ω sweeps from 0 to ∞, the curve traces out the system’s gain and phase characteristics. The critical point for stability, per the Nyquist stability criterion, is the point –1 on the real axis. If the curve encircles this point a certain number of times (determined by the number of open-loop poles in the right half-plane), the closed-loop system is unstable. This elegant graphical test is one reason the Nyquist plot remains a cornerstone of classical control theory.

Constructing a Nyquist Plot

Building a Nyquist plot by hand involves evaluating the magnitude and phase of G(jω) at key frequencies. Low frequencies often produce a response near the DC gain; high frequencies typically approach the origin as gain rolls off. Intermediate frequencies reveal peaks, resonances, and the all-important crossing of the negative real axis. Modern tools such as MATLAB’s nyquist function automate the process, but the engineer must still interpret the plotted curve. The number of encirclements of –1, the location of gain and phase margins, and the shape of the curve near critical frequencies all contribute to the stability verdict.

For a more detailed introduction to constructing and interpreting Nyquist plots, see the Control Tutorials for MATLAB and Simulink.

The Challenge of Nonlinear Systems

Common Nonlinearities

Every physical control system contains nonlinear elements. Some are intentional—like the dead zone in a thermostat that prevents chatter—while others are unavoidable side effects of hardware limits. The most frequently encountered nonlinearities include:

  • Saturation: Amplifiers, valves, and actuators cannot produce infinite output. When the input attempts to drive the output beyond its physical limits, the output clips. This hard limit is a classic nonlinearity.
  • Dead zone: Many actuators require a minimum input before any output occurs. This gap in response, common in hydraulic valves and gear trains, creates a region where system gain is effectively zero.
  • Hysteresis: The output depends not only on the current input but also on the history of inputs. Hysteresis appears in ferromagnetic materials, relays, and mechanical play. It introduces a memory effect that can cause oscillations and limit cycles.
  • Backlash: Mechanical gears and linkages have play or “slop.” When the direction of motion reverses, the output briefly fails to respond until the slack is taken up. Backlash reduces positioning accuracy and can lead to instability in position control loops.

These nonlinearities break the fundamental assumption of linear system theory: that the output is proportional to the input and that superposition holds. As a result, the frequency response measured at one amplitude may differ dramatically from the response at another.

How Nonlinearities Distort Frequency Response

In a linear system, the output spectrum contains only the input frequency (plus, possibly, its harmonics if the system is nonlinear). When nonlinearities are present, the output can contain higher harmonics, subharmonics, and even chaotic components. The Nyquist plot, which implicitly assumes a single-frequency sinusoidal response, cannot capture this spectral spreading.

For example, consider a saturating amplifier. For small inputs, the amplifier operates in its linear region and the Nyquist plot is well-behaved. As the input amplitude increases, the amplifier clips the peaks of the sinusoid. The output becomes a distorted waveform that, when analyzed by a standard frequency-response measurement (which typically looks at the fundamental Fourier component), can show a lower effective gain. This reduction in gain shifts the Nyquist curve inward, making the system appear “more stable” than it actually is—a dangerous illusion.

Similarly, hysteresis can introduce a phase lag that varies with amplitude. The Nyquist curve may shift in ways that are not predicted by a linear model, potentially crossing the –1 point in a non‑obvious manner. Engineers who rely solely on a Nyquist plot obtained from a linearized model risk missing the true stability boundaries.

Impact on Nyquist Plot Accuracy

False Stability Margins

The Nyquist stability criterion is mathematically precise for linear systems. For nonlinear systems, applying the same criterion to a Nyquist plot obtained from a single input amplitude may lead to incorrect conclusions. A classic example is the “jump resonance” phenomenon in systems with saturation or other gain-nonlinear effects. The Nyquist plot drawn from a small-signal test might show a gain margin of 10 dB, yet the actual closed-loop system can still enter a limit cycle when the disturbance amplitude exceeds a threshold. The plot does not reveal the amplitude-dependent nature of the nonlinearity.

Conversely, a nonlinearity such as dead zone can reduce the effective gain at low amplitudes, causing the Nyquist plot to indicate a smaller phase margin than actually exists. This can lead to over‑conservative controller designs that sacrifice performance unnecessarily.

Harmonic Generation and Describing Functions

Because a nonlinear system under sinusoidal excitation produces harmonics, the conventional Nyquist plot—which only portrays the fundamental component—is incomplete. To bridge the gap between linear Nyquist analysis and nonlinear reality, engineers often use describing functions. A describing function is an amplitude-dependent transfer function that approximates the nonlinear element’s response at the fundamental frequency. By inserting the describing function into the loop, a modified Nyquist plot can be constructed that accounts for the nonlinearity’s effect on gain and phase at the given input amplitude.

The describing function approach is not exact—it assumes that higher harmonics are filtered out by the rest of the system, which is often true for low‑pass plants—but it is far more informative than a simple linear Nyquist plot. For a comprehensive discussion, refer to the MathWorks documentation on describing functions.

Strategies for Accurate Interpretation

Linearization Around Operating Points

The simplest strategy is to linearize the nonlinear system around one or more steady‑state operating points. By deriving small‑signal models at each point, you can construct a family of Nyquist plots, one per operating condition. This approach works well when the system remains near the chosen operating point most of the time. However, it cannot capture transient excursions into nonlinear regions or limit-cycle behavior that spans multiple regimes.

When using linearization, always verify that the actual input amplitudes stay within the small‑signal range over which the linear model is valid. If the system regularly encounters saturation or other hard limits during normal operation, linearization alone is insufficient.

Using Describing Functions

Describing functions provide a more quantitative tool for analyzing how a specific nonlinearity modifies the Nyquist plot. The nonlinear element is replaced by an equivalent complex gain N(A, ω), where A is the input amplitude. The open‑loop transfer function becomes G(jω) · N(A, ω). The critical condition for a limit cycle (sustained oscillation) is that the locus of –1/N(A, ω) intersects the Nyquist plot of the linear plant G(jω). This intersection point indicates the amplitude and frequency at which the nonlinear system may self‑oscillate.

Engineers can overlay the –1/N locus on the standard Nyquist plot to predict the existence and characteristics of limit cycles. This technique is especially valuable for systems with saturation, hysteresis, or dead zones. For a practical walkthrough, see the Lund University notes on describing functions.

Multiple Operating Point Analysis

Because nonlinearities are amplitude‑dependent, a single Nyquist plot taken at one input amplitude cannot tell the whole story. A systematic approach is to run frequency‑response tests at several amplitudes—from very small (linear regime) up to the maximum expected disturbance. Plot each resulting Nyquist curve on the same axes. The envelope of these curves reveals how the stability margins shrink or expand as amplitude varies. If the curves for different amplitudes converge, the system is essentially linear over that range. If they diverge significantly, the nonlinearities dominate the response.

This multi‑amplitude Nyquist plot is a powerful diagnostic tool. It can expose hidden instabilities that only appear at certain amplitudes and can guide the selection of anti‑windup or gain‑scheduling strategies.

Time‑Domain Simulation

No frequency‑domain technique can fully replace time‑domain simulation when nonlinearities are present. Simulating the system’s response to step, ramp, or sinusoidal inputs—especially with disturbances—provides a direct look at transient behavior, overshoot, settling time, and limit cycles. Use the simulation results to validate any Nyquist‑based stability conclusions. For example, if your Nyquist analysis predicts a gain margin of 8 dB but the time‑domain simulation shows growing oscillations after a large disturbance, trust the simulation. The Nyquist plot of the linearized model was misleading.

Combine frequency‑domain and time‑domain analyses iteratively. Run a Nyquist test, adjust the model based on the shape of the curve, then simulate to confirm. This hybrid approach is the industry standard for nonlinear control system design.

Combining Graphical and Analytical Methods

No single tool can guarantee correct stability analysis for every nonlinear system. The most reliable approach uses a layered methodology:

  • Start with linear Nyquist analysis to get a baseline understanding of the system’s stability when operating near a chosen point.
  • Identify dominant nonlinearities through physical reasoning, data sheets, and testing.
  • Apply describing functions to model the effect of each nonlinearity on the frequency response, and produce an amplitude‑dependent Nyquist plot.
  • Perform multi‑amplitude frequency sweeps to experimentally validate the describing function predictions.
  • Simulate the full nonlinear system in time to check for limit cycles, transient instability, and robustness to parameter variations.
  • Iterate the controller design until both the Nyquist‑based margins and the time‑domain performance meet the specifications.

This systematic workflow prevents the common mistake of treating a Nyquist plot as an infallible truth. Instead, the plot becomes one piece of a larger puzzle.

Practical Examples

Saturation in a Servo System

Consider a DC motor position control loop with a PI controller. The motor’s voltage amplifier saturates at ±10 V. For small step commands, the amplifier stays in its linear range, and the Nyquist plot shows a healthy phase margin of 50°. For larger steps, the amplifier saturates during the transient, effectively reducing the loop gain during the clipping period. The describing function for saturation shows that the effective gain drops as the command amplitude increases. Plotting –1/N(A) on the Nyquist diagram reveals that for sufficiently large amplitudes, the locus intersects the G(jω) curve, predicting a limit cycle. Time‑domain simulation confirms a sustained oscillation around the final position. The linear Nyquist plot alone would never have predicted this instability.

Hysteresis in a Relay

Relays and on–off controllers exhibit hysteresis: the switch‑on point is higher than the switch‑off point. The describing function for a relay with hysteresis introduces an amplitude‑dependent phase lag. The Nyquist plot of the linear plant may indicate a stable system, but the –1/N(A) locus can intersect the Nyquist curve at a frequency where the plant’s phase exceeds –180°. This intersection predicts a limit cycle whose amplitude and frequency can be read directly from the plot. In practice, this limit cycle may be acceptable (e.g., a small ripple around a setpoint) or catastrophic (e.g., chattering in a machining tool). Understanding the Nyquist plot in the presence of hysteresis allows the engineer to design a dead zone or dither signal to suppress the oscillation.

Conclusion

System nonlinearities are not mere nuisances—they are inherent properties of every physical control system. Their effect on Nyquist plot accuracy can be profound, leading to erroneous stability conclusions if only a single linearized model is used. By acknowledging these nonlinearities and employing strategies such as describing functions, multi‑amplitude testing, linearization around multiple operating points, and time‑domain simulation, engineers can extract reliable stability information from Nyquist plots even in the presence of saturation, dead zones, hysteresis, and backlash.

The Nyquist plot remains a valuable tool, but its interpretation must be tempered with an understanding of its limitations. A robust control design does not rely on a single graphical test; it integrates frequency‑domain analysis with nonlinear system theory and simulation. In doing so, engineers ensure that their controllers not only look stable on paper but perform reliably under the full range of real‑world operating conditions.