REINFORCEMENT LEARNING Learning control algorithms for power electronic systems: from zero to hero in minutes

Related Vendors

dSPACE GmbH

CISSOID S.A.

PCIM

Reinforcement learning-based control design opens up the door to a new world of optimal controllers, which can be learned in minutes without human intervention. Traditional model-based control design techniques require significant development effort and, therefore, this article explores control design automation opportunities for power electronic systems.

The world of power electronics is heterogeneous and extensive: it includes AC-DC, DC-AC, AC-AC, DC-DC conversion for various load and source combinations, numbers of phases and covers a power range from milliwatts to megawatts. Many of those power electronic converts require proper control engineering to ensure a safe, reliable and efficient operation. Due to the heterogeneity of the field, this requires an enormous amount of engineering effort across all industrial applications, both in the control development and control tuning phase for specific converters.

Traditionally, linear feedback controllers, such as the classic PID controller, are used for this purpose.

However, this comparatively simple approach often proves to be difficult in practice, as non-linear system behavior, input action and state limitations as well as parasitic influences lead to significant deviations between the theoretical linear control design and the controlled system behavior in the real application. Especially for more complex converters, e.g. in the area of highly utilized drive systems or uninterruptible power supplies, corresponding control concepts have become more and more complex and include heuristic adaptation controllers with nested control loops, which can add up to a three-digit setting of tunable control parameters. Those historically evolved control concepts are cumbersome to improve as well as adapt to new converter hardware design, that is, they tie up enormous personnel resources. And yet these generally do not exhibit optimum control behavior due to their linear feedback approach.

Model predictive control (MPC), on the other hand, is an interesting alternative because it interprets control as an optimization problem and can thus theoretically achieve optimal control performance. By considering the possibly non-linear system behavior, the system limitations and the future system behavior, MPC is completely different from classic feedback controllers and has been a major topic in academic research since roughly 40 years.

Although MPC could achieve tremendous merits in the academic world and has proven its versatility in almost every power electronic converter type, it could not achieve a breakthrough in industry. Besides very limited, niche applications (e.g., ABB’s megawatt drive series), heuristic feedback control concepts are still the backbone of industrial control systems in the active field. Main reasons for this observation are likely the increased complexity of MPC versus linear feedback control, which requires more development and tuning effort as well as engineers with a higher level of education. The resulting increase of development effort and time-to-market are crucial reasons contrast with the widespread use of MPC in the power electronics industry.

POWER ELECTRONICS

A brief history of power electronics and why it's important

Fortunately, reinforcement learning (RL)-based control approaches offer an alternative way of achieving optimal control. Instead of solving a complex and model-based optimization problem in real time, RL learns optimal control actions from past data. Similar to human learning, RL evaluates past actions and adapts its policy such that favorable control actions, e.g., leading to better tracking performance or higher converter efficiency in steady state, are more likely in the future. To prevent critical errors, especially during the early training phase of this data-driven controller, an additional safe guard is deployed to override unsafe actions, which would likely lead to critical overcurrent or overvoltage situations.

In this way, an explicit control policy, typically encoded within an artificial neural network, is learned. Thanks to embedded hardware accelerators, like field-programmable gate arrays (FPGAs) or neural processing units (NPUs), those policies are executable in the microsecond range rendering them feasible for many power electronic applications. The learning process itself does not be executed in real time, but can be either outsourced to a prior simulation-based offline training or an edge-learning based training at test benches. Both approaches can be fully automated and do not require human intervention, which is favorable regarding development resources.

While RL-based control for power electronic applications is still in the early research stage, recent experimental proof-of-concept studies for different power electronic converter applications show that learning optimal control policies is possible within just a few minutes opening up the door to a completely new paradigm for control design (see example video below). These encouraging intermediate results motivate further research in this direction, particularly with a view to transfer to other power electronics applications and further methodological improvement, e.g. to reduce both training times and numerical effort on the embedded control devices. The prospect is that optimal control performance, as theoretically possible with MPC, can be achieved with minimal (human) development effort in heterogenous industrial power electronic applications.

Coffee machine vs. machine learning: who is quicker? Watch this video to find out:

References

• M. Schenke, B. Haucke-Korber and O. Wallscheid, "Finite-Set Direct Torque Control via Edge-Computing-Assisted Safe Reinforcement Learning for a Permanent-Magnet Synchronous Motor," in IEEE Transactions on Power Electronics, vol. 38, no. 11, pp. 13741-13756, Nov. 2023, doi: 10.1109/TPEL.2023.3303651

• D. Weber, M. Schenke and O. Wallscheid, "Safe Reinforcement Learning-Based Control in Power Electronic Systems," 2023 International Conference on Future Energy Solutions (FES)2023 International Conference on Future Energy Solutions (FES), Vaasa, Finland, 2023, doi: 10.1109/FES57669.2023.10182718

• S. Zhang, O. Wallscheid and M. Porrmann, "Machine Learning for the Control and Monitoring of Electric Machine Drives: Advances and Trends," IEEE Open Journal of Industry Applications, vol. 4, pp. 188-214, 2023, doi: 10.1109/OJIA.2023.3284717

PCIM Expo 2025: Become an exhibitor now

Let’s elevate your company’s global presence within the power electronics industry! Showcase your solutions, connect with the international community and shape the future at the PCIM Expo from 6 – 8 May 2025 in Nuremberg, Germany.

Learn more

(ID:50357429)