Prototypical recurrent unit

Abstract Despite the great successes of deep learning, the effectiveness of deep neural networks, such as LSTM/GRU-like recurrent networks, has not been well understood. Not only attributed to their nonlinear dynamics, the difficulty in understanding LSTM/GRU-like recurrent networks also resides in the highly complex recurrence structure in these networks. This work aims at constructing an alternative recurrent unit that is as simple as possible and yet also captures the key components of LSTM/GRU recurrent units. Such a unit, if available, can then be used as a prototype for the study of LSTM/GRU-like networks and potentially enable easier analysis. Towards that goal, we take a system-theoretic perspective to design a new recurrent unit, which we call the prototypical recurrent unit (PRU). Not only having minimal complexity, PRU is demonstrated experimentally to have comparable performance to GRU and LSTM over a range of modelling tasks. This establishes PRU networks as a prototypical example for future study of LSTM/GRU-like recurrent networks. The complexity advantage of PRU may also make it a favourable alternative to LSTM and GRU in practice.