Soft Actor Critic (SAC)

Related to Actor Critic Methods.

SAC is an off-policy actor-critic algorithm but adds an entropy term to the reward, encouraging the policy to explore more by remaining stochastic during training.

It uses two critic networks like TD3 to reduce overestimation in bias and improve stability.

$${\pi }^{\ast }=\arg \underset{\pi }{\max }{\mathbb{E}}_{\tau \sim \pi }\left[\sum _{t=0}^{\infty }{\gamma }^t\left(R(s_t,a_t,s_{t+1})+\underset{\text{Entropy}}{\underbrace{\alpha H(\pi (\cdot |s_t))}}\right)\right]$$

SAC qualities:

• stable training by reducing overestimation
• exploration: entropy regularization prevents early convergence.
• sample efficiency: off-policy learning (replay buffer) improves data usage.
• automatic tuning: learns $\alpha$ to balance exploration and exploitation.
• continuous actions: naturally handles high-dimensional action spaces.