Advertisement
Guest User

Untitled

a guest
Apr 27th, 2017
181
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.80 KB | None | 0 0
  1. def sgd_momentum(w, dw, config=None):
  2.     """
  3.    Performs stochastic gradient descent with momentum.
  4.  
  5.    config format:
  6.    - learning_rate: Scalar learning rate.
  7.    - momentum: Scalar between 0 and 1 giving the momentum value.
  8.      Setting momentum = 0 reduces to sgd.
  9.    - velocity: A numpy array of the same shape as w and dw used to store a
  10.      moving average of the gradients.
  11.    """
  12.     if config is None: config = {}
  13.     config.setdefault('learning_rate', 1e-2)
  14.     config.setdefault('momentum', 0.9)
  15.  
  16.     v = config.get('velocity', np.zeros_like(w))
  17.     learning_rate = config.get('learning_rate')
  18.     mu = config.get('momentum')
  19.  
  20.     v = mu * v - learning_rate * dw # integrate velocity
  21.     next_w = w + v # integrate position
  22.     config['velocity'] = v
  23.  
  24.     return next_w, config
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement